Intuitively and Exhaustively Explained

Real-Time Visualization of Agentic Interactions— Intuitively and Exhaustively Explained

Daniel Warfield — Mon, 24 Aug 2026 10:03:29 GMT

“Constellations” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. Brought to you by the subscribers of Intuitively and Exhaustively Explained.

For the last few months I’ve been working on a project called the Agent Harnesses Standard, which is a way of defining how information is structured so agentic tools like Claude can more quickly understand large bodies of information. You can read more about that here if you’re curious:

Share this article with friends and colleagues.

One of the goals of the standard is to allow an agent to quickly and consistently understand a large directory, like a codebase. It’s a fine goal, but without any ability to observe what “quickly and consistently understanding” looks like, it’s impossible to understand what approaches work well and what approaches don’t. That’s why I created ahar-visualizer, a way to visualize how an agent navigates through a directory in real time. I made ahar-visualizer an easy-to-install VSCode extension allowing a user to see how an agent navigates through large and complex systems.

A timelapse of me interacting with a large codebase, while using the ahar visualizer and the Claude CLI

In this article we’ll explore how to install ahar-visualizer and how it can be used to visualize how Claude interacts with the documents in your projects.

Let’s start with installation.

Installing ahar-visualizer

The most up-to-date version of the visualizer can be found here:

That link catalogs all of the releases. Click the ahar-visualizer-x.y.z.visx file in the most recent release to download it.

Then open up a terminal, navigate to the directory it was downloaded to, and install it to VSCode by running this (substituting X.Y.Z for whatever version you downloaded):

code --install-extension ahar-visualizer-X.Y.Z.vsix

Once you run that, if you open VSCode, you should see an icon for the ahar visualizer.

Click the icon; that will open up a side panel. Click “Open Tree Visualization” and, provided VSCode is open at an interesting directory, you might see a cool tree.

With that, the visualizer is installed. Let’s explore how it works.

Exploring the Visualizer

The visualizer looks at two things to build a view of your directory and how Claude is interacting with it.

First, it looks at your directory structure where VS Code is currently open. The structure of the tree is built based on the files and folders in your directory and responds dynamically when folders are created or deleted.
When you run Claude via the Claude CLI, the transcript of the conversation is stored in ~/.claude/projects//*.jsonl, which allows Claude to keep track of the chat context. The visualizer looks at the most recent chat session and monitors for reads, writes, edits, and updates to files, updating the visualization as necessary to highlight recent changes. This is how the visualizer shows you what Claude is working on.

Thus, wherever the visualizer is loaded, it’s always observing the most recent session on your computer by default, though this is configurable. Currently, the Claude CLI is the only agentic tool this visualizer works with. The plan is to extend this functionality to tools like Claude Desktop, but I started with the CLI because it’s convenient for some of the tests I’m currently conducting.

The intended workflow is that you open up VSCode in some directory, open up the Claude CLI, open up ahar-visualizer, and get to work.

What my view typically looks like when interacting with Claude. I sometimes re-orient these windows, hide the file browser, and generally change this setup depending on the specifics of what I’m working on.

As you work with Claude, you might notice that some nodes are getting brighter, some are fading, and some of the lines in the tree are going from a dark gray to a brighter color.

An example of nodes highlighted through traversal.

These denote the agent visiting certain files. The more recently in the chat a file has been visited, the brighter and bolder the outline. Over time, these become less bold. Regardless of how long ago a file was visited, the fact that it was visited at some point in the present chat is denoted by a thin orange line. If you start a new Claude session, these lines will reset as the visualizer represents that new session.

Within the tree, folders are denoted with squares, while files are denoted with circles. You can click on either to see information about them. If you click a file, you can open that file from the visualizer.

opening a file from the tree

If you click a folder, you can collapse or expand the tree at that location, allowing you to pare down the visualization to something you’re particularly interested in.

expanding and collapsing a tree

There are also a few quality of life features. A legend on the bottom left, denoting information on the nodes and their types.

The legend.

There’s also the idea of “sub-harnesses”, which are essentially just special folders. We’ll revisit this when we discuss the agent harnesses standard in the next section.

The sub-harnesse UI, which we’ll discuss later.

There are tools that allow you to expand or collapse the tree, play with the view, and mess with settings.

nav and setting tools

And that’s it for the general-purpose stuff. You can set up the ahar visualizer right now and use it to visualize how your agent is working with your directory.

There are a few powerful features we glossed over that I want to talk about, and those have to do with the agent harnesses standard.

Agent Harnesses Standard Specific Visualizations

I’ve been working on a variety of Git repos that define tools to support the agent harnesses standard

To accelerate the development of these tools, I have a “meta repo” called agentdev, which encapsulates all of these repos into one big monolithic repo using git submodules. The point of this “meta repo” is to define guidance so an agent can work through all of the pieces of the standard simultaneously. the meta repo has all of my repos as sub-modules, as well as tools and instructions that inform how an agent should interact with those submodules. I’ve found this workflow to be a powerful approach to working on many tightly interrelated projects, a topic which I want to explore in depth in a future article.

This repo actually consists of many indepdentently versioned sub-modules.

Because I launch Claude at this “meta repo” level, if I discover that a core idea is missing from the standard when working on one piece, I can work with Claude to improve all of the other repos based on that insight. Because the agent harnesses standard provides robust contextual information, and Claude is so powerful, this scheme is yet to cause confusion in terms of what Claude should be interacting with.

There are a variety of ideas within the standard that help to enable this workflow, but the most fundamental idea is that of “routing files”. These are special markdown files that are designed to instruct an agent how it should navigate through a directory. At the top level of a directory is HARNESS.md, which describes the directory from a high level, then different subdirectories have routing files like TOOLS.md, REFERENCES.md, DOCS.md, SRC.md, whatever. These files are named based on the directory structure itself and serve to provide routing information to an agent. Critically, their naming convention is standardized, meaning summarization about an entire directory can be retrieved via a single function call. You can read more about the specifics here if you’re interested:

The simple but powerful idea of routing files has been the biggest impact of the agent harnesses standard, as it allows for an efficient yet thorough description of an entire repo that an agent can observe upfront then act on throughout the chat continuum.

On the visualizer you might notice that some folders have a color associated with them. The orange/brown ones contain these routing files.

If you click one of these folders, you can open the associated routing file. If the agent reads this file, it will highlight the folder in the visualizer. Because these routing files are a first-order construct, they’re not represented as their own nodes but as modifiers to folders.

Recently, the standard added support for nested harnesses. Normally, each directory has a single HARNESS.md file at the root folder that allows an agent to understand that repo in totality. With nesting, one can also put harnesses within harnesses, allowing for a HARNESS.md powered repo to be used within another HARNESS.md powered repo.

The visualizer detects these nested structures and allows you to navigate to and isolate certain sub-harnesses via the sub-harness UI.

focusing on and isolating various sub-harnesses

It’s worth noting, in this particular repo, the idea of sub-harnesses is closely tied to git submodules, but that is by no means a requirement. A sub-harness is just a directory that has an isolated namespace from the agent harnesses’ perspective. If you were building one big agent that could satisfy numerous roles, each of those roles could be a sub-harness within a larger harness.

The agent harnesses standard also has a concept called a “leaf”. A “leaf” is a folder structure that should be considered as a termination point. Skills are the prototypical example of this; a skill has a SKILL.md file up top that describes the tools and resources relevant to that skill. When defined as a leaf, the contents of a skill are seen outside of the scope of the agent harness. All the nitty-gritty details in the skill are not exposed at the harness level; implying that the agent should go read the skill if it thinks it might be relevant. This idea of hierarchical summarization, and allowing the agent to dig deeper as necessary, is called “progressive disclosure”, and is central to the agent harnesses standard.

These leaf nodes are configurable within the agent harnesses standard, so you can specify what you consider to be a leaf. You might imagine a large directory of files that you don’t want the agent to dig through, and instead you want an agent to use a skill to interface with those documents. You might define this big directory of documents as a “leaf”, so the harness doesn’t consider it in scope. Typically leaf nodes have a markdown file associated with them, which describes to the agent how this information should be interacted with.

Leaves are represented by the visualizer as blue nodes within the graph. The squares are the folders that denote the leaf, and the circles are the files that the agent should be looking for to understand what the leaf is about.

Conclusion

The ahar visualizer, I think, is a fantastic general tool for understanding agentic work. I find myself watching the visualizer while the agent is working, allowing me to build a spatial mental map of certain pieces of functionality and where they live. It’s helping me tremendously to maintain a conceptual understanding of a codebase I’m controlling the creation of but have written very little of. For that reason, the visualizer has become a core part of my workflow.

The original reason I created it, to allow me to visualize A/B tests, is an article that I’m currently working on. Stay tuned for that. In the meantime, feel free to install the ahar cli to create and convert repos to the agent harnesses standard and the ahar visualizer to view how Claude is interacting with your documents. All of these repos are open source, and I welcome contributions if you’re interested.

A Brief Correction to Previous Articles

When initially releasing the Agent Harnesses Standard, I was a bit bullish and contrarian about what the term “harness” means for AI. Since then, I’ve revised my opinions, perhaps not in their essence but in their rigidity.

In this article I described a harness as a directory of files.

Then, in this article, I argued against calling tools like the Claude CLI and Claude Code a “harness”, and instead argued that they should be called something like an “Agentic Client.”

In retrospect, this stance was excessively contrarian. The term “harness” is a generic term that denotes abstracting things. Very commonly, in software and in other contexts where the word “harness” is used, there are harnesses within harnesses. In reality, I don’t think the term is actually all that specific, so bickering about its specific usage is thus ridiculous.

Personally, I still think in terms of “an agentic client with some functionality connecting to a harness that contains tools and resources”, but if you want to call Claude Code a harness, which is the standard verbiage, then the agent harnesses standard is a standard way to connect your harness to tools and context.

What, in this chain of abstraction, you call a “harness” is actually of little consequence. The important thing is that there’s some code that turns an LLM into an agent that can do things, sets of tools and context that allow that agent to satisfy a role, and something in the middle to ensure that the agent can understand those tools and context consistently and reliably. That “thing in the middle” is what the agent harnesses standard is about.

Towards Self-Repairing and Repeatable AI Systems

Daniel Warfield — Wed, 05 Aug 2026 19:02:02 GMT

“Roles” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. Brought to you by the subscribers of Intuitively and Exhaustively Explained.

I’ve been working on the Agent Harnesses Standard for a few months, and have been using it to automate key tasks within my workflow. I’d like to share how I use it, the tools I’ve built around it, their features, and the practical impact of approaching agentic role definitions in a standardized way.

Share this article with friends and colleagues.

The Goal of The Agent Harnesses Standard

Fundamentally, the idea is to standardize how information is organized throughout a folder hierarchy to make information easily discoverable by AI systems like Claude. As the project matures, that goal is beginning to crystalize into a set of subgoals:

People work with agents to satisfy several tasks throughout the day. Often, agents get confused when they need to reconcile with multiple roles simultaneously. One goal of the agent harnesses standard is to make the creation, maintenance, and isolation of these roles easier and more intuitive.
When an agent is designed to handle a complex role with multiple responsibilities, the context and tools necessary to fulfill that role can become numerous. This can lead to confusion as critical tools being misused. Another goal of the agent harnesses standard is to allow for greater organization of context and tools.
Regardless of the organisational patterns of information, it’s useless if an agent can’t find that information efficiently. Another goal of the agent harnesses standard is to structure the context and information necessary to satisfy a role in a way that can be effectively, efficiently, and inexpensively understood by AI systems.
Agentic systems can quickly deviate from human understanding if left to update themselves. A goal of the agent harnesses standard is to encourage AI systems to follow a structure that a human can readily understand, edit, adjust, and converse with an agent about, preserving human comprehension.

I don’t think any of these are solved problems, but the Agent Harnesses Standard has been a useful construct in allowing me to iterate on these problems and wrangle them significantly more easily. I’d like to briefly describe the standard, then describe some of the tools I’ve built to accelerate workflows around it.

The Standard in a Nutshell

The Agent Harnesses Standard is, in its essence, an extension of the skills standard.

If you’re familiar with the skills standard, it boils down to having a single markdown file, called SKILL.md, which describes context and tools the agent can use to do stuff. You might have a directory called send-email that contains a SKILL.md file describing the skill, and some code the agent can execute to send the email.

send-email/
├── SKILL.md
└── scripts/
     └── send_email.py

The actual SKILL.md file has information about all of the scripts and references in the send-email directory, allowing an agent to read a markdown file to get a gist of how to use the resources within the skill, rather than needing to read all the files and understand the code for itself. This saves time, and makes execution of the skill more consistent. If you want to know more about the specifics, I have a whole article on the subject.

This standard is simple, but it’s great. It’s great because it allows for a few key things:

Efficient abstraction of large batches of information, allowing an AI model to quickly understand sophisticated functionality.
A single entry point. You know the agent will look at SKILL.md before doing other stuff with the skill, so you can introduce critical prompting information just in time, rather than trying to get the agent to understand everything about a whole project in a system prompt.
A shared standard. Skills aren’t model or provider specific, a skill defined for one agent will likely work for another, as the skill standard is ubiquitously adopted.

However, skills aren’t perfect.

Claude, for instance, organizes skills into a flat directory structure. This is fine if you only have a handful of skills, but if you have many skills, or if skills are meant to exist in some type of sequential workflow, it can be cumbersome to try to get the model to consistently execute on high-level tasks consistently. You might have skills for send-email, look-up-user-preferences , and view-purchasing-history, but if you want the model to execute those skills in a particular way, like looking up the user and viewing their purchasing history before sending an email, there’s no standard way of defining that intention. The Agent Harnesses Standard attempts to solve that problem via a hierarchical organizational structure.

The Agent Harnesses Standard allows you to specify skills within a hierarchical structure, and to create key routing documents to describe to an agent how information should be used. For instance, we could have the following structure

email-client/
├── EMAIL-CLIENT.md
├── send-email/
├── look-up-user-preferences/
└── view-purchasing-history/

Where send-email, look-up-user-preferences, and view-purchasing-history are all skill folders with their own SKILL.md file. The EMAIL-CLIENT.md file is from the Agent Harnesses Standard, and it’s called a “routing file”. It serves as an entry point, allowing the agent to understand how these skills should be used within the context of emailing a client.

These organizations can take on high-order structures. Say you also need to call or send physical mail to a customer. Those could be their own directory each with routing files allowing the AI system to quickly understand how and when these high level functionalities, and their respective skills, should be used.

interactin-skills/
├── INTERACTION-SKILLS.md
├── email-client/
|   ├── INTERACTION-SKILLS.md
|   ├── send-email/
|   ├── look-up-user-preferences/
|   └── view-purchasing-history/
├── call-client/
|   ├── INTERACTION-SKILLS.md
|   ├── send-call/
|   └── look-up-number/
└── mail-client/
    ├── INTERACTION-SKILLS.md
    ├── send-mail/
    └── look-up-address/

Exactly how you organize a harness is up to you, and should be designed as a function of the project you’re working on and what you want an agent like Claude to do within that project.

You might notice that each of the skills has an INTERACTION-SKILLS.md. To make routing decisions more obvious to both agents and humans, each routing file within a subdirectory inherits the name of the top-level subdirectory. All the routing files in interactin-skills/ are INTERACTION-SKILLS.md . Here’s a more complex example from the agent harnesses docs that has two top level subdirectories.

my-harness/
├── HARNESS.md
├── tools/
│   ├── TOOLS.md
│   ├── backend/
│   │   ├── TOOLS.md        
│   │   └── create-api/
│   └── frontend/
│       ├── TOOLS.md
│       └── build-ui/
└── data/
    ├── DATA.md             
    ├── schemas/
    │   ├── DATA.md
    │   └── table-definitions.md
    └── quirks/
        ├── DATA.md
        └── known-issues.md

At the very top of the harness there’s a HARNESS.md file, which serves as the entry point to the entire harness. Like how SKILL.md is the single entry point into a skill, HARNESS.md is the single entry point into a harness, and can provide top-level routing information, allowing an agent to navigate through the harness from a particular starting point.

Since defining the structure of the agent harnesses standard, I’ve been refining tools to allow Claude to efficiently interact with an manage this structure. I want to talk about those tools, and how I work with them, next.

Tools and Workflow

One of the biggest quality of life changes I’ve made recently is a tool called ahar, which allows one to initialize repos that follow the agent harnesses standard. It’s pip installable via the following:

pip install agentharnesses-cli

This installs ahar which can be used to do all sorts of stuff. Chiefly, initialize an agent harnesses standard directory.

ahar init

Currently, this creates the following structure. This can be modified based on the nature of your project, the agent harnesses standard is agnostic to folder names, though I’ve found that having a skills/ and references/ directory along with HARNESS.md has worked well for me.

my-harness/
├── HARNESS.md                       # entry point and agent identity
├── README.md                        # human-facing description
├── .gitignore
├── .claude/settings.json            # registers the harness as a Claude Code plugin
├── skills/
│   └── SKILLS.md                    # skill index
└── references/
    └── REFERENCES.md                # reference index

If you type in claude in the configuration, it will also create the “agent harnesses metaskill”, which is a skill that claude can use to interact with the harness.

├── .claude/skills/agent-harnesses/  # metaskill for progressive harness exploration
└── skills/
    └── maintenance/
        ├── SKILLS.md
        └── modify-harness/
            └── SKILL.md

Pretty much any time I start a new repo any more I

git initialize
call ahar init and specify claude
then run claude in the command line.

When claude is running, I just tell it load harness and it loads the metaskill, observes that the harness is empty, and asks if I want to do anything. I then tell it what I’m working on, and it begins building out the structure of the project for me.

Because Claude knows about the agent harnesses standard through the metaskill, it understands the intent behind the routing files, and has tools to programatically interact with them. This allows Claude to do things like:

Create an efficient summary of the entire structure of the project in a single command line argument (using the metaskill), allowing claude to quickly understand an entire project at a glance.
Trace references of functionality throughout the hierarchy of a project, and adjust that referencing based on user feedback, allowing for modifications that remain discoverable across sessions, improving consistency.
Organize and structure information hierarchically, meaning I can let Claude make substantial changes, and the results tend to be both reasonable and easily understandable by me.

I’ve been rather impressed with Claud’s ability to do this. It works nicely with the standard, and I find that it can create well thought out organizational systems efficiently. This lets me spool up multiple sub-agents, update prompting information, and cache key information in these routing documents so that Claude behaves more consistently across different sessions. I think the biggest benefit is the hierarchical caching of key prompting information, which allows the system to achieves the “Self-Repairing and Repeatable” qualities referenced in the title.

Conclusion

I’ve been enjoying the Agent Harnesses Standard. Regardless of if it sees widespread adoption, it’s been a significant boost to my productivity, and I’ll continue working on it. Check it out if it interests you.

Infrastructure on Google Cloud—Intuitively and Exhaustively Explained

Daniel Warfield — Tue, 28 Jul 2026 12:00:28 GMT

“Mountain” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. Brought to you by the subscribers of Intuitively and Exhaustively Explained.

This is a big topic. Too big to be truly exhaustive. Instead, the goal of this article is to be “exhaustively intuitive.” We’re going to explore key tools in the Google Cloud Platform, how they function, and how they can be used to build large-scale, complex applications. By the end of this article, you’ll have a practical understanding of the fundamentals, a general understanding of the lay of the land, and an overall intuition of how Google Cloud can be used to solve large-scale problems in real-world settings.

To galvanize our understanding, we’ll be exploring seven projects throughout this tutorial, all of which explore fundamental ideas in common architectural design.

testproject1 — A simple “Hello World” application
chat-app — An application with a front end and database
diary — An app with user authentication, allowing for secure accounts.
pdf-reader — An event driven application which processes data and exposes it to users
collatz — A task designed for processing heavy workloads
collatzquota —The same as the previous one, but with caching and API-level rate limiting using Redis
PubSub — A high level exploration of a building an event driven architecture to process orders in an ecommerce context

This will give us a fairly comprehensive understanding, but doesn’t cover all of Google Cloud’s 150+ services. This article is the first in a series exploring more advanced topics like global networking, database technologies, and GenAI on Google cloud.

Share this article with friends and colleagues.

Who is this useful for? Anyone interested in understanding robust, cloud-scale applications.

How advanced is this post? This article is designed to introduce cloud architecture and build fundamental understanding. Advanced architectural experience is not required.

Prerequisites: Realistically, you’ll probably need some software development experience to get everything out of this article. If you’re a complete beginner, just read critically, google a lot, and ask questions to a chat assistant.

A Note on How to Read This Article: Learning how the cloud works is hard for one key reason; it requires both a high-level and low-level understanding. If you try to learn cloud concepts from a high level, the concepts feel detached and floaty. If you try to learn cloud concepts from a low level, it’s hard to know what’s going on and why things are happening. We’ll be exploring various examples from both high-level and low-level prospectives. If you find yourself confused, consider forging on regardless so that both levels of granularity can have an opportunity to wash over you.

The Core Building Blocks of Google Cloud

Google Cloud (often abbreviated as GCP, for “Google Cloud Platform”) is a platform with all the bells and whistles necessary to build enterprise-scale applications. Depending on how you count, it exposes over 200 named products or services, allowing you to do global information networking, spool up large compute resources, build databases, and do whatever else you might need to build a large-scale application.

This size is, to a large extent, why getting into cloud architecture is so difficult. There are a billion considerations one can make when building on the cloud, and knowing where to start can be challenging.

From a high level, the offerings in Google Cloud can be divided into the following macroscopic categories:

Compute — Run code on VMs, containers, or functions
Storage & Databases — Store and query your data
Networking — Connect, route, and protect traffic
Data & Analytics — Process and visualize data at scale
AI & ML — Build and deploy models and AI-powered features
Identity, Security & IAM — Control access and protect resources
DevOps & Developer Tools — Build, deploy, and ship code
Management & Monitoring — Observe, debug, and control costs

We’ll be touching each of these buckets, in one way or another, throughout the article.

Google has a free tier that users can employ when creating a new account. It gives you $300 worth of credits and a range of products that are free if you remain under a set utilization.

I’ve consumed a lot of tutorials on cloud architecture in my life, and I want to save you the pain of watching someone dig through all of the resources on GCP one-by-one. It’s a horrific experience. Instead, we’re going to be building several applications that highlight the core pieces of GCP so that we can build a practical understanding of how these things tick. Let’s start by setting up an account.

Setting Up a GCP Account

To do anything, we’re going to need an account.

Click the “Get started” button.

This will lead you to a setup page. You’ll have to put in your credit card to get started, but Google Cloud is pretty good about not allowing you to incur accidental charges unless you explicitly enable your account as a paid account.

Once you answer a few questions, you’ll be in.

Projects on GCP

Arguably the first and most fundamental concept on GCP is that of a project. When you create a new account, a project will already be created, called “My First Project”

and if you click the project, you can view a project browser.

Projects are a base-level container that organizes everything on GCP. Compute resources, databases, all of the stuff that incurs a charge and needs to communicate with one another. “Projects” allow you to organize different projects into different buckets, both for isolating them for security purposes and to keep billing information organized.

We can click the new project on the top right to create a new project.

Then, once you create a project, it’ll end up in the browser.

Let’s build a hello world application in our new test project.

Exploring Cloud Run

Go ahead and search “Cloud Run”, and select it.

After doing that, we’ll end up here, on the dashboard for cloudrun, the first GCP product we’ll be diving into.

Various cloud providers have similar offerings to Cloud Run. The essential idea is that you have code, you point Cloud Run to it, and it runs it on the cloud.

I’m going to start by creating a function in Python, and calling it “testfunction”

Then I’ll set up some configuration. There’s a few different runtimes, but I’ll be using Python. I’ll also expose the function to the public internet, so that it can be easily accessed. In a real-world situation this would require careful consideration from a cost and security perspective, but we’re just experimenting for now.

In billing, I’m doing “request-based”, which means I pay a small fee per request and for the CPU and memory needed to process the request.

Costs for request based billing of cloudrun in us-centeral1. Source.

If I, instead, chose instance-based billing, I would be renting out a slice of a server and paying for the resources I provisioned over time.

Costs for running instance based billing. Source.

We’ll keep service scaling at the default; this setting will make more sense as we move on to discussing VMs and serverless. We’ll also set Ingress to “All,” allowing access to our function by anyone on the internet. And with that, we can go ahead and click “create.”

Then, you’ll get this popup. The Cloud Build API is a fundamental API used to programmatically build resources on your Google Cloud account. Various workflows around creating resources on GCP use the Cloud Build API under the hood to turn code into an actually runnable application. So, go ahead and “enable.”

You might find it silly that such a fundamental idea would require enablement, but least privileges is a critical concept used to keep the cloud secure. We’ll be exploring that throughout the article.

Then, once our function gets built, we’ll see this dashboard.

By default, a placeholder exists at the URL tied with this function. I’m going to click “Save and redeploy.” Then, after a moment it will go through the deployment steps and deploy my function.

If I then click the URL listed, I’ll see the following output

which corresponds to the function I deployed; the default setup when I created the function.

import functions_framework

@functions_framework.http
def hello_http(request):
    “”“HTTP Cloud Function.
    Args:
        request (flask.Request): The request object.
        
    Returns:
        The response text, or any set of values that can be turned into a
        Response object using `make_response`
        .
    “”“
    request_json = request.get_json(silent=True)
    request_args = request.args

    if request_json and ‘name’ in request_json:
        name = request_json[’name’]
    elif request_args and ‘name’ in request_args:
        name = request_args[’name’]
    else:
        name = ‘World’
    return f”Hello {name}!”

Et voilà, we published a function that’s accessible online. In reality a human probably wouldn’t access this cloud function directly; typically a function like this is for doing odd jobs to support a larger application, but we can make this function return HTML with some styling and JavaScript to serve a static website.

import functions_framework

HTML = “”“


  
  
  Intuitively and Exhaustively Explained
  
  


  
  
    Welcome
    Hello
    from
    Intuitively and
Exhaustively Explained
    
    Where every concept finds its clarity.
  

  

”“”


@functions_framework.http
def hello_http(request):
    return HTML, 200, {”Content-Type”: “text/html; charset=utf-8”}

After replacing the code for our function with this, saving and deploying, and waiting for the deployment to go through, we’ll see a static website when we hit the URL.

Little hoaky for my taste, but I’m glad Claude thinks so highly of IAEE.

In theory you could make an entire website like this. For instance, when a user sends a request, we can parse arguments out of that request and use those arguments to decide what to return. The user can specify arguments directly in the URL itself. This, for instance,

https://testfunction-613513089702.us-central1.run.app/?page=home

hits our website with the argument page=home. We can change our website to output different content when page=home vs when page=about, and add into our website buttons that redirect us to different pages. If we build and run this code

import functions_framework

FONTS = ‘’

COMMON_CSS = “”“
* { box-sizing: border-box; margin: 0; padding: 0; }
body {
  min-height: 100vh;
  display: flex;
  align-items: center;
  justify-content: center;
  background: #0f0f0f;
  font-family: ‘Inter’, sans-serif;
}
.stars { position: fixed; inset: 0; z-index: 0; }
.star {
  position: absolute;
  border-radius: 50%;
  background: white;
  animation: twinkle var(--dur) ease-in-out infinite;
  opacity: 0;
}
@keyframes twinkle {
  0%, 100% { opacity: 0; }
  50% { opacity: var(--peak); }
}
.card {
  position: relative;
  z-index: 1;
  text-align: center;
  padding: 4rem 3.5rem;
  max-width: 640px;
  width: 90%;
  border: 1px solid rgba(255,255,255,0.08);
  border-radius: 20px;
  background: rgba(255,255,255,0.03);
  backdrop-filter: blur(12px);
}
.eyebrow {
  font-size: 11px;
  letter-spacing: 0.2em;
  text-transform: uppercase;
  color: #6b7cff;
  margin-bottom: 1.5rem;
}
.divider {
  width: 40px; height: 2px;
  background: linear-gradient(90deg, #6b7cff, #a78bfa);
  border-radius: 2px;
  margin: 2rem auto;
}
.nav-link {
  display: inline-block;
  margin-top: 2rem;
  font-size: 0.8rem;
  letter-spacing: 0.12em;
  text-transform: uppercase;
  color: #6b7cff;
  text-decoration: none;
  border: 1px solid rgba(107,124,255,0.3);
  border-radius: 999px;
  padding: 0.45rem 1.2rem;
}
“”“

STAR_JS = “”“

“”“


def wrap(title, css, body):
    return (
        “”
        “”
        “”
        “” + title + “”
        + FONTS
        + “”
        “”
        “”
        + body
        + STAR_JS
        + “”
    )


def home_page():
    css = “”“
    .hello {
      font-family: ‘Playfair Display’, serif;
      font-weight: 700;
      font-size: clamp(2.8rem, 8vw, 4.5rem);
      color: #fff;
      line-height: 1.1;
      margin-bottom: 0.25rem;
    }
    .from {
      font-family: ‘Playfair Display’, serif;
      font-style: italic;
      font-size: clamp(1rem, 3vw, 1.35rem);
      color: rgba(255,255,255,0.45);
      margin-bottom: 0.6rem;
    }
    .brand {
      font-family: ‘Playfair Display’, serif;
      font-weight: 700;
      font-size: clamp(1.1rem, 3.5vw, 1.55rem);
      background: linear-gradient(90deg, #6b7cff, #a78bfa, #f0abfc);
      -webkit-background-clip: text;
      -webkit-text-fill-color: transparent;
      background-clip: text;
    }
    .tagline { font-size: 0.875rem; color: rgba(255,255,255,0.3); line-height: 1.7; }
    “”“
    body = (
        “”
        “Welcome”
        “Hello”
        “from”
        “Intuitively and
Exhaustively Explained”
        “”
        “Where every concept finds its clarity.”
        “”
        “
”
    )
    return wrap(”Intuitively and Exhaustively Explained”, css, body)


def about_page():
    css = “”“
    .card { text-align: left; max-width: 680px; }
    .page-title {
      font-family: ‘Playfair Display’, serif;
      font-weight: 700;
      font-size: clamp(2rem, 6vw, 3.2rem);
      color: #fff;
      line-height: 1.15;
    }
    .subtitle {
      font-family: ‘Playfair Display’, serif;
      font-style: italic;
      font-size: clamp(0.95rem, 2.5vw, 1.15rem);
      color: rgba(255,255,255,0.4);
      margin-top: 0.25rem;
    }
    .body-text {
      font-size: 0.9rem;
      color: rgba(255,255,255,0.55);
      line-height: 1.85;
      margin-top: 1rem;
    }
    .pill-row { display: flex; flex-wrap: wrap; gap: 0.5rem; margin-top: 1.5rem; }
    .pill {
      font-size: 0.75rem;
      letter-spacing: 0.1em;
      text-transform: uppercase;
      color: #a78bfa;
      border: 1px solid rgba(167,139,250,0.25);
      border-radius: 999px;
      padding: 0.3rem 0.85rem;
    }
    “”“
    body = (
        “”
        “About”
        “Our mission”
        “Making the complex feel inevitable.”
        “”
        “Intuitively and Exhaustively Explained exists for the person “
        “who isn’t satisfied with a surface-level answer. We dig until the idea clicks “
        “— then we dig a little further, just to be sure.”
        “Every explanation starts from first principles and builds up, “
        “so by the end you don’t just know the what — you know the why, the how, “
        “and the \”oh, of course.\””
        “”
        “First principles”
        “Deep dives”
        “No hand-waving”
        “Built to last”
        “”
        “”
        “
”
    )
    return wrap(”About — Intuitively and Exhaustively Explained”, css, body)


PAGES = {
    “home”: home_page,
    “about”: about_page,
}


@functions_framework.http
def hello_http(request):
    page = request.args.get(”page”, “home”)
    handler = PAGES.get(page, home_page)
    return handler(), 200, {”Content-Type”: “text/html; charset=utf-8”}

We have a functional website where, if we click the “About Us” button

It redirects us to the about page by way of ?page=about at the end of the URL.

Aw, shucks Claude, love you too.

And, in theory, that’s it. You could make a whole product using a single cloud function and a whole lot of tender love and care. In reality this isn’t how it’s done, but you could.

Let’s go back to the Cloud Run dashboard and deploy a website the right way via GitHub.

I’m going to choose Cloud Build.

and, this time, we’re going to connect Cloud Run to a GitHub account with a more robust website built in. I set up some source code here; it’s basically just a Vue app.

When we click “Set up with Cloud Build,” we’re going to be directed to authenticate with GitHub

After a few steps setting up Github, installing stuff, and granting permissions

I can select my repository from my GitHub account.

If we click next, we’ll see build configuration.

There are two ways to build, by specifying a Dockerfile, or a buildpack. Dockerfiles are the standard and recommended way of building on GCP, and our repo is set up with a Dockerfile, so we’ll choose that. After clicking “Save”, we can then click through the rest of the settings (just like the function we created previously), and set to “Allow public access” so we can see our website once it’s up.

Once we’re happy with all our settings, we can click the “create” button. After a little while our app will be built and deployed. We can see both of the services we created are ready to rock in the “Services” panel.

And if we click into gcp-example1-cloudrun we can see the URL and high-level health metrics.

Clicking on the URL, we can see that the website has been deployed

Neat!

The inquisitive among you might be wondering about the website itself, what Docker’s role is in making this site available, and some of the settings we glossed over around request and instance-based billing. Let’s take a moment to discuss the deployment of this website from a more conceptual perspective.

Core Concepts Behind Deployment

Covering even a small chunk of GCP is going to take a lot of time. To avoid going too far off track, we’re going to have to keep our exploration of the various applications we’re deploying high-level. If you want a more granular exploration of Docker, and how applications fit into it, I have an article on the subject:

To truly understand what’s going on, though, we’re going to have to form a high-level understanding about Docker.

Docker allows you to package your application and everything it needs to run into a portable and self-contained unit called a container. Instead of configuring servers on the cloud to run your application specifically, the servers are designed to be able to run Docker containers. So, if you package your code and its dependencies into a Docker container, you can be confident it can run on the cloud.

This is super useful because it alleviates the “but it worked on my machine” issues that are common to trying to get software off of a developer’s computer and onto the cloud. A common workflow is for a developer to build software and test it on their computer, then deploy it onto the cloud. Docker makes that process straightforward.

Containerization also allows a server to conveniently subdivide its resources across numerous applications simultaneously. You might imagine a server running several different people’s Docker applications simultaneously, running several parallel instances of your application on a single server, or even running the same application across many servers.

When people talk about “serverless architecture,” they’re usually talking about this paradigm. There’s still a server, but it’s not up to you as a developer to manage it. You just tell GCP to run some code, and it runs it in a containerized manner.

This also relates to the choice of billing being request-based or instance-based. Docker containers are pretty fast to set up and get running. If you only have to run a function once every couple of hours, it might be worth while to spool up your dockerized application when you need it, run it, and then de-prioritize it. You don’t need the code sitting there ready to go 24/7. Thus, you might choose request-based billing, which does that under the hood.

On the other hand, while it’s relatively efficient to spool up and tare down instances, there is some cost, and there’s a cost to managing the overhead of orchestrating how they’re spooled up or torn down. If you know you’ll get consistent traffic, it might make sense to rent the compute resources themselves, so you can park your code there over an extended period of time. That’s instance-based billing, in a nutshell.

Regardless of if you’re doing instance or request-based billing, there’s a concept of load-balancing and parallelization. If you’re doing instance-based billing, you might need several instances of your code running to keep up with the traffic. If you’re doing request-based billing, you might find yourself spinning up several containers at once to handle a burst of simultaneous requests. Either way, the system responds to increased load by running more copies of your code in parallel and spreading traffic across them.

While it’s relatively fast to set up a new container to run for request-based billing, and a bit slower but still relatively fast to spool up a new instance, it does take time. The word for this is “cold-start”, and includes all of the time required to get your code available to run.

Recall earlier in the article, when we spooled up our hello world function. There was an option to specify a minimum and maximum number of instances.

from earlier in the article.

Setting the minimum number of instances to 1 ensures that your code is always ready to run, meaning you don’t have to wait for things to spool back up if your instances scaled down to zero due to a lack of usage. Naturally, there’s a cost to having some compute always ready to run.

One of the critical ideas allowing all this to run is the idea of statelessness. You can build stateful applications on GCP, we’ll explore that later, but the most common paradigm is that of statelessness.

Statelessness means that the servers running your code don’t contain any session-based information. If a user calls a function running on one computer in the cloud, and then calls the same function on a different computer on the cloud, they won’t know the difference because the response will be identical. This allows a user’s traffic to be balanced across many instances of an application, their traffic being switched between instances without them knowing. We’ll touch on that to a greater degree later in the article.

Exploring Firestore

We’re going to make a new project called “chat-app” which will consist of a website where users can register an account, create or select a chat room to join, and both view and contribute to that chat.

Generally, development of cloud resources happens locally before it gets deployed onto the cloud. Before we deploy Firestore on the cloud, we’re going to use local emulation to experiment with using a database on our machine.

Before we go further, it’s worth pausing on a question that’s about to get confusing: we set up a GCP project, and now we’re going to be using something called firebase. What gives?

The short version is that Firebase and GCP aren’t separate things. Firebase was an independent startup Google acquired in 2014; over the years the two have been steadily stitched together. Today a Firebase project is a Google Cloud project, but viewed through two different consoles. Firebase is best thought of as a developer-friendly layer sitting on top of GCP, aimed at app developers who want client SDKs, easy auth, hosting, and good local tooling without having to think about infrastructure.

A lot of the products blur across this line. Firestore, the database we’re about to use, originally came out of the Firebase world but is now a first-class GCP product. Firebase Authentication and GCP’s Identity Platform are similarly two faces of the same underlying login system, which is why later in the article we’ll authenticate with “Firebase Auth” and “Identity Platform” somewhat interchangeably.

So why touch Firebase tooling at all? Purely for convenience. The Firebase CLI ships a local emulator suite that’s the smoothest way to run Firestore on your own machine, which lets us build and test everything locally before spending a cent on the cloud. That’s all we’re borrowing it for here.

So, we’re going to create a new repo to hold our chat-app project, and run

 npm install -g firebase-tools

If you get permission errors, you might need to use superuser permissions to install it correctly

sudo npm install -g firebase-tools

Once that’s set up, you can run

firebase login

to log into the CLI tool. You might need to answer a few questions, I just replied with all the defaults by pressing enter, then it should open up your file browser so that you can finish authenticating.

The application is going to be running fastapi in python. I’m going to manage local development by using uv, which I cover thoroughly in this tutorial.

This is application and local development stuff, and we can’t cover everything. Feel free to read that article if you want to learn more about uv. It’s great if you’re doing local Python development.

Anywho, we can initialize uv via uv init

uv init

then add the dependencies

  uv add fastapi uvicorn google-cloud-firestore

This will essentially give us a self-contained Python project where we can build our application, with all of the dependencies necessary to have that application run.

Now we need the database it’ll eventually talk to, so let’s set up Firestore in our project. Searching for “Firestore” in the console

you should see something like this.

After clicking the “Get started with Firestore’s free tier” button, we can specify some key configuration fields. I’m naming the app chat-app-db, and using the “Standard Edition,” which is less expensive. The enterprise edition has some bells and whistles that we won’t need.

I’m going to launch Firestore in “Native mode”, which is the recommended mode. Restrictive is the default, and it’s a good idea to limit permissions as much as possible, so we’ll work with that. I’m also setting up Firestore in a single region; we don’t need a ton of multi-region resiliency in this demo.

And with that, we can go ahead and click the create button to create our database.

We get redirected to this dashboard, which we could use to play around with data if we wanted to, but we’ll instead let our application create all the data we need through user interaction.

To actually develop our application, we’re going to build everything locally using the firestore local emulator, which we can run using the command

gcp-example2-chatapp % firebase emulators:start --only firestore --project chat-app-500821

To actually get this to run, I had to set up some dependencies like Java on my computer. But, now that it ran properly, we can see

⚠  Could not find config (firebase.json) so using defaults.
i  emulators: Starting emulators: firestore
⚠  firestore: Did not find a Cloud Firestore rules file specified in a firebase.json config file.
⚠  firestore: The emulator will default to allowing all reads and writes. Learn more about this option: https://firebase.google.com/docs/emulator-suite/install_and_configure#security_rules_configuration.
i  firestore: Firestore Emulator logging to firestore-debug.log
✔  firestore: Firestore Emulator was started in standard edition.
✔  firestore: Firestore Emulator UI websocket is running on 9150.

┌─────────────────────────────────────────────────────────────┐
│ ✔  All emulators ready! It is now safe to connect your app. │
│ i  View Emulator UI at http://127.0.0.1:4000/               │
└─────────────────────────────────────────────────────────────┘

┌───────────┬────────────────┬─────────────────────────────────┐
│ Emulator  │ Host:Port      │ View in Emulator UI             │
├───────────┼────────────────┼─────────────────────────────────┤
│ Firestore │ 127.0.0.1:8080 │ http://127.0.0.1:4000/firestore │
└───────────┴────────────────┴─────────────────────────────────┘
  Emulator Hub host: 127.0.0.1 port: 4400
  Other reserved ports: 4500, 9150

Issues? Report them at https://github.com/firebase/firebase-tools/issues and attach the *-debug.log files.

And, if we click the link, we can see that we have an emulation of Firestore running locally.

I don’t want to dig into the actual application code too much; here’s the repo.

It’s a FastAPI application in Python that serves a static website and can communicate with our local database. It can be downloaded via

git clone https://github.com/DanielWarfield1/gcp-example2-chatapp.git

Then we can CD into the repo and run the application.

FIRESTORE_EMULATOR_HOST=localhost:8080 uv run uvicorn main:app --reload

Once it’s running, we can pull up localhost:8000/docs#/, which is the docs for out FastAPI api.

And, now that we’ve confirmed it’s working, we can hop right into localhost. We can register an account, log in

Create chat rooms, and chat with our friends.

And, after a little bit of debugging to make sure everything is properly configured, we can see the data in our local emulation of Firestore.

Firestore is a NoSQL database like MongoDB. I’ll probably do an article on MongoDB at a later date, but it’s basically just a bunch of JSON objects (called documents) within an indexed list (called collections). Here we have a collection for messages, rooms, sessions, and users, each has information based on the usage thus far. Our stateless FastAPI grabs this data, based on the request by the user, in order to respond with the page the user asked for, along with the data in that page.

This is cool, but it’s all running on my machine. The idea is to get it to run in the cloud so that anyone can chat via a public URL. I went ahead and made a docker file that containerizes the code, and sets everything up by running some commands in uv. You’ll have a solid idea of what’s going on here if you read my articles on uv and docker.

FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev

COPY . .

CMD [”.venv/bin/uvicorn”, “main:app”, “--host”, “0.0.0.0”, “--port”, “8080”]

So, our repo should be ready to roll to deploy with Cloud Run, like we did before. Slide over to cloud run with our chat-app active

Choose GitHub.

Then go through all the same configuration steps we went through to set up our previous application. Once it’s set up, it should be available on a public URL, and two people should be able to chat with one another.

Me sending emojis to my partner, who was logged in simultaneously on a different computer.

Twitter eat your heart out. New technocrat inbound.

Jokes aside, we glossed over pretty much all of the application logic to get to this point. Most of it’s out of scope, but I do think it would be pertinent to discuss the code that communicates with the database or the database emulation.

If we take a look at main.py we can see that we use the firestore library to connect with the database.

from google.cloud import firestore
...
db = firestore.Client(project=”chat-app-500821”, database=”chat-app-db”)

Because the application running on cloud run is in the same project as the chat app, authentication to connect securely to the database happens automatically, allowing our server to talk to our database despite the database being locked down from outside traffic.

Recall that, when we set up the database, we set it up with restrictive rules. This bars any external actor from interacting with the database, but allows programs running internally within the project to connect.

When we ran the application locally on our computer, we used the following command.

FIRESTORE_EMULATOR_HOST=localhost:8080 uv run uvicorn main:app --reload

FIRESTORE_EMULATOR_HOST is a special environment variable that the firestore library automatically looks for. If that’s set, then whatever is specified in

db = firestore.Client(project=”chat-app-500821”, database=”chat-app-db”)

is automatically discarded. This allows us to use an environment variable to run our application locally and point it at the emulator.

In the page returned by the server, there’s javascript that causes the website to request an update in three seconds, allowing for a simple implementation of real-time streaming. There’s also the HTML, CSS, and scripts necessary to run the application.

When you press various buttons, it calls endpoints in the FastAPI, like the login button calling the login endpoint.

async function login() {
    const u = document.getElementById(’login-username’).value;
    const p = document.getElementById(’login-password’).value;
    const res = await fetch(’/login’, { method: ‘POST’, headers: {’Content-Type’: ‘application/json’}, body: JSON.stringify({username: u, password: p}) });
    const data = await res.json();
    if (!res.ok) { document.getElementById(’auth-error’).textContent = data.detail; return; }
    token = data.token;
    username = u;
    showApp();
  }

That endpoint might return a token signifying that login was successful, allowing the user to authenticate with different endpoints

@app.post(”/login”)
def login(req: LoginRequest):
    doc = db.collection(”users”).document(req.username).get()
    if not doc.exists or doc.to_dict()[”password_hash”] != hash_password(req.password):
        raise HTTPException(status_code=401, detail=”Invalid credentials”)
    token = secrets.token_hex(32)
    db.collection(”sessions”).document(token).set({
        “username”: req.username,
        “created_at”: datetime.now(UTC),
    })
    return {”token”: token}

List the rooms that are available on the app, using the token from login as authentication.

@app.post(”/rooms”)
def create_room(req: RoomRequest, authorization: str = Header(...)):
    username = get_username_from_token(authorization)
    _, ref = db.collection(”rooms”).add({
        “name”: req.name,
        “created_by”: username,
        “created_at”: datetime.now(UTC),
    })
    return {”room_id”: ref.id, “name”: req.name}

Or whatever.

It’s worth noting that authentication is not done correctly in this application. This was a simple proof of concept, but the usernames and passwords are stored in plain text. In reality, you would want to use a more robust identity and access management system to make authentication to your app more robust. In fact, that seems like a good thing to discuss next.

Exploring Identity Platform

Cool, let’s make another project: a diary application. You log in, you can create and read your own diary entries. We’ll use a proper approach to auth and play around with using a frontend framework like Vue to make it a bit more human maintainable. Setup will be as follows:

Set up a new project, calling it diary
Set up a firestore in that project to function as the database
Set up Cloud Run to host the server

Very similar to the previous project we made, except, this time, the user will log into the application by authenticating with “Firebase Authentication” rather than with our own API.

First let’s set up the project and database

Setting up the project

Building a firestore database in us-central-1

Essentially, when a user accesses our application, the server sends the login page. That login page will then direct the user to create or log into an account via “Identity Platform”, Google Cloud’s managed service for user management and authentication. Identity Platform manages cryptography and security, allowing us to identify users with user IDs.

To get it set up, search “Identity Platform.”

and enable it

After a moment, the identity platform will be enabled in our project.

In providers, we’ll click “Add a provider”, and select “email/password”

Next, we’ll go back to “providers” and select “application setup details.”

Here we can see the code that we’ll need to inject into our front-end to enable identity platform authentication.

It’s worth noting, despite there being an API key, this is essentially public information. Users of our application will use this API key to talk to the authentication service. If they’re properly authenticated, they’ll receive a token that they can use to authenticate with our application.

Before we deploy our application, we can clone the repo and take a look at the code locally.

git clone https://github.com/DanielWarfield1/gcp-example3-diary

It has a very similar core structure to the previous application, except there’s some code in the website that the user uses to authenticate with Firebase Auth (AKA GCP Identity Provider).

mounted() {
  firebase.auth().onAuthStateChanged(async (fbUser) => {
    if (fbUser) {
      this.user = { uid: fbUser.uid, email: fbUser.email };
      this.idToken = await fbUser.getIdToken();
      this.error = ‘’;
      await this.fetchEntries();
    } else {
      this.user = null;
      this.idToken = null;
      this.entries = [];
    }
  });
},
methods: {
  async submitAuth() {
    this.error = ‘’;
    try {
      if (this.authMode === ‘login’) {
        await firebase.auth().signInWithEmailAndPassword(this.email, this.password);
      } else {
        await firebase.auth().createUserWithEmailAndPassword(this.email, this.password);
      }
      this.password = ‘’;
    } catch (err) {
      this.error = err.message;
    }
  },
  async logout() {
    await firebase.auth().signOut();
  },
  authHeaders() {
    return { Authorization: `Bearer ${this.idToken}`, ‘Content-Type’: ‘application/json’ };
    },

When a user talks to our server, they get this code that runs in their browser. This code talks to Firebase Auth directly. Thus, we don’t need to manage user credentials at all. They talk to Firebase and get back a token they can use to talk with our server.

This token is called a “JWT” token, and it’s a cryptographic token generated by Firebase. It’s practically impossible to make a valid JWT token without knowing the private key, which the identity platform keeps in isolation. Whenever a user communicates our server, they have to send this token that they got from identity provider. We can use this information to decode a user ID, and confirm it with Firebase to make sure the token is valid.

def get_uid_from_token(authorization: str) -> str:
    if not authorization.startswith(”Bearer “):
        raise HTTPException(status_code=401, detail=”Invalid authorization header”)
    id_token = authorization.removeprefix(”Bearer “)
    try:
        decoded = firebase_auth.verify_id_token(id_token)
    except Exception:
        raise HTTPException(status_code=401, detail=”Invalid or expired token”)
    return decoded[”uid”]

Thus, we have a unique ID for our user, and can ensure every request has valid credentials associated with it, without needing to manage any critical authentication information ourselves.

We can run a local firestore emulation so our server has a database to talk to

gcloud emulators firestore start --host-port=localhost:8080

and we can run our server

FIRESTORE_EMULATOR_HOST=localhost:8080 .venv/bin/uvicorn main:app --host 0.0.0.0 --port 8000

This spools up the server locally on localhost 8000, which hosts our application.

This user ID that we derive from the identity provider token is a unique identifier assigned to the user, which we can use to access user-specific data through the server, thus making our diary app private. If we create a new account

and add an entry

Then log out and log into a different account; we won’t see the entry from the other account.

It’s worth noting we aren’t and don’t need to emulate authentication. Because the authentication code is nested into the client code directly, and talks directly with the Google identity provider, it works with the real system regardless of if we’re running a local instance or if we deploy it on the cloud.

If we navigate to the providers section, we can see the ability to add multiple other providers if we wanted to support multiple signon methods

Now that we’ve confirmed that this stuff works, let’s deploy it. Just like last time, it’s as simple as spooling up a Cloud Run instance and configure it to point to the repos Docker File

And after a few minutes, it’s live and functional.

As a quick editorial note, I wanted to use firebase to make this example “easier”, which I think it is in some respects. However, I think the abstraction around firebase auth and firestore adds a level of confustion in this section. Future examples will use more GCP native approaches, which might be more intuitive.

Google Cloud Storage

In most cloud environments there are three key, fundamental types of storage: object storage, block storage, and file storage.

Block storage is essentially computer drives but on the cloud. You can rent discrete hard drive or SSD space and connect those to some compute resource. It’s essentially like making a big USB stick in the cloud, allowing you to connect it to different computers by mounting it directly to the operating system. It’s possible to use it directly, but it’s more common for it to be used behind the scenes, powering the storage used in your database or virtual machine.

The defining feature of a “block storage device” is that data is stored into fixed-size blocks throughout the storage medium. You might remember, back in the day, you had to “defragment” your hard drive, which involved moving data around to make your hard drive more efficient.

That’s because the hard drive on your computer is a block storage medium and, critically, back in the day, this data was read via a head that physically needed to move around

source

Block storage works by saving all data into fixed-size blocks. If you have a large file, it’s saved across numerous blocks. They don’t have to be close to each other, but when you have a hard drive that takes time to move around in physical space, having your data fragmented around your hard drive makes reads slower. Defragmentation is the process of moving stuff around so that data representing the same thing are next to each other. This isn’t practically required anymore, as storage has largely shifted to solid-state drives.

File storage is essentially an abstraction of block storage. You may have gotten a USB stick and needed to “format” it in the past. Essentially, raw block storage is great, but you might have several drives and want to organize the data onto your computer into a file system. Thus, you need to format block storage to hold data in a structured manner. File storage, on the cloud, is that at a cloud scale. Google File Store is a “network-attached storage” (NAS) product, where multiple machines can mount to the same network-attached storage device and access a shared pool of data simultaneously. This has some fun benefits and is used to do cool stuff, like allowing large sets of computers to train an AI system to quickly and efficiently read from a shared dataset without networking overhead.

GCS, Google Cloud Storage, is neither of these things. It’s a distributed object store that exposes an abstraction called “Blob Storage.” In blob storage, there is no folder structure. Instead, it’s a flat “key-value” storage system, where everything you upload has a specific name. These names can look like a folder structure, like “images/designs/set-design-1.pdf”, but in reality that entire URI is a key, not a specific subdirectory.

Because blob storage has a flat key-value organizational structure, it has some key advantages when storing certain types of data at scale:

Massive scalability. Since there is no hierarchical file system to manage, object stores can efficiently scale to billions or even trillions of objects spread across many storage servers.
High durability. Cloud providers automatically replicate objects across multiple disks, servers, and often multiple facilities, making it extremely unlikely for data to be lost.
Simple access patterns. Objects are retrieved directly by their key rather than traversing a directory tree. This makes blob storage ideal for applications that know exactly what object they want.
Cost efficiency. Blob storage is generally much cheaper than persistent disks or network file systems because it is optimized for storing large amounts of relatively static data.

As a result, blob storage is often used to store large files on a website, which might otherwise be expensive or cumbersome to store in a database, which is designed and optimized for a very different use case.

Because blob storage is chiefly designed for larger files, many cloud providers expose functionality allowing large files to be uploaded and downloaded from blob storage directly on the public internet. This is typically done with a “pre-signed URL”. Basically, if a user wants to upload a large file they can send a lightweight request to our server stating their intentions, then we can talk to blob storage and request a spot our user can put the data. Blob storage responds with a URL and a set of credentials, which are timeboxed and very restrictive, only allowing the user to upload that specific file. Our server can then pass that URL and the credentials to the user, and the user can upload files directly to blob storage.

This gets around a lot of nasty issues, HTTP timeouts, tying up server resources, or paying a high cost for network bandwidth. It also allows the storage provider to handle retries, multipart uploads, and other optimizations that make transferring large files more reliable. The result is a faster upload for the user, lower infrastructure costs, and a more scalable system overall.

Let’s explore a practical implementation. Let’s make a reading app that allows you to upload large PDF files, and read those PDF files. We’ll have Cloud Run serving as our core server, Firestore to keep track of application logic, and Google Cloud Storage to handle bulk file upload and download. We’ll also throw in a Cloud Run function that responds to upload events and generates a thumbnail for each of the PDFs so we can begin dabbling with event-driven architecture. Before we do that, though, let’s talk about the GCP CLI

The GCP CLI—gcloud

Our projects are starting to get a bit more complicated, and as a result, picking through the console UI to set up each resource is going to prove cumbersome. Plus, in our modern wave of AI-accelerated development, needing to navigate through a complex UI restricts the ability for AI assistance. Thus, in this example we’ll be using the command-line tool, gcloud, to set everything up.

Most cloud services have a command-line tool, and GCP is no exception. Instead of clicking a bunch of buttons, you can use the gcloud tool to create and manage resources via textual commands. This is useful for a whole bunch of reasons

commands can be copy-pasted, making re-production of similar resources simpler
AI can spit out these commands, accelerating our development
These can be automated, allowing us to manage our infrastructure using code
Things are generally more explicit and granular, allowing us to form a more thorough conceptualization of what’s going on under the hood

So, not only is the CLI convenient for educational purposes, it’s also practically very useful when managing large systems or keeping up with the AI-accelerated Joneses.

There are various ways to configure the CLI. You can set it up on your local machine or do it through the GCP dashboard. The dashboard is easier to set up and a simple way to get started, so we’ll use that until we have a reason to do otherwise. In our GCP dashboard, we can look to the top right and see a little terminal icon.

Then, after a moment and a bit of authentication stuff, we see a terminal.

We can kick things off by running

gcloud projects list

which gets me all of the projects I have on my account

PROJECT_ID: chat-app-500821
NAME: chat-app
PROJECT_NUMBER: 669756088814
ENVIRONMENT: 

PROJECT_ID: diary-500912
NAME: diary
PROJECT_NUMBER: 421502150200
ENVIRONMENT: 

PROJECT_ID: pdf-reader-502816
NAME: pdf-reader
PROJECT_NUMBER: 104419232710
ENVIRONMENT: 

PROJECT_ID: project-d946312c-060c-4b6d-83a
NAME: My First Project
PROJECT_NUMBER: 894485926663
ENVIRONMENT: 

PROJECT_ID: testproject1-500807
NAME: testproject1
PROJECT_NUMBER: 613513089702
ENVIRONMENT:

As you can see, I have a project called pdf-reader-502816 which I can set as my default project in the CLI. This will save us from needing to repeat this project ID over and over again for every command.

gcloud config set project pdf-reader-502816

You may recall that, for new projects, we needed to enable a few APIs to allow us to even set anything up. We can do that with the following command:

gcloud services enable run.googleapis.com firestore.googleapis.com storage.googleapis.com \
  cloudfunctions.googleapis.com eventarc.googleapis.com pubsub.googleapis.com \
  artifactregistry.googleapis.com cloudbuild.googleapis.com iam.googleapis.com \
  iamcredentials.googleapis.com identitytoolkit.googleapis.com

Often when a CLI gets used, things get much more explicit, which can be daunting at first. What are these services, and why are they necessary for this application? Well, here’s why:

run.googleapis.com:
Let’s us use cloudrun, which our API layer for the application will use

firestore.googleapis.com: 
Let’s use use firestore for storing user informaiton and metadata about our
books

storage.googleapis.com:
Let’s us use GCS, which is the whole point

cloudfunctions.googleapis.com:
Let’s us create cloudrun functions, which we’ll be using to create thumbnails

eventarc.googleapis.com:
This lets you connect a function call to some event on the cloud, which we’ll
be using to generate a thumbnail

pubsub.googleapis.com:
We haven’t talked about pubsub yet, but it’s used under the hood in eventrac,
so we need it

artifactregistry.googleapis.com:
When we use both cloudrun and cloud functions, we’re using containers.
Artifactregistry lets us store and manage containers, which is practically
required

cloudbuild.googleapis.com:
Our code needs to actually get turned into containers. We could do this ourselves
but we’ve been letting google take care of building our actual containers.
This is the resource that’s been doing that work.

iam.googleapis.com:
We need this for creating something called a “service account”, which is a
set of credentials allowing resources in our project to talk to one another

iamcredentials.googleapis.com:
This is required for creating presigned URLs

identitytoolkit.googleapis.com:
Required for using identity platform, which we’re using to manage our users
login credentials

Behind the scenes when we created stuff on the dashboard, a lot of these APIs were created when we enabled some service.

It’s worth noting we’re not using Firebase any more. After Google released and made popular Firebase, they migrated many of the offerings of Firebase into GCP. Firestore, the database technology underpinning Firebase, is now offered as a standalone product within GCP. Firebase Auth, the technology used for login, is now compatible with Identity Platform, which is GCP’s native identity system.

Next, we can set up a firestore database

gcloud firestore databases create --database=pdfreaderdb --location=us-central1

--database gives it a named ID that we’ll be using to connect with it later
--location sets the region for the database. Generally, resources that need to work together should be set in a similar location to reduce latency. I’m using us-centeral1 in this example.

Based on my research, setting up identity platform is a bit fiddly via the CLI, so we’ll do this step through the dashboard. To set it up we can search for identity platform

Enable it

Add a provider

Set up an email provider

and make sure it’s enabled

Then, once we’re all set up, we can get the authentication setup details. This gets us the API key and domain our front end will use to authenticate with Google’s identity provider.

note, it’s not enabled. That was a mistake I had to go back and fix. It should be enabled for authentication to work

If you’re playing along at home, These credentials were added into the front end code via firebaseConfig

const firebaseConfig = {
  apiKey: “AIzaSyBDoSerpprHIvB-eJn9siVfDlfp15C03nQ”,
  authDomain: “pdf-reader-502816.firebaseapp.com”,
};
firebase.initializeApp(firebaseConfig);

Next, we need to create a service account. We’ll talk about permissions later, but a service account is a method for us to apply specific permissions onto services within GCP, so we can control what talks to what.

gcloud iam service-accounts create pdfreader-runtime

After we create a google cloud storage bucket, which is that blob storage technology we discussed previously

gcloud storage buckets create gs://pdf-reader-502816-books --location=us-central1

We can add permissions to our service account, allowing it to access both Firestore and the GCS instance we created. This will allow our application layer to talk with these services, creating an explicit level of security in our application.

gcloud projects add-iam-policy-binding pdf-reader-502816 \
  --member=”serviceAccount:pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com” \
  --role=”roles/datastore.user”

gcloud storage buckets add-iam-policy-binding gs://pdf-reader-502816-books \
  --member=”serviceAccount:pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com” \
  --role=”roles/storage.objectAdmin”

In a real production setting it’s a good policy to make service accounts least privileged and to have specific service accounts for specific things. We’re going to talk about permissions to a greater degree of depth throughout the article. For now, I’m glossing over the specifics so we can form a general understanding.

Next, we need to add this to the service account that connects to our bucket so that we can generate signed URLs and pass them to our user.

gcloud iam service-accounts add-iam-policy-binding pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com \
  --member=”serviceAccount:pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com” \
  --role=”roles/iam.serviceAccountTokenCreator”

This allows our service account pdfreader-runtime to request the generation of a presigned URL. We’ll use this to create a URL that has a cryptographic key tied to it and allows the user with that key to do a specific action on a specific file within our cloud storage bucket. By passing a signed URL to the user, we can let GCS handle bulky uploads and downloads, rather than needing to manage that ourselves. This is a common paradigm in processing large files on the cloud.

Now we can set up Cloud Run. Recall that we previously set up Cloud Run to build directly with a GitHub integration. This could work perfectly fine for this use case, but I wanted to explore deploying in a slightly more explicit way so we can build an understanding of how containers actually get deployed under the hood. It will also be a bit easier to set up, though it won’t have some of the conveniences like automatic redeploy on updates to the codebase.

First, we need the code in our Cloud Shell so we can actually deploy it. You might need to fork the repository so you can configure it for your actual cloud environment.

git clone https://github.com/DanielWarfield1/gcp-example4-PDFreader.git
cd gcp-example4-PDFreader

Then we can run the following, which will tell GCP we want to create a cloud run compute instance based on the source code in the repo.

gcloud run deploy pdfreader --source . \
  --service-account=pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com \
  --region=us-central1

This will ask me if I want to create a registry for Docker containers, which I do. We’ll use this to store our dockerized application. Under the hood, this will use CloudBuild to build our docker container, then that build will be stored in the Artifact Registry. CloudRun will pull the build from the Artifact Registry to then deploy it.

Deploying from source requires an Artifact Registry Docker repository to store built containers. A repository named 
[cloud-run-source-deploy] in region [us-central1] will be created.

Do you want to continue (Y/n)?  Y

Then it asked me if I want to allow unauthenticated invocations. I said yes, because I want this to be exposed on the public internet. The application itself does authentication, but I want traffic from all sources to be accepted.

Allow unauthenticated invocations to [pdfreader] (y/N)?  Y

In the console output we’ll get a link to Cloud Build, where we can see GCP building our containerized application based on the dockerfile in the codebase. Again, I’ve covered docker in depth if you want to learn more about it.

Now, we can set up our cloud function that generates thumbnails for our uploaded PDFs. To do that, we’re going to need to set up some policies allowing the function to be triggered by key events in the bucket, like files being uploaded. This will trigger thumbnail generation when a file is uploaded.

gcloud projects add-iam-policy-binding pdf-reader-502816 \
  --member=”serviceAccount:pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com” \
  --role=”roles/eventarc.eventReceiver”

And we need to allow Google Cloud Storage to actually publish events that will trigger our function.

gcloud projects add-iam-policy-binding pdf-reader-502816 \
  --member=”serviceAccount:service-104419232710@gs-project-accounts.iam.gserviceaccount.com” \
  --role=”roles/pubsub.publisher”

So, now our function can respond to events, and our bucket can generate events. Once all those permissions are set up, we can go ahead and deploy our function.

gcloud functions deploy generate-thumbnail --gen2 --runtime=python312 \
  --region=us-central1 --source=functions/thumbnail --entry-point=generate_thumbnail \
  --trigger-bucket=pdf-reader-502816-books \
  --service-account=pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com \
  --memory=1Gi --timeout=300s

Breaking down the arguments for this command and what they do:

- --gen2 — use 2nd-generation Cloud Functions, which run on top of Cloud Run under the hood (rather than the older gen1 runtime). This matters because gen2 functions get Eventarc-based triggers instead of gen1’s more limited built-in trigger system — that’s why all the Eventarc/Pub/Sub IAM plumbing you just fought through is even involved.
- --runtime=python312 — the language runtime to build the function’s container with (Python 3.12). This determines things like which base image and package installer gets used during the Cloud Build step.
- --region=us-central1 — where the function itself runs, matching your bucket, Firestore database, and Cloud Run service, so nothing is talking cross-region.
- --source=functions/thumbnail — the directory to package and deploy. Points at the folder containing this main.py and its own requirements.txt/dependencies, not the repo root (this function is a self-contained deployable unit, separate from the FastAPI backend).
- --entry-point=generate_thumbnail — which Python function inside that source directory actually gets invoked. Matches def generate_thumbnail(cloud_event: CloudEvent) at functions/thumbnail/main.py:14 — the @functions_framework.cloud_event decorator is what makes it invocable as an event-driven function at all.
- --trigger-bucket=pdf-reader-502816-books — shorthand that wires up the whole Eventarc pipeline you just debugged: creates a Pub/Sub topic, configures the bucket to publish “object finalized” notifications into it, and creates an Eventarc trigger routing those events to this function. Every time an object finishes uploading to this bucket, cloud_event.data[”bucket”]/[”name”] (lines 15-16) get populated with that object’s info.
- --service-account=pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com — the identity the function runs as. This is why it needed roles/eventarc.eventReceiver (to receive the trigger) plus the existing roles/storage.objectAdmin and roles/datastore.user from step 6 (to read the uploaded PDF, write the thumbnail back, and update the Firestore doc — lines 31, 36-38, 40-46).
- --memory=1Gi --timeout=300s — resource limits: 1 GiB of memory and a 5-minute max execution time, sized generously for PDF rendering (fitz.open, get_pixmap) which can be memory-hungry for larger files.

And now that the generate-thumbnail function is created, we need to give the service account for pdfreader-runtime permission to invoke it.

gcloud run services add-iam-policy-binding generate-thumbnail \
  --region=us-central1 \
  --member=”serviceAccount:pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com” \
  --role=”roles/run.invoker”

Before we actually run this thing, there’s one other configuration we have to make now that everything is set up.

SERVICE_URL=$(gcloud run services describe pdfreader --region=us-central1 --format=”value(status.url)”)
echo “$SERVICE_URL”

cat > /tmp/cors.json <

And with that, everything is set up. We have a database, application layer, authentication, cloud storage, and a cloud function that automatically generates thumbnails when a PDF has been uploaded. We can find the URL for our application by finding the public URL of our pdfreader cloudrun service

gcloud run services describe pdfreader --region=us-central1 --format=”value(status.url)”

Then we can click on that URL, and see our application running online. We can create an account or log in

And see that we can upload PDFs. After a moment it will be populated with a thumbnail.

and if we click the PDF, we can read the whole file.

The objective of this article is to talk about the infrastructure rather than the application code, but I think it might be nice to explore the code, just a bit, to see how all this is working under the covers.

The application layer is defined in main.py. This serves as the core server that lives in the center of the application. The base path serves a static html page, which serves as the font end of the website

@app.get(”/”)
def index():
    return FileResponse(”static/index.html”)

Then there’s also a bunch of endpoints for manipulating data. The uid is associated with the user’s authorization credentials and saved in the database along with key data

@app.post(”/books”)
def create_book(req: CreateBookRequest, authorization: str = Header(...)):
    uid = get_uid_from_token(authorization)
    doc_ref = db.collection(”books”).document()
    book_id = doc_ref.id
    pdf_path = f”pdfs/{uid}/{book_id}.pdf”
    doc_ref.set({
        “uid”: uid,
        “title”: req.filename,
        “status”: “processing”,
        “pdf_path”: pdf_path,
        “thumbnail_path”: None,
        “page_count”: None,
        “created_at”: datetime.now(UTC),
    })
    return {”book_id”: book_id, “upload_url”: generate_upload_url(pdf_path)}

so when the user attempts to retrieve data, we only give back the data associated with the user

@app.get(”/books”)
def list_books(authorization: str = Header(...)):
    uid = get_uid_from_token(authorization)
    docs = db.collection(”books”).where(”uid”, “==”, uid).stream()
    books = []
    for doc in docs:
        data = doc.to_dict()
        thumbnail_url = (
            generate_download_url(data[”thumbnail_path”])
            if data.get(”thumbnail_path”)
            else None
        )
        books.append({
            “book_id”: doc.id,
            “title”: data[”title”],
            “status”: data[”status”],
            “thumbnail_url”: thumbnail_url,
            “created_at”: data[”created_at”].isoformat(),
        })
    return sorted(books, key=lambda b: b[”created_at”], reverse=True)

There’s also a few helper functions for getting presigned URLs for uploading and downloading, which are used to pass this information to the client.

def generate_upload_url(object_name: str) -> str:
    blob = storage_client.bucket(BUCKET_NAME).blob(object_name)
    return blob.generate_signed_url(
        version=”v4”,
        expiration=dt.timedelta(minutes=15),
        method=”PUT”,
        content_type=”application/pdf”,
        credentials=_signing_credentials,
    )


def generate_download_url(object_name: str) -> str:
    blob = storage_client.bucket(BUCKET_NAME).blob(object_name)
    return blob.generate_signed_url(
        version=”v4”,
        expiration=dt.timedelta(minutes=15),
        method=”GET”,
        response_disposition=”inline”,
        credentials=_signing_credentials,
    )

The function that generates thumbnails, and what we deployed as a cloud function responding to upload events, looks like this:

import functions_framework
from cloudevents.http import CloudEvent
from google.cloud import firestore, storage
import fitz  # PyMuPDF

PROJECT_ID = “pdf-reader-502816”
FIRESTORE_DATABASE = “pdfreaderdb”

storage_client = storage.Client(project=PROJECT_ID)
db = firestore.Client(project=PROJECT_ID, database=FIRESTORE_DATABASE)


@functions_framework.cloud_event
def generate_thumbnail(cloud_event: CloudEvent):
    bucket_name = cloud_event.data[”bucket”]
    object_name = cloud_event.data[”name”]

    # Ignore anything that isn’t a freshly-uploaded PDF, including the
    # thumbnails this function itself writes back into the same bucket.
    if not object_name.startswith(”pdfs/”):
        return
    parts = object_name.split(”/”)
    if len(parts) != 3 or not parts[2].endswith(”.pdf”):
        return
    uid, book_id = parts[1], parts[2].removesuffix(”.pdf”)

    doc_ref = db.collection(”books”).document(book_id)
    bucket = storage_client.bucket(bucket_name)

    try:
        pdf_bytes = bucket.blob(object_name).download_as_bytes()
        pdf = fitz.open(stream=pdf_bytes, filetype=”pdf”)
        pixmap = pdf.load_page(0).get_pixmap(matrix=fitz.Matrix(0.5, 0.5))

        thumbnail_path = f”thumbnails/{uid}/{book_id}.png”
        bucket.blob(thumbnail_path).upload_from_string(
            pixmap.tobytes(”png”), content_type=”image/png”
        )

        doc_ref.update({
            “status”: “ready”,
            “thumbnail_path”: thumbnail_path,
            “page_count”: pdf.page_count,
        })
    except Exception:
        doc_ref.update({”status”: “error”})
        raise

Naturally, with the function reacting to upload events, and this function also triggering upload events, there’s an opportunity for an infinite loop as the system reacts to the uploads it itself triggered. We nip that in the bud by only reacting to freshly uploaded PDF files, and not the thumbnails we’re uploading.

if not object_name.startswith(”pdfs/”):
  return

If it is a pdf, we generate a thumbnail and add it to the bucket. We also update the database as to whether we were successful or not. The frontend picks up the new thumbnail via polling. When we upload a new PDF we poll every 4 seconds while any of the books we’re uploading are in the processing state on the database.

startPolling() {
  if (this.pollTimer) return;
  this.pollTimer = setInterval(async () => {
    if (this.books.some((b) => b.status === ‘processing’)) {
      await this.fetchBooks();
    } else {
      this.stopPolling();
    }
  }, 4000);
},

In the front end code, we find their state by using the fetchBooks function

async fetchBooks() {
  const res = await fetch(’/books’, { headers: this.authHeaders() });
  if (res.status === 401) return firebase.auth().signOut();
  if (res.ok) {
    this.books = await res.json();
  } else {
    this.error = ‘Failed to load books’;
  }
},

Which is hooked up to the list_book function on the API layer, which we previously discussed.

This app is pretty simple, but it could scale to a very large user base very easily. Because the users carry all of the state with them (i.e. who they are and what they want), the API layer is completely stateless. That means a user could make an API request from one server and then another server, and it wouldn’t matter, as both servers are talking to the database. As a result, the API layer could scale horizontally, adding more and more computers, and be able to serve a practically unlimited number of users.

Firestore and GCS are both designed to handle this type of workload at a massive scale, so they can handle a huge amount of throughput as well. Same with our cloud function; we can always provision the cloud function to be able to execute a large amount of parallel execution. If we have the pockets deep enough, any of these systems can scale pretty much infinitely.

The main issue for this system, in terms of reliability, isn’t scale but provisioning time. We can scale all these things up if we want to, allowing these resources to provision more computers to handle the load, but that takes time. If we get a huge flush of uploads all at the same time, it will likely crash our servers. The next topic, queuing, is designed to help us solve that problem.

Queuing

Sometimes you have a lot of work to do and not enough compute to do it. For instance, imagine a user uploads a stack of thousands of markdown files that you need to convert to PDF. If you push all these requests to a server simultaneously, forcing it to hold all these documents in memory and process them at the same time, you will likely run out of memory and crash that server.

Instead, you can upload all of the documents onto a task queue. A task queue is a system that can hold onto a list of tasks, acting as a buffer between the thing making requests and the thing fulfilling requests. If a lot of requests get made all at once, too much for our service to fulfill, we can add them to a queue to store the requests, which feeds them to our system at a steady rate. This context is commonly referred to as a “producer-consumer” system, where the “producer” is the thing making the requests, and the “consumer” is the thing fulfilling the requests.

A Task Queue stores tasks and releases them at a steady rate; allowing a system to smooth out traffic spikes.

There are many ways to achieve the idea of queuing, which can generally be broken down into two families: push queues and pull queues. In a push system, the queue decides when to push requests to the consumer, while pull systems require the consumer to request tasks then mark them as completed. Cloud Tasks is a push-based system (Pub/Sub, which we’ll explore later, supports both push and pull delivery), which means Cloud Tasks decides when to send tasks to consumers based on configurable rate limits.

Let’s explore cloud tasks by making a system to compute the “Collatz conjecture”, otherwise known as 3n+1. If you’re not familiar, the Collatz conjecture has two key rules:

if the input number is even, divide by two
if the number is odd, multiply by three and add one

The idea is that, by following these rules over successive iterations, any input number will eventually converge to 1. It’s a pretty famous problem in math, and is somewhat of a boogeyman in the field

Mathematics is not yet ripe enough for such questions. — Paul Erdos

To help our mathematician friends out, we’re going to make a handy online tool that allows users to upload a bunch of numbers, and we’ll tell them how many cycles it takes for the number to converge to 1.

The issue with this system is that I expect some spikey traffic. It’s likely a user will upload a large quantity of numbers simultaneously, and I want to smooth that out so we can handle a large amount of intermittent traffic with a single machine. We’re going to use a queue to help us smooth out this traffic. We’re also going to be deploying everything this time with a single command.

Recall, in previous examples, we used console commands to deploy the infrastructure. Well, we can wrap the commands necessary to launch everything in a single bash file, then just deploy everything by running the bash file.

It’s the same stuff going on behind the scenes, but now everything get’s built in a single line on the command line, which is pretty cool. This is the full file, in all its glory.

#!/usr/bin/env bash
set -euo pipefail

cd “$(dirname “$0”)/..”

if [[ -f .env ]]; then
  set -a
  # shellcheck disable=SC1091
  source .env
  set +a
fi

# --- Configuration (env vars or .env; see .env.example) ---
PROJECT_ID=”${PROJECT_ID:?Set PROJECT_ID (export it or put it in .env)}”
REGION=”${REGION:-us-central1}”
QUEUE_NAME=”${QUEUE_NAME:-collatz-queue}”
WORKER_SERVICE=”${WORKER_SERVICE:-collatz-worker}”
API_SERVICE=”${API_SERVICE:-collatz-api}”
TASKS_SA=”${TASKS_SA:-collatz-tasks-invoker}”
CHUNK_SIZE=”${CHUNK_SIZE:-25}”
MAX_DISPATCHES_PER_SECOND=”${MAX_DISPATCHES_PER_SECOND:-5}”
MAX_CONCURRENT_DISPATCHES=”${MAX_CONCURRENT_DISPATCHES:-10}”

# Runs a gcloud “create” command; treats “already exists” as success and
# anything else as a real failure instead of silently continuing.
create_if_missing() {
  local err
  if ! err=”$(”$@” 2>&1)”; then
    if echo “$err” | grep -qi “already exists”; then
      echo “    (already exists, continuing)”
    else
      echo “$err” >&2
      exit 1
    fi
  fi
}

gcloud config set project “$PROJECT_ID”
gcloud auth application-default set-quota-project “$PROJECT_ID” 2>/dev/null || \
  echo “    (skipping ADC quota project — run ‘gcloud auth application-default login’ if source deploys fail with permission errors)”

echo “==> Enabling required APIs”
gcloud services enable \
  run.googleapis.com \
  cloudtasks.googleapis.com \
  firestore.googleapis.com \
  cloudbuild.googleapis.com \
  artifactregistry.googleapis.com

echo “==> Ensuring a Firestore (Native mode) database exists”
create_if_missing gcloud firestore databases create --location=”$REGION”

echo “==> Creating service account used by Cloud Tasks to invoke the worker”
create_if_missing gcloud iam service-accounts create “$TASKS_SA” \
  --display-name “Cloud Tasks -> Collatz worker invoker”
TASKS_SA_EMAIL=”${TASKS_SA}@${PROJECT_ID}.iam.gserviceaccount.com”

echo “==> Deploying worker service (private — only Cloud Tasks may invoke it)”
gcloud run deploy “$WORKER_SERVICE” \
  --source ./worker \
  --region “$REGION” \
  --no-allow-unauthenticated
WORKER_URL=”$(gcloud run services describe “$WORKER_SERVICE” --region “$REGION” --format ‘value(status.url)’)”

echo “==> Granting the Cloud Tasks service account permission to invoke the worker”
gcloud run services add-iam-policy-binding “$WORKER_SERVICE” \
  --region “$REGION” \
  --member “serviceAccount:${TASKS_SA_EMAIL}” \
  --role “roles/run.invoker”

echo “==> Creating the Cloud Tasks queue with a dispatch rate limit”
create_if_missing gcloud tasks queues create “$QUEUE_NAME” \
  --location “$REGION” \
  --max-dispatches-per-second “$MAX_DISPATCHES_PER_SECOND” \
  --max-concurrent-dispatches “$MAX_CONCURRENT_DISPATCHES”

echo “==> Deploying API service (public entry point)”
gcloud run deploy “$API_SERVICE” \
  --source ./api \
  --region “$REGION” \
  --allow-unauthenticated \
  --set-env-vars “PROJECT_ID=${PROJECT_ID},LOCATION=${REGION},QUEUE_NAME=${QUEUE_NAME},WORKER_URL=${WORKER_URL}/process,WORKER_SERVICE_ACCOUNT=${TASKS_SA_EMAIL},CHUNK_SIZE=${CHUNK_SIZE}”

API_SA_EMAIL=”$(gcloud run services describe “$API_SERVICE” --region “$REGION” --format ‘value(spec.template.spec.serviceAccountName)’)”

echo “==> Granting the API service’s runtime service account permission to enqueue tasks”
gcloud tasks queues add-iam-policy-binding “$QUEUE_NAME” \
  --location “$REGION” \
  --member “serviceAccount:${API_SA_EMAIL}” \
  --role “roles/cloudtasks.enqueuer”

API_URL=”$(gcloud run services describe “$API_SERVICE” --region “$REGION” --format ‘value(status.url)’)”
echo “==> Done.”
echo “    API URL: ${API_URL}”
echo “    Try:     python client/collatz_client.py --url ${API_URL} --count 200”

It starts off with some Bash-specific stuff. Taking it section by section:

This makes the script fail loudly if it bumps into issues, and exit immediately, which is a good idea if you’re deploying things on the cloud. Otherwise you might burn a bunch of time and create a bunch of resources that don’t work.

set -euo pipefail

Then, this makes the script locationally independent. It doesn’t matter where you are when you call this script; the contents refer to the location of the bash file itself.

cd “$(dirname “$0”)/..”

Then, there’s an optional .env file that can be used to define key values that this script uses. This checks if the file exists then, if it does, sets the key-value fields in the .env file to actual environment variables so that the script can use them.

if [[ -f .env ]]; then
  set -a
  # shellcheck disable=SC1091
  source .env
  set +a
fi

After that, we get all our environment variables and save them as variables that we can use throughout the script.

# --- Configuration (env vars or .env; see .env.example) ---
PROJECT_ID=”${PROJECT_ID:?Set PROJECT_ID (export it or put it in .env)}”
REGION=”${REGION:-us-central1}”
QUEUE_NAME=”${QUEUE_NAME:-collatz-queue}”
WORKER_SERVICE=”${WORKER_SERVICE:-collatz-worker}”
API_SERVICE=”${API_SERVICE:-collatz-api}”
TASKS_SA=”${TASKS_SA:-collatz-tasks-invoker}”
CHUNK_SIZE=”${CHUNK_SIZE:-25}”
MAX_DISPATCHES_PER_SECOND=”${MAX_DISPATCHES_PER_SECOND:-5}”
MAX_CONCURRENT_DISPATCHES=”${MAX_CONCURRENT_DISPATCHES:-10}”

The following is chiefly a quality of life feature:

# Runs a gcloud “create” command; treats “already exists” as success and
# anything else as a real failure instead of silently continuing.
create_if_missing() {
  local err
  if ! err=”$(”$@” 2>&1)”; then
    if echo “$err” | grep -qi “already exists”; then
      echo “    (already exists, continuing)”
    else
      echo “$err” >&2
      exit 1
    fi
  fi
}

This function runs an input command and captures any errors. If the error is that the resource already exists, it treats it as a non-error. If the error is about something else, the error is raised. This makes this script “declarative,” meaning it doesn’t define a set of procedures but rather declares what ought to exist. The idea of declaration comes up a lot in infrastructure as code, which I imagine we’ll cover to a greater degree of depth in future articles.

Next, we do some configuration

gcloud config set project “$PROJECT_ID”
gcloud auth application-default set-quota-project “$PROJECT_ID” 2>/dev/null || \
  echo “    (skipping ADC quota project — run ‘gcloud auth application-default login’ if source deploys fail with permission errors)”

The first one, we’ve seen before. This sets our project as the default project, using whatever we specified in the .env file. In this case, that’s collatz-503002. The next one is a practical requirement for local deployment, and has to do with quirks about uploading local source code when errors like PERMISSION_DENIED: … API requires a quota project, which is not set by default. We need to set our project id to be the “quota project” to get around this issue.

Now we get to some familiar territory. We enable a few APIs

echo “==> Enabling required APIs”
gcloud services enable \
  run.googleapis.com \
  cloudtasks.googleapis.com \
  firestore.googleapis.com \
  cloudbuild.googleapis.com \
  artifactregistry.googleapis.com

This requires similar APIs we defined in the previous example, with the notable addition of cloudtasks, which we’ll use to make our task queue.

After that we create a firestore database

echo “==> Ensuring a Firestore (Native mode) database exists”
create_if_missing gcloud firestore databases create --location=”$REGION”

Then we need to create a worker, create a service account allowing cloud tasks to invoke that worker, and create the task queue itself. We’ll start by creating the service account for our task queue, which we can attach permissions to.

echo “==> Creating service account used by Cloud Tasks to invoke the worker”
create_if_missing gcloud iam service-accounts create “$TASKS_SA” \
  --display-name “Cloud Tasks -> Collatz worker invoker”
TASKS_SA_EMAIL=”${TASKS_SA}@${PROJECT_ID}.iam.gserviceaccount.com”

And we’ll deploy the worker, not exposing it to the public internet. We’ll also find the URL and store it so other services can talk to it, provided they have a properly configured service account.

echo “==> Deploying worker service (private — only Cloud Tasks may invoke it)”
gcloud run deploy “$WORKER_SERVICE” \
  --source ./worker \
  --region “$REGION” \
  --no-allow-unauthenticated
WORKER_URL=”$(gcloud run services describe “$WORKER_SERVICE” --region “$REGION” --format ‘value(status.url)’)”

Now that we have a worker, we can grant our service account access to it. This command binds the role of run.invoker for the $WORKER_SERVICE , which we just created, to the TASKS_SA_EMAIL service account. This “email”, not an email at all just a quirk of naming, references the service account in charge of authenticating between the task queue and the worker.

echo “==> Granting the Cloud Tasks service account permission to invoke the worker”
gcloud run services add-iam-policy-binding “$WORKER_SERVICE” \
  --region “$REGION” \
  --member “serviceAccount:${TASKS_SA_EMAIL}” \
  --role “roles/run.invoker”

Now we can create the queue itself. This will have dispatch rate limits defined, which allows us to achieve our smoothing effect over spikey input traffic.

echo “==> Creating the Cloud Tasks queue with a dispatch rate limit”
create_if_missing gcloud tasks queues create “$QUEUE_NAME” \
  --location “$REGION” \
  --max-dispatches-per-second “$MAX_DISPATCHES_PER_SECOND” \
  --max-concurrent-dispatches “$MAX_CONCURRENT_DISPATCHES”

We’ll also create the “API Service”, which is the public server that actually accepts requests to our system. This service’s only job is to accept requests and rapidly fulfill them, either by querying the database or by forwarding jobs to the queue.

echo “==> Deploying API service (public entry point)”
gcloud run deploy “$API_SERVICE” \
  --source ./api \
  --region “$REGION” \
  --allow-unauthenticated \
  --set-env-vars “PROJECT_ID=${PROJECT_ID},LOCATION=${REGION},QUEUE_NAME=${QUEUE_NAME},WORKER_URL=${WORKER_URL}/process,WORKER_SERVICE_ACCOUNT=${TASKS_SA_EMAIL},CHUNK_SIZE=${CHUNK_SIZE}”

Notice that there are a variety of environment variables assigned on the deployment. These allow the code in the API service to leverage these environment variables to reference these resources and configurations.

We then need to define a service account that allows our API service to add things into the queue. We define that binding here.

API_SA_EMAIL=”$(gcloud run services describe “$API_SERVICE” --region “$REGION” --format ‘value(spec.template.spec.serviceAccountName)’)”

echo “==> Granting the API service’s runtime service account permission to enqueue tasks”
gcloud tasks queues add-iam-policy-binding “$QUEUE_NAME” \
  --location “$REGION” \
  --member “serviceAccount:${API_SA_EMAIL}” \
  --role “roles/cloudtasks.enqueuer”

This actual command is some AI-generated shenanigans that I think are worth discussing. Instead of explicitly creating a service account and assigning it to our API service, we’re instead just adding a policy to the default service account assigned to our API service on creation.

--member “serviceAccount:${API_SA_EMAIL}” \

This is fine for a little demo, but it has some major security concerns. GCP doesn’t create a unique service account for every new instance of Cloud Run, but instead shares the same default service account. That means, by creating this policy binding, we allowed any cloud function, GKE cluster, or other Cloud Run service to enqueue to the task.

This approach isn’t recommended, as it has obvious security concerns. It’s much more advisable to create dedicated service accounts to facilitate communication between specific resources. For this application, though, it’s fine.

Finally, this gets the URL of the public API service, which serves as the entry point into the system.

API_URL=”$(gcloud run services describe “$API_SERVICE” --region “$REGION” --format ‘value(status.url)’)”
echo “==> Done.”
echo “    API URL: ${API_URL}”
echo “    Try:     python client/collatz_client.py --url ${API_URL} --count 200”

There’s no UI tied to this project. Instead, I created a python client that can send a bunch of numbers to a URL, and waits for the backend to finish processing the job.

import time

import requests


class CollatzClient:
    def __init__(self, base_url, poll_interval=2.0, timeout=120.0):
        self.base_url = base_url.rstrip(”/”)
        self.poll_interval = poll_interval
        self.timeout = timeout

    def submit(self, numbers):
        resp = requests.post(f”{self.base_url}/submit”, json={”numbers”: numbers})
        resp.raise_for_status()
        return resp.json()[”job_id”]

    def status(self, job_id):
        resp = requests.get(f”{self.base_url}/status/{job_id}”)
        resp.raise_for_status()
        return resp.json()

    def wait(self, job_id):
        start = time.monotonic()
        while True:
            data = self.status(job_id)
            if data[”status”] == “complete”:
                return data[”results”]
            if time.monotonic() - start > self.timeout:
                raise TimeoutError(f”job {job_id} did not complete within {self.timeout}s”)
            time.sleep(self.poll_interval)

    def compute(self, numbers):
        job_id = self.submit(numbers)
        return self.wait(job_id)


if __name__ == “__main__”:
    import argparse
    import random

    parser = argparse.ArgumentParser(description=”Submit numbers to the Collatz queue API”)
    parser.add_argument(”--url”, required=True, help=”Base URL of the API service”)
    parser.add_argument(”--count”, type=int, default=100, help=”How many random numbers to send”)
    parser.add_argument(”--max”, type=int, default=1_000_000, help=”Max value for random numbers”)
    args = parser.parse_args()

    numbers = [random.randint(1, args.max) for _ in range(args.count)]
    client = CollatzClient(args.url)

    print(f”Submitting {len(numbers)} numbers...”)
    results = client.compute(numbers)
    for r in results:
        print(f”{r[’number’]}: {r[’steps’]} steps”)

We can run that function, with some arguments specifying the url we want to hit and a count for the number of random numbers we want to compute, and get an answer in a few seconds.

oxanne@roxanne-AMD-Ryzen-7-9700X-8-Core-Processor:~/Documents/Github/gcp-example5-collatz$ python3 client/coll
atz_client.py --url https://collatz-api-142557322943.us-central1.run.app --count 200
Submitting 200 numbers...
107618: 141 steps
750052: 110 steps
901988: 232 steps
397743: 117 steps
404336: 68 steps
347530: 166 steps
147499: 219 steps
...

We can do this for a bunch more numbers and still get a result, just after waiting a bit longer.

roxanne@roxanne-AMD-Ryzen-7-9700X-8-Core-Processor:~/Documents/Github/gcp-example5-collatz$ python3 client/collatz_client.py --url https://collatz-api-142557322943.us-central1.run.app --count 20000
Submitting 20000 numbers...
288359: 114 steps
361443: 65 steps
571680: 146 steps
735096: 149 steps
894310: 95 steps
151479: 108 steps
174097: 72 steps
306038: 202 steps
...

One issue with this system is rate limiting and a lack of backpressure. We’re protecting our infrastructure, but anyone can blast an arbitrary number of requests to our system on the public internet, slowing our system down and exposing us to undue financial risk. We know we can implement authentication to restrict who is talking to our system, but there’s no way to deal with one authenticated person sending too much traffic.

Rate Limiting with Redis

It’s worth noting that Redis isn’t always going to be the right choice for implementing rate limiting. Before we dive into implementing rate limiting with Redis, I want to discuss what Redis is, how rate limiting is commonly implemented, and when we might opt not to use Redis and instead use the database we already have.

I’m planning on doing an in-depth guide to Redis; it’s actually many technologies with a variety of interesting applications, but the most common application of Redis is in caching. Redis is essentially a very, very fast database that you can use to store information you need to read from and update frequently, saving the time and computational load in querying a larger database or triggering some downstream process by instead returning a value that’s been stored close to the API layer itself.

Imagine our users of our “Collatz conjecture” calculation system. They noticed their kids saying the number 67 over and over a few months ago, and they’re still coming to terms with the number’s implications from a mathematical perspective. So, they ask our system how many iterations it takes for the number to converge to 1. The answer is 27 steps. Hundreds of confused mathematicians grappling with the mystery of gen alpha are asking the same exact question over and over again.

A natural idea would be to put our answers in the database. So, any time we get a query of 67 we check if we’ve already solved it and return that. That’s actually a really good solution, but as the number of users grows, this puts a significant strain on our database. Millions of people, from around the globe, are hammering our database trying to find out how many iterations it takes of the Collatz conjecture for 67 to converge to 1.

It’s in these types of environments that caching is useful. The idea is to store the hottest values in their own highly efficient in-memory database, then check that database before checking the big, expensive database. Redis is specifically designed to satisfy these low-latency applications.

Redis is a “single-threaded in-memory data structure store.” Again, I don’t want to go too far in the weeds, but in a nutshell, it’s super good at quickly reading and writing small files. Because it’s in memory (on RAM) it’s not particularly resilient (if the Redis instance goes down, you lose your data, unless you’re using a fancy persistent tier). Thus, Redis shouldn’t generally be used as a primary database, but it’s great to use as a caching layer for things that don’t matter if we lose them, or things that are stored in the database but we just want a faster way to look them up.

For rate limiting, Redis is great. Any time a user sends a query, we can increment a counter on Redis for that user. When that user’s been allocated more resources, we can decrement that counter. Thus, we can keep track of specific users’ usage rates and reject traffic if their request frequency is too high. If Redis goes down once in a blue moon, whatever, people get their usage rate reset; no big deal (especially because we have a task queue defending our system generally).

One drawback with this approach is in the nature of Redis. We’ve been using cloud run with autoscaling, which has the nice quality of being able to scale down to zero instances. Both our servers and our functions only run when we need them to. Redis can’t function like this; its entire job is to be an in-memory database, meaning it can’t be scaled down to zero. As a result, we need to provision and constantly pay for a certain allocation of resources. This means using Redis isn’t the guaranteed right move for every application. In fact, for this one it’s probably overkill (we could just use the database to keep track of usage, or we could use GCS, either would probably work perfectly. Hell, the API layer is so light, we could probably use a single machine and just keep the rate-limiting information in memory and just scale our application vertically if needed), but for big applications with a lot of traffic and horizontal scaling Redis caching is a common paradigm, so I figured we’d give it a shot for educational purposes.

Here’s the code we’ll be working with

https://github.com/DanielWarfield1/gcp-example6-collatzquota

Because we’re using authentication, we need to set that up through the dashboard. So, head over to identity provider and set up a new email provider, just like we’ve done a few times.

We’re going to need the application setup details

I opted to just hard-code these in the frontend. In reality it’s probably sleeker to define these as environment variables, but I won’t tell anyone if you don’t.

This project is very similar to the project we discussed previously; we have the same core architecture and application code, but we also deploy a caching layer. The full deployment code can be found here.

Going through some of the changes, we have a few new environment variables

RATE_LIMIT_CAPACITY=”${RATE_LIMIT_CAPACITY:-50}”
RATE_LIMIT_WINDOW_SECONDS=”${RATE_LIMIT_WINDOW_SECONDS:-60}”
API_KEY_CACHE_TTL_SECONDS=”${API_KEY_CACHE_TTL_SECONDS:-60}”
LAST_USED_FLUSH_INTERVAL_SECONDS=”${LAST_USED_FLUSH_INTERVAL_SECONDS:-60}”
REDIS_INSTANCE=”${REDIS_INSTANCE:-collatzquota-redis}”
REDIS_TIER=”${REDIS_TIER:-basic}”
REDIS_SIZE_GB=”${REDIS_SIZE_GB:-1}”
VPC_CONNECTOR=”${VPC_CONNECTOR:-collatzquota-connector}”
VPC_CONNECTOR_RANGE=”${VPC_CONNECTOR_RANGE:-10.8.0.0/28}”

These define the following:

RATE_LIMIT_CAPACITY: The quantity of numbers an individuals can upload within a given rate-limiting window
RATE_LIMIT_WINDOW_SECONDS: The amount of time it takes for a user’s rate-limiting capacity to be refilled. It’s also used in clearing data from Redis, which we’ll discuss later.
API_KEY_CACHE_TTL_SECONDS: A cache to store API key authentication, limiting round-trip passes to the database. This, specifically, stores how long that cache exists before the database needs to be checked.
LAST_USED_FLUSH_INTERVAL_SECONDS: When we’re using an API key, we’re updating the database with when the API key was last used. We don’t want to write to the database for every single request; this serves as a rate limiter, minimizing how often we update the last used field for the API.
REDIS_INSTANCE: Just the name of the instance
REDIS_TIER: We’re using Google Cloud Memorystore, which hosts Redis for us. Memorystore has two tiers: basic, which just has one Redis node with no failover, and standard, which employs multiple Redis instances with failover. Standard is best when making sure Redis is maximally available, but we’ll use basic because failure of the cache doesn’t significantly harm our application and it costs less money. In a real production setting, the choice of exactly what to use would be based on the product itself and the needs of the customer.
REDIS_SIZE_GB: The amount of memory tied to the Redis cluster. This is, in effect, the size of the Redis database.
VPC_CONNECTOR: Necessary for getting our API service running on Cloud Run to talk to our Redis cache. We’ll talk about this more in a bit.
VPC_CONNECTOR_RANGE: Also used in communication; we’ll talk about this in a sec

Then, the next thing that changed is the API enablement

echo “==> Enabling required APIs”
gcloud services enable \
  run.googleapis.com \
  cloudtasks.googleapis.com \
  firestore.googleapis.com \
  cloudbuild.googleapis.com \
  artifactregistry.googleapis.com \
  identitytoolkit.googleapis.com \
  redis.googleapis.com \
  vpcaccess.googleapis.com \
  compute.googleapis.com

New additions are

identitytoolkit.googleapis.com: For hooking up identity toolkit and allow for user authentication
redis.googleapis.com: For creating Redis clusters
vpcaccess.googleapis.com: For getting our API to talk to Redis
compute.googleapis.com: Also for getting our API to talk to Redis

The next addition is for actually creating Redis


echo “==> Creating the Memorystore Redis instance (first run can take several minutes)”
create_if_missing gcloud redis instances create “$REDIS_INSTANCE” \
  --region “$REGION” \
  --tier “$REDIS_TIER” \
  --size “$REDIS_SIZE_GB” \
  --redis-version redis_7_0

This spools up a Redis instance and connects it to the VPC of the project. We haven’t talked about VPCs, so we should probably get into that now.

Previously, we’ve been using managed services that have an API and IAM roles built in. When we create a service account to connect two services together, Google Cloud manages making sure the connection between those two services is secure. Redis is an open-source software and doesn’t have those conveniences built into it. so we need to connect it to other resources ourselves.

A VPC, or virtual private cloud, allows us to do that securely. It’s a private connected cloud, where services can talk to one another directly. When we create a project, a VPC is automatically created. Redis is automatically exposed to the VPC, like we plugged an Ethernet cable into it on our home network. This is essentially required to make Redis do anything; because it’s not a fully fledged service with an API, it needs to be connected to some VPC so it can talk to other resources.

CloudRun, on the other hand, does not connect to our VPC by default. CloudRun is a serverless managed service that has an API, compute resources, and networking built in and thus doesn’t necessarily need to be connected to a VPC to do something. To get it to talk to Redis, we need to plug it into the same VPC that Redis is connected to. We do that configuration by creating a “VPC-access connector,” which is like plugging CloudRun into the same VPC our Redis account was connected to by default.

echo “==> Creating the Serverless VPC Access connector (Cloud Run’s route to Memorystore)”
create_if_missing gcloud compute networks vpc-access connectors create “$VPC_CONNECTOR” \
  --region “$REGION” \
  --range “$VPC_CONNECTOR_RANGE”

REDIS_HOST=”$(gcloud redis instances describe “$REDIS_INSTANCE” --region “$REGION” --format ‘value(host)’)”
REDIS_PORT=”$(gcloud redis instances describe “$REDIS_INSTANCE” --region “$REGION” --format ‘value(port)’)”

echo “==> Deploying API service (public entry point; needs the VPC connector to reach Redis)”
gcloud run deploy “$API_SERVICE” \
  --source ./api \
  --region “$REGION” \
  --allow-unauthenticated \
  --vpc-connector “$VPC_CONNECTOR” \
  --vpc-egress private-ranges-only \
  --set-env-vars “PROJECT_ID=${PROJECT_ID},LOCATION=${REGION},QUEUE_NAME=${QUEUE_NAME},WORKER_URL=${WORKER_URL}/process,WORKER_SERVICE_ACCOUNT=${TASKS_SA_EMAIL},CHUNK_SIZE=${CHUNK_SIZE},REDIS_HOST=${REDIS_HOST},REDIS_PORT=${REDIS_PORT},RATE_LIMIT_CAPACITY=${RATE_LIMIT_CAPACITY},RATE_LIMIT_WINDOW_SECONDS=${RATE_LIMIT_WINDOW_SECONDS},API_KEY_CACHE_TTL_SECONDS=${API_KEY_CACHE_TTL_SECONDS},LAST_USED_FLUSH_INTERVAL_SECONDS=${LAST_USED_FLUSH_INTERVAL_SECONDS}”

There are some networking specifics in setting this up, which I think are useful to discuss.

Under the hood, there’s a computer running Redis somewhere on the cloud, and our API needs to talk to it. They’re essentially plugged into the same home router and can talk to each other. To do that, though, they need an IP and a port.

An IP address consists of four numbers from 0–255 that serve as the street number of a computer. There are public IPs and private IPs, but because we’re in a virtual private cloud, we’re only concerned with private IPs. Different computers on a network talk to each other by IP.

When a program runs on one of those computers, it can expose itself to connected computers by connecting to a port. Multiple programs can be operating on multiple different ports on a single computer. If I have a computer, and I want to talk to a specific program on another computer, I can find it by using the computer’s IP address, and the program’s port.

When we create a vpc-access connector, we essentially create a few tiny virtual machines that get attached to our VPC, and have an IP address assigned to them. When we connect Cloud Run to that vpc connector, we’re telling Cloud Run to route traffic to those tiny computers, which then make a request as a proxy on Cloud Run’s behalf. Those talk to Redis, then forward the response back to Cloud Run.

At this point, our API service needs to talk to two networks; the public internet and our private VPC. More specifically, it needs to accept incoming requests from the public internet, and make outgoing requests to Redis to read or write information. By setting these values:

--vpc-connector “$VPC_CONNECTOR” \
--vpc-egress private-ranges-only \

We’re connecting our API to the VPC, and routing outgoing traffic to that VPC (which has Redis).

And those are all of the changes on the infrastructure side. There are some fundamental changes to the software running behind the scenes, which I want to briefly explore before we move on to another topic. All the interesting stuff is happening in the API.

In this folder there’s two new files: redis_client.py and rate_limiter.py. redis_client.py is pretty simple; it just creates a Redis client using the Redis library.

import os

import redis

redis_client = redis.Redis(
    host=os.environ[”REDIS_HOST”],
    port=int(os.environ.get(”REDIS_PORT”, “6379”)),
    decode_responses=True,
)

The rate_limiter.py, on the other hand, is more sophisticated. I don’t want to get into it too much because it does some Redis-specific things, but it maintains a count of how many tokens a given user has and returns that information to the client. You can explore the implementation here, I expect to cover Redis in the near future.

And with that we have rate limiting and caching API tokens in Redis. Again, kind of overkill for this application (database lookups would probably be more than fine), but a powerful general strategy when building larger applications.

Before we talk about more technologies, I want to take a moment to formalize our understanding of a few concepts we’ve been discussing.

Formalizing some Fundamental Ideas

Levels of Compute Abstraction

Throughout the article we’ve been talking about “little computers running on the cloud,” which is true. Under the hood, there are tiny computers being spooled up for pretty much everything: routing traffic, querying databases, and running cloud functions. It’s networking, compute, and storage. The big differentiator is the level of abstraction.

The least abstract level is probably “Compute Engine”, a service where you can provision specific classes of computers with CPUs, memory, and network-attached storage.

A sample of some of the computers you can provision on GCP via compute engine

There’s still plenty of abstraction; it’s not just a computer sitting on a desk but a provisioning of resources sitting on a massive server farm. However, it walks and talks just like a computer. If properly configured, you can SSH into it and do pretty much whatever you want. You can even set up a remote desktop system and literally treat it like a desktop computer, play games on it, whatever.

The next level of abstraction is “Kubernetes Engine” (GKE), a system for managing clusters of computers. I have an in-depth article on Kubernetes, which I hope to expand in future installments

But, in a nutshell, kubernetes is a way to manage clusters of computers. While Compute Engine has you managing individual computers, GKE abstracts some of that control around provisioning and managing computers, allowing you to define high-level provisioning through the Kubernetes control plane.

One rung above that is Cloud Run, which we’ve been using. It’s built on technology similar to GKE and abstracts it significantly. You define containers, and Cloud Run handles the rest. Under the hood, depending on your settings, Cloud Run might manage one computer or a cluster of computers. It might choose to expose some of those computers to the internet or not. It might choose to scale compute based on some key metrics like CPU load or request frequency. It’s still a bunch of computers under the hood, but we have to do very little management of resources, which is great.

Cloud Run functions are yet another level of abstraction above using Cloud Run with containers. You don’t even have to define the container; you just define the code that gets run on the container, which Google manages. Yet another layer above that are technologies like App Engine or Firebase itself. These abstract away many of the needs to define infrastructure at all and instead allow you to build applications that just run on predefined architectures.

I suppose you can think of all of the abstract services in GCP as another layer of abstraction. In these, you don’t even define the code. When you set up a task queue, for instance, there is a compute service under the that manages scale by itself, has code to accept and work on your requests, has code to manage the queue, an API to interact with it, etc.

At the end of the day, there is no “right” or “wrong” in terms of abstraction, but there are better choices. Those choices are a function of the project you’re working on and its needs. If you’re working on a system that requires deep integration with core Linux configurations, you might need to work with provisioned instances in Compute Engine. If you have an existing set of documents and you want to turn them into a blog in five days, maybe use some abstract service like App Engine. The balance between control and agility through abstraction is fundamentally application-specific.

IAM

We touched on this topic adjacently throughout the article, but I want to take a moment to discuss what IAM is, what it does, and how it relates with ideas like service accounts.

IAM allows GCP to answer a fundamental question: “Is the entity making a request allowed to make that request?”, a question that’s tied to pretty much everything that happens on GCP. Any time you, as an administrator on GCP make a request via the console or to the CLI, your request is compared to the IAM permissions tied to your account. If you’re not allowed to do that operation, then the operation fails with a permission error.

Individual services like Cloud Run don’t have an account like humans do, so we give them a “Service account”. That allows us to bind IAM permissions to that service by way of the service account. It’s like Cloud Run has its own login, just like we do, with certain rules allowing it to access certain things.

Each IAM policy has the following core structure:

Who — the identity making the request (called a principal or member)
What — the role, which is a bundle of permissions
Which — the resource being acted on

By assigning specific permissions with service accounts, we can reference who they are, and assign what they can do to which resource, by building an IAM Policy.

The fundamental component of IAM is individual permissions. Everything on Google has a list of permissions with defined names. These generally map to a specific API method. Different services have different permissions, because they have different methods in their API.

storage.objects.create
storage.objects.get
storage.objects.delete
firestore.documents.get
firestore.documents.create
run.services.invoke
cloudtasks.tasks.create

Here’s a list of permissions for just cloud storage. There’s a lot, because there’s a lot of stuff you can do with cloud storage. These are so granular to be ungainly for the majority of use cases. In most contexts it makes more sense to employ a “role”.

A “role” in IAM is a bundle of permissions. We used these throughout the previous examples, things like:

roles/datastore.user
roles/storage.objectAdmin
roles/iam.serviceAccountTokenCreator
roles/eventarc.eventReceiver
roles/pubsub.publisher
roles/run.invoker

These bundle a bunch of granular IAM permissions together into a convenient-to-use and easy-to-conceptualize grouping. Under the hood, they’re defined something like this:

title: “PDF Reader Object Writer”
description: “Create and read book objects; no delete.”
stage: “GA”
includedPermissions:
  - storage.objects.create
  - storage.objects.get
  - storage.objects.list

If we saved that file as role-definition.yaml we could register our own role, like so

gcloud iam roles create pdfReaderObjectWriter \
  --project=pdf-reader-502816 \
  --file=role-definition.yaml

Generally, it’s preferable to use Roles managed by Google rather than build your own, as Google manages these roles for us. It’s really only recommended to build a custom role if there is a meaningful security improvement that justifies the level of ownership required to maintain that role.

Whether we’re using our own defined roles, or roles defined for us, we’ve been applying them via the add-iam-policy-binding throughout the article, like so:

gcloud projects add-iam-policy-binding pdf-reader-502816 \
  --member=”serviceAccount:pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com” \
  --role=”roles/datastore.user”

This function defines the “who”, “what”, and “which” necessary to define an IAM policy.

Who: The service account tied to pdfreader-runtime is being granted a permission
What: It’s being granted the permission defined in the role datastore.user, which grants full CRUD permissions for Firestore (and Datastore, the legacy thing firestore was built off of, hence the role name).
Which: This permission is being granted to all firestore/datastore instances in the pdf-reader-502816 project. So, resources tied to the pdfreader-runtime service account will be able to access all firestore instances in the project.

Permissions don’t have to be granted project-wide. This, for instance, binds a service account to a specific resource.

gcloud storage buckets add-iam-policy-binding gs://pdf-reader-502816-books \
  --member=”serviceAccount:pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com” \
  --role=”roles/storage.objectAdmin”

Here the pdfreader-runtime service account is being granted the storage.objectAdmin role, but only to the pdf-reader-502816-books bucket.

A little gotcha: we actually can’t do that with Firestore because, under the hood, Firestore is several sub-services all working together. That’s why it’s common to apply an IAM policy on the project level for Firestore. If we did have multiple Firestore instances in a single project and wanted to make access more granular, we could specify a condition that, despite the scope of the permission being on the project level, it programmatically restricts that permission to certain namespaces.

gcloud projects add-iam-policy-binding pdf-reader-502816 \
  --member=”serviceAccount:pdfreader-runtime@pdf-reader-502816.iam.gserviceaccount.com” \
  --role=”roles/datastore.user” \
  --condition=’expression=resource.name==”projects/pdf-reader-502816/databases/pdfreaderdb”,title=pdfreaderdb-only,description=Restrict to the pdfreaderdb database’

That’s a bit more advanced, though. Like custom roles, the degree of granularity that should be adopted is a function of the necessities of security for your specific project.

IAM Policies are typically additive (though they can be subtractive through “deny policies”). Everything starts with no permissions to do anything, then we add policies across different scopes. Multiple policy bindings can be applied to the same account, allowing us to define different permissions for different scopes.

# Grant 1 — project altitude: read-only on ALL objects in ALL buckets
gcloud projects add-iam-policy-binding pdf-reader-502816 \
  --member=”serviceAccount:pdfreader-reporting@pdf-reader-502816.iam.gserviceaccount.com” \
  --role=”roles/storage.objectViewer”

# Grant 2 — single-bucket altitude: full control of ONE bucket
gcloud storage buckets add-iam-policy-binding gs://pdf-reader-502816-reports \
  --member=”serviceAccount:pdfreader-reporting@pdf-reader-502816.iam.gserviceaccount.com” \
  --role=”roles/storage.objectAdmin”

In GCP, the hierarchy of organization of resources is as follows:

Organization: your entire company/domain. A grant here covers everything you own on GCP.
Folder: an optional grouping of projects (can nest), usually by team or environment. A grant here covers all projects inside it.
Project: the standard container your resources live in. A grant here covers every resource in that project
Resource: A grant here covers just that resource, if its type carries its own IAM policy. This works for things that are atomic, like cloud storage, but not for things that don’t really have a policy of their own, like firestore. Some resources are classified as “policy bearing resources”, and some are not, and instead are made up of “policy bearing resources”. The more abstract a services tend not to be policy bearing and instead combine several sub-resources.

Those were the topics I wanted to inject. I want to briefly touch on a few more technologies before we wrap up.

Pub/Sub

Publish/Subscriber technologies are a way to fan-out jobs to a variety of workers, decoupling producers and consumers and enabling event-driven architecture. We actually used Pub/Sub under the hood by implementing a task queue; which uses Pub/Sub to broadcast tasks out to worker nodes.

Essentially, with pubsub, you can have a system “publish” “messages”, with consumers that “subscribe” to those messages. The back and forth between subscribers and the Pub/Sub mechanism can either be pull-based or push-based, meaning you can either broadcast a message as soon as it’s ready, or hold the message until a subscriber requests it. Subscribers register with the Pub/Sub mechanism via a “subscription” to a particular “topic”.

Imagine you have a retail chain and, any time a user places an order, you want to trigger microservices that update inventory, send a confirmation email, and communicate with shipping companies. In a Pub/Sub context, your order processing service might publish a message to the order_event topic, which the downstream services process.

A big advantage of Pub/Sub is that it decouples producers from consumers logically. If we wanted to add a new downstream task to our system, we would just make that task subscribe to the order_event topic. We wouldn’t have to modify the ordering system, nor any of the other downstream microservices.

To whip up a system like this, we can start by registering a new topic called order-events

gcloud pubsub topics create order-events

Then, on something like cloud run, we can make an order processing machine that runs some code like this, which publishes a message to the order-events topic.

from google.cloud import pubsub_v1
import json

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(”your-project-id”, “order-events”)

def publish_order(order_id: str, email: str, amount: float):
    data = json.dumps({”order_id”: order_id, “email”: email, “amount”: amount}).encode(”utf-8”)
    future = publisher.publish(topic_path, data, event_type=”order.placed”)
    return future.result()  # blocks until the message ID comes back

Each of our subscribers will do different stuff, but they’ll have some endpoint allowing for a post request. After the request has been processed, they’ll send a 200 status code (or something like 200, like 204) to acknowledge that the request has been received. It will send something else if there’s been a failure.

from fastapi import FastAPI, Request
import base64, json

app = FastAPI()

@app.post(”/”)
async def handle(request: Request):
    envelope = await request.json()
    message = envelope[”message”]
    payload = json.loads(base64.b64decode(message[”data”]).decode(”utf-8”))

    send_confirmation_email(payload[”email”], payload[”order_id”])

    return “”, 204  # 2xx = ack. non-2xx = nack, and Pub/Sub redelivers.

Acknowledgement is how the pub/sub system knows to clear a message from a given subscription, or to try again after waiting some period of time. In Pub/Sub, every subscription is handled in isolation, so if there are three subscribers to a topic and one fails, the pub/sub system will try that one failed subscription again.

Once we have workers and a pub/sub system created, we need to wire up permissions so the publisher can publish to pubsub, and so pubsub has permissions to invoke subscribers. That would look something like so (this is AI generated and unconfirmed, but for demonstrative purposes):

#!/usr/bin/env bash
#
# Deploys a Pub/Sub fan-out system:
#
#   client --> [API service] --publish--> (order-events topic)
#                                              |
#                        +---------------------+---------------------+
#                        |                     |                     |
#                  (email-sub)           (orders-sub)         (analytics-sub)
#                        |                     |                     |
#                 [email service]      [orders service]     [analytics service]
#
# One publish fans out to three independent subscribers. Each subscription
# tracks delivery on its own, so if one subscriber fails, only THAT
# subscription redelivers - the other two are unaffected.
#
set -euo pipefail
cd “$(dirname “$0”)/..”
 
if [[ -f deploy/.env ]]; then
  set -a
  # shellcheck disable=SC1091
  source deploy/.env
  set +a
fi
 
# --- Configuration ---
PROJECT_ID=”${PROJECT_ID:?Set PROJECT_ID (export it or put it in deploy/.env)}”
REGION=”${REGION:-us-central1}”
TOPIC=”${TOPIC:-order-events}”
DEAD_LETTER_TOPIC=”${DEAD_LETTER_TOPIC:-order-events-dlq}”
MAX_DELIVERY_ATTEMPTS=”${MAX_DELIVERY_ATTEMPTS:-5}”
 
# One identity that Pub/Sub authenticates AS when it pushes to the private
# subscriber services. (You could split this per-subscriber for finer control;
# a single push identity that may invoke all three is a common middle ground.)
PUSH_SA=”pubsub-push-invoker”
 
# Treats “already exists” as success so the script is safe to re-run.
create_if_missing() {
  local err
  if ! err=”$(”$@” 2>&1)”; then
    if echo “$err” | grep -qi “already exists”; then
      echo “    (already exists, continuing)”
    else
      echo “$err” >&2
      exit 1
    fi
  fi
}
 
gcloud config set project “$PROJECT_ID”
PROJECT_NUMBER=”$(gcloud projects describe “$PROJECT_ID” --format=’value(projectNumber)’)”
PUBSUB_AGENT=”@gcp-sa-pubsub.iam.gserviceaccount.com”>service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com”
 
echo “==> Enabling required APIs”
gcloud services enable \
  run.googleapis.com \
  pubsub.googleapis.com \
  firestore.googleapis.com \
  cloudbuild.googleapis.com \
  artifactregistry.googleapis.com \
  iam.googleapis.com
 
echo “==> Ensuring a Firestore (Native mode) database exists”
create_if_missing gcloud firestore databases create --location=”$REGION”
 
echo “==> Creating the main topic and the dead-letter topic”
create_if_missing gcloud pubsub topics create “$TOPIC”
create_if_missing gcloud pubsub topics create “$DEAD_LETTER_TOPIC”
 
# A dead-letter topic is only useful if something reads it. Attach a plain
# subscription so failed messages are retained and inspectable.
create_if_missing gcloud pubsub subscriptions create “${DEAD_LETTER_TOPIC}-monitor” \
  --topic “$DEAD_LETTER_TOPIC”
 
echo “==> Creating the shared push-auth service account”
create_if_missing gcloud iam service-accounts create “$PUSH_SA” \
  --display-name “Pub/Sub push invoker for order subscribers”
PUSH_SA_EMAIL=”${PUSH_SA}@${PROJECT_ID}.iam.gserviceaccount.com”
 
# Pub/Sub’s own service agent must be allowed to mint the OIDC tokens it
# attaches to authenticated push requests.
echo “==> Letting the Pub/Sub service agent create push auth tokens”
gcloud projects add-iam-policy-binding “$PROJECT_ID” \
  --member=”serviceAccount:${PUBSUB_AGENT}” \
  --role=”roles/iam.serviceAccountTokenCreator” >/dev/null
 
# For dead-lettering to work, the SAME service agent needs to publish to the
# dead-letter topic and subscribe on the source subscriptions. Miss this and
# dead-lettering silently does nothing - no error, messages just pile up.
echo “==> Granting the Pub/Sub service agent dead-letter permissions”
gcloud pubsub topics add-iam-policy-binding “$DEAD_LETTER_TOPIC” \
  --member=”serviceAccount:${PUBSUB_AGENT}” \
  --role=”roles/pubsub.publisher” >/dev/null
 
# ---------------------------------------------------------------------------
# Helper: deploy one private subscriber service + its push subscription.
#
#   $1  service name        (e.g. email-service)
#   $2  source directory     (e.g. subscribers/email)
#   $3  subscription name     (e.g. email-sub)
#   $4  runtime SA short name (e.g. email-runtime) or “” for the default SA
#   $5  runtime role to grant  (e.g. roles/datastore.user) or “” for none
# ---------------------------------------------------------------------------
deploy_subscriber() {
  local service=”$1” src=/__u/iaee.substack.com/”$2” sub=”$3” runtime_sa_name=”$4” runtime_role=”$5”
  local runtime_sa_flag=()
 
  if [[ -n “$runtime_sa_name” ]]; then
    echo “==> [$service] creating runtime service account $runtime_sa_name”
    create_if_missing gcloud iam service-accounts create “$runtime_sa_name” \
      --display-name “$service runtime”
    local runtime_sa_email=”${runtime_sa_name}@${PROJECT_ID}.iam.gserviceaccount.com”
    runtime_sa_flag=(--service-account “$runtime_sa_email”)
 
    if [[ -n “$runtime_role” ]]; then
      echo “==> [$service] granting $runtime_role to its runtime SA”
      gcloud projects add-iam-policy-binding “$PROJECT_ID” \
        --member=”serviceAccount:${runtime_sa_email}” \
        --role=”$runtime_role” >/dev/null
    fi
  fi
 
  echo “==> [$service] deploying (private - only Pub/Sub may invoke it)”
  gcloud run deploy “$service” \
    --source “$src” \
    --region “$REGION” \
    --no-allow-unauthenticated \
    --set-env-vars “PROJECT_ID=${PROJECT_ID}” \
    “${runtime_sa_flag[@]}”
 
  local url
  url=”$(gcloud run services describe “$service” --region “$REGION” --format ‘value(status.url)’)”
 
  echo “==> [$service] allowing the push SA to invoke it”
  gcloud run services add-iam-policy-binding “$service” \
    --region “$REGION” \
    --member=”serviceAccount:${PUSH_SA_EMAIL}” \
    --role=”roles/run.invoker” >/dev/null
 
  echo “==> [$service] creating push subscription $sub with dead-letter policy”
  create_if_missing gcloud pubsub subscriptions create “$sub” \
    --topic “$TOPIC” \
    --push-endpoint “${url}/” \
    --push-auth-service-account “$PUSH_SA_EMAIL” \
    --ack-deadline 60 \
    --dead-letter-topic “$DEAD_LETTER_TOPIC” \
    --max-delivery-attempts “$MAX_DELIVERY_ATTEMPTS” \
    --min-retry-delay 10s \
    --max-retry-delay 600s
 
  # The service agent also needs subscriber on each source subscription to
  # forward its dead-lettered messages.
  gcloud pubsub subscriptions add-iam-policy-binding “$sub” \
    --member=”serviceAccount:${PUBSUB_AGENT}” \
    --role=”roles/pubsub.subscriber” >/dev/null
}
 
# Three subscribers on ONE topic. Note the differing least-privilege IAM:
#   - email needs no data access at all
#   - orders + analytics each get datastore.user and nothing more
deploy_subscriber “email-service”     “subscribers/email”     “email-sub”     “”                “”
deploy_subscriber “orders-service”    “subscribers/orders”    “orders-sub”    “orders-runtime”   “roles/datastore.user”
deploy_subscriber “analytics-service” “subscribers/analytics” “analytics-sub” “analytics-runtime” “roles/datastore.user”
 
# ---------------------------------------------------------------------------
# The publisher goes LAST so the topic already exists. It’s the only public
# service. Its runtime SA needs permission to publish to the topic.
# ---------------------------------------------------------------------------
echo “==> Creating the API runtime service account”
create_if_missing gcloud iam service-accounts create “api-runtime” \
  --display-name “Order API runtime”
API_SA_EMAIL=”api-runtime@${PROJECT_ID}.iam.gserviceaccount.com”
 
echo “==> Granting the API publish rights on the topic”
gcloud pubsub topics add-iam-policy-binding “$TOPIC” \
  --member=”serviceAccount:${API_SA_EMAIL}” \
  --role=”roles/pubsub.publisher” >/dev/null
 
echo “==> Deploying the public API service”
gcloud run deploy “api-service” \
  --source ./api \
  --region “$REGION” \
  --allow-unauthenticated \
  --service-account “$API_SA_EMAIL” \
  --set-env-vars “PROJECT_ID=${PROJECT_ID},TOPIC_ID=${TOPIC}”
 
API_URL=”$(gcloud run services describe api-service --region “$REGION” --format ‘value(status.url)’)”
echo “==> Done.”
echo “    API URL: ${API_URL}”
echo “    Try:     python client/orders_client.py --url ${API_URL} --count 5”

If you read through it, there are a lot of the similar suspects; cloud run, service accounts, enabling APIs, etc. I really just want to focus in on the pub/sub specific concepts.

First of all, there’s an important practical concept we haven’t explored; dead letters.

TOPIC=”${TOPIC:-order-events}”
DEAD_LETTER_TOPIC=”${DEAD_LETTER_TOPIC:-order-events-dlq}”
MAX_DELIVERY_ATTEMPTS=”${MAX_DELIVERY_ATTEMPTS:-5}”

When you have a pub/sub system or task queue system, it will retry sending events when an error occurs, which is great; this makes our system resilient to the intermittent outages that are practically common on large, internet-scale systems.

However, imagine we have a malformed message or task. If that were the case, the same message would be broadcast over and over, continually, eating up resources. As more malformed requests pile up, the task queue or pubsub topic would be saturated by the same malformed tasks being requested over and over again.

This idea is referred to as a “dead letter”, a message or taks in a queue or topic which has been retried a few times and needs to be quarantined for future analysis as to not cause downstream issues. In queues, this is called a “dead letter queue”, in pubsub it’s called a “dead letter topic”. The idea is that, after a message has been attempted some number of times, it will be moved to a dead-letter topic where a dedicated service can ingest and handle that message; likely storing it for future triage.

We can create those topics

echo “==> Creating the main topic and the dead-letter topic”
create_if_missing gcloud pubsub topics create “$TOPIC”
create_if_missing gcloud pubsub topics create “$DEAD_LETTER_TOPIC”

Next, we create a subscription, which does some interesting stuff.

# A dead-letter topic is only useful if something reads it. Attach a plain
# subscription so failed messages are retained and inspectable.
create_if_missing gcloud pubsub subscriptions create “${DEAD_LETTER_TOPIC}-monitor” \
  --topic “$DEAD_LETTER_TOPIC”

It might be useful to think of a topic as a queue of messages which consumers consume, but in reality topics don’t hold messages, “subscriptions” hold messages. Recall that multiple subscribers can be subscribed to an event, and failures and throughput are treated independently. Thus subscriptions function as a sort of queue which holds messages. Even though we don’t have a downstream process to ingest messages in our dead letter topic, we’ll need to create a subscription so they have somewhere to go. Otherwise, without a subscription, they’ll disappear into the void. By default, messages are held for 7 days on GCP, but this is configurable. Also, if we wanted, we could create a service that records this in a database for infinite retention.

After that, we have some permissions stuff. First we create a service account, which represents the permissions granted to our Pub/Sub system. Because our Cloud Run instances are not publicly exposable, Pub/Sub needs a service account that it can use to invoke downstream Cloud Run tasks. To do that, we create the service account

echo “==> Creating the shared push-auth service account”
create_if_missing gcloud iam service-accounts create “$PUSH_SA” \
  --display-name “Pub/Sub push invoker for order subscribers”
PUSH_SA_EMAIL=”${PUSH_SA}@${PROJECT_ID}.iam.gserviceaccount.com”

And give it the role of run.invoker on a particular service

echo “==> [$service] allowing the push SA to invoke it”
  gcloud run services add-iam-policy-binding “$service” \
    --region “$REGION” \
    --member=”serviceAccount:${PUSH_SA_EMAIL}” \
    --role=”roles/run.invoker” >/dev/null

There’s some fun functional logic going on in this particular example. Any time we create a new subscriber, we’re doing it with the deploy_subscriber() function, which simultaneously creates and connects a subscriber. The code above is run for each service being created, granting the pubsub push service account permission to invoke each subscriber.

After a particular service is created, and a permissioned service account is defined, we can set up the subscription

echo “==> [$service] creating push subscription $sub with dead-letter policy”
create_if_missing gcloud pubsub subscriptions create “$sub” \
  --topic “$TOPIC” \
  --push-endpoint “${url}/” \
  --push-auth-service-account “$PUSH_SA_EMAIL” \
  --ack-deadline 60 \
  --dead-letter-topic “$DEAD_LETTER_TOPIC” \
  --max-delivery-attempts “$MAX_DELIVERY_ATTEMPTS” \
  --min-retry-delay 10s \
  --max-retry-delay 600s

This does a few things:

defines the name of the subscription, here defined as the variable $sub
topic: says what topic the subscription will be to, in this case all of them observe the order-events topic.
push-endpoint: The URL of the subscribing Cloud Run. Pubsub will push messages to this URL.
push-auth-service-account: The service account granting permission for pubsub to invoke the cloudrun service
ack-deadline: How long a worker has to complete and acknowledge the message as complete before we consider it a failure and try again
dead-letter-topic: The topic where messages that repeatedly fail end up
max-delivery-attempts: The number of retries we get before we consider a message to be a dead letter
The two retry delays define the bounds for exponential backoff. If there’s a failure we re-try again in 10 seconds. If there’s another failure we wait double the time; 20 seconds. Another failure 40 seconds. That number keeps doubling every time, waiting longer and longer after trying again (approximately. In reality there’s some randomized jitter applied so that concurrent requests are naturally staggered). We cap the delay period we’re willing to wait with max-retry-delay.

Now we get into some of the more counterintuitive aspects. The principle of least permissions, that everything should be granted minimal permissions by default, is a necessity in making the cloud secure at scale. However, the price for that security is an inevitable hunt for esoteric and counterintuitive permissions that need to be enabled to get everything working. Take for example the following:

PUBSUB_AGENT=”@gcp-sa-pubsub.iam.gserviceaccount.com”>service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com”

# Pub/Sub’s own service agent must be allowed to mint the OIDC tokens it
# attaches to authenticated push requests.
echo “==> Letting the Pub/Sub service agent create push auth tokens”
gcloud projects add-iam-policy-binding “$PROJECT_ID” \
  --member=”serviceAccount:${PUBSUB_AGENT}” \
  --role=”roles/iam.serviceAccountTokenCreator” >/dev/null

Here PUBSUB_AGENT is a Google-managed service account that moves messages around under the hood for PubSub. Our subscriber services are private, so Cloud Run only lets a request in if it carries a token proving the caller is from the PUSH_SA service account, the invoker identity we created and granted run.invoker. But the thing actually making the push request is Pub/Sub, i.e. PUBSUB_AGENT, which is a different account. So for the push to work, PUBSUB_AGENT has to attach a token that says “I am PUSH_SA” essentially, impersonating another account. serviceAccountTokenCreator is precisely the permission to mint a token as another service account. That’s what this grant does: it lets Google’s Pub/Sub agent generate tokens in the name of PUSH_SA, so that when it pushes to a subscriber, Cloud Run’s door check sees PUSH_SA and opens. Without doing this, PubSub can’t actually properly authenticate messages sent to our private workers.

This agent also needs permission to actually publish to the dead letter topic


# For dead-lettering to work, the SAME service agent needs to publish to the
# dead-letter topic and subscribe on the source subscriptions. Miss this and
# dead-lettering silently does nothing - no error, messages just pile up.
echo “==> Granting the Pub/Sub service agent dead-letter permissions”
gcloud pubsub topics add-iam-policy-binding “$DEAD_LETTER_TOPIC” \
  --member=”serviceAccount:${PUBSUB_AGENT}” \
  --role=”roles/pubsub.publisher” >/dev/null

And needs access to each subscription so it can manage the messages within them

# The service agent also needs subscriber on each source subscription to
  # forward its dead-lettered messages.
  gcloud pubsub subscriptions add-iam-policy-binding “$sub” \
    --member=”serviceAccount:${PUBSUB_AGENT}” \
    --role=”roles/pubsub.subscriber” >/dev/null

This is all required because the principle of least permissions is applied by default, but it means we might need to track down these Google-managed service accounts and modify them, especially when dealing with abstract services that have multiple elements going on under the hood.

After all that is configured, of course, the publisher needs to be wired up. with permissions to publish to the order-events topic.

gcloud pubsub topics add-iam-policy-binding “$TOPIC” \
  --member=”serviceAccount:${API_SA_EMAIL}” \
  --role=”roles/pubsub.publisher”

And, with that, we’ve reviewed what it takes to create a functional pub/sub system. There’s plenty more technologies I’m planning on exploring in following articles, but I think that covers the lion’s share I wanted to discuss in this article. Let’s briefly discuss some relevant concepts before concluding.

Brief Mentions

I’m planning on making some dedicated follow-ups to this article describing more specific technologies in GCP around ML, security, cost management, observability, and networking. There’s a lot to cover, and I can’t hope to explore everything in this article. However, I do want to discuss some technologies and concepts from a high level before concluding.

Idempotency

Both task queues and pub/sub systems tend to guarantee “at least once delivery”, meaning a message (that’s not flagged as a dead letter due to excessive failures) will be processed at least once, but may be processed more than once. This is because it’s possible for a consuming service to accept a message/task, then finish it after the acknowledgement deadline. From the queue’s perspective, the request failed, but it was actually successful, meaning the queue is sent it to another consumer while the first consumer was still running.

Practically, it’s very hard to guarantee exactly once delivery, and is typically too expensive to justify. Instead of trying to force single delivery, the way this is handled is with “idempotent” design. The idea of idempotency is to design systems such that, even if you send the same request over and over, you still get the same result as if you did the operation once.

We can use our three microservices for order processing as an example. Consider the three cases:

Recording order information in the database
Recording revenue information in the database, based on the order
Sending emails to users

In the first case, it’s pretty easy. If we write the following record more than once, it’s of no consequence

db.collection(”orders”).document(order_id).set({
    “order_id”: order_id,
    “email”: payload[”email”],
    “amount”: payload[”amount”],
    “status”: “recorded”,
})

Because we’re writing to the document defined by order_id, if we re-write the same data multiple times, we’ll end up with an equivalent record

Things get more complicated when we record metrics. Let’s say we’re trying to increment revenue and order count on a per-day basis. If we do that multiple times, we’ll end up incrementing the order count and revenue multiple times for a single order.

db.collection(”daily_revenue”).document(day).set({
    “revenue”: firestore.Increment(payload[”amount”]),
    “order_count”: firestore.Increment(1),
}, merge=True)

We can fix this issue by keeping track of the orders we’ve used for incrementation previously. If that already exists, then don’t do the incrementation.

from datetime import datetime, UTC
from google.cloud import firestore

order_id = payload[”order_id”]          # the idempotency key
amount = payload[”amount”]
day = datetime.now(UTC).strftime(”%Y-%m-%d”)

processed_ref = db.collection(”analytics_processed”).document(order_id)
counter_ref = db.collection(”daily_revenue”).document(day)

@firestore.transactional
def apply_once(txn):
    # Firestore requires all reads before any writes.
    already = processed_ref.get(transaction=txn).exists
    if already:
        return False                    # seen this order, do nothing
    txn.set(processed_ref, {”order_id”: order_id, “at”: datetime.now(UTC)})
    txn.set(counter_ref, {
        “revenue”: firestore.Increment(amount),
        “order_count”: firestore.Increment(1),
    }, merge=True)
    return True

apply_once(db.transaction())
return “”, 204

Firestore, for instance, allows us to execute several interactions in our database as a “transaction”. We wrap checking if a transaction has already been used, updating that the transaction has been used, and incrementing the counters as a single transaction. If one of these happens, they all happen. There’s a term called “ACID compliance”, which gives us certain guarantees about transactions, allowing us to design systems that are idempotent. Look up ACID transactions, or stay tuned for an upcoming article on databases.

The email-sending use case is a bit trickier. Generally speaking, if you’re sending an email, you’ll be triggering a request to a downstream email provider. Without having control of the email provider, it’s very hard to absolutely guarantee only one email will be sent. We can try our best to record what emails have been sent to mitigate duplicates, but what happens if the email system crashes after we record that we sent the email in our database? What happens if the database crashes as we attempt to send an email? Naturally, with more robust systems we can mitigate these issues with idempotent-like designs, but guaranteeing exactly-once delivery in all circumstances is very challenging. So difficult that it’s worth asking “is it that big of a deal if there’s a fringe possibility we send two confirmation emails occasionally?”

Sharding

We discussed statelessness throughout the article. Some things, like DataBases and caching layers, can’t be stateless for obvious reasons; the whole reason they exist is to record information and recall it efficiently. Because of this, databases are harder to scale horizontally; in a stateless application, it doesn’t matter which computer processes a request. Thus, you can just spool up more computers and load balance between them to achieve near-infinite scale. In stateful contexts, you need to use “sharding.”

Sharding is the idea of breaking up your database into different sub-databases. If you want to keep track of people’s birthdays, you might choose to divide that data, storing the birthdays for the people with the first name “Daniel” in one database, and birthdays for people with the first name “Rachael” in another database. These would be considered shards of one larger database. The effect is that you can divide your work across different computers in charge of managing different shards in the database.

The thing you elect to divide by, in this case the name of the person, is what’s called the “shard key”. In reality, using a name as a shard key is horrible, as there’s so many names you would need a bunch of shards. Choosing the right shard key has a massive impact on the distribution of load throughout a database, and choosing the right one is incredibly important. In some context it’s handled automatically, like in Firestore, but in some contexts, like Redis, the developer needs to choose their own shard key.

Billing

The cloud can become very expensive very quickly, and it’s important to continually monitor

You can generate reports breaking down the services you’re using and how much they cost

You can define budget alerts when you exceed that budget

and generally see how much your projects or products cost. It’s worthwhile to continually monitor costs as scale grows, and set up alerts notifying you when costs reach a threshold you expect.

Logging, Monitoring, and Tracing

When you’re trying to manage hundreds of computers across an internet-scale application, it can be difficult. Generally, Logging, Monitoring, and tracing are how developers understand performance at scale and identify key issues.

The Log Explorer allows you to view and query logs across the various machines running your applications.

These logs are a capture of the stdout of the machine. There’s also a variety of metrics posted throughout various services

With a little bit of work, it’s also possible to wire up monitoring so all of the critical resources are organized into a single place

If you’re using tools like OpenTelemetry to trace requests throughout your application, you can also use CloudTrace to monitor how specific requests flow through your application. I’m hoping to cover that in a future installment.

Ingress

Ingress isn’t a service in itself, but there are a variety of services in GCP designed to help secure and handle inbound requests from the internet.

For what we’ve been doing, having a server publicly exposed as our API endpoint is a solid strategy. Tools like “API Gateway” really shine when you want to make a publicly accessible API. It does a lot of great stuff for us; authenticating users via their API token, rate limiting, that kind of stuff. It’s in the weeds for this article, but definitely worth exploring if you’re building an online API rather than an application.

“Load Balancers” are another form of ingress management that we’ve already used indirectly. We talked about scaling horizontally in previous sections, where we increase our capacity by creating several computers that can accept incoming requests. The thing that routes those requests to different computers is a load balancer.

A load balancer functions as a single entry point that then fans out requests to different services based on a variety of rules. A properly configured load balancer can distribute load across a scaling cluster, allow you to phase load between destinations to facilitate incremental rollout, and define routing rules based on the URL; routing different paths to different services.

Ingress management is a big topic, and we’ve already covered a lot. I’m planning on making a follow up article that discusses networking to a greater degree of depth.

Security

Naturally, we talked about security throughout the entire article. Setting IAM policies, defining service accounts, enabling authentication, and restricting public exposure. These are all fundamentally security decisions. In addition to those, GCP has a variety of tools specifically made to create more secure products.

Some notable examples are:

Secret Manger: Allows you to store and distribute API keys securely.
Security Command Center: Scans through projects and identifies misconfigurations and security vulnerabilities
Cloud Armor: This is a Web Application Firewal DDOs system, rate limiter, Captcha enabler, etc. This can sit infront of your entire application to protect it from attack and malicious use
Identity-Aware Proxy: A complement to identity provider, allowing granular monitoring and blocking of traffic based on authentication.

Conclusion

The general topics we described are the beating heart of most large-scale cloud applications. Naturally, we couldn’t cover everything on Google Cloud, but with this knowledge you can make scalable backends that are secure, and use this knowledge to explore more advanced and application-specific technologies. Stay tuned for more idea-specific topics like GenAI, database technologies, and networking in future installments.

The Documentation is the Code

Daniel Warfield — Tue, 30 Jun 2026 05:01:35 GMT

“Inversion” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained.

I’ve been working on a project that attempts to standardize an aspect of developing agentic systems. This article isn’t about that project, but some of the feedback I’ve received as I’ve exposed the project to the world. Specifically, feedback that has the following flavor:

It’s just markdown files. Real *AI related thing* is actual, running code

I’d like to take a moment to unpack what this sentiment implies, the context in which it exists, and how it relates to where software development is going as a discipline.

Share this article with friends and colleagues

The Fight Between Documentation and Code

Software developers have been hating creating documentation since software has been a thing. In my article on agile project management I discuss, among many things, the “Agile Manifesto”. This document started the agile software development mentality that dominates modern software development. Its second tenant is to prioritize functional software over documentation.

The Agile Manifesto, with “working software over comprehensive documentation” highlighted. source

Software is an evolving and dynamic thing. Over several revisions a project can drastically change, meaning documentation around that software has to be updated and maintained constantly to keep up with changes in the codebase. This puts a strain on those maintaining the codebase; do they prioritize their time maintaining documentation that will inevitably be made antiquated or focus on working on the code that actually drives the project forward?

The balance between these two things is a contentious debate, and the correct approach depends on your role, the project you’re working on, the size of the team you’re working in, and the dynamics of that team.

If you’re a solo founder cranking away at a new product, you don’t have time to document every function, every interface, and every API. You have bigger fish to fry, and the documentation is superfluous because you, the solo founder, are the only person who would be looking at the code anyway.

For a small size team of experienced developers, some documentation is important, but chiefly in coordinating high-level initiatives and aligning the team in the right direction. Schematics, specs for key APIs where two developers are working on opposite ends simultaneously, and tickets to keep the team aligned towards the same goal is pretty much all you need.

As a project grows, things become more complicated. Developers might join and leave the company, marketing teams might need information about specific aspects of how the product functions, and you might need to bring on a technical support team to assist customers. Quickly, “ask the guy who built it” becomes infeasible if not impossible. At this point, weather the technical team likes it or not, they’re going to have to sit down and write some docs.

And this is the way it’s gone, more or less, for over 40 years. Different developers might have different opinions as to when you start writing docs, how much you write, and why, but generally documentation gets written out of necessity as a project matures. With AI, this status quo completely changes.

The Changing Landscape of Software Development

Agentic systems are already outpacing human development. Is the code quality good? Not always, but it seems like it’s starting to be good enough, especially when guided by a competent developer.

source

The paradigm of a single person sitting down over the course of months and fleshing out a prototype by hand, frankly, is the past. The modern paradigm of software development is people who know what they’re doing delegating their understanding to AI systems and making sure those AI systems are implementing robust solutions that are faithful to their vision.

In essence, the role of software developer is becoming more like the role of technical product owner/project manager. Doing a good job in complex environments still requires the hard skills of software development; you need to know what you’re doing to manage a team of AI agents effectively, but sitting down and writing individual lines of code is becoming a rarity.

With the fundamental bedrock of software development changing so significantly, it’s no surprise that the role of documentation is changing along with it.

AI, and the Role of Documentation

Every software project is now a small to mid-sized team. Every developer is using AI to augment their work. Soon, most knowledge workers will be in the same situation. As AI grows and evolves, most humans will be augmented, assisted, and supported by AI systems in some way. I’m both excited and afraid about the prospect, like most people are. Regardless of one’s feelings, though, it’s the obvious reality.

When we look to software development, specifically, I feel there’s a cognitive dissonance between the work people do, vs the work they imagine that they do. Many developers have been writing code for decades, and they still imagine themselves as writing code. They use AI to assist them to be faster, but they’re ultimately the ones creating it. However, as AI begins to generate 50%, 60%, 90%, 99% of a codebase, this conceptualization becomes less and less accurate. More and more, saying you “wrote the code” on a project you worked on with an AI system feels more like a quirk of language than a truthful statement.

Developers who have spent their careers writing code now, in plain truth, don’t. They’re still getting something done, but they don’t know how to describe what they’re doing and why they’re necessary. I feel like some developers have yet to make the leap in terms of how they conceptualize themselves. In reality they are a manager, not a developer. A manager that’s managing very fast, fairly competent, but oddly inconsistent AI developers. They can understand large amounts of information quickly, and leverage that information to write a significant amount of code rapidly, but also ignore entire critical pieces of documentation, make odd assumptions that are clearly out of alignment with the subtext of the project they’re working on, and completely reset their understanding of a problem every time you close and open a terminal.

In this environment, documentation is not some superfluous line item; it is fundamental and absolutely critical to being an effective developer in the modern era. Naturally, as LLMs remain inconsistent, strong software development skills remain fundamentally important. As LLMs and agentic systems continue to evolve, though, the instructions used to get agentic systems to understand a codebase and make changes begin to become a more important fundamental artifact than the code itself. Increasingly, code is made based on documentation, rather than the other way around.

Thus, we go back to the sentiment that inspired this article

It’s just markdown files. Real *AI related thing* is actual, running code

I see this sentiment a lot throughout the internet: that agentic systems are complex and sophisticated pieces of software, and the various prompts and context that guide them are of marginal interest. There’s some wisdom in it; it seems like every day someone posts an article about how they “made an AI agent to help them make a billion dollars online,” and that “AI agent” is Claude, pointed to three poorly worded markdown files. At this point, “a bunch of markdown files.” is indicative of hoaky, hacky, and hypy software, and the frequency and cheapness of this form of content have a natural erosive quality to the idea of structuring information for agentic systems, which already inherited a bad flavor from its father, “prompt engineering.”

While true and justified in a dimension, there’s also a significant untruth in the statement as well. That the structuring of information such that it can be easily accessed and understood by an agent is not of importance. As agentic systems evolve, and skills, instructions, MCP servers, and other paradigms become bundled into larger and larger projects, the efficient organization of information becomes a critical and not-so-trivial task.

I worry that some developers don’t understand that documentation is increasingly shifting from a byproduct of software to becoming its fundamental artifact. Defining systems, how they work, and the intent behind them is of critical importance to the modern AI-augmented developer. Defining that information in a consistent manner that agentic systems can continually reference over the lifetime of a project is becoming a fundamental skill. Thus, the ability to write and maintain good documentation (perhaps more so than writing good code) is becoming a fundamentally important skill.

The Agent Harnesses Standard

The ideas in this article are why I’m writing the Agent Harnesses Standard, the goal of which is to define how information can be structured such that it can be more robustly leveraged by agentic systems. For it to be successful, I can’t build it alone. It’s the type of thing that requires a diversity of opinions to flesh out correctly. If you’re interested in this topic, check out the github for information on contributing.

Agent Harnesses with Claude — Intuitively and Exhaustively Explained

Daniel Warfield — Sat, 27 Jun 2026 06:02:15 GMT

“Suiting Up” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained.

A few weeks ago I released a piece called “Agent Harnesses”. The objective of that piece was to soft release and understand other people’s perspectives on a fledgling project I’ve been working on; a new standard for agentic design called “Agent Harnesses”.

In my career thus far, I’ve experienced the following truths:

No great things are made in a vacuum
Negative feedback is often the most productive feedback

My goal in creating agent harnesses is to design a standard that democratizes the creation of sophisticated and robust agentic systems. To do that effectively, I knew I needed to expose my ideas, fledgling as they were, to an environment of criticism. I had a wider vision of “Agent Harnesses” than what was outlined in my previous article, but I needed feedback to give that vision a form so I could be confident moving forward.

Since writing that piece, I’ve been working hard to flesh out my understanding, have been exploring other peoples approaches to developing modern agentic systems, and have spent much time reflecting on other people’s perspectives. I now feel comfortable peeling another layer off of agent harnesses, exposing it at a greater depth, and sharing how it can be used to practically build robust and efficient agentic systems with Claude.

Before I get into the meat of it, though, I want to take a moment to address a fundamental question.

Share this article with friends and colleagues.

What is a Harness?

The term “harness” is being passed around by different cutting edge AI organizations, kind of like how hip cool kids talk about a hangout spot you’ve never been invited to. You’re not sure what they’re talking about, or if they’re even talking about the same thing, but you’re sure there’s something going on.

If you ask Google what a harness is, you’ll get this answer:

In AI, an agent harness is the supporting software infrastructure that wraps around an AI model (like an LLM), giving it the tools, memory, and environment to take real-world actions

So, from Google’s perspective, a harness is the supporting software that allows LLMs to do agentic things, though exactly how the code wraps around the model is somewhat vague.

This post by Anthropic discusses a “harness for long-running agents”.

We developed a two-fold solution to enable the Claude Agent SDK to work effectively across many context windows: an initializer agent that sets up the environment on the first run, and a coding agent that is tasked with making incremental progress in every session, while leaving clear artifacts for the next session.

In that article, Anthropic links to this github repo. It uses two “agents”, an initializer and a coding agent, to solve problems. These “agents” both use the same python script, but employ different starting prompts; one for coding and one for initialization. So, in this post, a harness contains agents, and coordinates them to solve problems.

This article by LangChain thinks of a harness differently.

Building useful agents is largely about customization: connecting your agent to the right context, data, and environment(s) for the task at hand…. The job of a harness is to get the model the right context at the right time for the given task.

They define a harness as being the code that turns an AI model into an agent

agent = model + harness

which is a bit confusing because the anthropic example uses two agents to build a harness. Are harnesses used to build agents, or are agents used to build harnesses?

If you look through the various mentions of harnesses online, the high level idea of a harness will be largely equivalent, but if you ask probing questions you’ll find that two people using the term “harness” might actually be discussing two totally different things.

The most common sentiment I’ve found is that a harness is a set of rigid programs that enable an LLM to do things other than respond to text, like understand state, execute tools, and operate in complex environments

An actual harness is the infrastructure around the LLM. These are things like memory, state, persistence, guardrails, sanboxing. This is deterministic software that quite literally harnesses the probabalistic language model. It’s not some markdown files, it’s big infrastructure.
— A response I got on my previous article which, by the way, I was very happy to receive. I quote this with great appreciation to the comment and commenter. If you would like to be mentioned by name, I would be more than happy to edit this quote.
A true harness lives outside the context window: It is the heavy, deterministic infrastructure wrapping the LLM and includes things like isolated sandboxes, state persistence engines, memory routers, and hard guardrails.
— Another response I received with gratitude, same offer applies as the previous.
A harness provides an agentic looping environment for use cases where the AI is supposed to loop until it solves the given task (like coding). — Source

The issue I have with this definition is that it bears a significant overlap with the definition of an agent. An “agent,” by definition, is a set of rigid programmatic tools that connect an LLM with state and abilities so that it can execute tasks in complex environments.

If “it’s code that lets LLM do things” were to become the working definition of a harness, I don’t feel it would benefit the community, but instead lead to confusion by rebranding an already established idea. That’s what an agent is.

Thankfully, there is another popular definition of a harness.

most “harnesses” are still thin wrappers around prompts, tools, and state, and the hard part is not structure, it’s evals, failure handling, and keeping behavior stable after one small model change. that part is where most agent stacks die
— Another quote I received, with tremendous gratitude, from a fellow writer on Medium.

Along with chats like this, and this, there’s an implication that a harness isn’t so much programmatic agentic code, but rather something that is applied to an agent to get it to do something specific.

This definition, that a harness is something that agents connect to that applies them to specific problems, is something that I resonate with. I’m personally of the opinion that, if there is a functional definition of a harness, it is that.

A harness is information and tools that allow a general purpose agentic system to do specific, complex tasks in a repeatable and maintainable manner.

I like this definition, partly because it doesn’t step on the toes of the definition of an agent, but also because I think it’s a practically vital concept on the cutting edge of AI application development.

The Context in Which Harnesses are Being Defined

I think there are two fundamental ideas that are critical in understanding why harnesses matter, and that is “agentic constraint” and the “agent skills standard”.

Context: Agentic Constraint

“Agentic constraint” is a phrase I made up, but it’s been bouncing around conceptually for years now. It can be boiled down into a simple question: how granularly do you constrain your agent?

Do you let it do whatever it wants by thinking through problems, like a ReAct style agent?

a ReAct style agent, on the right, thinking through a problem and invoking tools. source from my article on agents.

Or do you lock it down, and define a rigid structure that an agent has to navigate within?

A heavily constrained agent forced to make specific choices and do specific things. From my article on conversations as directed graphs.

Generally, when thinking about agents over the last few years, I’ve put them into two buckets: “constrained agents”, which follow a human interpretable and editable logical path, and “unconstrained agents”, which are allowed to think through problems on their own. There are costs and benefits to both approaches; constrained agents are more deterministic and understandable and are therefore easier to maintain. Unconstrained agents are flexible, which can allow them to do arbitrary tasks and also to fail in arbitrary ways.

Context: Agent Skills

Agent Skills are a big deal. It’s a standard that’s changed the way agentic systems work on a fundamental level. I cover it in this article

and the spec can be found here

A skill is, in its essence, a folder with a markdown file. That’s it.

Literally, that’s it.

It’s so simple that it can feel laughable to call it a standard, a stance I took when covering it initially

Brief aside, I haven’t been impressed with how “vibey” specifications around AI, LLMs, and Agents often are. If we don’t know where files are, or have a rigid specification around them, how can we enforce progressive exposure (which is one of the major objectives of the agent skills specification)? Also, should all of the output specifications be in FORMS.md? Should some of it be in REFERENCE.md, or other more specific references within the references directory? How can we enforce output formats if we don’t even know where the output formats are supposed to be? — from my article on skills

Frankly, I was wrong. That’s been happening more as I’ve been covering the bleeding edge, I suppose it’s something I’ll just have to get used to.

I stand by the opinion (at least for now) that there are attributes like FORMS.md that are vaguely defined and hardly adopted, but the core of the skills specification is inspired. It’s inspired because of one simple thing:

It allows the people writing skills, and the agents using skills, to leverage a shared agreement on structure.

This is fundamental and absolutely critical. Simple things, like the markdown file that defines a skill having a name and a description, allows agentic systems to index, scan through, make decisions on, and leverage resources within multiple skills efficiently and (somewhat) predictably. Skills make functionality in agents portable, maintainable, and efficient, and are thus gaining adoption pretty much anywhere agents run in the real world.

Just the other day, my partner asked Alexa if it could play rainforest noises, and alexa said it “encountered an error when running the rainforest noises skill”. The biggest productionalized AI systems are using skills under the hood.

In a nutshell, constraints are a fundamentally important tool in designing AI systems, and skills are a monumental leap forward in making agents practically useful through standardization. I don’t think it’s enough, though, which is why I’m working on the agent harnesses standard. Before I explain the standard I’m working on, I want to take a moment to explain limitations around the current status quoe of agentic design.

Limits of Constraints and Skills

The only time I’ve talked publicly about agent constraints is in a reddit post where I break down my approach to making AI systems improvable. The essential idea is that graph-based agents are easier to make robust when working on complex tasks, whereas unconstrained “ReAct-esque” agents can be finicky to improve reliably.

This unconstrained approach isn’t so great, I’ve found, when you build an AI agent to do something specific and complicated. If you have some logical process that requires a list of steps and the agent messes up on step 7, it’s hard to change the agent so it will be right on step 7, without messing up its performance on steps 1–6. It’s hard because, the way you define these agents, you tell it how to behave, then it’s up to the agent to progress through the steps on its own. Any time you modify the logic, you modify all steps, not just the one you want to improve. I’ve heard people use “whack-a-mole” when referring to the process of improving agents. This is a big reason why.
I call graph based agents “constrained agents”, in contrast to the “unconstrained agents” we discussed previously. Constrained agents allow you to control the logical flow of the agent and its decision making process. You control each step and each decision independently, meaning you can add steps to the process as necessary. — From my Reddit post on improvable AI

This is great; if you want to make a super reliable agent just use a graph. The issue is that, for even the most simple of tasks, a decent graph can be incredibly difficult to build. Later we’ll be discussing a harness that can be loaded by the Claude CLI to help someone apply to jobs. In theory this is a simple job that could be defined as a graph:

look for job opportunities
write a resume
help the user answer questions and help with application

Great… accept what about the user wanting to review resumes? What about the user having specific requests for specific resumes for specific job opportunities? What about if the job listing requires JS rendering to be viewed properly? What about if some job listings have an option for a cover letter, and some don’t? What if the user has specific rules they want the agent to follow when crafting a resume? What if the user wants to skip all this stuff and just use the agent to help them answer questions for a form in a job posting?

As humans we experience this all the time. Chefs, drivers, artists, writers, project managers, software developers. We all have a little graph of what our job is in general: you do this, then this, then this, then that. Except when you have to do this. Also sometimes that happens. The graph of procedures we use to govern our lives exists, but it’s fuzzy. We have a structure to help us, but that structure must be supplemented with contextual decision-making that exists outside of any structure. So, building super rigidly constrained agents isn’t enough.

The skills standard also isn’t enough. let me be clear, I think skills are vital and fundamental, but we need something else on top of them.

Right now, modern agentic systems typically look something like this:

A .claude directory
A bunch of skills in the .claude directory
A README.md, and some other markdowns, that describe how the agent ought to behave.

I didn’t know this until recently, but this approach was proposed and coined by Karpathy in the form of an “LLM Wiki”:

The idea here is different. Instead of just retrieving from raw documents at query time, the LLM incrementally builds and maintains a persistent wiki — a structured, interlinked collection of markdown files that sits between you and the raw sources. When you add a new source, the LLM doesn’t just index it for later retrieval. It reads it, extracts the key information, and integrates it into the existing wiki — updating entity pages, revising topic summaries, noting where new data contradicts old claims, strengthening or challenging the evolving synthesis. The knowledge is compiled once and then kept current, not re-derived on every query. — source

This is great, and actually works remarkably well; people are using a bunch of skills and a bunch of markdown files to automate all sorts of stuff right now.

Agree. We are using a version of Karparthy’s LLM Wiki as a harness and it works perfectly at the scale we need. It builds the graph by itself as needed and also restructures it by itself while it grows. — Another response I gratefully received on my agent harnesses article.

However, if you use this approach for long enough on sufficiently complex tasks, you start to identify a few problems:

Every time you start a new session with an agent, it completely forgets your entire wiki. You might have things written down, but if an agent doesn’t find and load that into context, it will ignore what’s written.
For big tasks you need a lot of skills, reference files, assets, MCP servers, and other stuff. A flat folder structure containing an exhaustive list of skills and MCPs becomes difficult to maintain, just like having all of your files on your desktop rather than organized by folders would become difficult to maintain. In complex systems, you need to organize tools and information in complex hierarchical ways.

By employing skills and an LLM Wiki alone, the following issues have a tendency to occur:

It can take forever for an agent to understand the environment it’s working in on initialization
Or, the agent doesn’t take the time to understand its environment, and completely ignores documentation
If you use an LLM to maintain a Wiki, it forgets where it wrote things and creates disorganized, duplicate, and contradictory information
It’s difficult to configure an LLM Wiki, as the agent has a tendency to put whatever, wherever. On large LLM Wiki’s, making minor adjustments in an agent’s decision-making can be challenging

When working in claude, you’ll find further issues

Skills must obey a flat structure within .claude, and can’t be organized
Thus, skills can’t be packaged within a greater context. If you have high level prompts that describe a role in a directory structure, you have to pair that with a .claude directory with the correct corresponding skills

The goal of the “Agent Harnesses” spec is to alleviate these issues by:

Making the degree of constraint the harness proposes on the agent easily configurable by a maintainer.
Creating a defined routing system within a directory structure that both humans and agents can understand to make decisions and maintain information.
Allowing all types of context and tools to be bundled arbitrarily within that directory structure.
Expose all of this information in an efficient and predictable manner, allowing agents to perform a task consistently and efficiently.

As I said before, the Agent Harnesses spec is currently in active development. It’s still in soft-release territory. That said, I want to briefly share some of the new constructs within it, and an example of an efficient harness allowing for complex yet maintainable functionality using claude CLI.

Concepts in Agent Harnesses

Agent Harnesses is heavily inspired by both MCP and Agent Skills, but it’s particularly inspired by agent skills. The structure of the project is virtually a copy of Agent Skills, down to the service running the website.

The most fundamental inspiration Agent Harnesses takes from Agent Skills is that of “progressive disclosure”. Progressive disclosure is the idea of allowing an agent to peek at a summary of information before considering a large amount of information. When you make a skill, you define a high level description. When the agent is looking for a skill to solve a problem, it can use these descriptions to peek at many skills efficiently, then choose which ones are worth looking into more.

skill: send_email
  desc: Send emails via SMTP or Gmail API. Handles attachments,
        HTML formatting, and bulk sending.

skill: query_database
  desc: Run SQL queries against PostgreSQL or MySQL. Returns
        structured results and handles connection pooling.

skill: resize_image
  desc: Resize, crop, or convert images between formats (PNG,
        JPEG, WebP). Supports batch processing.

skill: parse_pdf
  desc: Extract text, tables, and metadata from PDF files.
        Handles scanned docs via OCR.

skill: call_rest_api
  desc: Make authenticated HTTP requests to external APIs.
        Supports OAuth, API keys, and retries.

The Agent Harnesses standard uses key files to achieve a similar effect, but across an entire directory. Just like Agent Skills, this simplicity is what makes Agent Harnesses powerful.

All of the files defined in the Agent Harnesses standard are designed to do two things:

Provide information as it’s needed to the agent
Provide references to where further information can be found

The core of a harness is a HARNESS.md file, which is at the root of the harness directory. It’s essentially a README.md file designed for agents, providing high-level information so the agent can understand the fundamentals of the harness at a glance.

Within the root of a harness there will likely be various sub-directories. These “top-level subdirectories” are important to the agent harnesses specification as they define the macroscopic buckets used to organize the agent. These top-level subdirectories can be arbitrarily defined, but should be considered based on the application your working on. You might have a harness that looks like this

marketing-assistant-harness/
├── HARNESS.md
├── capabilities/
├── brand/
└── campaigns/

or this

data-analyst-harness/
├── HARNESS.md
├── tools/
├── data/
└── outputs/

or this

tech-support-harness/
├── HARNESS.md
├── playbooks/
├── integrations/
└── environment/

whatever works for the project you’re working on.

Agent harnesses uses these top-level subdirectories as keywords used to define markdown files that route within them. If you have a tools/ top-level subdirectory, the agent harnesses spec expects markdown files called TOOLS.md to be used to provide routing information throughout the subdirectories within tools/.

Here’s an example of a harness structure. Notice how TOOLS.md exists in tools/, tools/database/, tools/files/ , tools/compute/ , and tools/external/. because tools/ is a top-level subdirectory within the harness, it looks to TOOLS.md to provide routing information within. Also, notice how data/ and outputs/ has a similar structure.

data-analyst-harness/
├── HARNESS.md
├── tools/
│   ├── TOOLS.md
│   ├── database/
│   │   ├── TOOLS.md
│   │   ├── query-database/
│   │   │   ├── SKILL.md
│   │   │   ├── scripts/
│   │   │   └── references/
│   │   └── write-database/
│   │       ├── SKILL.md
│   │       └── scripts/
│   ├── files/
│   │   ├── TOOLS.md
│   │   ├── read-spreadsheet/
│   │   │   ├── SKILL.md
│   │   │   └── scripts/
│   │   ├── read-csv/
│   │   │   ├── SKILL.md
│   │   │   └── scripts/
│   │   └── read-json/
│   │       ├── SKILL.md
│   │       └── scripts/
│   ├── compute/
│   │   ├── TOOLS.md
│   │   ├── run-python/
│   │   │   ├── SKILL.md
│   │   │   └── scripts/
│   │   └── run-sql/
│   │       ├── SKILL.md
│   │       └── scripts/
│   └── external/
│       ├── TOOLS.md
│       ├── call-rest-api/
│       │   ├── SKILL.md
│       │   └── scripts/
│       └── fetch-warehouse/
│           ├── SKILL.md
│           └── scripts/
├── data/
│   ├── DATA.md
│   ├── relational/
│   │   ├── DATA.md
│   │   ├── schema-overview.md
│   │   ├── table-relationships.md
│   │   └── field-definitions.md
│   ├── warehouse/
│   │   ├── DATA.md
│   │   ├── dataset-catalog.md
│   │   └── partition-strategy.md
│   └── quirks/
│       ├── DATA.md
│       ├── known-data-quirks.md
│       └── null-handling.md
└── outputs/
    ├── OUTPUTS.md
    ├── visual/
    │   ├── OUTPUTS.md
    │   ├── generate-chart/
    │   │   ├── SKILL.md
    │   │   └── scripts/
    │   └── generate-table/
    │       ├── SKILL.md
    │       └── scripts/
    ├── written/
    │   ├── OUTPUTS.md
    │   ├── write-summary/
    │   │   └── SKILL.md
    │   └── write-report/
    │       └── SKILL.md
    └── export/
        ├── OUTPUTS.md
        ├── export-csv/
        │   ├── SKILL.md
        │   └── scripts/
        └── export-pdf/
            ├── SKILL.md
            └── scripts/

Like the agent skills standard, agent harnesses expects these files to have key markdown information to assist in routing decisions. HARNESS.md must have a name and description as key-value front matter along with the body, like so:

---
name: job-application-harness
description: Assists a job seeker through the full application workflow — from researching roles and companies through tailoring materials, tracking applications, and maintaining the harness.
---

This harness supports a job seeker managing an active job search. It covers researching target companies and roles, crafting and tailoring application materials, tracking the application pipeline, and maintaining the harness itself as the search evolves.
...

Then the various routing files, like the TOOLS.md file discussed previously, must have a description along with a body.

---
description: Technical tools for DB admins, files, compute, and external integrations
---

- **database/** — Tools for interacting with the database
...

This allows an agent to peek at these files before making a routing decision.

The practical result of the agent harnesses spec is that a harness ends up containing a fuzzy tree-like structure, where HARNESS.md is the root node, and various key markdown files can be used to encourage certain routing decisions as the agent propagates down the tree.

Structurally, the agent harnesses standard defines a harness as a directory structure with key routing files within it.

These files contain natural text and thus can reference anything, not only their children, allowing a harness to articulate the edge cases and nuances endemic to complex agentic problems.

Because these files are written in natural text, they can communicate the complex and subtle routing information that is often required in complex agentic systems.

A natural follow-up question is this: Where does a harness end?

You might want to define a harness that can explore every nuance of a codebase for helping developers work on code. In that case, every code file should be in the view of the harness.

It’s worth noting, while the agent harnesses standard specific names for routing files, they are not mandatory. Thus, where to put them throughout a project is at the discretion of the developer. Here, these folders define a skill. The standard for a skill is to have a SKILL.md file, which describes the content.

However, if you have code that you’re happy with and you want to build a harness that helps non-technical people leverage that code, you might just want the harness to know that skills exist and not to explore beneath that level.

In a different example, you might imagine a large store of knowledge, which you’ve designed a specific skill to traverse. You might want to make it so the harness considers the large store of knowledge out of scope so that an agent is forced to use a tailored skill to access it.

To enable these types of use cases, the agent harnesses spec employs both a terminator document, .harnessleaf, and terminator patterns (like skill.md) to know when a folder should not be explored further. This allows developers to constrain where agents should explore, limiting their scope to what is necessary.

So that’s the agent harnesses standard. HARNESS.md, some intermediary markdown files for routing, and key files for terminating scope. The point isn’t to define a structure for its own sake, but to leverage this structure to make harnesses better. Let’s explore how we can use the agent harnesses “meta skill” to connect Claude with harnesses.

Using Agent Harnesses with Claude

The idea of the agent harnesses standard is that the authors of the agent, and the author of the harness, might agree on a shared upon specification.

However, agent harnesses is new, so Claude has no knowledge of the standard, it’s structure, or optimization around that standard. Thus I created the agent harnesses “metaskill”, which allows agents that don’t have native support for the agent harnesses standard, but do have support for skills, to understand agent harnesses. You can find the implementation of the agent harnesess metaskill here:

It essentially functions as an entry point into the agent harnesses standard, and has scripts and prompts that are optimized to guide an agent to efficiently leverage a standardized agent harness.

I’m still working on the metaskill, and improving it to deal with different harnesses, agentic clients, and applications. At its present, it has the following core functionality:

summarize.py — Briefly summarizes the project structure for an agent, so that it can quickly understand the harness and its contents.
disclose.py — Allows the agent to traverse through the harness via progressive disclosure to find resources relevant to a task
map_references.py — produces a structural overview of the harness, allowing the agent to view structural information at a glance
reverse_disclose.py — allows the agent to see routing documents before a given file, making maintenance of routing information more robust.

The skill also has a directory for exploration sessions, which allows an agent to keep track of how it’s explored a harness when trying to find specific information.

I plan on making agents that have native support for the agent harnesses spec in the near future, and I hope the standard proves useful enough that it might be adopted by popular agentic systems natively, but for now the metaskill is a convenient bridge that allows popular existing agents to use harnesses with a fair degree of robustness.

With the metaskill, we can now build a harness and use it with Claude.

Exploring a Job Application Assistant using Agent Harnesses

I’m working on a few example harnesses that explore how the agent harnesses standard can be employed to satisfy a variety of use cases. I’d like to take a moment to walk through one, a harness designed to assist people in finding a job.

It’s design operates under a fundamental ethos: AI agents are bad at writing, but they’re good at understanding large amounts of information and aligning that information towards a goal. Thus, the core of the system is the “tailor” skill, which tells the agent to look at existing resumes and use the information therein to create new resumes and cover letters tailored to a particular job posting.

Another fundamental idea is lead management. The harness maintains a leads.csv file that keeps track of job postings, the resumes tailored to them, their status, links, their level of priority based on user defined criteria, etc. The resume tailoring skill, and skills around exploring the internet to find new job postings, all use the leads.csv file as a central source of truth to coordinate searches and job applications.

It’s worth noting; if you look through the repo you won’t actually see leads.csv, nor resumes, job descriptions, etc. However, there are instructions on how to set these things up for a new user, which the harness has access to. Thus, when an agent loads the harness for the first time, it will help the user set up the requisite information.

To facilitate these various high-level workflows of finding jobs, writing resumes, and doing other supporting work, the harness has some functionality built into it. Notably in dealing with docx files

and a config file telling the agent to connect to playwright, which is an MCP server that allows for browser automation. This is necessary in viewing and confirming job listings.

{
  “mcpServers”: {
    “playwright”: {
      “command”: “npx”,
      “args”: [”@playwright/mcp@latest”]
    }
  }
}

This convention is compatible with Claude. Similar files could be use to expose Playwright to agents like Cursor, Windsurf, and Codex.

If we clone this repo and navigate into the job-application-harness directory

git clone https://github.com/agentharnesses/exampleharnesses.git
cd exampleharnesses/job-application-harness

we can go ahead and initialize claude.

danielwarfield@Mac job-application-harness % claude
╭─── Claude Code v2.1.191 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                    │ Tips for getting started                                                                  │
│                Welcome back Daniel!                │ Run /init to create a CLAUDE.md file with instructions for Claude                         │
│                                                    │ ───────────────────────────────────────────────────────────────────────────────────────── │
│                       ▐▛███▜▌                      │ What’s new                                                                                │
│                      ▝▜█████▛▘                     │ Added `/rewind` support for resuming a conversation from before `/clear` was run          │
│                        ▘▘ ▝▝                       │ Fixed scroll position jumping to the bottom while reading earlier output during a stream… │
│                                                    │ Fixed background agents resurrecting after being stopped — stopping an agent from the ta… │
│     Sonnet 4.6 · Claude Pro · Daniel Warfield      │ /release-notes for more                                                                   │
│ ~/…/exampleharnesses-fresh/job-application-harness │                                                                                           │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯


──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
❯ Try “edit  to...”
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ? for shortcuts · ← for agents

Recall, Claude has no understanding of the agent harnesses spec, nor of this harness in particular. This harness comes packaged with a pre-defined .claude directory which contains the metaskill, though, so we can fix both those issues by telling claude to load, load harness, initialize, or whatever.

after telling it to load, Claude uses the metaskill to get a summary of the harness structure

load the harness                                                                                                                                

⏺ Skill(agent-harnesses)
  ⎿  Successfully loaded skill  

⏺ Let me start by summarizing the harness structure.

⏺ Bash(python
      /Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/.claude/skills/agent-harnesses/scripts/summarize.py
      /Users/danielwa…)
  ⎿  job-application-harness/
     ├── [harness]  HARNESS.md  — Assists a job seeker through the full application workflo...
     │
     … +80 lines (ctrl+o to expand)

It then realizes this is a new initialization, and decides to use the progressive-disclosure skill in the meta-skill to explore the harness to a greater degree of depth.

Now let me start a disclosure session and load the HARNESS.md to get the full context.

⏺ Bash(python
      /Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/.claude/skills/agent-harnesses/scripts/disclose.py
      /Users/danielwar…)
  ⎿  {
       “status”: “exploring”,
       “session”: “8f15d70e”,
     … +25 lines (ctrl+o to expand)

  Read 1 file (ctrl+o to expand)

⏺ Bash(python
      /Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/.claude/skills/agent-harnesses/scripts/disclose.py
      --session 8f15d7…)
  ⎿  {
       “status”: “exploring”,
       “session”: “8f15d70e”,
     … +31 lines (ctrl+o to expand)

...

I think the reason it’s doing progressive disclosure is to make sure it properly understands the repo. Specifically, it’s trying to understand that this is a fresh install. It doesn’t usually do this on a more fleshed out instance of the harness that has resumes, job listings, and other stuff.

Progressive disclosure takes time to run. If we wanted to discourage the agent from doing an exhaustive search on initialization, we could specify that in HARNESS.md to discourage that functionality.

...
# How to Find Information for Claude

Use the `agent-harnesses` skill to explore the harness. On initial load, run `summarize.py` to get a full picture of the harness structure — this is sufficient to understand what skills and references are available without needing a full disclosure session. Reserve `disclose.py` for when you need to read specific skill or reference content to act on a user request.
...

Re-initializing a new session, we can see that Claude formed a high-level understanding of the repo in 14 seconds, without using progressive disclosure.

anielwarfield@Mac job-application-harness % claude
╭─── Claude Code v2.1.191 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                    │ Tips for getting started                                                                  │
│                Welcome back Daniel!                │ Run /init to create a CLAUDE.md file with instructions for Claude                         │
│                                                    │ ───────────────────────────────────────────────────────────────────────────────────────── │
│                       ▐▛███▜▌                      │ What’s new                                                                                │
│                      ▝▜█████▛▘                     │ Added `/rewind` support for resuming a conversation from before `/clear` was run          │
│                        ▘▘ ▝▝                       │ Fixed scroll position jumping to the bottom while reading earlier output during a stream… │
│                                                    │ Fixed background agents resurrecting after being stopped — stopping an agent from the ta… │
│     Sonnet 4.6 · Claude Pro · Daniel Warfield      │ /release-notes for more                                                                   │
│ ~/…/exampleharnesses-fresh/job-application-harness │                                                                                           │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯


❯ load                                                                                                                                            

⏺ Skill(agent-harnesses)
  ⎿  Successfully loaded skill  

⏺ Starting with a harness summary to get oriented.

⏺ Bash(python
      /Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/.claude/skills/agent-harnesses/scripts/summarize.py
      /Users/danielwa…)
  ⎿  job-application-harness/
     ├── [harness]  HARNESS.md  — Assists a job seeker through the full application workflo...
     │
     … +80 lines (ctrl+o to expand)

⏺ The job-application-harness is loaded and ready. Here’s what’s available:

  Skills:
  - explore/find-listings — search for and source new job leads
  - explore/job-description — load/scrape a job posting into the harness
  - lead-management/apply — open a listing in the browser to submit an application
  - lead-management/recommended-apply — pick the best lead to apply to next
  - materials/answer — write answers to open-ended application questions
  - materials/docx — create/edit Word documents (.docx)
  - materials/tailor — tailor resume and cover letter to a specific job

  References:
  - references/setup.md — setup instructions for a fresh clone
  - skills/explore/find-listings/references/currently-working-sources.md — notes from 68 leads collected as of 2026-06-24

  What would you like to do?

✻ Cogitated for 14s

If we ask it a question, like how to get started, it will be able to quickly find and reference the correct source information.

❯ how do we get started?                                                                                                                          

⏺ Reading the setup instructions and harness root to give you an accurate answer.

  Read 2 files (ctrl+o to expand)

⏺ Here’s what you need to do to get started:

  1. Create references/leads.csv — the pipeline tracker:
  company,job_title,posting_url,status,applied_date,resume_file,compensation,location,job_description_file

  2. Create references/target-roles.md — your personal role profile. Suggested sections:
  - What I’m Looking For
  - Role Types (primary targets + acceptable alternatives, prioritized)
  - Core Differentiators to Emphasize
  - Industries / Companies
  - What Makes a Strong Fit
  - What to Avoid

  3. Add a base resume template — copy your .docx resume into references/resumes/ as TEMPLATE_*.docx. The tailor skill uses this as the base for
  all tailored resumes.

  The three data directories (references/resumes/, references/job-descriptions/, references/answers/) already exist from the clone — no action
  needed there.

  Once those three files are in place, the full workflow is available: find listings → load job descriptions → tailor materials → apply.

  Want help drafting target-roles.md, or do you have an existing resume to drop in first?

✻ Crunched for 9s

I want it to go ahead and draft up something to get started.

❯ yeah draft up what you need.                                                                                                                    

⏺ Let me check what’s already in the repo before drafting anything.

  Searched for 1 pattern (ctrl+o to expand)

⏺ Nothing exists yet beyond the empty directories. I need a few things from you to draft target-roles.md — the leads.csv I can create now since
  it’s just a header row, but the role profile needs your input.

  A few quick questions:

  1. What roles are you targeting? (titles, seniority level — e.g. “Senior ML Engineer”, “Head of Data Science”)
  2. What’s your background? (short version — what you’ve done, what makes you stand out)
  3. Any industry preferences or hard avoids? (e.g. “prefer fintech, avoid defense”)
  4. Location / remote preference?

  The resume template I can’t generate — you’ll need to drop your .docx into references/resumes/ yourself. But I can create the CSV and a draft
  target-roles.md as soon as I have the above.

✻ Cooked for 13s

The whole idea is that this system takes existing resumes and refines them, so instead of following these setup instructions, I’ll tell it my resume is in the downloads folder.

❯ my resume is in my downloads folder, use that.                                                                                                  

  Searched for 1 pattern (ctrl+o to expand)

⏺ Found it. Let me read it to understand your background before drafting the role profile.

⏺ Bash(python -c “
      from docx import Document…)
  ⎿  Daniel Warfield
     Austin, TX | hire@danielwarfield.dev
     Website
     … +43 lines (ctrl+o to expand)

⏺ Good picture of your background. Let me also check the tailor skill’s template requirements, then I’ll create everything at once.

  Read 1 file (ctrl+o to expand)

⏺ The tailor skill expects a specific template file: TEMPLATE_DanielWarfield_Resume_cover.docx with a cover letter page prepended. Your Downloads
  resume is the resume half — I’ll copy it over, but flagging now that you’ll need to add the cover letter page to it before the tailor skill
  works end-to-end.

  Creating the three files now:

⏺ Bash(cp “/Users/danielwarfield/Downloads/2026_04_17_DanielWarfield_Resume (1).docx”
      “/Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-ha…)
  ⎿  Done

⏺ Write(references/leads.csv)
  ⎿  Wrote 1 lines to references/leads.csv
      1 company,job_title,posting_url,status,applied_date,resume_file,compensation,location,job_description_file

⏺ Write(references/target-roles.md)
  ⎿  Wrote 55 lines to references/target-roles.md
       2 # Target Role Profile — Daniel Warfield
       3 ## What I’m Looking For
       4 
       5 Senior technical roles at the intersection of AI/ML and product delivery — where I can own meaningful systems end to end, drive re
         venue outcomes, and operate with autonomy. The sweet spot is hands-on technical leadership: still building, but shaping direction 
       6 and managing delivery.
       7 ## Role Types
       8
       9 **Primary targets (in priority order):**
      10 1. Technical Lead / Principal Engineer — AI/ML, agentic systems, or LLM infrastructure
     … +47 lines (ctrl+o to expand)

⏺ Done. Here’s what was created:

  ┌──────────────────────────────────────────────────────────────┬──────────────────────────────────────────────┐
  │                             File                             │                    Status                    │
  ├──────────────────────────────────────────────────────────────┼──────────────────────────────────────────────┤
  │ references/leads.csv                                         │ Created with header row, ready for leads     │
  ├──────────────────────────────────────────────────────────────┼──────────────────────────────────────────────┤
  │ references/target-roles.md                                   │ Drafted from your resume — review and adjust │
  ├──────────────────────────────────────────────────────────────┼──────────────────────────────────────────────┤
  │ references/resumes/TEMPLATE_DanielWarfield_Resume_cover.docx │ Copied from Downloads                        │
  └──────────────────────────────────────────────────────────────┴──────────────────────────────────────────────┘

  One thing to follow up on: The tailor skill expects the template to have a cover letter page prepended with specific paragraph IDs
  (00000004–0000000C). Right now the template is just your resume. Before you tailor materials for the first time, that cover letter structure
  needs to be added to the template. We can tackle that when you’re ready to tailor.

  Take a look at target-roles.md and let me know what to adjust — I made some assumptions about role priorities and company stage preferences.

✻ Brewed for 1m 5s

And, now that it has an idea of what jobs I’m looking for, I can have it scan around for some jobs.

❯ Find five jobs                                                                                                                                  

  Read 2 files (ctrl+o to expand)

⏺ Running parallel searches across high-signal sources now.

⏺ Web Search(”site:builtinaustin.com “AI engineer” OR “ML engineer” OR “technical lead” 2026”)

⏺ Web Search(”site:builtinaustin.com “director of engineering” OR “head of AI” OR “staff engineer” AI”)
 
  Web Search(”site:job-boards.greenhouse.io “machine learning” OR “AI engineer” Austin OR remote 2026”)

⏺ Web Search(”site:builtinaustin.com “director of engineering” OR “head of AI” OR “staff engineer” AI”)
  ⎿  Did 1 search in 8s

⏺ Web Search(”site:job-boards.greenhouse.io “machine learning” OR “AI engineer” Austin OR remote 2026”)
  ⎿  Did 1 search in 8s

⏺ Web Search(”site:ashbyhq.com “technical lead” OR “staff engineer” OR “principal engineer” AI OR LLM”)
  ⎿  Did 1 search in 10s

⏺ Web Search(”site:wellfound.com “AI engineer” OR “ML engineer” Austin OR remote senior 2026”)
  ⎿  Did 1 search in 8s

⏺ Good haul. Filtering and ranking against your profile now.

  Here are the top 5 candidates:

  ---
  Technical Lead / Staff / Principal — AI
  
  1. TRM Labs — Staff Software Engineer, Agent Engineering (US Remote)
  Fraud/compliance tech company building LLM-powered agentic systems — direct overlap with FraudX. Staff level with real ownership.
  https://jobs.ashbyhq.com/trm-labs/eb560943-3adb-4839-bf64-4cbdeee9ae57

  2. Hippocratic AI — Senior Staff AI Engineer (Remote)
  Sets direction across RAG, agent orchestration, evaluation, real-time voice, and safety. Matches almost every line of your resume —
  observability, agentic design, LLM eval.
  https://jobs.ashbyhq.com/Hippocratic%20AI/af39c865-a9ab-403e-8746-79dec87d6ad0

  3. Glean — ML Engineer, AI Assistant & Autonomous AI Agents (Remote)
  Enterprise AI search platform, autonomous agents focus. Well-funded, high technical bar.
  https://job-boards.greenhouse.io/gleanwork/jobs/4605215005

  ---
  Director / Head of Engineering

  4. SEON — Director of Engineering (Austin/Remote)
  Fraud detection SaaS — directly adjacent to FraudX. Director-level, manages engineering strategy and teams. ⚠️  SEON is mid-size; worth checking
  comp.
  https://www.builtinaustin.com/job/director-engineering/9615304

  5. Dropbox — Director of Engineering, New AI Initiatives (Remote)
  Leading new AI productivity initiatives, people management + technical direction. Large-company autonomy risk, but strong brand and likely
  strong comp.
  https://www.builtinaustin.com/job/engineer/director-engineering-new-ai-initiatives/135271

  ---
  Which of these would you like to add to the pipeline?

It knows I’m based out of Austin, and it’s referencing the file

skills/explore/find-listings/references/currently-working-sources.md

to prioritize certain websites. If I give the harness high-level information like “I’m moving to California, change the harness accordingly,” it can update the relevant content because it has a strong idea of what information exists and where it exists.

❯ I’m moving to California, change the harness accordingly                                                                                        

⏺ Two files need updating: target-roles.md (Austin comp reference) and currently-working-sources.md (builtinaustin.com is the wrong surface now).

⏺ Update(references/target-roles.md)
  ⎿  Added 1 line, removed 1 line
      44  - Company is past “can we use AI?” and into “how do we scale this?”
      45  - Technical bar is high; teammates I’d learn from
      46  - Problem space I’d want to write or talk about publicly
      47 -- Compensation competitive with Austin/remote senior IC or director market                                                       
      47 +- Compensation competitive with California/remote senior IC or director market                                                   
      48  
      49  ## What to Avoid
      50  

  Read 1 file (ctrl+o to expand)

⏺ Update(skills/explore/find-listings/references/currently-working-sources.md)
  ⎿  Added 5 lines, removed 5 lines
       4  
       5  ## High Signal
       6  
       7 -**builtinaustin.com**                                                                                                            
       8 -- Best discovery surface for Austin-specific and Austin-remote roles                                                             
       9 -- Reliable location filtering — rarely surfaces SF-only or NYC-only postings labeled as Austin                                   
      10 -- Two search waves (16 candidates total) had ~87% signal rate (14/16 active, 2 expired)                                          
      11 -- Use queries like: `site:builtinaustin.com “AI engineer” OR “ML engineer”`, `site:builtinaustin.com “solutions architect” AI`   
       7 +**builtinsf.com / builtinla.com**                                                                                                
       8 +- builtinaustin.com was previously high signal (87% rate over 68 leads) but is no longer the target market — user has moved to Ca
         +lifornia                                                                                                                         
       9 +- builtinsf.com covers the Bay Area; builtinla.com covers LA — use both depending on target sub-region                           
      10 +- Expect similar signal quality to builtinaustin.com; location filtering is reliable across Built In sites                       
      11 +- Use queries like: `site:builtinsf.com “AI engineer” OR “ML engineer”`, `site:builtinla.com “technical lead” AI`                
      12  - Compensations are usually present and accurate
      13  
      14  **Direct company career pages**

⏺ Update(skills/explore/find-listings/references/currently-working-sources.md)
  ⎿  Added 2 lines, removed 2 lines
      22  **Greenhouse (greenhouse.io / job-boards.greenhouse.io)**
      23  - Reliable for mid-size and larger startups (Anthropic, Hightouch, LaunchDarkly, Avride, Apptronik)
      24  - Direct ATS URLs don’t expire as fast as aggregator mirrors
      25 -- Use: `site:job-boards.greenhouse.io “machine learning” OR “AI engineer” Austin OR remote`                                      
      25 +- Use: `site:job-boards.greenhouse.io “machine learning” OR “AI engineer” California OR remote`                                  
      26  
      27  **Ashby (ashbyhq.com / jobs.ashbyhq.com)**
      28  - Strong coverage of frontier AI and well-funded startups (Baseten, Cognition, H Company, webAI, Writer, Cohere, Eigen Labs)
     ...
      31  **Workday (myworkdayjobs.com)**
      32  - Covers larger enterprises (NVIDIA, Guidewire, CrowdStrike, Zoom)
      33  - Postings tend to stay live longer than aggregators
      34 -- Hard to search directly; best reached via Google: `site:myworkdayjobs.com “AI” Austin OR remote`                               
      34 +- Hard to search directly; best reached via Google: `site:myworkdayjobs.com “AI” California OR remote`                           
      35  
      36  ## Low Signal — Use Sparingly
      37  

⏺ Update(skills/explore/find-listings/references/currently-working-sources.md)
  ⎿  Added 1 line, removed 1 line
      41  - Avoid unless testing a very specific query
      42  
      43  **Lever (lever.co / jobs.lever.co)**
      44 -- 4 leads collected; 2 expired before application, 1 non-Austin, 1 applied (Palantir)                                            
      44 +- 4 leads collected; 2 expired before application, 1 non-California, 1 applied (Palantir)                                        
      45  - Higher expiration rate than Greenhouse or Ashby    
      46  - Still worth including in parallel searches but don’t over-index
      47  

⏺ Update(skills/explore/find-listings/references/currently-working-sources.md)
  ⎿  Added 2 lines, removed 2 lines
      49  
      50  **LinkedIn Jobs**
      51  - Not yet used; likely largest volume for established companies
      52 -- Austin + Remote filter is reliable                                                                                             
      52 +- California + Remote filter is reliable                                                                                         
      53  - Best for: Microsoft, Meta, Snowflake, Confluent, Anyscale, and other large-company ML/AI roles
      54 -- Query shape: search LinkedIn Jobs for “machine learning engineer” or “solutions architect AI” in Austin TX or Remote           
      54 +- Query shape: search LinkedIn Jobs for “machine learning engineer” or “solutions architect AI” in California or Remote          
      55  
      56  **Wellfound (wellfound.com, formerly AngelList)**
      57  - Not yet used; strong coverage of funded AI startups

⏺ Done. Changes made:

  - references/target-roles.md — comp benchmark updated from Austin to California market
  - skills/explore/find-listings/references/currently-working-sources.md — replaced builtinaustin.com with builtinsf.com / builtinla.com, updated
  all Austin location references to California throughout

  One thing I didn’t touch: your resume still says “Austin, TX” — you’ll want to update that before tailoring materials. Want me to do that now?

This isn’t foolproof, admittedly, but the structure imposed by agent harnesses makes these types of general edits both more reliable, and makes fixing issues around partial or incomplete edits more consistent. Paired with Claud’s implicit abilities to maintain complex code bases, misalignments between different documents are easily resolved.

❯ ensure that all localtion references align across the harness                                                                                   

  Searched for 1 pattern, read 1 file (ctrl+o to expand)

⏺ Three changes needed in recommended-apply/SKILL.md: the status soft-note example, the location ranking criterion, and the example output.

⏺ Update(skills/lead-management/recommended-apply/SKILL.md)
  ⎿  Added 1 line, removed 1 line
      15  
      16  ## Step 2: Filter to Pending Leads
      17  
      18 -A lead is pending if its `status` field is blank, or contains a soft note like “Verify Austin availability” or “Verify remote ava
         -ilability”. Exclude any lead where status is one of:                                                                             
      18 +A lead is pending if its `status` field is blank, or contains a soft note like “Verify California availability” or “Verify remote
         + availability”. Exclude any lead where status is one of:                                                                         
      19  
      20  - `Applied` (any variant, including “Applied - ...”)
      21  - `No Interview`

⏺ Update(skills/lead-management/recommended-apply/SKILL.md)
  ⎿  Added 1 line, removed 1 line
      28  Score each pending lead against the criteria in `target-roles.md`. In priority order:
      29  
      30  1. **Role type** — ML/AI Engineer scores highest; FDE/Solutions Architect is acceptable; SA/PM are lower
      31 -2. **Location** — Austin TX (on-site or hybrid) > Fully Remote > Out-of-state on-site                                            
      31 +2. **Location** — California (on-site or hybrid) > Fully Remote > Out-of-state on-site                                           
      32  3. **Company quality** — Large established company or well-known funded startup > smaller/earlier stage
      33  4. **Technical fit** — Agentic AI, production ML, LLMs, end-to-end ownership match the background best
      34  5. **Interest signal** — Robotics, frontier AI labs, and novel problem spaces rank up

⏺ Update(skills/lead-management/recommended-apply/SKILL.md)
  ⎿  Added 4 lines, removed 4 lines
      46  ```
      47  My recommendation:
      48  
      49 -1. **Cognite — ML Engineer, Atlas AI** (Austin TX)                                                                               
      50 -   Best available Austin ML engineer role — LLM agents + RAG, strong company fit.                                                
      49 +1. **Cognite — ML Engineer, Atlas AI** (California)                                                                              
      50 +   Best available California ML engineer role — LLM agents + RAG, strong company fit.                                            
      51  
      52 -2. **Avride — Machine Learning Engineer** (Austin TX)                                                                            
      53 -   Autonomous driving perception + LLMs, well-funded, Austin.                                                                    
      52 +2. **Avride — Machine Learning Engineer** (California)                                                                           
      53 +   Autonomous driving perception + LLMs, well-funded, California.                                                                
      54  
      55  3. **Cohere — Applied AI Engineer, Agentic Workflows** (Remote)
      56     Top-tier AI lab, pure ML engineering, agentic pipelines — best remote option.

⏺ Now fix the remaining “BuiltIn Austin” reference in the sources file:

⏺ Update(skills/explore/find-listings/references/currently-working-sources.md)
  ⎿  Added 1 line, removed 1 line
      56  **Wellfound (wellfound.com, formerly AngelList)**
      57  - Not yet used; strong coverage of funded AI startups
      58  - Many frontier AI companies appear here before BuiltIn or Greenhouse
      59 -- Good complement to BuiltIn Austin for startup-tier leads                                                                       
      59 +- Good complement to BuiltIn SF/LA for startup-tier leads                                                                        
      60  
      61  **Direct pages not yet checked**
      62  Companies in the target profile tier that haven’t been sourced yet:

⏺ Let me do a final check to confirm nothing was missed:

  Searched for 1 pattern (ctrl+o to expand)

⏺ Clean. All location references across the harness now point to California. Files updated:

  - references/target-roles.md — comp benchmark
  - skills/explore/find-listings/references/currently-working-sources.md — builtinaustin → builtinsf/builtinla, all query examples
  - skills/lead-management/recommended-apply/SKILL.md — status soft-note, location ranking criterion, example output

  Your resume still says “Austin, TX” — that’s outside the harness but worth updating before you apply anywhere.

✻ Baked for 38s

For big changes that fundamentally change the structure of the harness itself, the metaskill should invoke reverse_disclose.py, allowing the agent to quickly find key resources that need to be modified. Unfortunately, because Claud doesn’t natively understand the agent harnesses standard, it doesn’t know about its structure and may not think to invoke reverse disclosure on its own. So, you might have to give it some hints up front to use tooling in the harness to assist in maintenance.

❯ get rid of the resumes directory. I want all resumes to be generated on the fly. Follow harness maintinance.                                    

⏺ Following harness maintenance — running reverse disclosure on the resumes directory first to find every routing file that references it.

⏺ Bash(python /Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/.claude/skills/agent-harnesses/scripts/reverse
      _disclose.py /Users/d…)
  ⎿  {
       “status”: “complete”,
       “target”: “references/resumes”,
     … +12 lines (ctrl+o to expand)

⏺ Two routing files to update: REFERENCES.md and HARNESS.md. I already know the skill files that reference resumes/ from my earlier grep. Now I’ll
  make all changes, then move the template and delete the directory.

  Read 4 files (ctrl+o to expand)

⏺ Now I have everything I need. Making all changes in parallel:

⏺ Update(references/REFERENCES.md)
  ⎿  Added 2 lines, removed 1 line
       5  - **setup.md** — Setup instructions for a freshly cloned repo: what files and directories to create before the harness is usable
       6  - **leads.csv** — Application pipeline tracker: company, role, posting URL, status, applied date, resume used, compensation, loca
          tion
       7  - **job-descriptions/** — Saved job postings, one markdown file per role. Used as input for tailoring materials and assessing fit
          .
       8 -- **resumes/** — Tailored resumes, one per target role or application                                                            
       8 +- **background.md** — Canonical work history and competencies pool; read by the tailor skill to generate resumes on the fly      
       9 +- **TEMPLATE_DanielWarfield_Resume_cover.docx** — Base .docx template used by the tailor skill for all generated resumes         
      10  - **answers/** — Saved responses to application questions, including standard fields (name, email, phone, location, LinkedIn) and
           open-ended responses. Used as source material for future applications.
      11  - **target-roles.md** — Profile of target roles, preferred industries, company size, and criteria for evaluating fit

⏺ Update(HARNESS.md)
  ⎿  Added 2 lines, removed 1 line
      25  
      26  - **leads.csv** — Application pipeline tracker (company, role, posting, status, applied date, resume, compensation, location). **
          When an application is submitted, the `status` and `applied_date` columns must be filled in immediately** — blank status means pe
          nding, so an un-updated row will be re-recommended as a lead.
      27  - **job-descriptions/** — Saved job postings in markdown, used as input for tailoring materials and fit assessment
      28 -- **resumes/** — Tailored resumes, one per target role or application                                                            
      28 +- **background.md** — Canonical work history and competencies pool; source of truth for all generated application materials      
      29 +- **TEMPLATE_DanielWarfield_Resume_cover.docx** — Base .docx template used for all generated resumes                             
      30  - **answers/** — Saved responses to application questions, including standard fields (name, email, phone, location, LinkedIn) and
           open-ended responses. Used as source material for future applications.
      31  - **target-roles.md** — Profile of target roles, industries, and criteria for a good fit

⏺ Update(references/setup.md)
  ⎿  Added 1 line, removed 1 line
       8  
       9  ## Directories
      10  
      11 -The three reference directories (`resumes/`, `job-descriptions/`, `answers/`) are preserved in git via `.gitkeep` files and will 
         -exist immediately after cloning. No action needed.                                                                               
      11 +The two reference directories (`job-descriptions/`, `answers/`) are preserved in git via `.gitkeep` files and will exist immediat
         +ely after cloning. No action needed.                                                                                             
      12  
      13  ## Files to Create
      14  

⏺ Update(references/setup.md)
  ⎿  Added 2 lines, removed 2 lines
      34  - **What Makes a Strong Fit** — specific signals to look for
      35  - **What to Avoid** — hard or soft exclusions
      36  
      37 -### `references/resumes/TEMPLATE_*.docx`                                                                                         
      37 +### `references/TEMPLATE_DanielWarfield_Resume_cover.docx`                                                                       
      38  
      39 -The tailor skill uses a template `.docx` file as the base for all new resumes. Copy your base resume template into `references/re
         -sumes/` before tailoring any materials. The template must match the paraId structure expected by the tailor skill — see `skills/m
         -aterials/tailor/SKILL.md` for the required section IDs.                                                                          
      39 +The tailor skill uses this template `.docx` file as the base for all generated resumes. Copy your base resume template here befor
         +e tailoring any materials. The template must match the paraId structure expected by the tailor skill — see `skills/materials/tail
         +or/SKILL.md` for the required section IDs.                                                                                       

⏺ Update(skills/materials/tailor/SKILL.md)
  ⎿  Added 3 lines, removed 9 lines
      11  
      12  Before starting, confirm you have:
      13  - The job listing (pasted text or URL)
      14 -- Access to existing resumes in `references/resumes/`                                                                            
      14 +- Access to `references/background.md`                                                                                           
      15  
      16  If the job listing needs to be fetched and the page is JavaScript-rendered (WebFetch returns only the title), do not ask the user
           to paste it. Instead, provide the posting URL from `leads.csv` so the user can open it themselves and paste the text.
      17  
      18 -Read **every** resume in `references/resumes/` before drafting anything — not just the most obviously relevant ones. All existing
         - resumes are approved and contain valid source material; the goal is to find the best language for each point across the full set
         -. Convert all of them in parallel with `pandoc`:                                                                                 
      18 +Read `references/background.md` before drafting anything. It contains the canonical work history, bullets, and competencies pool 
         +— all source material for generated resumes. Do not invent content not present there.                                            
      19  
      20 -```bash                                                                                                                          
      21 -ls references/resumes/*.docx | grep -v TEMPLATE | xargs -I{} bash -c ‘pandoc --track-changes=all “{}” -o “/tmp/$(basename “{}” .d
         -ocx).md”’                                                                                                                        
      22 -```                                                                                                                              
      20 +When producing a new docx, always use `references/TEMPLATE_DanielWarfield_Resume_cover.docx` as the base. It has the correct form
         +atting, page break, and structure. Replace only the `` text nodes — do not rebuild runs or paragraphs. The paraIds for varia
         +ble sections are:                                                                                                                
      21  
      24 -Then read all the converted markdown files before proceeding.                                                                    
      25 -                                                                                                                                 
      26 -When producing a new docx, always use `references/resumes/TEMPLATE_DanielWarfield_Resume_cover.docx` as the base. It has the corr
         -ect formatting, page break, and structure. Replace only the `` text nodes — do not rebuild runs or paragraphs. The paraIds f
         -or variable sections are:                                                                                                        
      27 -                                                                                                                                 
      22  | Section | ParaIds |
      23  |---|---|
      24  | Cover letter opening | `00000004` |

⏺ Update(skills/materials/tailor/SKILL.md)
  ⎿  Added 3 lines, removed 3 lines
      59  **Core content:** Keep the work history, project descriptions, and accomplishments the same. Do not invent new bullets.
      60  
      61  **What to vary across versions:**
      62 -- **Core competencies** — select and reorder from across all existing resumes to match what the listing emphasizes. Keep each com
         -petency string to **30 characters or fewer** — the resume renders them in a 3-column layout and longer strings wrap, adding heigh
         -t that pushes content to a third page.                                                                                           
      63 -- **Skills / Tools** — pull from whichever existing resume frames the most relevant tools. Update the list to reflect both real e
         -xperience and what the listing calls out.                                                                                        
      64 -- **Framing** — where multiple resumes describe the same role differently, choose the framing that best aligns with the listing’s
         - language.                                                                                                                       
      62 +- **Core competencies** — select and reorder from the competencies pool in `references/background.md` to match what the listing e
         +mphasizes. Keep each competency string to **30 characters or fewer** — the resume renders them in a 3-column layout and longer st
         +rings wrap, adding height that pushes content to a third page.                                                                   
      63 +- **Skills / Tools** — pull from the skills listed in `references/background.md`. Update the list to reflect both real experience
         + and what the listing calls out.                                                                                                 
      64 +- **Framing** — where the background document offers multiple ways to describe a role, choose the framing that best aligns with t
         +he listing’s language.                                                                                                           
      65  
      66  ## Output Format
      67  

⏺ Update(skills/materials/answer/SKILL.md)
  ⎿  Added 1 line, removed 1 line
      21  - **Theme overlap** — does the existing answer address the same underlying capability (e.g., data pipelines, safety-critical depl
          oyment, evaluation systems, stakeholder communication)?
      22  - **Language reuse potential** — are there sentences or phrases that can be carried over with minimal change?
      23  
      24 -If no existing answers are relevant, fall back to the resume content in `references/resumes/` using the same approach as the tail
         -or skill.                                                                                                                        
      24 +If no existing answers are relevant, fall back to `references/background.md` using the same approach as the tailor skill.        
      25  
      26  ## Step 3: Draft the Answer
      27  

⏺ Update(skills/lead-management/apply/SKILL.md)
  ⎿  Added 1 line, removed 1 line
      15  
      16  Read `references/leads.csv` and find the matching row. Extract:
      17  - `posting_url` — the job listing URL to open
      18 -- `resume_file` — relative path to the tailored resume (e.g., `resumes/19_webAI_FDE_DanielWarfield_Resume_cover.docx`)           
      18 +- `resume_file` — relative path to the generated resume (e.g., `19_webAI_FDE_DanielWarfield_Resume_cover.docx`)                  
      19  
      20  If `resume_file` is blank, warn the user that no resume is on file for this role and ask whether to proceed without one or run th
          e tailor skill first.
      21  

⏺ Now move the template and delete the directory:

⏺ Bash(mv “/Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/references/resumes/TEMPLATE_DanielWarfield_Resume
      _cover.docx” “/Users/…)
  ⎿  Done

⏺ Now verify with reverse disclosure that background.md and the template are correctly referenced:

⏺ Bash(python /Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/.claude/skills/agent-harnesses/scripts/reverse
      _disclose.py /Users/d…)
  ⎿  {
       “status”: “complete”,
       “target”: “references/background.md”,
     … +30 lines (ctrl+o to expand)

⏺ Both are correctly referenced. Final sanity check — no stale resumes/ references anywhere:

  Searched for 1 pattern (ctrl+o to expand)

⏺ One stale reference in README.md:

  Read 1 file (ctrl+o to expand)

⏺ Update(README.md)
  ⎿  Added 1 line, removed 1 line
      27  
      28  - `references/leads.csv` — pipeline tracker (see `references/setup.md` for the header row)
      29  - `references/target-roles.md` — your role profile, used to rank and recommend leads
      30 -- `references/resumes/TEMPLATE_*.docx` — your base resume template                                                               
      30 +- `references/TEMPLATE_*.docx` — your base resume template                                                                       
      31  
      32  See `references/setup.md` for full setup instructions.
      33  

⏺ Clean. Here’s what changed:

  - Deleted references/resumes/ directory
  - Moved template to references/TEMPLATE_DanielWarfield_Resume_cover.docx
  - references/background.md — canonical content source, wired into routing
  - tailor/SKILL.md — reads background.md instead of scanning stored resumes; updated template path
  - answer/SKILL.md — fallback updated to background.md
  - apply/SKILL.md — example path updated
  - REFERENCES.md, HARNESS.md, setup.md, README.md — all routing and docs updated, no stale paths remaining

✻ Baked for 1m 54s

This issue of needing to say “make sure you do this”, “don’t forget to do that”, is a problem that the agent harnesses standard is attempting to solve. Without native integration into agentic clients, though, there will need to be some manual effort on the part of the user.

Now that we just changed how the harness fundamentally works, we can generate some materials to apply to some jobs and see what we get.

❯ looks good, make a resume                                                                                                                       

⏺ Which one — TRM Labs, Hippocratic AI, or SEON?

✻ Cooked for 8s

❯ all three                                                                                                                                       

  Read 1 file (ctrl+o to expand)

⏺ The approach is: unpack the template, edit the XML paragraphs at the known paraIds, repack as three separate files. Let me check the scripts
  exist first, then unpack.

  Searched for 1 pattern, listed 1 directory (ctrl+o to expand)

⏺ Scripts are in skills/materials/docx/scripts/office/. Unpacking the template now to inspect its structure.

⏺ Bash(python skills/materials/docx/scripts/office/unpack.py references/TEMPLATE_DanielWarfield_Resume_cover.docx
      /private/tmp/claude-501/-Users-danielwarfield-Documen…)
  ⎿  Unpacked references/TEMPLATE_DanielWarfield_Resume_cover.docx (15 XML files), simplified 0 tracked changes, merged 0 runs

  Searched for 2 patterns (ctrl+o to expand)

⏺ The template is the plain resume (no cover letter page yet). Let me see the full structure before proceeding.

  Searched for 1 pattern (ctrl+o to expand)

⏺ The template is a plain resume — no cover letter page yet. I have the full paraId map. Writing a generation script to produce all three .docx
  files with tailored competencies and updated location.

⏺ Write(/private/tmp/claude-501/-Users-danielwarfield-Documents-GitHub-exampleharnesses-fresh-job-application-harness/b6cf4d26-1a58-4195-af1f-2d40
       40537723/scratchpad/generate_resumes.py)
  ⎿  Wrote 134 lines to ../../../../../../private/tmp/claude-501/-Users-danielwarfield-Documents-GitHub-exampleharnesses-fresh-job-applicatio
     n-harness/b6cf4d26-1a58-4195-af1f-2d4040537723/scratchpad/generate_resumes.py
       2 import osutil
       3 import re
       4 import subprocess
       5 
       6 scratchpad = “/private/tmp/claude-501/-Users-danielwarfield-Documents-GitHub-exampleharnesses-fresh-job-application-harness/b6cf4d
         26-1a58-4195-af1f-2d4040537723/scratchpad”
       7 template_dir = scratchpad + “/template_unpacked”
       8 harness = “/Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness”
       9
      10 # competency paraIds in table order (positions 1-9)
     … +125 lines (ctrl+o to expand)

⏺ Bash(python /private/tmp/claude-501/-Users-danielwarfield-Documents-GitHub-exampleharnesses-fresh-job-application-harness/b6cf4d26-1a58-4195-af1
      f-2d4040537723/scratc…)
  ⎿  Error: Exit code 1

     Generating TRMLabs...
     Traceback (most recent call last):
       File “/private/tmp/claude-501/-Users-danielwarfield-Documents-GitHub-exampleharnesses-fresh-job-application-harness/b6cf4d26-1a58-4195-af1f
     -2d4040537723/scratchpad/generate_resumes.py”, line 118, in 
         result = subprocess.run(
             [”python”,
         ...<3 lines>...
             capture_output=True, text=True
         )
     … +16 lines (ctrl+o to expand)

⏺ Bash(python3 /private/tmp/claude-501/-Users-danielwarfield-Documents-GitHub-exampleharnesses-fresh-job-application-harness/b6cf4d26-1a58-4195-af
      1f-2d4040537723/scrat…)
  ⎿  Error: Exit code 1

     Generating TRMLabs...
     Traceback (most recent call last):
       File “/private/tmp/claude-501/-Users-danielwarfield-Documents-GitHub-exampleharnesses-fresh-job-application-harness/b6cf4d26-1a58-4195-af1f
     -2d4040537723/scratchpad/generate_resumes.py”, line 118, in 
         result = subprocess.run(
             [”python”,
         ...<3 lines>...
             capture_output=True, text=True
         )
     … +16 lines (ctrl+o to expand)

⏺ Update(/private/tmp/claude-501/-Users-danielwarfield-Documents-GitHub-exampleharnesses-fresh-job-application-harness/b6cf4d26-1a58-4195-af1f-2d4
        040537723/scratchpad/generate_resumes.py)
  ⎿  Added 1 line, removed 1 line
      116  
      117      output_path = f”{harness}/references/{config[’filename’]}”
      118      result = subprocess.run(
      119 -        [”python”,                                                                                                              
      119 +        [”python3”,                                                                                                             
      120           f”{harness}/skills/materials/docx/scripts/office/pack.py”,
      121           work_dir, output_path,
      122           “--original”, f”{harness}/references/TEMPLATE_DanielWarfield_Resume_cover.docx”],

⏺ Bash(python3 /private/tmp/claude-501/-Users-danielwarfield-Documents-GitHub-exampleharnesses-fresh-job-application-harness/b6cf4d26-1a58-4195-af
      1f-2d4040537723/scrat…)
  ⎿  Generating TRMLabs...
       OK: /Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/references/TRMLabs_DanielWarfield_Resume_202
     6-06-25.docx
     … +13 lines (ctrl+o to expand)

⏺ All three built. Now verifying page counts and updating leads.csv.

⏺ Bash(for f in TRMLabs HippocraticAI SEON; do
        bash /Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/skills/materials/tailor/scr…)
⏺ Bash(python3 /private/tmp/claude-501/-Users-danielwarfield-Documents-GitHub-exampleharnesses-fresh-job-application-harness/b6cf4d26-1a58-4195-af
      1f-2d4040537723/scrat…)
  ⎿  Generating TRMLabs...
       OK: /Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/references/TRMLabs_DanielWarfield_Resume_202
     6-06-25.docx
     … +13 lines (ctrl+o to expand)

⏺ All three built. Now verifying page counts and updating leads.csv.

⏺ Bash(for f in TRMLabs HippocraticAI SEON; do
        bash /Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/skills/materials/tailor/scr…)
  ⎿  2
     TRMLabs: OK
     2
     … +3 lines (ctrl+o to expand)

⏺ All three are exactly 2 pages. Updating leads.csv with the file paths.

⏺ Write(references/leads.csv)
  ⎿  Added 3 lines, removed 3 lines
      1  company,job_title,posting_url,status,applied_date,resume_file,compensation,location,job_description_file
      2 -TRM Labs,Staff Software Engineer - Agent Engineering,https://jobs.ashbyhq.com/trm-labs/eb560943-3adb-4839-bf64-4cbdeee9ae57,,,,,$2
        -00k-$275k,Remote,job-descriptions/TRMLabs_staff-swe-agent-engineering_2026-06-25.md                                               
      3 -Hippocratic AI,Senior Staff AI Engineer,https://jobs.ashbyhq.com/Hippocratic%20AI/af39c865-a9ab-403e-8746-79dec87d6ad0,,,,, Palo A
        -lto (in-office),job-descriptions/HippocraticAI_senior-staff-ai-engineer_2026-06-25.md                                             
      2 +TRM Labs,Staff Software Engineer - Agent Engineering,https://jobs.ashbyhq.com/trm-labs/eb560943-3adb-4839-bf64-4cbdeee9ae57,,,TRML
        +abs_DanielWarfield_Resume_2026-06-25.docx,$200k-$275k,Remote,job-descriptions/TRMLabs_staff-swe-agent-engineering_2026-06-25.md   
      3 +Hippocratic AI,Senior Staff AI Engineer,https://jobs.ashbyhq.com/Hippocratic%20AI/af39c865-a9ab-403e-8746-79dec87d6ad0,,,Hippocrat
        +icAI_DanielWarfield_Resume_2026-06-25.docx,,Palo Alto (in-office),job-descriptions/HippocraticAI_senior-staff-ai-engineer_2026-06-
        +25.md                                                                                                                             
      4  Glean,Machine Learning Engineer - AI Assistant & Autonomous AI Agents,https://job-boards.greenhouse.io/gleanwork/jobs/4605215005,,
         ,,,Remote,
      5 -SEON,Director of Engineering,https://www.builtinaustin.com/job/director-engineering/9615304,,,,, Austin Hybrid,job-descriptions/SE
        -ON_director-engineering_2026-06-25.md                                                                                             
      5 +SEON,Director of Engineering,https://www.builtinaustin.com/job/director-engineering/9615304,,,SEON_DanielWarfield_Resume_2026-06-2
        +5.docx,,Austin Hybrid,job-descriptions/SEON_director-engineering_2026-06-25.md                                                    
      6  Dropbox,Director of Engineering - New AI Initiatives,https://www.builtinaustin.com/job/engineer/director-engineering-new-ai-initia
         tives/135271,,,,,Remote,

⏺ Opening all three in LibreOffice for review.

⏺ Bash(open -a “LibreOffice” \
        “/Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/references/TRMLabs_DanielWarfield_Resume_2026-0…)
  ⎿  Waiting…

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
      5 -SEON,Director of Engineering,https://www.builtinaustin.com/job/director-engineering/9615304,,,,, Austin Hybrid,job-descriptions/SE
        -ON_director-engineering_2026-06-25.md                                                                                             
      5 +SEON,Director of Engineering,https://www.builtinaustin.com/job/director-engineering/9615304,,,SEON_DanielWarfield_Resume_2026-06-2
        +5.docx,,Austin Hybrid,job-descriptions/SEON_director-engineering_2026-06-25.md                                                    

⏺ Bash(open -a “LibreOffice” \
        “/Users/danielwarfield/Documents/GitHub/exampleharnesses-fresh/job-application-harness/references/TRMLabs_DanielWarfield_Resume_2026-0…)
  ⎿  (No output)

⏺ All three resumes are generated and open in LibreOffice. Each is exactly 2 pages with:
  - Location updated to “California” throughout
  - Tailored competencies for the role
  - Full work history unchanged

  leads.csv has been updated with the resume file paths for TRM Labs, Hippocratic AI, and SEON. Take a look and let me know if anything needs
  adjusting.

✻ Crunched for 3m 21s

※ recap: Job search harness set up and three tailored resumes generated for TRM Labs, Hippocratic AI, and SEON. Review the open LibreOffice docs 
  and say if anything needs adjusting before applying.

I want to zoom out a little bit. By saying the following commands over the course of a few minutes:

> load
> how do we get started?
> yeah draft up what you need.
> my resume is in my downloads folder, use that.
> Find five jobs
> I’m moving to California, change the harness accordingly
> ensure that all localtion references align across the harness
> get rid of the resumes directory. I want all resumes to be generated on the fly. Follow harness maintinance.
> apply to something 
> yeah add those to the pipeline and apply
> make a resume for all three   
> looks good, make a resume 
> all three

The harness

makes itself understood by the agent
initialized itself with the relevant information to get started
allowed the user to change key information across the harness
allowed the user to fundamentally changed a core aspect of how the harness operated
found five jobs online that align with my experience
downloaded the job descriptions of those listings
drafted a resume tailored for those jobs in text
after confirming, created the resumes themselves as docx files

et voila, it makes me three resumes.

If you look at these resumes, they’re not perfect. It moved some of my experience over to California, which is a bit of an odd design choice. Recall that I told the harness to re-work itself to generate resumes on the fly rather than using a template. I did this to demonstrate how the harness can maintain itself, but the original approach of the harness using and refining a template file alleviates these types of issues.

Despite its flaws, if I really wanted to maintain this new paradigm, I can use the harness to help me figure out exactly what needs to change.

❯ I don’t like some of the aspects of the final resumes. What should I modify in the harness to make the output better?                          

⏺ What specifically don’t you like? The answer depends on which part of the output is wrong:

  ┌─────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │      If you dislike...      │                                                 Change this                                                 │
  ├─────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ The competencies chosen     │ references/background.md — tighten the competency pool, or add guidance to skills/materials/tailor/SKILL.md │
  │                            │ on how to rank them                                                                                          │
  ├────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ The bullet text / phrasing │ references/background.md — rewrite the bullets directly; that’s the source of truth                          │
  ├────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ The professional summary   │ references/background.md — it’s currently the same for every resume; add per-role-type variants if you want  │
  │                            │ it tailored                                                                                                  │
  ├────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Which bullets appear /     │ skills/materials/tailor/SKILL.md — add framing guidance for specific role types                              │
  │ ordering                   │                                                                                                              │
  ├────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ The cover letter           │ The template doesn’t have a cover letter page yet — that needs to be built                                   │
  ├────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ Formatting / layout        │ references/TEMPLATE_DanielWarfield_Resume_cover.docx — edit the base template                                │
  └────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

  What are you seeing that you want to change?

In an unstructured “LLM Wiki” approach, figuring out what information to change in a large harness can be challenging. Because the agent harnesses standard imposes a referencing structure, we can leverage that structure to make modifications.

Let’s take a moment to discuss this harness and its implications from a high level. Then I’ll show you how to initialize your own harness for Claude, and then we can conclude.

A Discussion of Implications

The harness above is doing work that requires both constraint and open-endedness. It requires context, tools, and consistency over numerous sessions. The agent harnesses standard, paired with the metaskill, allows claude to leverage that structure consistently and repeatedly.

One of the great things about a harness is that it’s project-configurable. In Claude, plugins (which are the closest similar construct) can only be installed at a global level, which makes it hard to isolate different skills and contexts to different roles.

If you’re a content creator, for instance, you might want to want to consider creating tiktok videos, vs drafting video scripts, vs creating blog posts as different roles with very different rules. However, if you need to install all of these as a high-level plugin that’s always exposed, claude might choose to use the wrong plugin at the wrong time. Thus, it can be difficult to reign claude into the functionality you want to satisfy a specific role.

Because harnesses are a directory, you can cd into that directory, run claude, and then tell it to load the harness. Because HARNESS.md is a relative entry point, it’s easy to ensure that claude is obeying the proper context and using the correct reference material. Thus you can have many, many harnesses on a single machine, all specifying different roles, with little risk of cross-contamination.

One thing you might be thinking is that this is a fairly simple demo, and it is. It’s more complex than you’d think, iterating on a harness to output good quality content consistently is challenging, and the structure of agent harnesses makes that feasible, but job application isn’t exactly rocket surgery. I’m excited to discuss configurable leaves later, which allows for abstraction around information in a harness that I think will enable very sophisticated use cases. You can expect a follow-up article that goes into that in depth.

Another thought you might be considering is how harnesses are structured within a larger agentic system. Can an agent use more than one harness? Can harnesses be nested in some way? To be honest, I’m still thinking about what these questions mean practically, but I’m excited to begin working on agentic clients that leverage the agent harnesses standard with the hopes of understanding these questions more thoroughly. The agent harnesses standard makes no constraints around how harnesses interact with each other intentionally, so that developers can leverage them in ways that suit their needs. Stay tuned for follow up articles that explore that as well.

Anywho, agent harnesses is still a work in progress, and there will be follow ups describing that progress. Let’s wrap this up by exploring how you can create a harness yourself.

Creating a Harness

There are two pip-installable repos I’m maintaining that can aid you in creating harnesses that follow the agent harnesses standard. These are both early constructs that will continue to mature over the coming months.

agentharnesses-cli exposes a variety of command line tools around creating and evaluating harnesses

And harnesses-ref is a specific Python library that allows a user to evaluate if a harness obeys the agent harnesses standard.

The agentharnesses-cli uses harnesses-ref under the hood as a dependency, and exposes all of its functionality, so we can go ahead and simply install the agent harnesses CLI.

pip install agentharnesses-cli

that will be exposed as a command line tool called ahar (short for Agent Harnesses). we can call ahar init to create a new harness. That will ask us if we want to create a bare harness, or one pre-loaded with tools to be made easily compatible with Claude.

danielwarfield@mac test % ahar init
Initialized harness ‘test’ in /Users/danielwarfield/Desktop/test
  HARNESS.md
  README.md
  .gitignore
  .claude/settings.json
  skills/SKILLS.md
  references/REFERENCES.md

Claude Code preset:
  claude   Full Claude Code setup: metaskill plugin + maintain-harness skill
  empty    Bare minimum: no additional plugins or skills
Preset [claude]:

I’ll just use the default, and boom, we have a harness. I can run ahar validate . to check if the harness obeys the agent harness standard, which obviously it does.

danielwarfield@mac test % ahar validate .
✓ . is valid

and we can run ahar show . to print out all the key files in our harness.

danielwarfield@mac test % ahar show
/
├── HARNESS.md [root]
├── references/
│   └── REFERENCES.md [routing]
└── skills/
    ├── SKILLS.md [routing]
    └── maintenance/
        ├── SKILLS.md [routing]
        └── modify-harness/
            └── SKILL.md [leaf]

We can run claude , type in load, and Claude will know exactly what’s up in a few seconds.

danielwarfield@mac test % claude
╭─── Claude Code v2.1.193 ─────────────────────────────────────────────────────────────────────────────╮
│                                               │ Tips for getting started                             │
│              Welcome back Daniel!             │ Run /init to create a CLAUDE.md file with instructi… │
│                                               │ ──────────────────────────────────────────────────── │
│                    ▐▛███▜▌                    │ What’s new                                           │
│                   ▝▜█████▛▘                   │ Added `autoMode.classifyAllShell` setting to route … │
│                     ▘▘ ▝▝                     │ Added auto-mode denial reasons to the transcript, t… │
│                                               │ Added `claude_code.assistant_response` OpenTelemetr… │
│   Sonnet 4.6 · Claude Pro · Daniel Warfield   │ /release-notes for more                              │
│                ~/Desktop/test                 │                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯


❯ load                                                                                                  

  Read 1 file (ctrl+o to expand)

⏺ This is a stub HARNESS.md that hasn’t been configured yet. Let me check what else is in the project
  directory.

  Searched for 1 pattern (ctrl+o to expand)

⏺ Skill(agent-harnesses)
  ⎿  Successfully loaded skill

⏺ Let me summarize the harness to get a full picture of what’s here.

⏺ Bash(python /Users/danielwarfield/Desktop/test/.claude/skills/agent-harnesses/scripts/summarize.py
      /Users/danielwarfield/Desktop/test)
  ⎿  Waiting…

────────────────────────────────────────────────────────────────────────────────────────────────────────
 Bash command

   python /Users/danielwarfield/Desktop/test/.claude/skills/agent-harnesses/scripts/summarize.py

⏺ Bash(python /Users/danielwarfield/Desktop/test/.claude/skills/agent-harnesses/scripts/summarize.py
      /Users/danielwarfield/Desktop/test)
  ⎿  test/
     ├── [harness]  HARNESS.md  — TODO describe what this harness does and the role it give...
     │
     … +31 lines (ctrl+o to expand)

⏺ This is a freshly scaffolded harness — all the stubs are in place but nothing has been filled in yet.
  Here’s what exists:

  - HARNESS.md — root entry point (stub, needs description)
  - README.md — stub
  - references/REFERENCES.md — empty reference index
  - skills/SKILLS.md — empty skill index
  - skills/maintenance/ — one real skill already present: modify-harness, which helps keep routing files
  in sync when you add/move/rename things

  The only functional content is the modify-harness skill. Everything else is TODO placeholders.

  What would you like to do with this harness? For example:
  - Describe its purpose and fill in the stubs
  - Add skills or references to it
  - Start building it out for a specific use case

✻ Baked for 19s

Now we can do pretty much anything we want. I told Claude to make this harness represent all of the articles on IAEE.substack.com, organized by topic, with summaries and key ideas.

After Claude did its thing for a while, scraping my content online and sticking it in the harness, I quit out of claude and then ran ahar validate .

danielwarfield@mac test % ahar validate .
⚠ references/agentic-ai.md: markdown file should have a ‘description’ in frontmatter
⚠ references/data-and-tools.md: markdown file should have a ‘description’ in frontmatter
⚠ references/retrieval-and-rag.md: markdown file should have a ‘description’ in frontmatter
⚠ references/software-engineering.md: markdown file should have a ‘description’ in frontmatter
⚠ references/training-and-fine-tuning.md: markdown file should have a ‘description’ in frontmatter
✓ . is valid

The agent harnesses standard recommends having descriptions in all markdown files so that they can be peeked at via progressive disclosure, but it’s not a strong requirement. It also makes sense that they might not be relevent in this context. Seems like claude made a harness that obeys the standard.

we can run ahar show . to see what structure it came up with.

danielwarfield@mac test % ahar show .
/
├── HARNESS.md [root]
├── references/
│   └── REFERENCES.md [routing]
└── skills/
    ├── SKILLS.md [routing]
    └── maintenance/
        ├── SKILLS.md [routing]
        └── modify-harness/
            └── SKILL.md [leaf

It actually didn’t change the overall structure of the harness at all, just its contents.

If we use the flag -v allows us to view the tree with varying degrees of verbosity. it defaults to 0, the lowest verbosity, but we can take a peek at descriptions throughout the harness by calling ahar show -v 1

danielwarfield@mac test % ahar show -v 1
/
├── HARNESS.md [root] — Reference harness for all articles published on Intuitively and Exhaustively Explained (IAEE) by Daniel Warfield. Use this to find articles by topic when answering questions about ML, AI, data engineering, hardware, or software practice.
├── references/
│   └── REFERENCES.md [routing] — Index of all IAEE article reference files, organized by topic. Each file lists the articles in that topic with their URLs and descriptions.
└── skills/
    ├── SKILLS.md [routing] — Index of skill buckets in this harness. Currently contains only maintenance skills for keeping the harness structure up to date.
    └── maintenance/
        ├── SKILLS.md [routing] — Harness upkeep skills — for modifying the harness structure, adding skills and references.
        └── modify-harness/
            └── SKILL.md [leaf] — Update harness structure files — HARNESS.md, SKILLS.md indexes, REFERENCES.md — to keep routing and descriptions accurate as the harness evolves.

And a picture speaks a thousand words; the content seems pretty good.

One of the files generated.

If we hop back into a new session of claude and tell it to load harness, we’ll see that it understands the new structure quickly.

danielwarfield@mac test % claude
╭─── Claude Code v2.1.193 ─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                               │ Tips for getting started                                                             │
│              Welcome back Daniel!             │ Run /init to create a CLAUDE.md file with instructions for Claude                    │
│                                               │ ──────────────────────────────────────────────────────────────────────────────────── │
│                    ▐▛███▜▌                    │ What’s new                                                                           │
│                   ▝▜█████▛▘                   │ Added `autoMode.classifyAllShell` setting to route all Bash/PowerShell commands thr… │
│                     ▘▘ ▝▝                     │ Added auto-mode denial reasons to the transcript, the denial toast, and `/permissio… │
│                                               │ Added `claude_code.assistant_response` OpenTelemetry log event containing the model… │
│   Sonnet 4.6 · Claude Pro · Daniel Warfield   │ /release-notes for more                                                              │
│                ~/Desktop/test                 │                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯


❯ load harness                                                                                                                          

⏺ Skill(agent-harnesses)
  ⎿  Successfully loaded skill  

⏺ I’ll start by summarizing the harness structure to understand what’s available.

⏺ Bash(python /Users/danielwarfield/Desktop/test/.claude/skills/agent-harnesses/scripts/summarize.py /Users/danielwarfield/Desktop/test)
  ⎿  test/
     ├── [harness]  HARNESS.md  — Reference harness for all articles published on Intuitive...
     │
     … +83 lines (ctrl+o to expand)

⏺ The harness is loaded. Here’s what it contains:

  IAEE Article Reference Harness — A reference harness for Daniel Warfield’s Intuitively and Exhaustively Explained Substack, covering
  73 articles across 13 topic areas:

  References (topic indexes):
  - ai-fundamentals.md — Neural networks, training dynamics, core ML concepts
  - agentic-ai.md — AI agents, LangGraph, MCP, A2A, agent harnesses
  - attention-by-hand.md — Step-by-step math walkthroughs of attention/transformers
  - computer-vision.md — CNNs, YOLO, CLIP, Flamingo, VQA
  - data-and-tools.md — SQL, Python packaging, combinatorics, signal analysis
  - graph-ml.md — Graph theory, GCNs, applied graph ML
  - hardware-and-infrastructure.md — GPUs, Groq, Spark, Kubernetes
  - large-language-models.md — Transformer architecture, GPT/BERT/DeepSeek, positional encoding
  - retrieval-and-rag.md — RAG, CRAG, multimodal RAG, cache-augmented generation
  - software-engineering.md — Agile, MLOps, experiment tracking
  - training-and-fine-tuning.md — LoRA, self-supervised learning, evolutionary optimization
  - video-and-generative-models.md — Sora, VAE/beta-VAE 
  - opinion-and-commentary.md — Essays on AI ethics, industry trends

  Skills:
  - modify-harness — For updating the harness structure (HARNESS.md, SKILLS.md, REFERENCES.md)

✻ Brewed for 14s

and it can leverage that understanding to answer questions quickly and efficiently

❯ what’s the topic with the most articles?                                                                                              

⏺ Let me read all the reference files in parallel to count articles per topic.

  Read 13 files (ctrl+o to expand)

⏺ Agentic AI is the topic with the most articles at 10:

  ┌───────────────────────────┬──────────┐
  │           Topic           │ Articles │
  ├───────────────────────────┼──────────┤
  │ Agentic AI                │ 10       │
  ├───────────────────────────┼──────────┤
  │ Large Language Models     │ 6        │
  ├───────────────────────────┼──────────┤
  │ Computer Vision           │ 7        │
  ├───────────────────────────┼──────────┤
  │ Hardware & Infrastructure │ 6        │
  ├───────────────────────────┼──────────┤
  │ Retrieval & RAG           │ 6        │
  ├───────────────────────────┼──────────┤
  │ AI Fundamentals           │ 6        │
  ├───────────────────────────┼──────────┤
  │ Opinion & Commentary      │ 6        │
  ├───────────────────────────┼──────────┤
  │ Graph ML                  │ 4        │
  ├───────────────────────────┼──────────┤
  │ Training & Fine-Tuning    │ 4        │
  ├───────────────────────────┼──────────┤
  │ Data & Tools              │ 5        │
  ├───────────────────────────┼──────────┤
  │ Software Engineering      │ 4        │
  ├───────────────────────────┼──────────┤
  │ Attention (By Hand)       │ 5        │
  ├───────────────────────────┼──────────┤
  │ Video & Generative Models │ 3        │
  └───────────────────────────┴──────────┘

✻ Sautéed for 17s

There’s a lot more I want to talk about, but I need to work a bit more on agent harnesses before I’m ready to peel back another layer of the onion. I think it’s time to wrap it up.

Conclusion

In this article we started by discussing what a harness is. While divisive, we settled on a working definition based on common language and the needs of modern agentic design

A harness is information and tools that allow a general purpose agentic system to do specific, complex tasks in a repeatable and maintainable manner.

Practically, that ends up being a directory of resources.

The “Agent Harnesses” spec exists to create a lightweight but robust structure around that directory, allowing an agent to leverage this directory of information effectively, while minimizing some of the pitfalls of similar and less structured approaches.

Once we had a working idea of agent harnesses, we explored an example harness and then made our own.

Agent Harnesses is still in early and active development. The concrete is not yet dry on the ideas I outlined in this article. I hope to hear some more constructive feedback and criticism so that the Agent Harnesses spec can further evolve. That said, I do hope that it’s starting to take the shape of something useful. Feel free to check out the website, github repo, and download the CLI so you can use agent harnesses in your projects.

Also, GitHub stars are cool, and people who want to help maintain the standard in these early phases are even cooler. Hope to see you on GitHub.

Agent Harnesses — Intuitively and Exhaustively Explained

Daniel Warfield — Tue, 16 Jun 2026 22:01:42 GMT

“Connected” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained.

In this article we’ll discuss “Agent Harnesses,” a new standard in AI agent development. Harnesses allow you to build a structure around how AI systems act and think, allowing you to customize those AI systems to do a variety of tasks.

We’ll explore this paradigm by first covering fundamental ideas that harnesses build off of. Once we have a foundational understanding, we’ll discuss the idea of a harness and how one can be implemented.

Share this article with friends and colleagues.

Who is this useful for? Anyone who wants to learn about modern AI Agent development

How advanced is this post? This article is accessible to readers of all levels

Prerequisites: All prerequisite are covered in the “Building a Fundamental Understanding” section

Contributing: This article proposes a standard for developing agent harnesses, called “Agent Harnesses.” This is an open-source and publicly maintained project. If you’re interested in contributing, check out the contributing guidelines.

Building a Fundamental Understanding

Before we discuss harnesses, let’s cover some of the fundamental ideas harnesses build off of.

Review: LLMs

I have long and in-depth articles discussing LLMs, their various types, and how they function. Feel free to explore these if you’re curious:

For the purposes of this article, you can think of an LLM as a function. You put text in, you get a response out. That’s all they do: respond to text with text. Much of the modern AI technology that’s been coming out recently is built on top of this text response functionality.

Review: Agents

An “agent” is still a broad term. From an academic perspective, there are many types of agents that function fundamentally differently. Here are a few articles where I explore several of those approaches, from newest to oldest.

When you talk to a layman, however, there’s really only one type of agent. It’s a chat AI system that takes your response, does things, then responds. I typically refer to this as a ReAct-style agent, which is essentially some lightweight code that connects an LLM to functionality, allowing the LLM to reason, act, and observe.

ReAct allows a model to “reason” about things it will do, then choose to take “actions” to interact with the environment, then make “observations” about those actions. From my article on agents.

When I refer to “Agents” throughout this article, I am referring to ReAct, or “ReAct-esque” agents.

Review: Chain of Thought and Tool Use

To understand ReAct-style agents, it’s critical to understand how they approach “reasoning” and “action”.

Reasoning is typically performed with “chain of thought.” Basically, the idea of chain of thought is to encourage a model to think about its solution before the model outputs an answer. In the original paper that introduced chain of thought, this was done by giving the model an example of how it should answer in the prompt itself.

The example on the left failed with standard prompting, even though the model was exactly the same. From my article on agents. Source

As LLM-powered systems have evolved, this is such a core paradigm that, often, the models are trained to produce a chain of thought implicitly. Normal models have a tendency to think through problems before spitting out an answer, and now “reasoning models”, popularized by DeepSeek, have a specific step where the model thinks through a problem with a very long chain of thought.

Tool use is similar in that it’s been around for a while, has consistently proven useful, and is starting to become standardized. In its most simplistic form, tool use is essentially telling an LLM, “If you output this specific text, I’ll run some code and give you an answer”, allowing the model to decide when it should search the internet, look through your files, transfer all your money to a Nigerian prince, or whatever.

An example of a prompt, and a variety of answers from different prompting strategies. As can be seen, sometimes a combination of chain of thought and action is required to answer complex questions. From my article on agent. source

Review: Skills

As agents, tool use, and prompting have evolved over the last few years, standard practices have emerged as a natural consequence. If you want to get an agent to automate some process, maybe adding some fields to a database, you need to define tools allowing the model to see the database and interact with it. You also need to define prompts that describe to the model what the database looks like, and the tools the agent has access to.

Skills are a standardized way of structuring this information. It allows you to lump information and functionality into a single “skill” so the agent can leverage that information as a complete package.

my-skill/
├── SKILL.md          # Required: metadata + instructions
├── scripts/          # Optional: executable code
├── references/       # Optional: documentation
├── assets/           # Optional: templates, resources
└── ...               # Any additional files or directories

I covered skills in this article in depth:

In reality, skills are remarkably simple. Most skills are just a bunch of text files. Despite their simplicity, though, skills are incredibly powerful in allowing agents to understand and interact with complex systems, and provide a reusable and portable layer that allows developers working on agents to define robust and reusable plans and procedures.

With that, I think we’ve covered all the essential ideas. Let’s get into it.

The Fundamental Idea of a Harness

Harnesses can be best understood as an extension of skills. Skills allow you to package broad functionality, like modifying a website, database administration, and understanding customer feedback into a single “thing.” However, to do many complicated tasks, like to serve as an assistant to developers working on a product, an AI agent might need to use numerous different skills in tandem. A harness allows you to bunch together various skills, as well as the reference information necessary for the agent to understand its role and the environment it’s operating in.

The Agent Harnesses Standard

A Harness consists of three key parts: a markdown file describing the harness, a set of skills allowing the agent to interact with its environment, and a set of references an agent can use to better understand the environment it’s working in.

my-harness/
├── HARNESS.md        # Required: metadata + instructions
├── skills/           # Optional: a set of skills
└── references/       # Optional: a set of reference documents

HARNESS.md describes the harness from a high level. This is designed to be a lightweight document that an agent can use to quickly understand the most fundamental high-level information of the problem it’s working on and to understand where it can find more specific information within skills and references.

This paradigm follows the “progressive disclosure” paradigm, recently popularized by the Agent Skills standard, which encourages an agent to engage in the following:

Discovery: At startup, agents load only HARNESS.md, which describes the structure of the harness. When a prompt is provided to the agent, the agent compares the prompt to the definitions provided in the harness and incrementally explores the descriptions and names of skills and references in the harness until it finds the necessary material.
Activation: When relevant material is discovered, the full documentation on the relevant skills and references are loaded and read by the agent.
Execution: The agent follows information provided in the harness and optionally executes skills.

Let’s dig through each of these components individually so we can understand their rationale.

Defining HARNESS.md

Like skills, the harness employs frontmatter to allow for more efficient exploration of the pertinent assets. Frontmatter allows developers to assign key information at the top of a markdown file, which the agent can use to readily understand

---
name: 
description: 
---

In HARNESS.md, a “name” and “description” are required so that the agent can easily understand what the harness is. The body is a brief markdown file designed to explain key elements of the harness to the agent, allowing it to understand which skills or references it should employ.

Defining the Skills Directory

The skills directory is a directory containing a set of skills following the Agent Skills standard, which defines a skill as the following:

my-skill/
├── SKILL.md          # Required: metadata + instructions
├── scripts/          # Optional: executable code
├── references/       # Optional: documentation
├── assets/           # Optional: templates, resources
└── ...               # Any additional files or directories

Harnesses extend this functionality. Thus, the skills directory might look like the following:

my-harness/
├── HARNESS.md
├── skills/
|  ├── my-skill-1/
|  |  ├── SKILL.md
|  |  ├── scripts/
|  |  ├── references/
|  |  ├── assets/
|  |  └── ...
|  └── my-skill-2/
|     ├── SKILL.md
|     ├── scripts/
|     ├── references/
|     ├── assets/
|     └── ...
└── references/

Defining the References Directory

The references directory is a set of arbitrary documents that can be used by the agent to better understand its environment, role, or other high-level information.

Skills also have a concept of references. Generally, information in the references directory of the harness should be at a level of abstraction higher than skills. The references within skills should discuss how the specific skill should be employed, while the references at the harness level should describe information that is necessary across numerous skills.

For instance, if you were building an agent designed to assist with the creation of marketing content, you might have a skill for creating blog posts, a skill for generating images, and a skill for scanning social media.

my-mraketing-harness/
├── HARNESS.md
├── skills/
|  ├── create-blog-post/
|  |  ├── SKILL.md
|  |  └── ...
|  ├── generate-images/
|  |  ├── SKILL.md
|  |  └── ...
|  └── scan-social/
|     ├── SKILL.md
|     └── ...
└── references/

Each of these skills might have references necessary for their function. scan-social might have references pertaining to the social media accounts in the company, generate-images might have references about references describing where images are saved within the company after creation, and create-blog-post might have references about the publication process.

In this context, you might choose to use the harness-level reference directory to create things like style guides and brand priorities that can be used across all of the skills.

my-mraketing-harness/
├── HARNESS.md
├── skills/
|  ├── create-blog-post/
|  |  ├── SKILL.md
|  |  └── ...
|  ├── generate-images/
|  |  ├── SKILL.md
|  |  └── ...
|  └── scan-social/
|     ├── SKILL.md
|     └── ...
└── references/
   ├── style-guide.md
   └── brand-priorities.md

The harness standard allows arbitrary content to be added to the harness-level references directory. However, it’s highly recommended to use markdown files, and to add a high level description to each markdown file so that agents can quickly understand if a particular reference is relevant. style-guide.md, in this example, it might look like the following:

---
description: a style guide for the marketing website, which covers the creation
of all visual assets and standard typography rules
---

Structuring Large Harnesses

One of the most important aspects of a harness is that it can contain numerous skills and references. With sufficiently large sets of skills and references, maintenance can become confusing, and efficiency around injecting information into the agent can become problematic.

my-harness/
├── HARNESS.md
├── skills/
|  ├── skill1/
|  ├── skill2/
|  ├── skill3/
|  └── ...
└── references/
   ├── reference1
   ├── reference2
   ├── reference3
   └── ...

within both the skills and references directory, sub-directories can be created to organize skills and references into subsets.

my-harness/
├── HARNESS.md
├── skills/
│   ├── software-development/
│   │   ├── SKILLS.md
│   │   ├── skill1/
│   │   │   └── SKILL.md
│   │   └── skill2/
│   │       └── SKILL.md
│   ├── market-analysis/
│   │   ├── SKILLS.md
│   │   └── skill3/
│   │       └── SKILL.md
│   └── social-media/
│       ├── SKILLS.md
│       ├── skill4/
│       │   └── SKILL.md
│       └── skill5/
│           └── SKILL.md
└── references/
    ├── brand-assets/
    │   ├── REFERENCES.md
    │   ├── reference1.md
    │   └── reference2.md
    ├── software-infrastructure/
    │   ├── REFERENCES.md
    │   ├── reference3.md
    │   └── reference4.md
    └── product-information/
        ├── REFERENCES.md
        └── reference5.md

In order to allow the agent to fully but succinctly understand this structure, it should be communicated to the agent via HARNESS.md. To aid in exploration, each subdirectory should have a file describing it. SKILLS.md describes the skills within a subdirectory and REFERENCES.md describes the references within a subdirectory.

This structure enables something called “progressive disclosure.” An agent can use HARNESSES.md to understand the high-level structure of the harness, then choose to explore specific SKILLS.md and REFERENCES.md files as necessary to efficiently find relevant material.

Within both Skills and References, the folder structure can be nested arbitrarily, allowing for high-level organization of many skills and references. Each of these subdirectories should contain a SKILLS.md or REFERENCES.md file, allowing the agent to understand the structure within each layer.

...
└── references/
  └── data-sources/
      ├── REFERENCES.md
      ├── relational/
      │   ├── REFERENCES.md
      │   └── schema-overview.md
      └── warehouse/
          ├── REFERENCES.md
          └── dataset-catalog.md

Agent Skills — Intuitively and Exhaustively Explained

Daniel Warfield — Tue, 12 May 2026 14:03:06 GMT

“Portable Plan” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained.

“Agent skills” are emerging as an open standard for defining how developers can instruct agents, across numerous platforms, to solve problems.

some of the platforms that support agent skills. Source.

Building backend infrastructures, creating consistent styles in frontend applications, adopting a certain literary style, and more. Virtually anything can be defined in a skill, allowing an AI agent to use that skill to solve problems. In this article we’ll explore what “skills” are, how AI agents use them, and how we can implement them.

Share this article with friends and colleagues.

Who is this useful for: Anyone who wants to harness the power of modern AI agents.

How advanced is this post? This article is generally conceptual, and is accessible to readers of all levels.

Prerequisites: None

Core Concept Review

Before we get into the meat of it, I want to take a moment to review some core concepts. Feel free to skip this section if you feel confident, or dive into the supporting material if you want a more in-depth understanding.

Review: LLMs

I have long and in-depth articles discussing LLMs, their various types, and how they function. Feel free to explore these if you’re curious:

For the purposes of this article, you can think of an LLM as a function. You put text in, you get a response out. That’s all they do: respond to text with text.

Review: Agents

When I refer to “Agents” throughout this article, I am referring to ReAct, or “ReAct-esque” agents.

Review: Chain of Thought and Tool Use

To understand ReAct-style agents, it’s critical to understand how they approach “reasoning” and “action”.

The example on the left failed with standard prompting, even though the model was exactly the same. From my article on agents. Source

We’ll discuss more modern ways to invoke tools later, but that’s the essential idea. For now, I think we’ve covered prerequisites. Let’s get into it.

The Point

I have an article coming out soon that will build on this idea in greater depth, so I’ll get to the point directly. AI is exiting its phase of individualized bespoke solutions and is converging on a core set of paradigms that are shared across most AI applications. Agents are being defined the same way, are using the same tools, and are using models under the hood that operate in a similar manner.

Around this convergence is arising a set of AI standards that are agreed upon and shared by the biggest players in the industry. I’ve already discussed some examples in other articles:

In this article we’ll be discussing “Skills,” an emerging standard that allows developers to define structured guides and tools, allowing agents to execute tasks in a manner that’s more robust.

The Anatomy of a Skill

A “skill” is, in its essence, a folder structure with some key resources inside.

my-skill/
├── SKILL.md          # Required: metadata + instructions
├── scripts/          # Optional: executable code
├── references/       # Optional: documentation
├── assets/           # Optional: templates, resources
└── ...               # Any additional files or directories

The primary asset in a skill is SKILL.md, which is a markdown file that defines the skill. At its most simplistic, a skill is just a folder with a single SKILL.md file.

Here’s an example of a SKILL.md file, from the agent skills guide

---
name: roll-dice
description: Roll dice using a random number generator. Use when asked to roll a die (d6, d20, etc.), roll dice, or generate a random dice roll.
---

To roll a die, use the following command that generates a random number from 1
to the given number of sides:

```bash
echo $((RANDOM %  + 1))
```

```powershell
Get-Random -Minimum 1 -Maximum ( + 1)
```

Replace `` with the number of sides on the die (e.g., 6 for a standard
die, 20 for a d20).

Structurally, a SKILL.md file consists of two parts: “YAML frontmatter” and the “body”.

---

---

The front matter is designed to have high-level key-value information about the skill. This is used to communicate high level information to the agent.

One of the fundamental ideas of skills is that you can have many of them for doing many different things. An agent might have access to several hundred skills. In order to allow the model to efficiently reason about which skills are relevant, the agent can use the front matter to get a gist of what a skill is good for. If it decides a skill might be relevant for the task at hand, the “body” can then be used to instruct the agent through the skill in-depth. This idea of exposing different degrees of information at different times is referred to as “progressive exposure” and is a central concept in skills.

The frontmatter must contain a name and description, which makes sense because, without them, an agent couldn’t reference your skill by name or understand its general purpose. The frontmatter can have some other fields as well:

license: the software license
compatibility: what tools, environment, or resources does the skill need to function
metadata: arbitrary key-value pairs
allowed-tools: The tools the agent is allowed to use

Kind of like MCP, how exactly these are used depends on the agent you’re connecting to. Claude code might use these resources slightly differently than VS Code, for instance. The specification defines what information is in the SKILL.md, but it’s up to the developers implementing the agents to define how these specifications are integrated.

After the “frontmatter”, the “body” simply defines what the skill is for. This is just a big block of markdown text that, essentially, defines a prompt that explains the skill to the agent. Naturally, in being so loosely defined, there are practically innumerable ways the body can be created. The spec provides some recommended best practices for defining a high-quality body which, chiefly, break down into the following recommendations:

• Be task-focused (only include what the agent wouldn’t already know)
• Use step-by-step procedures
• Keep it short (core instructions only; move details to references/)
• Include examples (inputs/outputs/templates)
• Add gotchas and examples of common mistakes
• Provide defaults, and don’t list too many options
• Match strictness to task (flexible when safe, exact when fragile)
• Tell the agent how to check its work
• Use progressive disclosure (load extra context only when needed)

In order to keep the body properly brief, a reference directory can be used to provide additional information about specific ideas. For instance, if there is a script you want your agent to understand, with a ton of parameters and subtle functionality, you can create a reference describing it in depth. That will allow the agent to look at that reference if it decides that the information is necessary.

It’s recommended that references exist in a flat structure within the references subdirectory, like so:

my-skill/
├── SKILL.md
├── references/
  ├── REFERENCE.md
  ├── FORMS.md
  ├── referencefile1.md
  └── referencefile2.md
└── ...

Exactly what REFERENCE.md is for, when the reference directory can also contain arbitrary reference markdown files, I’m not sure. It appears a lot of agents don’t really know or care either. When asking ChatGPT, ChatGPT said REFERENCE.md belongs outside of the references directory. Claude didn’t even mention the reference directory at all and stuck reference.md, lowercase, within the top-level skill.

FORMS.md is similarly loosely defined. The agent skills documentation describes it, sparsely, as “Form templates or structured data formats”. My understanding is that most people are using this to describe output structures that should remain consistent. This allows the agent to, in theory, work within a greater context that expects a certain output structure.

Brief aside, I haven’t been impressed with how “vibey” specifications around AI, LLMs, and Agents often are. If we don’t know where files are, or have a rigid specification around them, how can we enforce progressive exposure (which is one of the major objectives of the agent skills specification)? Also, should all of the output specifications be in FORMS.md? Should some of it be in REFERENCE.md, or other more specific references within the references directory? How can we enforce output formats if we don’t even know where the output formats are supposed to be?

With agent skills, to a large extent, the agent is responsible for deciding what to look into. The only defined progressive exposure in agent skills is as follows:

Agents load skills progressively, pulling in more detail only as a task calls for it. Skills should be structured to take advantage of this:

Metadata (~100 tokens): The name and description fields are loaded at startup for all skills
Instructions (< 5000 tokens recommended): The full SKILL.md body is loaded when the skill is activated
Resources (as needed): Files (e.g. those in scripts/, references/, or assets/) are loaded only when required

Keep your main SKILL.md under 500 lines. Move detailed reference material to separate files. source

So, presumably, if you have rigid output structures or complicated references, you should probably describe where follow up information can be found in the SKILL.md body so the agent can know where to look for specific information.

One could imagine this loose approach to constraints to be great for more open ended or simplistic skills, but difficult to maintain for more complex skills that deal with more difficult problems.

Along with the reference directory is the scripts and assets directory. The scripts directory is described as:

Contains executable code that agents can run. Scripts should:

Be self-contained or clearly document dependencies
Include helpful error messages
Handle edge cases gracefully

Supported languages depend on the agent implementation. Common options include Python, Bash, and JavaScript. Source

and the assets directory is described as:

Contains static resources:

Templates (document templates, configuration templates)
Images (diagrams, examples)
Data files (lookup tables, schemas)

What’s the difference between a template and a form? I have no idea. As I read through this, I’m wondering what the distinction is between a specification and advice written in a blog post. I guess it’s adoption.

Despite its limited constraints, the skills specification is adopted to varying degrees across many platforms. There is also a case to be made that specifications should be increasingly specific as need for specificity emerges naturally through technical maturity. Agents are only a few years old, so maybe such a lightweight specification is appropriate.

Because skills are so simplistic, it should be easy to build our own. Let’s do it.

Making an IAEE reference skill

This skill will be simple. It will just tell an agent, in this case Claude Code, to reference “Intuitively and Exhaustively Explained” articles when answering questions about AI concepts.

Claude manages the location of skills in the following way:

source

So, we’re going to make a new “project”, which I’ll call “SkillsTest”, and that will have a .claude hidden directory which will, in tern, have a skills directory that will host our skill. In theory, that should mean the agent automatically has access to that skill in the context of the project, but it doesn’t pollute Claude across my entire computer.

SkillTest
  └── .claude
        └── skills
              └── ai-researcher
                    └── SKILL.md

I defined SKILL.md thus

You are an AI research assistant. When given an AI-related question, you must:

1. Use the WebFetch tool to search for relevant content on iaee.substack.com. Start by fetching https://iaee.substack.com and look for posts relevant to the question. Follow links to individual posts that seem relevant.
2. Cite specific posts or passages from iaee.substack.com in your answer when they apply.
3. Synthesize what you found with your own knowledge to give a complete, well-referenced answer.

The question or topic to research is: $ARGUMENTS

Always lead with what you found on iaee.substack.com before adding broader context.

we can confirm that the skill exists by opening up Claude code in the project, and calling /skills

╭─── Claude Code v2.1.136 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                               │ Tips for getting started                                                                                                            │
│              Welcome back Daniel!             │ Run /init to create a CLAUDE.md file with instructions for Claude                                                                   │
│                                               │ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │
│                    ▐▛███▜▌                    │ What’s new                                                                                                                          │
│                   ▝▜█████▛▘                   │ Added `CLAUDE_CODE_ENABLE_FEEDBACK_SURVEY_FOR_OTEL` to re-enable the session quality survey for enterprises capturing responses th… │
│                     ▘▘ ▝▝                     │ Added `settings.autoMode.hard_deny` for auto mode classifier rules that block unconditionally regardless of user intent or allow e… │
│                                               │ Fixed MCP servers configured in `.mcp.json`, plugins, and claude.ai connectors silently disappearing after `/clear` in the VS Code… │
│   Sonnet 4.6 · Claude Pro · Daniel Warfield   │ /release-notes for more                                                                                                             │
│       ~/Documents/LocalTests/SkillsTest       │                                                                                                                                     │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

❯ /skills                                                                                                                                                                              

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  Skills
  1 skill · Space to cycle, Enter to save, / to search, t to sort, Esc to cancel

  ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
  │ ⌕ Search skills…                                                                                                                                                                │
  ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
  ❯ ✔ on         ai-researcher · project · ~23 tok

Notice that the agent sees the ai-researcher skill. Now, when I ask the query “what is a VAE”, claude detects my ai-research skill and uses it.

roxanne@roxanne-AMD-Ryzen-7-9700X-8-Core-Processor:~/Documents/LocalTests/SkillsTest$ claude
╭─── Claude Code v2.1.136 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                               │ Tips for getting started                                                                                                            │
│              Welcome back Daniel!             │ Run /init to create a CLAUDE.md file with instructions for Claude                                                                   │
│                                               │ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │
│                    ▐▛███▜▌                    │ What’s new                                                                                                                          │
│                   ▝▜█████▛▘                   │ Added `CLAUDE_CODE_ENABLE_FEEDBACK_SURVEY_FOR_OTEL` to re-enable the session quality survey for enterprises capturing responses th… │
│                     ▘▘ ▝▝                     │ Added `settings.autoMode.hard_deny` for auto mode classifier rules that block unconditionally regardless of user intent or allow e… │
│                                               │ Fixed MCP servers configured in `.mcp.json`, plugins, and claude.ai connectors silently disappearing after `/clear` in the VS Code… │
│   Sonnet 4.6 · Claude Pro · Daniel Warfield   │ /release-notes for more                                                                                                             │
│       ~/Documents/LocalTests/SkillsTest       │                                                                                                                                     │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

❯ what is a VAE?                                                                                                                                                                       

● Skill(ai-researcher)
  ⎿  Successfully loaded skill                                                                                                                                          

● I’ll fetch the iaee.substack.com site to find relevant content on VAEs.

● Fetch(https://iaee.substack.com)
  ⎿  Received 107.9KB (200 OK)

● Fetch(https://iaee.substack.com/archive)
  ⎿  Received 270.2KB (200 OK)

● Found a relevant post. Let me fetch it for details.

● Fetch(https://iaee.substack.com/p/disentangled-variational-autoencoders)
  ⎿  Received 748.2KB (200 OK)

● Here’s what I found on iaee.substack.com, combined with broader context:

  ---
  Variational Autoencoder (VAE)
  
  From iaee.substack.com

  Daniel Warfield’s Intuitively and Exhaustively Explained has a post on Disentangled Variational Autoencoders that covers VAEs well.

  A VAE is a neural network that learns to compress data into a structured “latent space” — a compressed representation where similar things sit close together and the space can be
  smoothly sampled to generate new data.

  It works in three stages:

  1. Encoder — compresses input data into distribution parameters (a mean and variance), not a fixed point
  2. Reparameterization trick — samples from those distributions in a way that still allows gradient backpropagation during training
  3. Decoder — reconstructs (or generates) data from the sampled latent vector

  The loss function has two parts:
  - Reconstruction loss — how well the output matches the input
  - KL divergence — a penalty that keeps the learned distributions close to a standard normal, forcing the latent space to be smooth and organized

  The key distinction from a plain autoencoder: a regular autoencoder maps each input to a fixed point; a VAE maps it to a probability distribution. This makes the latent space
  continuous and generative — you can sample random points from it and decode them into plausible new data.

  ---
  Broader Context

  VAEs were introduced by Kingma & Welling in 2013 and are foundational to modern generative AI. They’re closely related to:

  - Diffusion models (e.g., Stable Diffusion uses a VAE to encode images into latent space before the diffusion process)
  - β-VAE — a variant that multiplies the KL term by β > 1 to encourage disentangled representations, where each latent dimension corresponds to a single interpretable factor (e.g.,
  one dimension controls “pose,” another controls “lighting”)

  VAEs are less sharp than GANs for image generation but are more stable to train and produce well-structured latent spaces, making them useful for representation learning,
  compression, and as components in larger systems.

✻ Brewed for 34s

to actually find my VAE article, Claude needed to dig through my archive. If I hard-coded the URLs of my articles into a resource, we could probably skip a few steps and get answers faster. Let’s do that.

I asked Claude to do the following:

can you add a resource to the skill `ai-researcher` that lists hardcoded URLs
for the currently published articles? Add a reference to it in the `SKILL.md`,
allong with a note  that the agent should query the archive if a match isn’t
found.

Claude code did its thing, fetched articles from the archive over a few fetch requests, made a file called articles.md (not in the resources directory, as is formally defined in the skill spec), and modified my SKILL.md. I’m not going to fight Claude about the resources directory. I doubt it matters, so we’ll just use the file structure it settled on.

the new SKILL.md file looks like this:

You are an AI research assistant. When given an AI-related question, you must:

1. Consult the article index in `articles.md` (same directory as this file) to find posts on iaee.substack.com relevant to the question. Fetch any matching article URLs directly with the WebFetch tool.
2. If no article in the index matches the query, fall back to fetching https://iaee.substack.com/archive to search for newer posts not yet listed in the index.
3. Cite specific posts or passages from iaee.substack.com in your answer when they apply.
4. Synthesize what you found with your own knowledge to give a complete, well-referenced answer.

The question or topic to research is: $ARGUMENTS

Always lead with what you found on iaee.substack.com before adding broader context.

and the articles.md looks like this (subset)

# Published Articles — iaee.substack.com

A hardcoded index of all articles published as of 2026-05-08. Use these URLs as a starting point when researching a topic. If no article here matches the query, fall back to fetching https://iaee.substack.com/archive to search for newer posts.

- [Developing Native Android Applications in Kotlin — Intuitively and Exhaustively Explained](https://iaee.substack.com/p/developing-native-android-applications)
- [Agile Project Management — Intuitively and Exhaustively Explained](https://iaee.substack.com/p/agile-project-management-intuitively)
- [Combinatorics in Probability — Intuitively and Exhaustively Explained](https://iaee.substack.com/p/combinatorics-in-probability-intuitively)
- [Don’t Be Distracted In 2026](https://iaee.substack.com/p/dont-be-distracted-in-2026)
- [Docker, Kubernetes, and Helm — Intuitively and Exhaustively Explained](https://iaee.substack.com/p/docker-kubernetes-and-helm-intuitively)
- [Apache Spark — Intuitively and Exhaustively Explained](https://iaee.substack.com/p/apache-spark-intuitively-and-exhaustively)
- [AI Generated In-Text Citations — Intuitively and Exhaustively Explained](https://iaee.substack.com/p/ai-generated-in-text-citations-intuitively)
- [Transformer XL Positional Encoding — By Hand](https://iaee.substack.com/p/transformer-xl-positional-encoding)
- [Positional Encoding — Intuitively and Exhaustively Explained](https://iaee.substack.com/p/positional-encoding-intuitively-and)
- [UV — Intuitively and Exhaustively Explained](https://iaee.substack.com/p/uv-intuitively-and-exhaustively-explained)
...

I exited Claude and then started it up again to reset the context. Asking my same query, “What is a VAE?” resulted in the following:

roxanne@roxanne-AMD-Ryzen-7-9700X-8-Core-Processor:~/Documents/LocalTests/SkillsTest$ claude
╭─── Claude Code v2.1.136 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                               │ Tips for getting started                                                                                                            │
│              Welcome back Daniel!             │ Run /init to create a CLAUDE.md file with instructions for Claude                                                                   │
│                                               │ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │
│                    ▐▛███▜▌                    │ What’s new                                                                                                                          │
│                   ▝▜█████▛▘                   │ Added `CLAUDE_CODE_ENABLE_FEEDBACK_SURVEY_FOR_OTEL` to re-enable the session quality survey for enterprises capturing responses th… │
│                     ▘▘ ▝▝                     │ Added `settings.autoMode.hard_deny` for auto mode classifier rules that block unconditionally regardless of user intent or allow e… │
│                                               │ Fixed MCP servers configured in `.mcp.json`, plugins, and claude.ai connectors silently disappearing after `/clear` in the VS Code… │
│   Sonnet 4.6 · Claude Pro · Daniel Warfield   │ /release-notes for more                                                                                                             │
│       ~/Documents/LocalTests/SkillsTest       │                                                                                                                                     │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

❯ what is a VAE?                                                                                                                                                                       
  ⎿  ⧉ Selected 1 lines from .claude/skills/ai-researcher/articles.md in Visual Studio Code

● Skill(ai-researcher) 
  ⎿  Successfully loaded skill

● Let me check the article index for relevant VAE content.

  Read 1 file (ctrl+o to expand)

● Found a directly relevant article on Disentangled VAEs. Fetching it now.

● Fetch(https://iaee.substack.com/p/disentangled-variational-autoencoders)
  ⎿  Received 748.2KB (200 OK)

● What is a VAE?

  Based on the article Disentangled Variational Autoencoders — Intuitively and Exhaustively Explained on iaee.substack.com:

  ---
  A Variational Autoencoder (VAE) is a generative neural network architecture that learns to compress data into a structured, continuous latent space — and then reconstruct it back
  out.
  
  Architecture

  VAEs have three parts:

  1. Encoder — compresses input into a condensed representation
  2. Latent Space — the intermediate learned representation
  3. Decoder — reconstructs the original data from the compressed form

  The key twist vs. a plain autoencoder: the encoder doesn’t output a single point — it outputs two vectors: a mean (μ) and a log-variance, which together define a probability
  distribution over the latent space.

  How It Works

  Instead of encoding “this image = this exact vector,” a VAE encodes “this image = this region of space.” It then samples from that region using the reparameterization trick (sample
  from a standard normal, then scale by μ and σ), which keeps the process differentiable so backprop still works.

  Loss Function

  The loss has two competing terms:

  - Reconstruction loss — how accurately did the decoder reproduce the input? (typically binary cross-entropy)
  - KL divergence — penalizes distributions that stray too far from a standard normal (mean=0, std=1), keeping the latent space organized and continuous

  These forces balance each other: reconstruction wants accuracy, KL wants smooth, overlapping distributions.

  Why It Matters

  The resulting latent space has a useful property: nearby points decode to similar outputs, enabling smooth interpolation. Individual dimensions can also encode interpretable
  features (e.g., one dimension = brightness, another = rotation).

  β-VAE Extension

  Multiplying the KL term by a hyperparameter β > 1 produces a β-VAE, which forces the model to pack information into fewer, more semantically meaningful dimensions — a property
  called disentanglement.

  ---
  In short: a VAE is an autoencoder that learns a probabilistic, navigable latent space rather than just a compressed lookup table — making it useful for generation, interpolation,
  and learning interpretable representations.

Pretty cool. I also have a GitHub, where I record some of my notebooks. You can check it out here.

You could imagine that as another skill, or more functionality defined in the same skill, that looks through code to help the agent create a more robust answer. The world’s your oyster.

This was a short article, but that’s because the specification is very light weight. I could play around with functions and resources, but I think you get the idea. Really, it’s a folder, a markdown file, then whatever you want. If you want to learn more, I recommend opening up Claude code and working with it to define a skill of your own.

Conclusion

Skills are cool. Instead of copy-pasting prompts all day, skills allow a convenient way to organize and re-use sets of prompts and reference material. I think the ease of use is perfect for what people are using agents for right now, and allow people to define rulesets locally so that they can keep an agent from continually repeating the same mistakes.

I posted on Reddit, a while ago, about the idea of a “constrained” and “unconstrained” agent. If you’re familiar with my work, a “constrained” agent is like LangGraph, while an “unconstrained” agent is implemented in a ReAct style, like Claude Code.

You can view reddit post here.

Skills are starting to blur the lines between constrained and unconstrained agents, but are still thoroughly in the unconstrained territory. I think it’s important for developers to appreciate the difference between the loose constraints of a skill and the more rigid constraint of a graphical agent. Personally, I see Skills as a convenience that makes unconstrained agents less wrong, but doesn’t mean I’ll now trust them to operate consistently or scale in a maintainable manner as an application becomes more complex. I think, for me, the most exciting use case is that I can create metadata to finecode how Claude Code interacts with specific projects.

Developing Native Android Applications in Kotlin — Intuitively and Exhaustively Explained

Daniel Warfield — Sat, 25 Apr 2026 14:03:17 GMT

“Team K” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained.

There are three popular approaches to developing an Android application; you can use some cross platform framework like React Native, or you can develop a truly native application in either Java or Kotlin. In this article, we’ll explore why developing native applications in Kotlin is a great choice.

We’ll start by breaking down the difference of developing an application natively or with a framework, then we’ll explore why native development in Android is moving towards Kotlin and away from Java.

After we get an idea of why Kotlin is the go-to choice for android development, we’ll install Android Studio and develop two Android applications in Kotlin to see how they tick. A “hello world” style app where we’ll unpack the basics using a view approach, and a highly interactive app that will allow us to go in-depth with complex UIs and user interaction using the more modern Jetpack Compose approach.

Share this article with friends and colleagues

Who is this useful for? Anyone who wants to understand app development on the world’s most popular consumer operating system.

Android has been the most popular consumer operating system for nearly a decade. Source

Most of my articles are geared towards data science. However, as the industry evolves the practical application of AI in real products becomes an increasingly useful skill. I think this article is relevant to virtually anyone interested in making a substantive impact on the world through software development, regardless of your specific discipline.

How advanced is this post? This article is designed to be a first exploration into Android app development.

Prerequisites: None, though prior programming experience would be very helpful, especially if you’ve spent some time in some explicitly typed programming language (like Java), in doing web development, or preferably both.

Native vs Framework

The choice of developing an app with native tools or using a framework is a tricky first decision. It’s tricky because there’s not a correct answer, you can do virtually anything with either approach, and regardless of the approach you’ll likely find some inconveniences at some point based on your selection.

Basically, In app development there’s two schools of thought:

I can make an app once that runs everywhere
I can make an app that’s optimized for the device I’m developing for

For many applications, the first option is incredibly compelling. If you want to make an app that runs on both Android and iOS, it’d be nice if you could develop (and maintain) a single app that works on both types of devices. Frameworks like React Native and Flutter allow you to do just that.

The basic idea of frameworks is that they function as a layer of abstraction. You might develop your application in JavaScript for React Native, or perhaps in Dart if you’re using Flutter. The framework then compiles these into components which can be understood by Android and iOS.

Native vs Framework based development. When using a framework you develop code for that framework, then the framework handles running that code on different device types. In native development, you develop separate applications for separate operating systems.

As you can see, there’s a layer of abstraction between the app you define and the app that actually runs on the device. This is both the blessing and the curse of frameworks.

If you’re building a simple web application, then it makes a lot of sense to use a framework. They’re as performant as native apps most of the time, require less effort to port to multiple device types, and might use approaches and programming languages that you’re familiar with outside of app development.

The abstraction of frameworks becomes less compelling if you need to interface closely with the device, however. Modern devices are complicated, and have incredibly complex APIs that are ever growing and evolving. Android and iOS devices have completely different operating systems that have different rules and constraints that a developer might need to account for.

If you have some functionality in your application that might require you to interact with a low level API on your device, perhaps to run some background task while the app is closed, interface with another device via bluetooth, do video encoding, or make some fancy widget on the OS’ main screen, you may be hard pressed to find functionality that implements that within the framework.

Cross platform frameworks do, generally, allow you to implement native modules, but these are native modules, meaning you have to implement them in the native language then bridge them with the framework. This is fine if 90% of the code can be within the framework. However, if you have an application that has a ton of OS specific functionality, the added complexity of a framework can be a big headache.

In the end there’s no one right answer. If you have a simple application that uses very little native functionality, then a cross platform framework might be a compelling choice. If your app requires heightened performance or OS specific functionality, native development may be a better choice. If you’re really serious about app development, it’s good to know both. In this article, we’re focusing on native Android development.

Java vs Kotlin

This one, based on my research, is not so nuanced. Native Android supports two languages; Java and Kotlin. If you can, use Kotlin. Kotlin is derivative of, similar to, and interoperable with Java, while being much less verbose and generally safer.

If you’ve ever programmed in Java, you might be familiar with its incredible verbosity. Here’s a data class in Java, for instance, which defines a User as a name and age.

public class User {
    private String name;
    private int age;

    public User(String name, int age) {
        this.name = name;
        this.age = age;
    }

    public String getName() { return name; }
    public void setName(String name) { this.name = name; }

    public int getAge() { return age; }
    public void setAge(int age) { this.age = age; }

    @Override
    public String toString() {
        return “User{name=’” + name + “’, age=” + age + “}”;
    }
}

And here’s the same data class in Kotlin

data class User(val name: String, val age: Int)

Kotlin takes care of a lot of boilerplate stuff, like defining the constructor, getters, etc.

Java also has some safety concerns that some people care about. I’ll be honest, I’m a solo Python developer and cowboy coder; safety at the language level has never been something I’ve been highly resonant with. If you’re working on critical infrastructure in a large codebase, though, it’s nice for a language to help you avoid common pitfalls.

This Java code, for instance, doesn’t throw an error at compile time but does throw an error at runtime.

String name = null;
int length = name.length(); // ❌ Throws NullPointerException

In Kotlin, however, you’re forced to acknowledge that a variable can be null with ? , then do something if the value is null with ?:

var name: String? = null
val length = name?.length ?: 0

Kotlin has a few such implementation details which make the language more robust to common pitfalls. (Also, fun fact, ?: is known as an “Elvis” operator because, when viewed sideways, it resembles the top of Elvis Presley’s head)

Generally speaking I’ve seen consistent sentiment that Kotlin is superior to Java in many respects, and the sentiment that Kotlin should be adopted rather than Java in new projects seems consistent. For this reason (plus some spicy legal battles), Android has transitioned to a Kotlin first approach.

source

So, if you’re developing on Android, the only reason you should choose Java is if you’re working in a company that uses an overwhelming amount of legacy Java code. If not, you should probably use Kotlin.

Installing Android Studio

Before we do anything, we’re going to need Android Studio installed. You can download Android studio here.

Downloading android studio, source.

Once you download Android Studio, you should have access to an install wizard.

I just did the standard installation.

Then, once you agree to the licenses (you may need to aggrege to more than one)

It’ll install everything.

Once it’s all said and done, and you open Android Studio, you should see something like this.

And, tadah, we can start playing around with Android app development! Before we do, though, let’s explore Kotlin by itself to get a feel for the language.

A Brief Exploration of Kotlin

This won’t be an exhaustive exploration, but if you already have some programming experience this should teach you enough to have an idea of what’s going on in Kotlin. First we’ll setup a Kotlin project in Android Studio, then we’ll make a few basic Kotlin scripts.

Kotlin 1) Setup

Before we explore building an Android application, let’s get a handle of Kotlin in general. We can make a new project so we can start playing around.

Click “New Project”

We’ll be prompted to select a template “activity”. An activity, essentially, is a single screen that allows for some type of functionality in the app. This is an app development specific concept which we’ll explore later. For now, we can select “No Activity”.

Now we can specify a Name for the Project and set the Language to Kotlin.

This will create an empty project that we can play around in.

We’ll explore the folder structure of android projects in-depth. For now, though, we just need to look at the kotlin+java folder.

This folder contains:

com.example.kotlinexploration : The source code of an actual Android application
com.example.kotlinexploration (androidTest) : Testing code, that can run on the Android device
com.example.kotlinexploration (test) : Testing code, that can run on your computer

We can play around with Kotlin in any of these, but for now we’ll use the first one. We can right click on com.example.kotlinexploration and create a new Kotlin file.

We’ll then make a new Kotlin file called helloWorld

This will then open an empty Kotlin file that looks something like this

package com.example.kotlinexploration

We can then get our application to print out hello world via the following:

package com.example.kotlinexploration

fun main(){
    println(”Hello World!”)
}

We can run that code by right clicking and pressing the Run button.

An output terminal should pop up, with the following content

“/Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java” ...
“Hello World!”

Process finished with exit code 0

Congrats, you just ran Kotlin code!

Kotlin 2) Variables, Arithmetic, and String Formatting

The most important thing in any programming language is to make variables. The second most important thing is to do things with those variables. The third most important thing is to see the result of what’s been done.

We can construct a variable using the var keyword, and then format that variable within a string by using $ followed by the name of the variable

fun main(){
    var name = “Daniel”
    println(”My name is $name!”)
}

My name is Daniel!

In Kotlin, variables are statically typed and inferred, meaning kotlin inferrs the type of the variable upon creation and doesn’t allow you to change it later. Here, name is a string, based on how it’s originally defined, so I can’t go and set it to an integer later.

fun main(){
    var name = “Daniel”
    name = 0
}

Line 3: Assignment type mismatch: actual type is ‘Int’, but ‘String’ was expected.

Kotlin has many of the usual suspects in terms of types. Strings, ints, floats, booleans, lists, etc. We’ll explore a few of them in this article.

As well as defining a variable with var, you can define a “value” with val. This is an immutable value that can’t be edited after it’s creation.

fun main(){
    val name = “Daniel”
}

If we do try to edit it, we’ll get an error.

fun main(){
    val name = “Daniel”
    name = “this throws an error”
}

Line 3: ‘val’ cannot be reassigned.

Naturally, Kotlin allows us to do all the basic arithmetic one might expect from a modern programming language.

fun main() {
    val num1 = 3.1415
    val num2 = 69

    val res1 = num1 + num2
    println(”Result 1: $res1”)

    val res2 = num1 * num2
    println(”Result 2: $res2”)

    val res3 = num1 / num2
    println(”Result 3: $res3”)
}

Result 1: 72.1415
Result 2: 216.76350000000002
Result 3: 0.04552898550724638

Kotlin 3) Functions and Data Classes

Functions are defined in Kotlin in the following way

fun add(a: Int, b: Int) = a+b

fun main(){
    println(add(1,2))
}

add is a single line function, allowing one to construct a function as being equal to some simple one-line expression.

main, as you might know from other programing languages, is itself a function. It accepts no inputs, and is used as an entry point in Kotlin to run your code.

We can define our own, more complex multi-line functions like so

data class ArithmaticResult(val sum: Double, val product: Double)

fun calculate(a: Double, b: Double): ArithmaticResult{
    val sum = a+b
    val product = a*b
    return ArithmaticResult(sum, product)
}

fun main() {
    val (s, p) = calculate(5.0, 10.0)
    println(”Sum: $s, Product: $p”)
}

Sum: 15.0, Product: 50.0

Here, the function calculate takes in two inputs, a and b, and outputs the result of adding and multiplying them together.

Because Kotlin is a statically typed language, we need to specify the types of outputs we will be providing. We do that by defining the “data class” ArithmaticResult. This class contains a sum which is a Double, and a product which is also a Double.

when we call val (s, p) = calculate(5.0, 10.0), we’re unpacking the attributes of the resulting ArithmaticResult in order to the values s and p. If we wanted to be a bit more explicit, we could code the same thing like this.

data class ArithmaticResult(val sum: Double, val product: Double)

fun calculate(a: Double, b: Double): ArithmaticResult{
    val sum = a+b
    val product = a*b
    return ArithmaticResult(sum, product)
}

fun main() {
    val result = calculate(5.0, 10.0)
    val s = result.sum
    val p = result.product
    println(”Sum: $s, Product: $p”)
}

Here, we’re explicitly extracting the sum and saving it to s, and the product and saving it to p.

Kotlin 4) Classes and Inheritance

data classes, which we previously discussed, are a particular flavor of “class”, a class being that famous object-oriented paradigm of marrying data with some functionality.

Here’s a simple implementation of a box:

class Box(val width: Double, val height: Double) {

    fun area(): Double = width * height

    fun perimeter(): Double = 2 * (width + height)

    fun isSquare(): Boolean = width == height

    fun scale(factor: Double): Box = Box(width * factor, height * factor)

     override fun toString(): String =
        “Box(width=$width, height=$height, area=${area()}, perimeter=${perimeter()})”
}

As you might observe, a Box is defined as a width and a height , and has certain functions defined within it. You can calculate the area, perimeter, evaluate if it’s a square or not, get a new box that’s the same as if this box was scaled by some scaling factor, and turn the box into a string.

You might notice, the toString function has an override. This is required because every new class in Kotlin inherits from the Any class. This is an arbitrary class that defines some basic functionality that any class has. Turning the class into a String is one such piece of functionality. If we want to implement a custom toString function, we need to override the original defined in the Any class.

We can go ahead and do stuff with this class

fun main() {
    val box = Box(5.0, 3.0)

    println(”Attributes:”)
    println(”Width: ${box.width}, Height: ${box.height}\n”)

    println(”toString result:”)
    println(”$box\n”)

    println(”Function result:”)
    println(”Is square? ${box.isSquare()}\n”)

    println(”Function result:”)
    val biggerBox = box.scale(2.0)
    println(”$biggerBox\n”)
}

Attributes:
Width: 5.0, Height: 3.0

toString result:
Box(width=5.0, height=3.0, area=15.0, perimeter=16.0)

Function result:
Is square? false

Function result:
Box(width=10.0, height=6.0, area=60.0, perimeter=32.0)

Here, you can see that because we implemented a toString function, each box instance is nicely printed out. We are computing things like area and perimiter each time those values are queried, though. If we wanted to pre-compute those values, we could do it at initialization like so:

class Box(val width: Double, val height: Double) {
    val area: Double
    val perimeter: Double
    val isSquare: Boolean

    init {
        area = width * height
        perimeter = 2 * (width + height)
        isSquare = width == height
    }

    fun scale(factor: Double): Box = Box(width * factor, height * factor)

    override fun toString(): String =
        “Box(width=$width, height=$height, area=$area, perimeter=$perimeter)”
}

Here, area, perimeter, and isSquare are defined as immutable attributes. We declare that they exist at the top of the class definition, then actually define their value within the init block.

Just like how Box inherits functionality from the default Any class, we can make a new class called Square that inherits from the Box class

class Square(side: Double) : Box(side, side) {
    val sideLength = side

    override fun toString(): String =
        “Square(side=$sideLength, area=$area, perimeter=$perimeter)”
}

Here, we’re saying that Square inherits from Box with the : symbol in the class definition, and that a Square is a Box where both width and height are equal to side.

Square, because it inherits from Box, Inherits all the same initialization logic and attributes like area, perimeter, and isSquare, while adding a new attribute called sideLength and overriding the toString function with one that’s more appropriate for a square.

Classes, and logic around classes, is a massive topic with a lot of subtlety. We’ll refine our understanding of this topic as necessary as we progress.

Kotlin 5) If and When

Similarly to other languages, if statements can be complex multi-line expressions

fun main(){
  val score = 85
  
  if (score > 90) {
      println(”Excellent!”)
  } else if (score > 70) {
      println(”Good job!”)
  } else {
      println(”Needs improvement.”)
  }
}

Good job!

or simple in-line expressions used to, for instance, assign a value to a variable.

String hotString = if (temperature > 30) “Hot” else “Not Hot”

Kotlin also has some pretty fancy functionality for multi-line if statements used in assignment. Notice how the value of the assignment is in the last line of each block, but other code is able to execute in the interim.

fun main(){
 val weatherDescription = if (temperature > 40) {
        println(”It’s extremely hot!”)
        “Scorching”
    } else if (temperature > 25) {
        println(”Warm day detected”)
        “Warm”
    } else {
        println(”Cool weather”)
        “Cool”
    }
    println(”Weather: $weatherDescription”)
}

Warm day detected
Weather: Warm

Kotlin has a very unique approach to scoping, which we’ll explore throughout the article. It can be counterintuitive if you’re not familiar with it, but very useful when developing complex apps.

When statements (which are similar to switch statements in some other languages) are a convenient way to chain numerous logical cases together.

val day = 3

when (day) {
    1 -> println(”Monday”)
    2 -> println(”Tuesday”)
    3 -> println(”Wednesday”)
    4 -> println(”Thursday”)
    5 -> println(”Friday”)
    6, 7 -> println(”Weekend!”)
    else -> println(”Invalid day”)
}

Also, like if statements, you can use when statements to do assignment.

val day = 3

val message = when (day) {
    1 -> “Monday”
    2 -> “Tuesday”
    3 -> “Wednesday”
    4 -> “Thursday”
    5 -> “Friday”
    6, 7 -> “Weekend!”
    else -> “Invalid day”
}
println(message)

Like other languages, When statements are functionally similar to If statements, but are a matter of convenience when many cases need to be covered.

Kotlin 6) Lists, Iteration, Ranges, and Lambda Expressions

Kotlin allows you to define either an immutable list, which can’t be changed after creation.

val fruits = listOf(”apple”, “banana”, “cherry”)

println(fruits[0])
println(fruits.size)
println(”banana” in fruits)

apple
3
true

or a mutable list, which can be modified after creation.

val numbers = mutableListOf(1, 2, 3)

numbers.add(4)
numbers.remove(2)
println(numbers)

[1, 3, 4]

You might notice that we’re modifying numbers despite numbers being defined as a val . This might seem strange to some; as you might recall, we use val to set immutable values, while we use var to specify mutable variables.

If you’re familiar with lower level languages like C, you might know that the value numbers is actually a pointer to the list in memory, not the actual list itself. thus, even when we modify the values within the list, we’re not modifying the pointer to the list, which is what the actual value numbers represents. Thus, even though a list is mutable, we can assign its reference to an immutable val. This happens a lot in strongly typed languages, You might have an immutable reference to a mutable object. In other words, you can’t change which object a val references, but you might be able to change the data contained in the thing the val references.

We can iterate over lists element by element

for (fruit in fruits) {
    println(”I like $fruit”)
}

I like apple
I like banana
I like cherry

We can also add indexes into the iteration by employing the withIndex method.

for ((index, fruit) in fruits.withIndex()) {
    println(”${index + 1}. $fruit”)
}

1. apple
2. banana
3. cherry

Kotlin also has while loops, which one might expect.

var count = 3
while (count > 0) {
    println(”Countdown: $count”)
    count--
}

Countdown: 3
Countdown: 2
Countdown: 1

If we want to do an iteration some number of times, or over some range of values, we can use ranges:

for (i in 1..5) print(”$i “)

1 2 3 4 5

.. is the “range” operator, and allows you to specify a range of numbers between two numbers inclusively.

The range operator assumes an ascending order, but we can flip the order by using the downTo operator

for (i in 5 downTo 1) print(”$i “)

5 4 3 2 1

If we don’t want the range operator to be inclusive (for instance, when iterating over indexes in a list based on the lists size). we can use the until operator.

for (i in 0 until 5) print(”$i “)

0 1 2 3 4

You can also dictate the step size of a range.

for (i in 1..10 step 2) print(”$i “)

1 3 5 7 9

A common application of iteration is to apply some function to each element in a list. This might be done for performing actions, filtering the list, or modifying the elements in that list. Each of these can be done with lambda functions, which are simply custom functions which one defines for the purposes of a single iteration.

val fruits = listOf(”apple”, “banana”, “cherry”)
fruits.forEach { println(it.uppercase()) }

APPLE
BANANA
CHERRY

the forEach function tells Kotlin we want to apply some function to each element in a list, and it (meaning “iterator”) is automatically assigned to hold the value of each element in the list throughout iteration. the function println(it.uppercase()), specified between the two curly brackets, is then applied to each element in the list.

If you weren’t happy with it, you can assign a variable name like so.

fruits.forEach { fruit -> println(fruit.uppercase()) }

You can also use functions like forEachIndexed to do a similar thing, but specifying the name of both the index and iterable

fruits.forEachIndexed { index, fruit -> println(”${index + 1}. $fruit”)}

If the llambda function returns a boolean function, you can use filter rather than forEach. This returns a new list, consisting of each element that evaluated as true.

val fruits = listOf(”apple”, “banana”, “cherry”)
val longNames = fruits.filter { it.length > 5 }
longNames.forEach { println(it) }

banana
cherry

You can also use map to return a new list that’s the result of applying the llambda function to each element.

val fruits = listOf(”apple”, “banana”, “cherry”)
val shout = fruits.map { it.uppercase() }
shout.forEach { println(it) }

APPLE
BANANA
CHERRY

Kotlin 7) Null Values, Safe Calling, Elvis Operators, and Smart Casting

As we discussed previously, explicitly handling null values is one of the things that makes Kotlin attractively robust.

By default, variables cannot hold null values

var name: String = “Daniel”
// name = null   // ❌ Compile-time error

To be able to hold null values, ? needs to be added explicitly.

var nickname: String? = “Dan”
nickname = null   // ✅ Allowed

In most languages, if a String was defined as null, trying to get its length would result in an error at runtime. However, ?. can be used in Kotlin to safely evaluate potentially null variables. If the variable exists, then the length will be retrieved. If the variable is null, the ?.length returns null.

var nickname: String? = null
println(nickname?.length)

null

If you’re trying to assign a variable that can be null, to a variable that can’t be null, you can use the Elvis (?:) operator to assign a default value for null

val length = nickname?.length ?: 0

Typically, you must use these safe evaluators for varaibles that can be null, with a major exception being smart casting.

Take this code, for example. It doesn’t work because I can’t assign a String? to a String.

var nickname: String? = “Dan”
var notNullNickname: String = nickname

Initializer type mismatch: expected ‘String’, actual ‘String?’.

However, if I create an if statement which checks if nickname is not null, then I can assign nickname as a string.

var nickname: String? = “Dan”

var notNullNickname: String
if (nickname != null) {
    notNullNickname = nickname
} else {
    notNullNickname = “was null”
}
println(notNullNickname)

Dan

This happens because Kotlin can smart cast null enabled variables intelligently based on surrounding logical components. Because I’m checking if nickname != null, it’s impossible for nickname to be null when I assign notNullNickname = nickname. Thus I don’t have to do any special null handling. This is called “smart casting”.

Fundamental Concepts in Android

Now that we have a working understanding of Kotlin, we can start applying it to Android app development. We’ll be building a few simple apps throughout this article, but before that let’s take a moment to cover some fundamental Android concepts.

For those of you who know a thing or two about Android app development, we’re going to start our exploration by way of views, then move onto Jetpack Compose later in the article.

Fundamental 1) Activities

Possibly the most fundamental idea in Android is that of “Activities”. Loosely speaking, a single screen within your application. You might have several “activities” in your application, for instance:

a login activity
a content browser activity
a chatroom activity

Each Activity is a self-contained unit responsible for drawing its UI and handling user input.

Fundamental 2) Views and View Groups

A view is an atomic component within your app. Buttons, text fields, images, switches, progress bars; These are all “views”.

What characterizes a view is that it:

occupies some bounding box on the screen
knows how to draw itself
knows how to react to user input

It’s often useful to group these views together. For instance, you might imagine a chat app with a text field and a send button. For this type of functionality, “view groups” exist. A ViewGroup is a special kind of View that can contain other Views.

Fundamental 3) Fragments

A fragment, is, essentially a reusable mini activity that can live within an actual activity. In other words, they’re like a window within your screen which can be re-used throughout the application.

In this particular example, a scrollable content browser is implemented as a fragment within an application. This is useful because it allows app developers to implement this major piece of functionality in a way that can be copied throughout the application. Source.

Fundamental 4) Services

A service is, essentially, code that you can register to the device that isn’t directly tied to your application. It’s used for tasks that need to keep running even when the user isn’t actively using your app, like:

Playing music while the user opens another app
Downloading files in the background
Syncing data periodically
Tracking location continuously

Fundamental 5) Lifecycle

Activities, views, fragments, and services follow a “lifecycle”, meaning a they have a predefined set of events that can happen to them.

An example of callbacks which can be used based on the various states of both fragments and views. These represent the possible “lifecycle” of fragments and views. source.

It’s common to add listeners which execute functions at certain points within the lifecycle of a particular element.

Fundamental 6) Intents

Intents are like messengers within Android, allowing different pieces of an app, or even different apps, to communicate with one another. Generally speaking, intents are divided into two categories:

explicit intents: When you want to do something fully defined, like navigating from one activity to another activity within the same app
implicit intents: When you want to do something, like open a website in a web browser, but you’re not sure what the users default browser is.

There’s a lot more to Android development than what was just covered here, but I wanted to cover the lions share of the core ideas as simply as possible.

The best way to begin is by beginning. Let’s make a simple app! In doing so, we’ll explore some other core Android app development ideas as we go.

Our First Android App: Hello/World App

Let’s start with something simple, but not so simple that we can’t learn a thing or two. This app will consist of two screens; a “hello” screen and a “world” screen. Each screen will have the definition of its respective word, as well as a button to switch to the other word.

The app we’ll be creating, with a fun changing background gradient and some animation (that’s hard to see in this video) blending between the two states.

We’ll start by hopping into Android studio and making a new project. We’ll start off with a blank template (No Activity) for this one.

I called this project Hello or World in plain english. This defines the actual name of the app as it would appear on the app store. The package name and save location are automatically updated based on the name I specified. Otherwise I’m just using defaults.

We covered the folder structure of an Android project briefly, but now that we’re making an actual app let’s go a bit more in-depth.

App 1.1) An In-Depth exploration of the project structure

There are two folders in our project, the app folder and the Gradle Scripts folder.

The app folder contains all of the code and assets used to define the app. The Gradle Scripts “folder” isn’t actually a folder at all, but a convenience of Android Studio. Gradle is the build system Android Studio uses to actually turn your code into a functioning app, and the Gradle Scripts “folder” contains the files one might need to access to configure how the app gets built. First, let’s unpack the app folder, then we’ll go back to exploring Gradle.

The app folder contains the following:

The manifests folder currently contains a single file, called AndroidManifest.xml

A “manifest” file contains key information used to register your application with the device. We can open up AndroidManifest.xml and see the following

The first line is probably the most complicated.

This is boilerplate, and for the vast majority of apps will never change. This defines two namespaces android and tools. android exposes key android attributes like android:label , android:theme, and android:permission which is used later in the manifest. tools exposes fancy developer tooling that can be used in Android Studio.

Then there’s this, which is also provided by default.

This defines the app-wide settings the application can use.

android:allowBackup - Lets the system back up your app’s data to the user’s Google account.
android:dataExtractionRules - Points to an XML file defining what data can be transferred or restored.
android:fullBackupContent - Specifies which files are included or excluded from full backups.
android:icon - Sets the main launcher icon shown on the home screen and app list.
android:label - Defines the app’s display name visible to users.
android:roundIcon - Provides a round version of the app icon for compatible launchers.
android:supportsRtl - Enables layout mirroring for right-to-left languages.
android:theme - Sets the global visual theme (colors, typography, style) for the app.

There’s a bunch of other things that could be added here, it’s worth giving them a skim if you’re interested. Besides application, there are several other tags that can be added to the manifest:

 - Declares what permissions your app needs (e.g., internet, camera, location).
 - Lists hardware or software features your app depends on (e.g., camera, GPS).
 - Specifies which external apps or intents your app can look up or interact with.
 /  - Defines custom permissions that other apps must request to interact with yours.
...

That’s just a subset. There’s a lot. There’s too much to go through in depth, nor do I think it’s necessary. As long as you know what the manifest file is about, you can search for how “feature x” or “permission y” might be configured based on your specific needs. For now, it’s enough to understand that the AndroidManifest file is used to configure how your app interacts with the phone, which we’ll explore more later.

As previously mentioned, the kotlin+java folder is where our actual app code exists.

immediately, we have some strangely named folders. The name of these folders comes from a Java convention, where folders have a “reversed domain name structure”, for some esoteric reason I have no interest in understanding. Basically these are like a domain name but backwards.

helloorworld.example.com

.com on the internet was defined from the word commercial, and is a web naming convention. However, in Java, this doesn’t have to be a valid url, it’s just for naming. The idea is that you might tie this to a url you have access to. For instance, I own danielwarfield.dev, so I might use

dev.danielwarfield.helloorworld

If I was google, and I was developing a series of android example apps, I might use something like

com.google.androidexamples.exampleapp1
com.google.androidexamples.exampleapp2
com.google.androidexamples.exampleapp3

The important thing is that, across all developers on the google play store, these are uniquely defined. You can name your app whatever you want when you’re developing locally, but when you publish to the app store there can never be more than one dev.danielwarfield.helloorworld. Google does enforce this upon uploading to the app store, so if I published an app like com.facebook.totalylegitapp facebook could then submit a complaint to Google and they’d take down the app, likely marking me as fraudulent in the process.

I can go ahead and change this to something like dev.danielwarfield.heloorworld by right clicking on the root folder appand clicking Open Module Settings

I can then navigate to the Default Config tab and change the Application Id

I renamed this to dev.danielwarfield.helloworld and… The folders didn’t change. Instead, this applicationId down in build.gradle.kts changed.

That’s because there’s actually two things that define the name of our app, the Package Name and Application ID. These used to be the same thing, but are now decoupled. Previously I was talking about how this reversed URL convention is important for ensuring that an app is unique on the app store. That’s relevant to the Application ID, which can’t be changed once an app is published. If you change the Application ID , you’re publishing a completely new app.

The Package Name, on the other hand, is for organizing your code. For instance, you might have different flavors of your app, like for free, paid, and debug. You could create different Package Names, but within the same Application ID

dev.danielwarfield.helloworld.free
dev.danielwarfield.helloworld.paid
dev.danielwarfield.helloworld.debug

Despite the Package Name having no real bearing, it’s probably a good idea for us to rename it so it matches with the Application ID we previously defined. To rename our package, first click these three dots up at the top right of the file browser

Then, deselect Compact Middle Packages

This will allow us to see the entire folder structure without intermediary folders being collapsed.

We can then right click on the com folder, select Refactor, then select Rename...

we can then click “All Directories”,

Then I’ll rename com to dev and press “Refactor”

I’ll then do the same thing with example, renaming it to danielwarfield. I can then re-enable the Compact Middle Packages setting, and voila; our Package Name is renamed to be inline with our Application ID.

We don’t have any code, but if we did, using refactor would automatically update our code. It doesn’t automatically update our build script (which we’ll be exploring in a bit) so hop into build.gradle.kts (module :app) and update the namespace

I’ve noticed, when refactoring the app name in Android Studio, it’s pretty common for the change not to propagate to everywhere it’s supposed to. I recommend settling on a name ASAP so you don’t have to go around chasing this in a fully developed app.

Otherwise, there’s the resources folder. We don’t have to go through each of these one by one. Basically, the resources folder is where you put everything that isn’t code.

Some common subfolders are

layout/ — UI layout XML files (screens, fragments, dialogs)
drawable/ — Images, vector icons, and shape graphics
mipmap/ — App launcher icons (for different screen densities)
values/ — Shared definitions: strings, colors, styles, dimensions
menu/ — Menu XMLs for app bars or context menus
anim/ — Animation definitions (tween or frame animations)
xml/ — Misc configuration files (backup, data rules, security, etc.)
raw/ — Arbitrary raw assets (audio, text, JSON, etc.)
font/ — Custom font files
color/ — Centralized color resources

In terms of Gradle Scripts, it contains the following:

You’ll rarely if ever touch most of these files. The most common exceptions are:

build.gradle.kts (Module :app) - we already modified this one. This is used to Add dependencies, tweak SDK/versions
libs.versions.toml - used to manage library versions

I know that was a lot, but hopefully when you look at the project folder structure you have a much better idea of what is there and why it exists.

App 1.2) Creating the Default Activity

If we press the play button at the top of the screen

we’ll be prompted to configure our run. We can then set Hello_or_world.app as our target Module.

If we press “ok”, android will then build our app, spool up an android emulator, and attempt to deploy our app to that emulator. However, we’ll see an error saying that there’s not a default activity.

This is because we chose a completely empty project without any activities (screens) defined. We can create a simple default activity by right-clicking on our app folder (in my case, dev.danielwarfield.helloorworld) and creating a new Kotlin Class/File

We then want to create a new class called MainActivity .

That will give us the following file

package dev.danielwarfield.helloorworld

class MainActivity {
}

We can change this code to the following (we’ll explore it line-by-line in just a second)

package dev.danielwarfield.helloorworld

import android.os.Bundle
import android.widget.TextView
import androidx.appcompat.app.AppCompatActivity

class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        // Create a TextView directly in code
        val textView = TextView(this).apply {
            text = “Hello World”
            textSize = 24f
            textAlignment = TextView.TEXT_ALIGNMENT_CENTER
        }
        
        setContentView(textView)
    }
}

and then hop over to our AndroidManifest.xml file and set this as the default activity.

This is the same as our original AndroidManifest.xml file, but with the addition of

The first line

declares an activity (screen) in your app. Each activity must be declared, but usually it’s a one-liner. This one’s an exception because it’s the main activity. We define it as the main activity by giving it an intent-filter, which defines when and how this activity can be started.

We have two intent filters.

the first line says “This is the main activity, start it when the app starts on its own”. The second says “allow this to be launched by clicking the icon on the home screen”. When you click an app icon on your home screen, Android registers an intent that looks like this:

action = android.intent.action.MAIN
category = android.intent.category.LAUNCHER

So, by setting our intent filters similarly, we thus make this the main activity that starts the app. You might be able to imagine other entry points, like clicking a notification, which might have other intents and thus could launch our app into other activities (aka other screens).

If we click run now, our app will launch.

Take a moment to appreciate this. I know it’s been a lot to get to this point, but you just built an Android app! Also, you know a thing or two about how it got built! That’s no mean feat!

Along with the Hello World we want this bar to render, we’re getting Hello or World , which is the name of the app. This is happening because our default style, defined in AndroidManifest.xml has an app bar built in. We can change this line in the manifest file

android:theme=”@style/Theme.HelloOrWorld”>

to this

android:theme=”@style/Theme.Material3.DayNight.NoActionBar”>

Rebuild, and voila. No bar.

Alright, let’s take a look at our MainActivity and see how it ticks.

package dev.danielwarfield.helloorworld

import android.os.Bundle
import android.widget.TextView
import androidx.appcompat.app.AppCompatActivity

class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        // Create a TextView directly in code
        val textView = TextView(this).apply {
            text = “Hello World”
            textSize = 24f
            textAlignment = TextView.TEXT_ALIGNMENT_CENTER
        }

        setContentView(textView)
    }
}

First of all, we need to declare the package that this activity belongs to.

package dev.danielwarfield.helloorworld

Then, we import a few necessities.

import android.os.Bundle
import android.widget.TextView
import androidx.appcompat.app.AppCompatActivity

Bundle is a lightweight key-value store built into Android. It’s critical because, at pretty much any time, Android can kill your app. For instance, when you switch apps or rotate your phone. Bundle allows you to save key values so that they can be recalled. It’s used all over the place in Android, including in the constructor for AppCompatActivity

AppCompatActivity is a modern and backwards compatible class for defining Activities, and it’s generally recommended over things like Activity. We’re using it to define the actual activity we’re creating.

Finally, TextView is a view (recall that a view is like an individual component in Android, like a button, image, slider, etc) that allows us to render some text onto the screen.

To actually create our MainActivity , we create a class that inherits from AppCompatActivity and then we override the onCreate function.

class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {

When Android creates an activity, it calls the onCreate function and passes a Bundle. So we need to define it this way, weather we use the Bundle or not.

We then call the default constructor for AppCompatActivity

super.onCreate(savedInstanceState)

and create our text view

// Create a TextView directly in code
val textView = TextView(this).apply {
    text = “Hello World”
    textSize = 24f
    textAlignment = TextView.TEXT_ALIGNMENT_CENTER
}

Here, we’re creating our TextView as a view that’s bound to this ,this being our MainActivity. When creating any view, a “context” is required, which gives the view access to key resources so it can function within the application. The MainActivity serves as this context, though there are other ways one can provide context to a view.

Once the TextView is created we can set key attributes. The less Kotlin-y way to do this might look like

val textView = TextView(this)
textView.text = “Hello World”
textView.textSize = 24f
textView.textAlignment = TextView.TEXT_ALIGNMENT_CENTER

However, Kotlin has this funky apply {...} syntax that allows you to apply several attributes without repeating the name of the thing. This is, essentially, the same code as the code block above. apply is an example of a “scoped function”. I don’t want to get caught in the weeds right now, feel free to read up on scoped functions if you’re curious. We might bump into more of them later.

Finally,

setContentView(textView)

tells Android to display the text view as the content of the activity’s screen. This is all fine and dandy, but it’s not exactly how most developers would define a screen like this.

App 1.3) Defining Layout with XML

Typically, we don’t define the layout of views within the code of an activity. Rather, we create an xml file that defines the layout for the activity, and reference it in the code.

It’s customary to do that in a folder called layout within the res directory. First, right click on the res folder and make a new Android Resource Directory .

A dialogue box will appear. If we set the Resource type to layout

our directory name will be defined automatically.

Pressing “ok” then results in a new folder called layout, which we can define our layout files in.

I’ll then right click the layout file, and create a new Layout Resource File .

We’ll then define the new resource file as activity_main.

It’s customary for classes to be in upper camel case.

MainActivity

and for the layout file to be in reversed snake case.

activity_main

Why? I have no idea. When you press “ok”, you’ll see a lot of stuff happen.

Android studio has a bunch of tools to help you work on styling. For me, these tools automatically opened in the design view

We’ll start by switching it over to split view.

That will allow us to see the xml code and the resulting layout.

If we replace that code with this code (which has a textview, like in the simple activity we just defined)

We’ll see a preview pop-up on the right.

We’ll talk about the various approaches to defining layouts later.

We can now replace our MainActivity code with this

package dev.danielwarfield.helloorworld

import android.os.Bundle
import androidx.appcompat.app.AppCompatActivity

class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
    }
}

Run it, and get “hello world”, but defined by an xml defined layout.

here, R is an index of resources that’s automatically generated by Android upon building the application. because we defined our layout in res/layout/activity_main.xml, we can reference it via R.layout.activity_main. It’s common in front-end development to decouple functionality, layout, and styling. This is how that’s done in native Android.

App 1.3) Creating Hello Or World

Alright, let’s make our app actually do something. We can adjust our template to have a text field and a button

and we can implement the functionality in our MainActivity to have our button swap between a screen that says “hello” and another screen that says “world”

package dev.danielwarfield.helloorworld

import android.os.Bundle
import androidx.appcompat.app.AppCompatActivity
import android.widget.Button
import android.widget.TextView

class MainActivity : AppCompatActivity() {

    private var showingHello = true

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        val textView = findViewById(R.id.definitionText)
        val button = findViewById

If we run it, it should look something like this:

gif of using the app, once it’s built.

Let’s unpack what’s going on here line-by-line. We’ll start with the layout, which itself starts with a LinearLayout

...

a LinearLayout is a “view group”, which is to say it’s a view (component) that can hold other views (components). Here, we’re saying:

android:orientation=”vertical” tells the content of the linear layout to stack vertically
android:gravity=”center” makes it so that the content of the linear layout centers in the middle of the screen
android:layout_width=”match_parent” every view needs a width. This sets the width to inherit from the parent, which is the screen.
android:layout_height=”match_parent” same as layout_width
android:padding=”24dp” adds padding which the content of the linear layout must exist within.

There are plenty of other view groups if you want to check them out. This “LinearLayout” then has two views within it

These are more-or-less self-explanatory, except perhaps a few details.

an id is assigned to each view. In Android, components in a layout can be referenced by id, both within the code and within the same xml file. the syntax @+id/ means we’re creating a new id. We could reference already created ids within the same xml file with the @id keyword. We won’t be referencing views in the xml for this example, but we will be referencing these in the code for the activity.
wrap_content means the width and height of the view will be big enough to contain the content, and no bigger
you might notice that the margin size for the button is defiled as 20dp, and the text size of the text is defined by 20sp. dp stands for “density-independent pixels”. It’s customary to define images on a screen based on pixels, but depending on the resolution of a screen, 20 pixels can represent wildly different things. dp accounts for this by representing 1dp as one pixel on a 160dpi screen, or around 0.006 inches. If you have a high resolution screen, 20dp corresponds with more pixels, if you have a lower resolution, it corresponds to less. This keeps app size physically consistent on different screen densities.
sp, on the other hand, stands for “scale-independent pixels”. Like dp, it assumes a 160dpi screen so, by default, a single sp is equivalent to 0.006 inches. However, it has the addition of a scaling factor for font size, which users can set based on their preferences in the accessibility settings of Android.

The formula for number of pixels based on dp is this:

px = dp × (dpi / 160)

The formula for number of pixels based on sp is this:

px = sp × (dpi / 160) × fontScale

That sets the layout of our code. To understand how it actually does stuff, we look at the code

package dev.danielwarfield.helloorworld

import android.os.Bundle
import androidx.appcompat.app.AppCompatActivity
import android.widget.Button
import android.widget.TextView

class MainActivity : AppCompatActivity() {

    private var showingHello = true

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        val textView = findViewById(R.id.definitionText)
        val button = findViewById

First, we set a variable called showingHello and tie it to the activity itself. This allows the activity to keep track of if the word “hello” or the word “world” is being rendered.

We then override the existing AppCompatActivity‘s onCreate function to define some new logic when the activity is created. This requires us to run

super.onCreate(savedInstanceState)

first, so that the AppCompatActivity can do any of its required startup stuff, then we set the content of the Activity based on our xml file

setContentView(R.layout.activity_main)

Previously, the elements we made were defined statically, but now we need to update the look of our app based on user interaction. To do that, we need to get references to our views out of our xml. We do that with the following code.

val textView = findViewById(R.id.definitionText)
val button = findViewById

Here, we’re calling the findViewById function on the id defined in our auto-populated resources within R. The id’s, as specified in activity_main.xml are global, meaning all our ids across all our screens get squashed together into a single global namespace. On more complex apps you might want to adopt a more rigorous naming convention to account for many different screens.

After that, we’re defining a function that can swap between our different screens. In actuality, they’re the same screen, it’s just when showingHello is true, we use certain text on the text view and button. Otherwise, we show other text.

fun updateScreen() {
    if (showingHello) {
        textView.text = “Hello\n/həˈlō/\nA greeting or expression of goodwill.”
        button.text = “Show World”
    } else {
        textView.text = “World\n/wərld/\nthe earth, together with all of its countries, peoples, and natural features.”
        button.text = “Show Hello”
    }
}

We can then set a listener to a function that swaps the boolean, and then updates the scene. That way, every time we click the button our screen will change.

button.setOnClickListener {
    showingHello = !showingHello
    updateScreen()
}

This simple block of code gets more complicated the longer you look at it. We’re calling a function setOnClickListener, but we’re calling the function with { , rather than (. The reason for this is because of a quality of life feature built into Kotlin called a “Single Abstract Method (SAM) interface”.

If you’re in straight-up java, you might do something like this to replace a listener.

// how this would work in normal java
button.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View v) {
        showingHello = !showingHello
        updateScreen()
    }
});

So, we’re creating a new OnClickListener anonymous class, which consists of the interface

public interface OnClickListener {
    void onClick(View v);
}

(An interface defines what a class can do, but not how it gets done. An anonymous class is a one-off implementation of an interface without giving it a name), and then passing that to the setOnClickListener of the button. This is a lot of boilerplate, which is problematic when you’re building an app and need to define about a billion buttons.

A SAM interface can be employed when the interface has exactly one method within it. If it does have one method (in this case, onClickListener has only one method, onClick) there’s no ambiguity around the function you’re trying to implement, so you can simply define the function. Kotlin takes care of the anonymous class stuff for you.

button.setOnClickListener {
    showingHello = !showingHello
    updateScreen()
}

Depending on your stylistic preferences, you might prefer the simple and explicit style of Java, or the more implicit style of Kotlin. I find Java to be longer, and also not exactly intuitive anyway. At least Kotlin is shorter.

Anywho, at the end of the onCreate of our activity, we call updateScreen to actually update the text when the app launches

updateScreen()

and, with all that, our App works!

Styling Our App

Virtually every tutorial on Android app development concludes with a functional yet incredibly ugly application, which is a shame. Visual appeal is a significant part of front end development. We just made our cool little “Hello or World” app, let’s make it look halfway decent.

Stylized app, with a fun changing background gradient and some animation (that’s hard to see in this video) blending between the two states.

There’s five core things working together to make this version of the app nicer on the eyes:

a better layout
some styling on the text
a fun translucent button
a background with an animated gradient
animated transitions between words

Let’s go through each one.

1) Better Layout

First, let’s adjust the layout to make the button occupy the bottom portion of the screen, and the definition occupy the top portion of the screen

Often, doing layout work is an iterative process, consisting of making adjustments and seeing how they impact the end result. For the first approach, let’s try thinking of our layout as two views which take up the center of the top and bottom half of the screen.

To do that, we might change this code, from our original layout

to this layout code

First, the LinearLayout was replaced with a RelativeLayout . Linear layouts are designed to simply render things in a list, like a checklist, while relative layouts allow you to place elements relative to each other and to the parent. Just using a relative layout, we can see that our elements start to position themselves based on ratios of the screen, rather than as a pure list.

When we add layout_centerInParent=”true” that centers the element both horizontally and vertically within its parent. If we set both the button and the text to be centerInParent=”true”, they would stack in the center of the screen.

However, if we set one of them to layout center, the centered element will be placed in the center of the remaining space relative to the other element.

Either approach will work. For instance, with the first approach, where the button was on top and shifted to the left, we can improve its position by centering it horizontally and aligning it to the bottom of the parent (the RelativeLayout, which has a vertical height of match_parent, so the entire screen).

android:layout_centerHorizontal=”true”
android:layout_alignParentBottom=”true”

This layout is better than it was, but it has an annoying tendency to have the button jiggle up and down when the text changes.

After a bit of playing around with different approaches, I concluded that it’s a fundamental issue with having the position of the button tied to the text box, which itself changes in size. I’m sure there’s some fancy way I could adjust the layout to get rid of the jiggling issue while following this general approach, but instead I decided to change how I thought about the layout itself.

Instead of defining the elements relative to each other, let’s try laying out the elements in the center horizontally, and relative to the top and bottom of the screen vertically.

notice how the location of the views don’t depend on one another, but only in terms of their location within the view group.

The following code does that:

Here, both views are set to android:layout_centerHorizontal=”true”, meaning they’re centered horizontally, but the text view is aligned to the top (alignParentTop=”true”) and the button is aligned to the bottom (alignParentBottom=”true” ). We set a margin with respect to either the bottom or the top of the screen, and voila, our button doesn’t move when we click it.

Which is great, except now our layout is much less responsive.

At the top right of our screen, when exploring a layout xml file, there’s an option to enable something called the “split view”.

If we click that, we can see a dynamic preview of the layout we’re currently working on.

You might have noticed, I snuck in android:text=”word” into the TextView, that’s so it’s easier to see the element in the layout preview.

This looks great, but if we click and drag these diagonal lines at the bottom of the screen

We’ll find that some screen sizes result in a bad result, like the elements overlapping.

Depending on which devices you support, this may or may not be an issue. Generally speaking, I think it’s a good policy to make layouts robust to arbitrary screen sizes, but if you’re only developing for a Pixel 8 Pro, for instance

The “Device for Preview” button, named “custom” if you set a custom aspect ratio, lets you choose a specific device to preview with.

You can confirm that your layout works on that device.

Well… At least if it’s in portrait mode.

switching the phone to landscape mode reveals that the layout no-longer works.

Again, it’s a good idea to make your layouts generally robust. Let’s cut to something that actually works.

We’re going to use something called a ConstrainedLayout, which is a view group similar to the LinearLayout and RelativeLayout we discussed previously, except it has more robust features around setting constraints relative to the position of a view within the screen. Specifically, it has the ability to define a views position as a percentage of the size of the parent container.

When the screen size doesn’t change, these elements will be locked in place. However, when the screen size does change (for different devices, or by rotating the orientation of the device) the locations will adjust in proportion to the screen size.

Here’s the code that makes that happen.

first of all, we swapped out our RelativeLayout with an androidx.constraintlayout.widget.ConstraintLayout , allowing us to use the constrained layout to set our layout based on percentages. Then, each view was constrained with the following:

app:layout_constraintStart_toStartOf=”parent” Android is designed to work both with languages that read left-to-right and languages which read right-to-left. Thus, sometimes in android, “left” and “right” are abstracted into “start” and “end”, where start could be either the left or right, depending on the language, allowing the UI flow of the app to follow the flow of the native language. This line says the “start” of this element should be constrained to the “start” of the parent. So, if you speak english, left side to left side.
app:layout_constraintEnd_toEndOf=”parent” Same as the previous, but with the “end” of the view being constrained to the “end” of the parent.
app:layout_constraintTop_toTopOf=”parent” There’s not really languages that read bottom up, so the top of a view is simply the top of the view. This constrains the top of the view with the top of the parent.
app:layout_constraintBottom_toBottomOf=”parent” Same as previous, but with the bottom of the view being constrained to the bottom of the parent.
app:layout_constraintVertical_bias= This value specifies how biased the view is to the top or bottom constraint. If 0, the view is biased to the top. If 1, the view is biased towards the bottom.

If we look at the layout of both elements, we can see that both elements are constrained to all four sides of the view group via squiggly lines.

In Android, it’s common to think of constraints as springs, which pull each element into equilibrium. Because the horizontal springs are “pulling” with the same force, each element is centered horizontally. When we introduce a Vertical_bias, it biases the force of the spring, allowing the element to reach equilibrium either higher or lower down in the screen.

As you can see, this works well with a bunch of aspect ratios.

We could stop here, but let’s do one more adjustment to make this even better.

Constrained layouts do their constraining based on the center of a view. This locks our button in place, but it means our text shifts up and down based on the text length.

notice how the top word shifts up and down. This is because the two text fields take up a different vertical space. The constraint is done relative to the center, so the different height causes the first word to shift up and down.

This is purely stylistic, but I would like for the text to appear fixed at the top of the text block. I think that would aid in readability, and make the application look cleaner and more professional.

To do that, I’m going to employ “Guidelines”. These are lines that you can place within a view group, which other views can be placed in respect to

So, we can set these guides at some vertical location on the screen, then align our elements to those guidelines. That way we get the percentage based spacing of the constrained layout, with more consistent text alignment regardless of updating text size.

Here’s the code.

Here, we set up two horizontal guide lines, specify their height within the screen with app:layout_constraintGuide_percent= , assign them an id, like android:id=”@+id/topGuide” then constrain our views to those guides via app:layout_constraintTop_toTopOf=”@id/topGuide”.

You might notice how some of these attributes start with android: and some of them start with app:. This is a bit of a rabbit whole, which we’ll explore more when we implement our own custom views. We can briefly touch on it now, though.

In essence, there’s a build tool called AAPT2 (Android Asset Packaging Tool 2). This is responsible for turning your high level resources (like layout files, images, etc.) into a binary format that’s actually usable by an android phone. To deal with different libraries having potentially similar naming conventions, Android uses “namespaces” in the xml file to contextualize attributes via some unique identifier. xmlns:android=”http://schemas.android.com/apk/res/android" , for instance. This isn’t a URL, but a URI (universal resource indicator) that references the core SDK for Android, and assigns it to the namespace android.

xmlns:app=”http://schemas.android.com/apk/res-auto" is a special “auto resolving” URI, that auto-resolves based on where the attribute is defined, in this case androidx.constraintlayout.widget.ConstraintLayout. xmlns:app="...res-auto" binds the prefix app: to a special namespace that lets AAPT2 automatically resolve attributes to the correct library resource package. Unlike android:, which points to the Android framework, res-auto tells the build tools: “Find the attribute definition in whatever library provides it,” which is why custom attributes from ConstraintLayout (like layout_constraintTop_toTopOf) appear under the app: namespace.

Android Jetpack is an official Android library which greatly improves Android development with modern design principles. Jetpack exists under the library named AndroidX. AndroidX defines custom XML attributes that are not part of the core Android framework. When we set xmlns:app="http://schemas.android.com/apk/res-auto", we’re telling AAPT2 to automatically resolve any app: attribute to whichever library defines it. In the case of ConstraintLayout, its custom attributes like layout_constraintTop_toTopOf come from the AndroidX ConstraintLayout library, so AAPT2 resolves the app: attributes to that library’s resources.

I’m not stating these things so that you understand it, but that you’re aware that, in modern Android app development, it’s common to use several different namespaces simultaneously, and you may need to account for that when building views.

Back to the task at hand, though, we used a constrained layout, defined two guidelines, and set our text and button view positions relative to those guidelines. Thus, we have a layout that serves for our purposes.

2) Text Styling

Now that we have a layout we’re happy with, let’s make the text look good. Recall that, on our improved app, we have a variety of fonts that we use for the main word, the phonetic pronunciation, the definition, and the font for the text.

Where we want to end up, in terms of styling.

There are various ways we can achieve this effect. We’ll make a few adjustments in our layout file, and also make some adjustments in our activity code. Let’s start with layout adjustments.

Our original XML defines our text style as the following

Which we’ll change to

We changed textSize to 24sp (24 scale-independent pixels, which is commonly used for fonts), to make it a bit bigger. We also set lineSpacingExtra to 8dp (density-independent pixels which is commonly used in layout to accommodate different screen sizes). This adds a small amount of space between each line which remains consistent across screens with different pixel densities. We also set textColor to “#FFFFFF” , which is hexadecimal for pure white, and we set the fontFamily to serif .

That will result in pretty text, which is difficult to see because our app currently has a white background.

We can set our constrained layout to have a background of blue, allowing us to actually see the text.

Already looking way better than before. Let’s give our button a bit of font treatment as well. We’ll keep the font family the same, but make the text bigger and set the font color to pure white.

This doesn’t look quite as spiffy as our end goal, largely because we’re still missing our cool translucent button. We’ll get around to that later. Another minor difference is that the text doesn’t have italicization on the phonetic pronunciation, and bold for the word.

Recall where we want to end up, in terms of styling. Notice how the word is bold, and the phonetic pronunciation is italicized.

We can achieve that by using HTML rendering within our activity. For those not familiar with HTML, it’s a language used throughout front end development to structure content on websites and applications.


  We can achieve that by using HTML rendering within our activity. For those not familiar with HTML, it’s a language used throughout front end development to structure content on websites and applications. The website you’re looking at to read this article uses HTML to build its own layout.

One of the most fundamental aspects of HTML is the “tag”, which is used to either encapsulate a section of text to have some styling, or inject some special component.

For instance, in HTML the tag means to “bold” some text. We denote when a section of text auto to start being bold with , then close out the tag with .

This text is bold! This text is not!

We can do something similar with italicization. html uses the tag , which stands for “emphasis” to control italicization.

This text is italicized! This text is not!

Some tags don’t require an open and a close. For instance, if we want to inject a newline in our sentence, we don’t need to specify anything “within” the newline, we just want to add a newline in a certain spot. To create a newline we can use the tag, which stands for line break, and we can make is “self closing” by adding a slash at the end, a la

This text is on line 1
this text is on line 2

Using bold, italics, and line breaks, we can turn our text into an HTML equivalent within our activity. We can then use Androids HtmlCompat (a backwards compatible HTML processor) to process the html.

package dev.danielwarfield.helloorworld

import android.os.Bundle
import androidx.appcompat.app.AppCompatActivity
import android.widget.Button
import android.widget.TextView
import androidx.core.text.HtmlCompat

class MainActivity : AppCompatActivity() {

    private var showingHello = true

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        val textView = findViewById(R.id.definitionText)
        val button = findViewById

Along with our current layout

This results in the following

Which is getting awfully close to our desired output. Next, let’s make our button look a bit more fancy.

3) A Fun Translucent Button

Recall where we want to end up in terms of styling. The button is larger, and has a transluscent quality which allows the background color to show through.

Views, like buttons, do a lot of things under the hood. They handle user interaction, logic, and how they render themselves on the screen. The actual visual part is a drawable, which you can think of as the logic within a view that is only relevant to how that view visually looks. In our resources folder, we can right-click drawable , create a new Drawable Resource File and call it translucent_button

In the resulting translucent_button.xml , we’ll use this code

The core drawable we’ll be building off of is called a ripple, which is a

Drawable that shows a ripple effect in response to state changes — Android docs

Basically, when the user clicks our button, it’ll make a cool subtle ripple effect.

You can think of the ripple as an effect that wraps around an item, where that item is a shape that’s a rectangle. The rectangle has a small radius on the corners, as specified by , and a color that’s specified by . This color is what defines the buttons translucency. I don’t want to go into hexadecimal color encoding in-depth, there are plenty of color pickers and converters you can use, but basically the first two characters say the button will be 20% opaque (80% transparent). The F’s say the button will be white. So, the button will be white and mostly transparent.

We can enable our button view to use this drawable as a background by simply referencing it with the @ symbol within our layout, and specifying the path based on the resources directory.

All put together, this code

res/drawable/translucent_button.xml

res/layout/activity_main.xml

MainActivity.kt

package dev.danielwarfield.helloorworld

import android.os.Bundle
import androidx.appcompat.app.AppCompatActivity
import android.widget.Button
import android.widget.TextView
import androidx.core.text.HtmlCompat

class MainActivity : AppCompatActivity() {

    private var showingHello = true

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        val textView = findViewById(R.id.definitionText)
        val button = findViewById

Yields this:

when we click the button, we get a fun little effect as a result of using ripple

That’s already looking pretty great, but we have even more improvements to make.

`4) Animated Gradient Background`

Another improvement we’ll make is drawing a subtle gradient of color that slowly changes over time.

To do this, we’ll implement another drawable, similarly to what we did for the button, but we’ll set it as the background of the entire activity. We’ll also need to implement animation so the color can change over time.

we’ll start by creating the drawable in a new xml, res/drawable/gradient_bg.xml

This defines a shape , that’s a rectangle , with a gradient inside. the gradient is a linear gradient (meaning there’s a steady shift from one color to the other along some line) with an angle of 45 (meaning the gradient is not a blend from side-to-side, which would be 0, or a blend from top to bottom 90, but a blend from the top right corner to the bottom left corner). It blends from the color #FF5F6D (a redish-pinkish color) to the color #FFC371 (a yellow-orange ish color).

We can get an idea of what that looks like by setting our constrained layout’s background to this drawable

\
 
    ...

To animate the background, we’ll need to reference this drawable via the main activity code, and implement animations that schedule when these colors should change.

To do that, first, we need to assign an id to our constraindLayout for our activity_main.xml so we can reference it’s background in the code.

Then, in our main activity, we’ll access our layout by the id. We’ll also extract the background and make sure it’s correctly set to a GradientDrawable type, allowing us to edit the gradients. While we’re in here, we’ll also define a few different gradients as an array of integer arrays.

...
import androidx.constraintlayout.widget.ConstraintLayout 
...

class MainActivity : AppCompatActivity() {
  ...
   override fun onCreate(savedInstanceState: Bundle?) {
        ...
        val root = findViewById(R.id.rootLayout)

        val gradient = root.background as GradientDrawable

        val colors = arrayOf(
            intArrayOf(0xFFFF5F6D.toInt(), 0xFFFFC371.toInt()), // pink → orange
            intArrayOf(0xFF2193B0.toInt(), 0xFF6DD5ED.toInt()), // blue → cyan
            intArrayOf(0xFFCC2B5E.toInt(), 0xFF753A88.toInt()), // magenta → purple
            intArrayOf(0xFFEE9CA7.toInt(), 0xFFFFDDE1.toInt())  // rose → light pink
        )
        ...
 }
 ...
}

Here, root is the root of the view, which is the ConstraintLayout where the entire screen exists within. We’re getting its background and making sure that gets cast as a GradientDrawable. Basically, at compile time, Kotlin can only infer that the background is a Drawable, not a GradientDrawable, and thus the compiler will throw an error when it analyzes our code and sees we’re messing with variables that don’t exist in the Drawable class. We need to cast it explicitly as a GradientDrawable, which inherits from Drawable, so we don’t get this issue.

Now we can set up and register our animation which modifies our gradient.

import android.animation.ArgbEvaluator
import android.animation.ValueAnimator
...

class MainActivity : AppCompatActivity() {
    ... 
    override fun onCreate(savedInstanceState: Bundle?) {

        ...
        val gradient = root.background as GradientDrawable

        val colors = arrayOf(
            intArrayOf(0xFFFF5F6D.toInt(), 0xFFFFC371.toInt()), // pink → orange
            intArrayOf(0xFF2193B0.toInt(), 0xFF6DD5ED.toInt()), // blue → cyan
            intArrayOf(0xFFCC2B5E.toInt(), 0xFF753A88.toInt()), // magenta → purple
            intArrayOf(0xFFEE9CA7.toInt(), 0xFFFFDDE1.toInt())  // rose → light pink
        )

        fun animateGradient(startIdx: Int = 0) {
            val (startA, startB) = colors[startIdx]
            val (endA, endB) = colors[(startIdx + 1) % colors.size]

            val animator = ValueAnimator.ofFloat(0f, 1f).apply {
                duration = 8000L
                addUpdateListener { valueAnimator ->
                    val fraction = valueAnimator.animatedFraction
                    val newStart = ArgbEvaluator().evaluate(fraction, startA, endA) as Int
                    val newEnd = ArgbEvaluator().evaluate(fraction, startB, endB) as Int
                    gradient.colors = intArrayOf(newStart, newEnd)
                }
                addListener(object : android.animation.Animator.AnimatorListener {
                    override fun onAnimationEnd(animation: android.animation.Animator) {
                        animateGradient((startIdx + 1) % colors.size)
                    }
                    override fun onAnimationStart(animation: android.animation.Animator) {}
                    override fun onAnimationCancel(animation: android.animation.Animator) {}
                    override fun onAnimationRepeat(animation: android.animation.Animator) {}
                })
            }
            animator.start()
        }

        animateGradient()
        ...

Alright, there’s a lot going on. At its heart, we’re using something called a ValueAnimator which blends between two colors in our list of gradients. When the ValueAnimator is done we create a new ValueAnimator that blends between the next pair of colors. We do that, indefinitely, while the current activity is running. When we call start on a ValueAnimator, Android registers it, keeps track of it, and and uses it to calculate what the screen should look like each time it draws a frame.

The animateGradient is a recursive function. The first time it’s called, it registers an animator that blends between the gradient specified at index 0 and the gradient specified at index 1 in our list of colors. Within the animator that gets created, we define a listener onAnimationEnd that creates a new animation for the next batch of colors once this animation ends.

    val colors = arrayOf(
            intArrayOf(0xFFFF5F6D.toInt(), 0xFFFFC371.toInt()), // pink → orange
            intArrayOf(0xFF2193B0.toInt(), 0xFF6DD5ED.toInt()), // blue → cyan
            intArrayOf(0xFFCC2B5E.toInt(), 0xFF753A88.toInt()), // magenta → purple
            intArrayOf(0xFFEE9CA7.toInt(), 0xFFFFDDE1.toInt())  // rose → light pink
        )    

    fun animateGradient(startIdx: Int = 0) {    
        ...
        fun animateGradient(startIdx: Int = 0) {
            val (startA, startB) = colors[startIdx]
            val (endA, endB) = colors[(startIdx + 1) % colors.size]

            val animator = ValueAnimator.ofFloat(0f, 1f).apply {
                ...
                addListener(object : android.animation.Animator.AnimatorListener {
                    override fun onAnimationEnd(animation: android.animation.Animator) {
                        animateGradient((startIdx + 1) % colors.size)
                    }
                    ...
                })
            }
            animator.start()
        }

        animateGradient()
        ...

Thus, we only need to create the first animation, then that animation will create the next, that animation will create the next, so on and so forth. Thus, virtually all of the functionality of our animated background functions within the animateGradient function, which then calls itself recursively.

Looking at that function, in its entirety:

fun animateGradient(startIdx: Int = 0) {
    val (startA, startB) = colors[startIdx]
    val (endA, endB) = colors[(startIdx + 1) % colors.size]
    
    val animator = ValueAnimator.ofFloat(0f, 1f).apply {
        duration = 8000L
        addUpdateListener { valueAnimator ->
            val fraction = valueAnimator.animatedFraction
            val newStart = ArgbEvaluator().evaluate(fraction, startA, endA) as Int
            val newEnd = ArgbEvaluator().evaluate(fraction, startB, endB) as Int
            gradient.colors = intArrayOf(newStart, newEnd)
        }
        addListener(object : android.animation.Animator.AnimatorListener {
            override fun onAnimationEnd(animation: android.animation.Animator) {
                animateGradient((startIdx + 1) % colors.size)
            }
            override fun onAnimationStart(animation: android.animation.Animator) {}
            override fun onAnimationCancel(animation: android.animation.Animator) {}
            override fun onAnimationRepeat(animation: android.animation.Animator) {}
        })
    }
    animator.start()

We get whatever colors we want to start and end with, and define them as (startA, startB) and (endA, endB). Then we define our animator, which is a ValueAnimator. At its core, a ValueAnimator interpolates between two values over time. The updating of this value is what ends up governing how other things should react. .ofFloat(0f, 1f) means we’re interpolating a float (decimal number) between 0 and 1. Then, we’re doing one of those fancy Kotlin “scope functions” with .apply { ; ValueAnimator.ofFloat(0f, 1f) creates a ValueAnimator , then calling .apply { on that ValueAnimator says that we’re going to work on that value animator by setting the valueAnimator to this.

Then, this code is using another Kotlin convenience. Within the scope function, Kotlin assumes that, if you specify duration = , you’re actually doing this.duration = if this does have a duration attribute. Same with addUpdateListener and addListener. It’s assumed that these are calling functions in this , where this is the ValueAnimator we just created.

If you’re anything like me, you might be skeptical of all this abstraction. Why do all this when you can just create animator = ValueAnimator then just set stuff like animator.duration = ? You certainly can do that. However, if you do wrap your head around these scoped functions, you might find that the strict scoping and encapsulation is practically convenient when managing complex apps with large numbers of things being defined and manipulated. I don’t know about you, but I’ve often had a sneaky bug where I copy-pasted some code that thought I was modifying button A, but I was actually modifying button B. Scoping can help clean up a lot of those issues, hence why Kotlin apps tend to be more reliable.

Anyway, so this weird syntax is really just setting the duration of animator , and calling its addUpdateListener and addListener methods. We’re defining the update listener via a Kotlin lambda with an implicit Single Abstract Method (SAM) conversion.

addUpdateListener { valueAnimator ->
    val fraction = valueAnimator.animatedFraction
    val newStart = ArgbEvaluator().evaluate(fraction, startA, endA) as Int
    val newEnd = ArgbEvaluator().evaluate(fraction, startB, endB) as Int
    gradient.colors = intArrayOf(newStart, newEnd)
}

Again, more weird Kotlin abstraction which is useful once you understand it. The addUpdateListener expects a single-method interface that looks something like this

public interface AnimatorUpdateListener {
    void onAnimationUpdate(ValueAnimator animator);
}

This allows us to define a lambda function which takes in the input valueAnimator, and pass that lambda function into addUpdateListener. Again, this might seem annoyingly abstract until you think about how many animations, buttons, toggles, switches, and other things need to have functions assigned to them. In a language like Java, that would require 6–8 lines of boilerplate just to set up, every single time. In Kotlin, there’s basically no boilerplate code.

Calling addListener has a similar story, except because it expects an interface that has more than one method, we need to implement it slightly differently. The interface which addListener expects, AnimatorListener, has four functions. We can create an anonymous class which satisfies that interface and then implement the functionality we want, which is to start a new animation when the animation ends.

addListener(object : android.animation.Animator.AnimatorListener {
    override fun onAnimationEnd(animation: android.animation.Animator) {
        animateGradient((startIdx + 1) % colors.size)
    }
    override fun onAnimationStart(animation: android.animation.Animator) {}
    override fun onAnimationCancel(animation: android.animation.Animator) {}
    override fun onAnimationRepeat(animation: android.animation.Animator) {}
})

So, we have our complete main activity

package dev.danielwarfield.helloorworld

import android.animation.ArgbEvaluator
import android.animation.ValueAnimator
import android.graphics.drawable.GradientDrawable
import android.os.Bundle
import androidx.appcompat.app.AppCompatActivity
import android.widget.Button
import androidx.constraintlayout.widget.ConstraintLayout
import android.widget.TextView
import androidx.core.text.HtmlCompat
import android.util.Log

class MainActivity : AppCompatActivity() {

    private var showingHello = true

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        val root = findViewById(R.id.rootLayout)
        val textView = findViewById(R.id.definitionText)
        val button = findViewById

Our layout file for our main activity

and our drawables for our button and background, respectively

And that gets our animated background done and dusted.

We have one more aesthetic adjustment I want to implement on this app.

`5) Animating Transitions Between Words`

Next, we’ll implement a nice blending animation that plays when switching between words.

This preview is choppy because I had to convert it to a gif, in reality the result is a nice fade away of the previous word, and a fade in for the word replacing it.

Our current button has a simple listener attatched to it that flips a boolean and calls updateScreen, which in turn updates the text based on that boolean.

button.setOnClickListener {
    showingHello = !showingHello
    updateScreen()
}

We can implement fading by simply assigning an animation to our text when we click the button.

button.setOnClickListener {
    showingHello = !showingHello

    val fadeOut = android.view.animation.AlphaAnimation(1f, 0f).apply {
        duration = 200
    }
    val fadeIn = android.view.animation.AlphaAnimation(0f, 1f).apply {
        duration = 300
        startOffset = 200
    }

    fadeOut.setAnimationListener(object : android.view.animation.Animation.AnimationListener {
        override fun onAnimationStart(animation: android.view.animation.Animation?) {}
        override fun onAnimationRepeat(animation: android.view.animation.Animation?) {}

        override fun onAnimationEnd(animation: android.view.animation.Animation?) {
            updateScreen()
            textView.startAnimation(fadeIn)
        }
    })

    textView.startAnimation(fadeOut)
}

Here we’re defining two animations, which are both AlphaAnimation. These animate the transparency of a view from fully opaque (1) to fully transparent (0). fadeOut makes the text transparent, and fadeIn makes the text opaque.

Similarly to what we did with the background, we’re controlling these sequential animations by calling setAnimationListener on our fadeOut animation. We’re using that to update the screen (swap the text) while the text is completely transparent, then call the fadeIn animation. So the text fades out, swaps without the user seeing, then fades in as the new text.

the fade out animation takes 200 milliseconds, then the text stays transparent for 200 more milliseconds (on account of the startOffset before fade in) before fading in over the course of 300 milliseconds.

And, with that, our fancy Hello or World app is done!

Now that we have a grasp of the basics, we can start building some pretty sophisticated stuff. Let’s get into it!

`Our Second App: Flag Crank App`

The idea of this app is to experiment with more complex user interaction, more complex UI design, and integrate more thoroughly into the phones resources. We’ll do that by building a simple game where you need to spin a crank to raise a flag. If the flag goes all the way up, you win. If it goes all the way to the bottom, the phone vibrates.

`App 2, Part 1 — Setup`

Just like the last app, I kicked this one off as a new blank project, this time called “Flag Crank”. I didn’t bother changing the whole com.example thing, so the package name for this project is com.example.flagcrank.

ignore that my “Save Location” is “…FlagCrank2”, I already had a FlagCrank directory on my computer. This shouldn’t change anything, as all of the resources within the project are relative to each other.

Then, in com.example.flagcrank, right click and make a new Kotlin Class, which we’ll call MainActivity

If we swing up to our AndroidManifest.xml file in the manifests folder, we’ll see the default manifest.

We need our new activity to be within the application, and we need to set it’s intent to be compatible with opening the application from the home screen, which (as previously discussed) we define via the following:

While I was in there, I also snuck in android:theme=”@style/Theme.Material3.DayNight.NoActionBar”>. This makes it so we don’t have a bar on the top of our screen with the app name.

It’s a good policy, when doing this type of configuration work, to try running the app often just to make sure everything’s set up right. We can define a super simple MainActivity, which just renders the text “Hello World” on the top left of the screen

package com.example.flagcrank

import android.os.Bundle
import android.widget.TextView
import androidx.appcompat.app.AppCompatActivity

class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        val tv = TextView(this).apply {
            text = “Hello World”
            textSize = 24f
            setPadding(50, 50, 50, 50)
        }

        setContentView(tv)
    }
}

Run it, and make sure everything works

Alright, let’s start building this FlagCrank app

`App 2, Part 2 — Planning The App`

unlike the previous app we made, we’re going to jump right to a functional solution, and we won’t spend a lot of time on UI prettification. On this app, we’re focusing on app logic and defining custom UI elements.

There are going to be two major elements, the flagpole and the crank. These will exist within the screen, which will manage how the two components communicate with one another. We’ll define these components so that, when the crank moves, the flag updates.

You might think the crank should be a view, the flag should be a different view, and we should implement a view group which contains both views. That’s one way to approach building an app like this, but we won’t be doing it. Instead, we’ll be using something called “Jetpack Compose”; a different and more modern approach to Android app development which replaces the concept of views.

`App 2, Part 3 — Exploring Jetpack Compose vs Traditional Views`

The traditional approach to Android development is to use views, which we discussed in the previous application; you define a layout in XML, and communicate with those layouts via Kotlin code. Each view has some state, can accept user input, and knows how to render itself to the user. Developing an app, then, is essentially defining these views and how they connect with one another.

The traditional approach to android app development. Each element is a “view” which has it’s own state information. App development, then, is a game of getting the states of these views to play nicely with one another.

One of the biggest issues with this approach is synchronization. There’s a lot of different sources of truth here, with different components reacting to user inputs and updating their own state independently. The user might press multiple buttons, each of which might impact multiple views, which in turn might impact multiple states. Generally the approach is to move state upward, so that container views hold the state that is relevant to the views it contains. Even when perfectly designed though, there’s a ton of edge cases a developer needs to account for.

Jetpack Compose is a modern framework for building applications in Android, which employs a different model than the view model. Instead of thinking of a component as having state that needs to be shared amongst one another, instead, components structure state in a strictly top-down manner. Instead of components needing to negotiate on state, state is passed from top to bottom, component to component, and that change in state is used to update the UI (a process called “recomposition”).

Using Jetpack Compose, state is shared between app components in a hierarchical manner.

Because different components don’t have to “agree” on the state, and instead modify and react to state based on a rigid top-down structure, handling complex state changes in a compose powered application is significantly more straightforward. This is made possible by parents passing callback functions to child elements. Children can call these callbacks, modifying state in the parent.

A conceptual diagram of state and callbacks in Jetpack Compose. State information flows in a top-down hierarchy, while callbacks allow granular events like user interaction to move up the chain.

It’s worth noting that you can achieve this in a view style app. In fact, a top down state management is encouraged, JetpackCompose just makes it more rigid and enforced based on the way the system works.

While some components need to keep track of state, most compose components don’t need to hold a state which changes over time, but instead react to a state change. When working with compose you define a function that defines the UI based on the state. When the state changes, the function is re-run, modifying the UI.

In Jetpack Compose, components are updated as functions which responed to changes in a shared app state.

There’s two conflicting ideas that start to make more sense when you get practical: hierarchical state management, and not managing state at all and treating the UI like a function. These are hard to understand without getting into the specifics, so let’s get composed set up in our build tools, then we can use it to begin implementing our app.

`App 2, Part 4 — Configuring Dependencies`

In order to use jetpack in our project, we need to set it up with Gradle (the build tool we’ve been using to turn our code into an actual running app). These types of configurations have a tendency to shift over time, but for me all I had to do is navigate to app/build.gradle.kts and add

buildFeatures {
        compose = true
    }

to it, like so:

plugins {
    alias(libs.plugins.android.application)
    alias(libs.plugins.kotlin.android)
    alias(libs.plugins.compose.compiler)
}

android {
    namespace = “com.example.flagcrank”
    compileSdk {
        version = release(36)
    }

    buildFeatures {
        compose = true
    }

    defaultConfig {
      ...

We can then replace our MainActivity.kt with the following code to confirm that we’re properly using Jetpack Compose

package com.example.flagcrank

import android.os.Bundle
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color

class MainActivity : ComponentActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        setContent {
            MaterialTheme {
                Surface(
                    modifier = Modifier.fillMaxSize(),
                    color = Color(0xFF101010)
                ) {
                    Box(
                        modifier = Modifier.fillMaxSize(),
                        contentAlignment = Alignment.Center
                    ) {
                        Text(
                            text = “Hello, Compose!”,
                            style = MaterialTheme.typography.headlineMedium,
                            color = Color.White
                        )
                    }
                }
            }
        }
    }
}

This is our first time seeing any Compose code, so I thought it might be productive to explore it in depth.

The first three lines of code are similar to what we’ve seen previously, but now we’re using import androidx.activity.compose.setContent. This is an integration detail, allowing our traditional activities to use the more modern Compose approach.

Then we have a bunch of imports to androidx.compose, which is where the core Jetpack Compose library lives. Within androidx.compose there are various modules, like:

ui: Contains foundational types and systems, for things like colors and layout measurement, which other modules use.
foundation: Un-styled and pre-built primatives, like boxes, rows, columns, as well as tools for understanding user interaction. These are primitive UI components that can be styled and modified to construct an app.
material3: This is built on top of foundation, and contains style information like color schemes, typography, and stuff like that.

There are other modules in jetpack compose as well, like animation and runtime, that can be used for more advanced functionality.

The component itself looks a lot like Android code we’ve explored previously. We have MainActivity that inherits from ComponentActivity(), and we have an onCreate function. However, instead of using setContentView, like we did previously, we’re using setContent. This is the bridge that connects Jetpack Compose to the outside application.

When you define something with JetpackCompose, the generall pattern looks something like this:

SomeComposable(                                                                                     
    parameter = value,                                                                              
    anotherParameter = value
) {                                                                                                 
    // children go here
}

Which allows you to nest composable elements inside one another. It uses a weird kotlin syntax called “trailing lambda syntax”. If the last argument to a function is another function, like so:

function(arg, arg, function_def)

You can create the function as a lambda function, using curly brackets after the function, and it will automatically resolve

function(arg, arg) {function_def}

If the function only takes in another function, you can skip the parantheses entirely

function{funciton_def}

Because Compose uses functions to define UI components, there’s a lot of nesting of those components. This kind of quirky syntax is super useful in doing that without having a lot of boilerplate.

So, taking another look at our code, the UI is a bunch of nested funcitons that feed into eachother. When the state of the app changes, jetpack compose calls these various functions to draw the app.

setContent {
    MaterialTheme {
        Surface(
            modifier = Modifier.fillMaxSize(),
            color = Color(0xFF101010)
        ) {
            Box(
                modifier = Modifier.fillMaxSize(),
                contentAlignment = Alignment.Center
            ) {
                Text(
                    text = “Hello, Compose!”,
                    style = MaterialTheme.typography.headlineMedium,
                    color = Color.White
                )
            }
        }
    }
}

setContent exists at the top to act as a bridge, then the MaterialTheme contains a Surface that contains a Box that contains Text.

The point of materials, is to create and then delineate design themes across child elements. It exists at the top, then all children can use it. This example doesn’t have a lot of details in that regard, I don’t want to get bogged down on styling, but the following example shows how material information might be propegated through a slightly more complex activity.

setContent {                                                                                        
      MaterialTheme(                                    
          colorScheme = darkColorScheme(                                                              
              primary = Color(0xFF6650A4),                                                            
              background=/__u/iaee.substack.com/Color(0xFF101010),                                                         
              onBackground = Color.White                                                              
          ),                                                                                          
          typography = Typography(
              headlineMedium = TextStyle(fontSize = 28.sp, fontWeight = FontWeight.Bold),             
              bodyMedium = TextStyle(fontSize = 16.sp),
              labelSmall = TextStyle(fontSize = 12.sp, color = Color.Gray)                            
          )       
      ) {                                                                                             
          Surface(                                                                                    
              modifier = Modifier.fillMaxSize(),
              color = MaterialTheme.colorScheme.background                                            
          ) {     
              Column(
                  modifier = Modifier.fillMaxSize(),
                  verticalArrangement = Arrangement.Center,
                  horizontalAlignment = Alignment.CenterHorizontally                                  
              ) {
                  Text(                                                                               
                      text = “Hello, Compose!”,
                      style = MaterialTheme.typography.headlineMedium,
                      color = MaterialTheme.colorScheme.onBackground
                  )                                                                                   
                  Text(
                      text = “This is body text.”,                                                    
                      style = MaterialTheme.typography.bodyMedium,
                      color = MaterialTheme.colorScheme.onBackground
                  )                                                                                   
                  Text(
                      text = “small label”,                                                           
                      style = MaterialTheme.typography.labelSmall
                  )
              }
          }
      }
  }

Inside the material is a Surface, which is pretty mugh just the background. Then there’s a Box, which helps us lign stuff up correctly, then Text, which renders the text on the screen. Give it a run, and we have a Jetpack Compose powered app.

I could describe all of the layout and styling ideas we’ve discussed previously, but many of them are directly applicable to Jetpack Compose with a modest amount of googling/LLMing.

`App 2, Part 5— Implementing the Flag in Jetpack Compose`

Recall, we’re trying to make this flag crank app

Let’s build it. First; right click, new, Kotlin Class/File

Then create a new file called FlagPole

and then define it like so

package com.example.flagcrank

import androidx.compose.foundation.Canvas
import androidx.compose.foundation.layout.*
import androidx.compose.runtime.*
import androidx.compose.ui.Modifier
import androidx.compose.ui.geometry.Offset
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.platform.LocalDensity
import androidx.compose.ui.unit.dp

@Composable
fun FlagPole(modifier: Modifier = Modifier) {
    val poleHeight = 300.dp
    val flagWidth = 90.dp
    val flagHeight = 60.dp

    var progress by remember { mutableStateOf(0.5f) }

    val density = LocalDensity.current
    val flagW = with(density) { flagWidth.toPx() }
    val flagH = with(density) { flagHeight.toPx() }
    val offsetX = with(density) { 14.dp.toPx() }

    Box(
        modifier
            .fillMaxWidth()
            .height(poleHeight)
    ) {
        Canvas(Modifier.fillMaxSize()) {
            val poleX = size.width / 2f
            drawLine(
                Color(0xFFCCCCCC),
                Offset(poleX, size.height),
                Offset(poleX, 0f),
                strokeWidth = 10f
            )

            val topY = 0f
            val bottomMargin = size.height * 0.10f
            val bottomY = size.height - flagH - bottomMargin
            val clampedBottomY = bottomY.coerceAtLeast(topY)
            val y = topY + (clampedBottomY - topY) * (1f - progress)

            drawRect(
                Color(0xFFFF5555),
                topLeft = Offset(poleX + offsetX, y),
                size = androidx.compose.ui.geometry.Size(flagW, flagH)
            )
        }
    }
}

and then use it in our main activity

package com.example.flagcrank

import android.os.Bundle
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color

class MainActivity : ComponentActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        setContent {
            MaterialTheme {
                Surface(
                    modifier = Modifier.fillMaxSize(),
                    color = Color(0xFF101010)
                ) {
                    Box(
                        modifier = Modifier.fillMaxSize(),
                        contentAlignment = Alignment.Center
                    ) {
                        FlagPole()
                    }
                }
            }
        }
    }
}

yielding

Let’s unpack the FlagPole composable. First of all, there’s a new piece of syntax @Composable, then a funciton definition. The FlagPole is the composable itself, a function that renders UI changes based on state. We put @Composable above it to tell the build system it’s a composable function. We didn’t need this before because we didn’t define a composable on its own, but just built a “hello world” one on-the-fly in our main activity.

@Composable
fun FlagPole(modifier: Modifier = Modifier) {

The modifier is a convention in Compose, allowing a composable to pass layout information to its children. It allows you to do things like making a child component half the width of the parent, for instance. It has a lot of uses, like:

Modifier.fillMaxSize() — make the component fill all available space
Modifier.fillMaxWidth() — fill available width only
Modifier.size(100.dp) — fixed size
Modifier.padding(16.dp) — add space around the component
Modifier.background(Color.Red) — set a background color
Modifier.clickable { … } — add a click handler
Modifier.align(…) — position within a parent like Box

In FlagPole.kt we then define a few parameters defining the size of the flag pole, then call this:

var progress by remember { mutableStateOf(0.5f) }

This is a mix of kotlin specific syntax and Compose functionality, that’s honestly more complex than it is enlightening to understand. by is a delegation keyword in kotlin that lets you skip some boilerplate, yadah yadah. Basically, though, the code block above allows you to remember a state, and interface with it via the variable progress.

  val current = progress      // read                                                                 
  progress = 0.7f             // write

This variable will persist across function calls, which is critical because compose uses functions to define UI components. This is how Compose allows you to use stateless things (functions) to modify a UI that requires state.

Anyway, we’ll explore state management in a bit.

Next, we’re drawing the actual flag itself. This takes place in a Canvas that is inside a Box. One quirk of a Canvas is that it draws in raw pixels. As we mentioned previously, different devices can hav drastically different pixel densities, which will change the size of the flag. Thus, before drawing, we want to scale our sizes based on the pixel density of our screen so that the size of the flag is consistent across devices. That’s where this code block comes in

val density = LocalDensity.current
val flagW = with(density) { flagWidth.toPx() }
val flagH = with(density) { flagHeight.toPx() }
val offsetX = with(density) { 14.dp.toPx() }

There’s some Kotlin syntax stuff mixed with Jetpack Compose stuff in this codeblock. To be honest, this article is getting so long that I’m experiencing a delay in my browser when writing it, so I’ll save understanding that as an exercise for the reader.

We then render the flag itself, which is a Box with a Canvas in it. The flag itself is simply a Rect and a Line within that Canvas.

One detail the astute might notice is that this function isn’t actually returning anything. If I create two Boxs, each with their own drawing, I’ll get two flag poles. In this example the second one is weird because I tweaked some numbers around to show both of them. Otherwise they’d draw on top of one another.

Box(
    modifier
        .fillMaxWidth()
        .height(poleHeight)
) {
    Canvas(Modifier.fillMaxSize()) {
        val poleX = size.width / 2f
        drawLine(
            Color(0xFFCCCCCC),
            Offset(poleX, size.height),
            Offset(poleX, 0f),
            strokeWidth = 10f
        )

        val topY = 0f
        val bottomMargin = size.height * 0.10f
        val bottomY = size.height - flagH - bottomMargin
        val clampedBottomY = bottomY.coerceAtLeast(topY)
        val y = topY + (clampedBottomY - topY) * (1f - progress)

        drawRect(
            Color(0xFFFF5555),
            topLeft = Offset(poleX + offsetX, y),
            size = androidx.compose.ui.geometry.Size(flagW, flagH)
        )
    }
}
Box(
    modifier
        .fillMaxWidth()
        .height(poleHeight/2)
) {
    Canvas(Modifier.fillMaxSize()) {
        val poleX = size.width / 2f+100
        drawLine(
            Color(0xFFCCCCCC),
            Offset(poleX, size.height),
            Offset(poleX, 0f),
            strokeWidth = 10f
        )

        val topY = 0f
        val bottomMargin = size.height * 0.10f
        val bottomY = size.height - flagH - bottomMargin
        val clampedBottomY = bottomY.coerceAtLeast(topY)
        val y = topY + (clampedBottomY - topY) * (1f - progress)

        drawRect(
            Color(0xFFFF5555),
            topLeft = Offset(poleX + offsetX, y),
            size = androidx.compose.ui.geometry.Size(flagW, flagH)
        )
    }
}

Composables, don’t “return” drawings, they “emit” them. When you draw a box, it doesn’t trickle up to your top-level entity, it gets drawn right there in the compose, and you can make as many of them as you want.

Practically, what you end up doing is drawing whatever you want, and have that thing exist within the thing that drew it by using the modifier to properly position it visually. State exists at the top, and drawing goes down the component tree until it hits things that actually draw stuff.

The point of the flag crank app is to have a crank raise and lower the flag. To do that, we’re going to need to implement the crank, but also have state information from the crank update the state that affects the flag. We’ll achieve that by creating a third component that contains both the crank and the flag.

Before we do anything, let’s define what “state” even is. There’s two things, the crank and the flag. For the flag, we care about how high up the flag is, called “progress” and for the crank we care about if the cranks rotation. Let’s create a new file called Models.kt that we can use to explicitly define these state variables.

package com.example.flagcrank

data class FlagState(
    val progress: Float = 0f
)

data class CrankState(
    val rotation: Float = 0f
)

Next, let’s modify our FlagPole. Before, it was in control of its own state, the progress of the flag. Now we want it to receive its state from its parent. We do that by simply changing the FlagPole to accept FlagState, and then use it to draw stuff.

package com.example.flagcrank

import androidx.compose.foundation.Canvas
import androidx.compose.foundation.layout.*
import androidx.compose.runtime.*
import androidx.compose.ui.Modifier
import androidx.compose.ui.geometry.Offset
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.platform.LocalDensity
import androidx.compose.ui.unit.dp

@Composable
fun FlagPole(
    flagState: FlagState,
    modifier: Modifier = Modifier
) {
    val poleHeight = 300.dp
    val flagWidth = 90.dp
    val flagHeight = 60.dp

    val density = LocalDensity.current
    val flagW = with(density) { flagWidth.toPx() }
    val flagH = with(density) { flagHeight.toPx() }
    val offsetX = with(density) { 14.dp.toPx() }

    Box(
        modifier
            .fillMaxWidth()
            .height(poleHeight)
    ) {
        Canvas(Modifier.fillMaxSize()) {
            val poleX = size.width / 2f
            drawLine(
                Color(0xFFCCCCCC),
                Offset(poleX, size.height),
                Offset(poleX, 0f),
                strokeWidth = 10f
            )

            val topY = 0f
            val bottomMargin = size.height * 0.10f
            val bottomY = size.height - flagH - bottomMargin
            val clampedBottomY = bottomY.coerceAtLeast(topY)
            val y = topY + (clampedBottomY - topY) * (1f - flagState.progress)

            drawRect(
                Color(0xFFFF5555),
                topLeft = Offset(poleX + offsetX, y),
                size = androidx.compose.ui.geometry.Size(flagW, flagH)
            )
        }
    }
}

We also need to define a crank, which will exist in the file Crank.kt

package com.example.flagcrank

import androidx.compose.foundation.Canvas
import androidx.compose.foundation.gestures.detectDragGestures
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.size
import androidx.compose.runtime.*
import androidx.compose.ui.Modifier
import androidx.compose.ui.geometry.Offset
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.StrokeCap
import androidx.compose.ui.graphics.drawscope.Stroke
import androidx.compose.ui.graphics.drawscope.rotate
import androidx.compose.ui.input.pointer.pointerInput
import androidx.compose.ui.platform.LocalDensity
import androidx.compose.ui.unit.dp
import kotlin.math.PI
import kotlin.math.sin
import kotlin.math.cos

@Composable
fun Crank(
    crankState: CrankState,
    onDrag: (Float) -> Unit,
    onDraggingChanged: (Boolean) -> Unit,
    modifier: Modifier = Modifier
) {
    val crankSize = 200.dp
    val density = LocalDensity.current

    val sizePx = with(density) { crankSize.toPx() }
    val radius = sizePx / 2.3f

    var theta by remember { mutableStateOf(0f) }

    Box(
        modifier
            .size(crankSize)
            .pointerInput(Unit) {
                detectDragGestures(
                    onDragStart = { onDraggingChanged(true) },
                    onDragEnd = { onDraggingChanged(false) },
                    onDragCancel = { onDraggingChanged(false) }
                ) { _, dragAmount ->
                    val dx = dragAmount.x
                    val dy = dragAmount.y

                    val angRad = theta * PI.toFloat() / 180f
                    val tx = -sin(angRad)
                    val ty =  cos(angRad)

                    val dot = dx * tx + dy * ty
                    val delta = dot * 0.4f

                    theta += delta
                    onDrag(delta)
                }
            }
    ) {
        Canvas(Modifier.fillMaxSize()) {
            rotate(crankState.rotation) {
                drawCircle(
                    Color(0xFF666666),
                    radius = radius,
                    style = Stroke(20f, cap = StrokeCap.Round)
                )

                drawLine(
                    Color(0xFFEEEEEE),
                    start = center,
                    end = Offset(center.x + radius, center.y),
                    strokeWidth = 14f
                )

                drawCircle(
                    Color(0xFFFF4444),
                    radius = 26f,
                    center = Offset(center.x + radius, center.y)
                )
            }
        }
    }
}

In a lot of ways it’s similar to the Flag. It accepts a modifier to help with positioning, receives a state, and draws stuff. There are some big differences though.

First of all, the Crank accepts two listeners, along with state and layout information.

@Composable
fun Crank(
    crankState: CrankState,
    onDrag: (Float) -> Unit,
    onDraggingChanged: (Boolean) -> Unit,
    modifier: Modifier = Modifier
) {
  ...

onDrag is a function that will receive a number representing the user dragging their hand on the phone then send information back to the parent, and onDraggingChanged is a function that receives if the user started or stopped dragging and send it back to the parent. These are functions that will be passed by the parent, allowing the Crank to tell the parent what’s happening.

The Crank has kind of a complicated job. It needs to keep track of rotation in a drag event, so that it can update its parent, but doesn’t really “own” that data, because we need that state information to exist such that it can impact the Flag. To make this easier, the Crank keeps track of its own internal angle measurement of rotation by a user during a “Drag Gesture”.

var theta by remember { mutableStateOf(0f) }

User interaction can be complicated, and it can be useful to decouple state for interaction vs state for drawing and logic. In this simple application, you could probably also just have one state variable and use it for both.

Regardless, we’re using androidx.compose.foundation.gestures.detectDragGestures to detect drag gestures by the user,

detectDragGestures(
    onDragStart = { onDraggingChanged(true) },
    onDragEnd = { onDraggingChanged(false) },
    onDragCancel = { onDraggingChanged(false) }
) { _, dragAmount ->
    val dx = dragAmount.x
    val dy = dragAmount.y

    val angRad = theta * PI.toFloat() / 180f
    val tx = -sin(angRad)
    val ty =  cos(angRad)

    val dot = dx * tx + dy * ty
    val delta = dot * 0.4f

    theta += delta
    onDrag(delta)
}

It has a few key callbacks, like a function that triggers when the drag starts, ends, or is cancelled. We can create a lambda function that calls onDraggingChanged for each of these events. Recall that onDraggingChanged is a listener passed by the parent of this component, so we can send those messages up the chain and the parent can update state based on that information. This is necessary in making the flag automatically fall when the user isn’t spinning the crank.

There’s also some trigonometry used to calculate how much we should spin the crank. Once we calculate that, delta, a certain rotation, we pass it to the onDrag listener that then calls it in the parent.

Notice that the actual drawing of the Crank has nothing to do with theta or delta. These are used to do calculations and pass data to the parent. The Crank receives the actual rotation data from the crankState, passed by the parent, to render.

...
    Canvas(Modifier.fillMaxSize()) {
        rotate(crankState.rotation) {
            drawCircle(
                Color(0xFF666666),
                radius = radius,
                style = Stroke(20f, cap = StrokeCap.Round)
            )

            drawLine(
                Color(0xFFEEEEEE),
                start = center,
                end = Offset(center.x + radius, center.y),
                strokeWidth = 14f
            )

            drawCircle(
                Color(0xFFFF4444),
                radius = 26f,
                center = Offset(center.x + radius, center.y)
            )
        }
    }
}

So, in a nutshell, the crank has some internal state to manage calculating information about user interaction across different UI refreshes, but that purpose is ephemeral and thus can be contained in the crank. That’s used to update the actual app state in the crankState, which is then fed back into the Crank to draw it correctly.

In a real app you might experiment with the costs and benefits of different state logic, but this approach results in functionality that works nicely.

So, the Flag and Crank both are designed to be children under something. This is that something, CrankWithFlag.kt

package com.example.flagcrank

import android.os.Build
import android.os.VibrationEffect
import android.os.VibratorManager
import android.widget.Toast
import androidx.compose.foundation.layout.*
import androidx.compose.runtime.*
import androidx.compose.ui.Modifier
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.unit.dp
import kotlinx.coroutines.delay

@Composable
fun CrankWithFlag() {
    val context = LocalContext.current

    var flagState by remember { mutableStateOf(FlagState()) }
    var crankState by remember { mutableStateOf(CrankState()) }
    var isDragging by remember { mutableStateOf(false) }

    var hasNotifiedTop by remember { mutableStateOf(false) }

    val vibrator = remember {
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
            context.getSystemService(VibratorManager::class.java).defaultVibrator
        } else {
            @Suppress(”DEPRECATION”)
            context.getSystemService(android.content.Context.VIBRATOR_SERVICE) as android.os.Vibrator
        }
    }

    LaunchedEffect(Unit) {
        while (true) {
            if (!isDragging) {
                flagState = flagState.copy(
                    progress = (flagState.progress - 0.0025f).coerceIn(0f, 1f)
                )
            }

            if (flagState.progress <= 0.02f) {
                if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
                    vibrator.vibrate(VibrationEffect.createOneShot(80, 180))
                } else {
                    @Suppress(”DEPRECATION”)
                    vibrator.vibrate(80)
                }
            }

            if (flagState.progress >= 1f && !hasNotifiedTop) {
                Toast.makeText(context, “Flag reached the top!”, Toast.LENGTH_SHORT).show()
                hasNotifiedTop = true
            }
            if (flagState.progress < 0.85f && hasNotifiedTop) {
                hasNotifiedTop = false
            }

            delay(16)
        }
    }

    Column(
        modifier = Modifier
            .fillMaxSize()
            .padding(40.dp),
        verticalArrangement = Arrangement.SpaceBetween
    ) {
        FlagPole(flagState)

        Spacer(modifier = Modifier.height(40.dp))

        Row(
            modifier = Modifier.fillMaxWidth(),
            horizontalArrangement = Arrangement.Center
        ) {
            Crank(
                crankState = crankState,
                onDraggingChanged = { isDragging = it },
                onDrag = { delta ->
                    crankState = crankState.copy(rotation = crankState.rotation + delta)
                    if (delta > 0f) {
                        flagState = flagState.copy(
                            progress = (flagState.progress + delta / 360f).coerceIn(0f, 1f)
                        )
                    }
                }
            )
        }
    }
}

This component handles state between the two Composables. It remembers the following things

var flagState by remember { mutableStateOf(FlagState()) }
var crankState by remember { mutableStateOf(CrankState()) }
var isDragging by remember { mutableStateOf(false) }
var hasNotifiedTop by remember { mutableStateOf(false) }

and also has some setup logic to allow the phone to vibrate when the flag reaches the bottom.

val context = LocalContext.current
...
val vibrator = remember {
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
        context.getSystemService(VibratorManager::class.java).defaultVibrator
    } else {
        @Suppress(”DEPRECATION”)
        context.getSystemService(android.content.Context.VIBRATOR_SERVICE) as android.os.Vibrator
    }
}
...
vibrator.vibrate(VibrationEffect.createOneShot(80, 180))
..
if (flagState.progress <= 0.02f) {
  if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
      vibrator.vibrate(VibrationEffect.createOneShot(80, 180))
...

Most of the actual logic of the application exists in LaunchedEffect, which is launched as an asynchronous coroutine in the background.

LaunchedEffect(Unit) {
    while (true) {
        if (!isDragging) {
            flagState = flagState.copy(
                progress = (flagState.progress - 0.0025f).coerceIn(0f, 1f)
            )
        }

        if (flagState.progress <= 0.02f) {
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
                vibrator.vibrate(VibrationEffect.createOneShot(80, 180))
            } else {
                @Suppress(”DEPRECATION”)
                vibrator.vibrate(80)
            }
        }

        if (flagState.progress >= 1f && !hasNotifiedTop) {
            Toast.makeText(context, “Flag reached the top!”, Toast.LENGTH_SHORT).show()
            hasNotifiedTop = true
        }
        if (flagState.progress < 0.85f && hasNotifiedTop) {
            hasNotifiedTop = false
        }

        delay(16)
    }
}

You might recall, way earlier in the article, we talked about lifecycles. This is created early in the lifecycle and persists, due to the while (true) loop, during the duration of the components existence. AKA, while the app is running.

You can see that the logic is pretty straight forward. While the user isn’t cranking, the progress is updated. If the flag hits the top, a notification says the top has been reached. There’s some threshold to that so the app doesn’t spam notifications when you reach the top.

Some more of the logic exists within the callbacks supplied to the Flag and Crank

Column(
    modifier = Modifier
        .fillMaxSize()
        .padding(40.dp),
    verticalArrangement = Arrangement.SpaceBetween
) {
    FlagPole(flagState)

    Spacer(modifier = Modifier.height(40.dp))

    Row(
        modifier = Modifier.fillMaxWidth(),
        horizontalArrangement = Arrangement.Center
    ) {
        Crank(
            crankState = crankState,
            onDraggingChanged = { isDragging = it },
            onDrag = { delta ->
                crankState = crankState.copy(rotation = crankState.rotation + delta)
                if (delta > 0f) {
                    flagState = flagState.copy(
                        progress = (flagState.progress + delta / 360f).coerceIn(0f, 1f)
                    )
                }
            }
        )
    }
}

Like before, the FlagPole and Crank exist within another composable for organization purposes, a Column for instance. The FlagPole is simple because it’s purely reactive, but the Crank has some complications due to callbacks being defined.

The Crank also has its own state. onDraggingChanged is a lambda function that has some clever and obscure Kotlin syntax in it. You could read this as

onDraggingChanged = { value -> isDragging = value }

if you wanted. Basically, it’s a function that sets isDragging to whatever the input value is. Because this is passed to the Crank, it allows the Crank to define what the value of isDragging is, which is contained in the higher order CrankWithFlag composable.

The onDrag callback is a bit more complicated, but does a very similar thing.

onDrag = { delta ->
    crankState = crankState.copy(rotation = crankState.rotation + delta)
    if (delta > 0f) {
        flagState = flagState.copy(
            progress = (flagState.progress + delta / 360f).coerceIn(0f, 1f)
        )
    }
}

When the onDrag callback is called within the Crank, it overwrites the crankState and the flagState (if the crank is spinning in the right direction). These state variables are then propegated to each component.

You might notice that we’re overwriting the state variables, rather than modifying them. To make apps more efficient, Jetpack Compose only renders components whos state changes. It compares old vs new values to do that. Thus, state variables are immutable and must be overwritten for Compose to do It’s job.

We can change our MainActivity to use our new component

package com.example.flagcrank

import android.os.Bundle
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color

class MainActivity : ComponentActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        setContent {
            MaterialTheme {
                Surface(
                    modifier = Modifier.fillMaxSize(),
                    color = Color(0xFF101010)
                ) {
                    Box(
                        modifier = Modifier.fillMaxSize(),
                        contentAlignment = Alignment.Center
                    ) {
                        CrankWithFlag()
                    }
                }
            }
        }
    }
}

and, with that, we have a Jetpack Compose App

The flag goes up and down when you spin the crank. There’s a notification that pops up when you get to the top. Take my word for it.

`Conclusion`

And now you know why I don’t often teach front end development. It’s a lot of information without a ton of core logic underneath. Necessarily, the domain is full of arbitrary rules and long lists of pre-defined tools. I’m more of a conceptual learner, so front end development has never really scratched an itch for me.

However, as I grow as a developer, manager, entrepreneur, and technical director, I find that front end presentation is incredibly important. You can make the most sophisticated thing under the sun, but if it doesn’t grab peoples attention or present them with an experience they understand, the value won’t be understood.

All developers, including data scientists, need to have some grasp of front end development. In that vein, I hope this article has proved useful. Do I really have no typos, or is this article simply too long for my proofreading system… Only time will tell.

Agile Project Management — Intuitively and Exhaustively Explained Daniel Warfield — Mon, 09 Mar 2026 14:03:07 GMT “Push” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained. If you’ve worked on complicated things, you may be aware of two fundamental truths: Some problems are too big for one person to solve and require a team Teams of people are sometimes inefficient Thus, teams can be a blessing and a curse. A team of people can move mountains, or a team can operate so inefficiently that it’s less productive than a single person working by themselves. If you’ve worked in software development for any length of time, you might be familiar with one of these problems: Developers not understanding what they should be working on, why it’s necessary, or how it should work Developers feel consistently stressed about deadlines, which impacts both their productivity and mental health Managers do not know how long tasks will take to finish A general feeling that the product is fragile, inconsistent, or quirky, and is getting worse rather than better over time Frustratingly slow progress across the board, with product managers blaming developers, and developers blaming product managers Problems like these mark the difference between a team that moves mountains and a team that’s stuck in a rut. A lot of people point the finger when problems arise in a team, with developers blaming managers and managers blaming developers. In my opinion, all of these are indicative of inadequate management. Management, not skill, money, clients, or time, is the biggest weakness in most software development projects. There is more than one way to be a successful project manager, and different projects sometimes require different styles, so it’s difficult to create a single unified framework for high-quality software project management. Still, through decades of trying different approaches, there are certain themes in software management that appear to be transcendent. The most popular style of project management falls under the name of “Agile”, with other words like “Scrum” and “Kanban” following closely behind. We’ll be discussing these approaches, as well as the approaches that preceded them, throughout this article. I often find that management is discussed from a very high level, making it hard to apply the lessons learned to actual teams. To alleviate that, this article will also provide some practical examples and tools that I’ve learned in my career thus far, to allow you to directly apply the theories discussed in this article to real-world problems. My objective is that, by the end of the article, you understand the point of project management, some of the challenges associated with the discipline, and how to do project management right, regardless of the project or team size. The focus of this article will be on software project management, but I believe the core ideas will be relevant to most disciplines that require multiple people to reason collaboratively over difficult problems. Subscribe To Intuitively and Exhaustively Explained. Share this article with friends and colleagues Share Who is this useful for? Anyone who works within teams to solve problems. How advanced is this post? This article is accessible to beginners. Prerequisites: None. A Request: IAEE articles are large in scope, and take a significant amount of time to produce. Despite this effort, IAEE articles are available for free. If you enjoy this style of content, and want to support a better and more educated internet, consider supporting Intuitively and Exhaustively Explained. If you are a supporting member, thank you. You keep the lights on. The Philosophy of Management Before we discuss agile management, I want to discuss some misconceptions around project management in general. It’s a divisive topic, with a tremendous potential impact on a team, and vague ideas of what being a good manager looks like. I find this combination of importance and vagueness lends an almost religious quality to how project management is taught. There are certain creeds you must abide by, certain rules you just don’t break, and certain sins you don’t transgress lest the project management gods smite you. Often, a fledgling project manager stumbles on a particular approach and treats it like gospel. This can lead to strong convictions without having a foundation of ideas to support them. Before we discuss Agile, I want to share the soul of my philosophy around project management. This philosophy is the result of learning from several great project managers, observing the failings of bad ones, and learning from my own successes and failures. Philosophy: The Goal is to Not Have a Team In my opinion, the first step to being a stellar manager is to recognize the reality of the situation; one perfect person is always better than a team. Building teams is a necessity, not a goal. This is a controversial statement, so I want to tackle it upfront. I have seen people mythologize teams. Some people cite differences in skill set, differences in experience, or simply a cooperative human spirit making teams better than the sum of their parts. I think this is idealistic and inaccurate. I want to convince you by way of a thought experiment. If you were building an app, for instance, you might require a front-end developer, a backend developer, a dev-ops engineer, a designer, and maybe a data scientist if you’re building fancy AI functionality. Any time a major change needs to happen, each of these people needs to sit down and talk to one another. No one person understands the product in its entirety, so each of them needs to advocate for their respective parts of the product. Coming to a conclusion can take hours of time, implementing a solution requires aligning people’s knowledge, and unknowns are likely to be encountered which the team will inevitably bump into. Imagine, instead, if there was one person working on the product. A super developer who is a world-class expert in front-end, backend, dev-ops, and data science. They had dedicated the last 40 years of their life to mastering each of these disciplines. They have several PHDs. They’ve built companies from scratch in a month that are valued at over $100M. They also have an artistic flair and can create beautiful app designs that have won international awards. They also know all of your intended users personally and understand how they think. Oh, and by the way, this super developer also speaks four of the most common languages and has lived around the globe. Did I mention they’re an acclaimed marketer too? If you ask a person like this to solve a problem, it’s possible they can do in minutes what it would take a team of developers days or weeks to do. The difference in productivity between these two situations is insanely large. When possible, you should always have a super-genius working perfectly in a codebase they completely understand. Better yet, have them build the whole company. That is the ideal, and is better than a team of developers trying to inefficiently discuss what an omnipotent mind can reason over virtually instantaneously. Trivially, if you think a team is so amazing, then imagine the capacity of that team within one brain (different personalities and everything, like they were plugged into the matrix together). It would be the same but faster. In theory, one brain with the capacity of a team will always be better than the same capacity spread out across multiple brains. The issue isn’t what’s better, but what’s realistic. Frankly, the super genius doesn’t exist. If they do exist, you can’t afford them. I’ve been writing IAEE for a while, with the goal of becoming more like the aforementioned super genius, a north star I know I’ll never hit despite a substantial amount of effort. Still, as I become more competent, the attractiveness of some job opportunities becomes less and less compelling. Why would I work for your company when I could build your company from scratch in a weekend? Will you give me 80% equity even though I’m generating 90% of the value to the company? Probably not. The majority of my early career consisted of a complete technical ownership of very young and small companies, where I would work directly with non-techncial business owners to build the bedrock of the company. As I gained more experienced, I found that these fledgling business owners don’t have the requisite knowledge to value or the requisite revenue to compensate my skills appropriately, and thus those jobs simply interest me less. When people discuss the rift between junior and senior developers and their job opportunities, this is why. brand-new developers are often a liability; no one wants to hire them because you need so many of them to solve a problem that the fragmentation makes the team incredibly inefficient. Senior developers are incredible assets, but they’re rare because most people don’t invest the time necessary to become one. Senior developers have a ton of selection, and few companies can afford the good ones. The last two paragraphs concern small companies working on small problems. Big problems in big companies, or cutting-edge startups working on groundbreaking things, are too complicated for any one person to wrap their head around, no matter how capable they are. The Manhattan Project, for instance, required a team of some of the most ingenious minds in the world. Each of them dedicated their life to gaining the knowledge they had. Creating a single human with the capability of all those minds would defy the realities of human intelligence and mortality. This catch 22 is why teams of developers exist; the utopian super-genius does not exist or is not available, so we have to make do with the manpower we have. We want to keep our teams small, and packed with the smartest people we can, but there will always be a breakdown in any large project that requires a team. Project management, thus, can be conceptualized as taking the super-genius’ mind and fragmenting it across different developers. The objective of project management is to minimize the derogatory impact of that division. This division of the super-genius mind, I think, is an incredibly useful mental model of team building and management. Sometimes you need to bring people on to complete the mind. Sometimes you can bring too many people on, and overfragment the mind so that it’s ineffective. Once you’ve built the team, it’s also a useful model for how to manage the team once it’s been built. Inefficiency happens when different fragments of the mind need to work together. Neurons in one developer can interact at will, but to interact with the neurons in another developer’s mind, they need to open Slack, book calls, and generally move at a snail’s pace relative to the potential of individual internal thought. Thus, the name of the game of good management is to break up a problem so it can be best distributed throughout the fragmented mind of a team. A project needs to be broken up into tasks that individual developers can work on, and done so in such a way that the pieces come back together to make one cohesive product. This, in a nutshell, is the task of software project management. Philosophy: People Work Alone on Tasks Teams work on projects, but individuals work on tasks. A task, in essence, is a portion of the project that one person can do within a reasonable timeframe. There is, in my opinion, no alternative. It’s healthy and productive for a team to have high-level conversations that align their vision of a product but, at the end of the day, things get done by one person doing them. Teams are collaborative in that they align and parallelize individual labor. The reality of collaboration; the parallelization and synchronization of individual labor I do not agree with some weird overcrowded nightmare where multiple people are working on the same thing, which is popular with some management styles. Images like this make me uncomfortable; the result of stress from working under managers who prioritize cooperation for its own sake. I find it funny that some people don’t understand that too much collaboration can be a bad thing, where images like this clearly show a situation where too much collaboration is drastically harming progress. Some managers, especially those who have been managers for their entire careers and have never built anything themselves, fetishize cooperation. Any artist, engineer, developer, health care professional, line cook, or otherwise productive member of society understands the most fundamental rule of creation. true creation is a solitary act — Charles Bukowski Thus, the idea is to map tasks throughout the fragmented genius which is a team. The name of the game is breaking up a project into bite-sized tasks that an individual can work on. If a task is too big, and requires multiple people, it needs to be divided. This doesn’t always happen on the management level. It is common to assign two developers with different expertise to a single task, and they might work together to solve that task. In-reality, though, they break the task into sub-tasks then individually work on those subtasks. At some point, whether it’s management or emergent within a teams dynamics, people work on individual things alone. Knowing how granularly to dice up a task is tricky, and requires experience to get right. We’ll talk about tricks to deal with that later. Philosophy: You Should Trust Your Team Implicitly I think one of the biggest failings of managers is a lack of trust. It’s the main driver behind micromanagement and manipulation. When talking with managers, either within a larger company or to the owners of companies themselves, I’ve encountered some of the following statements: “How do I know if people are actually working?” “I don’t understand why this is taking so long. They’re either dumb or lazy” These are existential problems that can be paralyzing for managers. The idea that you’re doling out high-quality plans, and they’re falling flat because of lazy or deceptive employees, can be draining. As a result, many managers implement complex tracking systems so they can audit productivity on a granular level. To assuage these concerns, and save you a bunch of unnecessary effort, I’d like to share a universal truth. Teams are inefficient because of management. There is no problem in a team that proper management can’t solve. Or, in other words, it’s always your fault. Because it’s always your fault, you don’t have to worry about trust. Workers are trustworthy when supplied with sufficient resources, are properly encouraged, sufficiently supported, and have a clear, compelling trajectory. People want to work on good things and be recognized for it. A team is not a group of people who work together. A team is a group of people who trust each other. — Simon Sinek If you get some unicorn sociopath that’s taking you for a ride, you can fire them. If someone doesn’t have the requisite skills to handle a job, you may need to fire or reposition them. Identifying such people is easy, and such instances are rare; thus, these issues are of marginal concern. As a manager, all major problems that actually happen in real life are your fault. The sooner you recognize that, the sooner you can be a good manager. Once you develop a muscle of trust for your team, magical things happen. Individuals can work efficiently on problems, they feel they have the individual capacity to solve problems as they come up, they respect their environment and want to support it, and they feel like contributors with agency. If you don’t trust the people in a team to do their job, they can’t work as individuals, which means they can’t get anything done. Robbing individuals of their agency saps a team from it’s most valuable asset. You have to trust your team, there’s no other option. I have never seen a genius idea coming from a team. Genius ideas always dawn on one person alone. To develop it, yes, a team is important. But to concoct a miraculous idea, it’s you — alone — Philippe Starck There are plenty more philosophical ideas that are important to robust project management, but I think these cover the fundamentals: Lean teams are important because they minimally fragment the genius mind Cooperation in solving a problem is done with the alignment and parallelization of individual labor Trust is vital in a team being able to operate effectively I believe, with that, we can begin discussing some of the high-level approaches to project management. We’ll start with waterfall, the predecessor to agile. Waterfall Management Waterfall management is possibly the most obvious and intuitive form of project management. It was the predominant management style for software projects in the 70s-90s. Essentially, the idea of waterfall management is to break up a problem into a series of steps, then execute those steps. If you’re having a hard time cleaning your room, you might decide to first put your clothes away, then make your bed, then vacuum the floor. You’ve taken the project of “clean your room” and turned it into a “waterfall” of steps put clothes away -> make bed -> vaccum floor Waterfall can be a compelling approach, especially in the beginning of a project. However, there are some demons bound to the approach that tend to rear their heads as the project progresses. I find waterfall has two distinct phases: the calm and the storm, which I’d like to describe by way of example. The Calm of Waterfall Let’s build a new app. We have a team ready to go, and the boss knows what they want. To be successful, we need to make their vision a reality. It’s a budget tracking app. Of course, like any app, it will go through several rounds of revisions, but the boss knows what’s going to make this a hit up front. An ability to connect to one or more banks Extraction of transactions from those banks, and categorize them into key transaction types: bills, automotive, food, and entertainment Create a dashboard that shows how each of these categories changes over time. An ability to compare your spending habits with other people in your region, anonymously, allowing you to honestly compare how well you’re budgeting relative to what’s common Tools allowing you to modify your purchasing habits based on your transaction types. A user might spend a lot of money on food, but others spend a lot less. The user can get recommendations based on what other people do in order to make better purchasing habits. The boss has been an accountant and financial advisor for 30 years. This is what they’re paid to do, and they know the business well. The app, essentially, is just automating what they already make money doing. The team sits down, and figures out all of the steps necessary to make this thing work. Here’s all of the steps they decide on: The first step, they figure, is fleshing out the requirements a bit more. They’ll need to make some designs of different screens, and identify if there are any security requirements needed from a regulatory perspective. The second step is to plan out the architecture that’s required to satisfy that vision. Folks start thinking about resources, database schemas, communication, environments, things like that. The third step is to figure out how to get transactions out of banks and into our database reliably. That means selecting integration methods and building the pipeline that pulls and stores transaction data. The fourth step is to plan out and build the algorithms required to categorize transactions into useful groups like bills, food, or entertainment, so the data becomes meaningful. The fifth step is to build the analytics layer that aggregates those categorized transactions over time so we can see trends and patterns in spending. The sixth step is to build the dashboard that visualizes those trends so users can easily understand how their spending changes over time. The seventh step is to implement the anonymous regional comparison feature, aggregating data so users can compare their spending habits to others nearby without exposing individual information. The eighth step is to build recommendation tools that help users adjust their spending habits based on what the data shows. The ninth step is testing and validation — making sure the system works end-to-end, is secure, and performs reliably. The final step is deployment and release, making the application available to users and monitoring it to ensure everything runs smoothly. Sounds like a plan. The team then estimates how long each of these steps will take, and lays them out relative to each other. Weeks → 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Requirements & Design ██████ Architecture Planning ██████ Bank Integration █████████ Transaction Categorization ██████ Analytics Development ██████ Dashboard UI ██████ Regional Comparison ██████ Recommendation Tools ██████ Testing & Validation ██████ Deployment ██ The graph you’re seeing is what’s called a “Gantt chart”, and is commonly used to lay out a waterfall style of project management (technically, this might be considered a “sashimi” style of timeline, rather than a formal “waterfall”. If we’re being nerds about project management, sashimi has overlap in the timeline, and waterfall does not. For our purposes, the distinction irrelevent. It gives clear steps, clear timelines, and a clear end goal. Now we just need to make it happen. The Storm of Waterfall The Requirements & Design phase went faster than expected, we got done in the first week, great. The architecture planning was done in 2 weeks, fantastic; we’re ahead of schedule. At week 4 we were already working on bank integration and transaction categorization. Unfortunately, this is where issues start creeping up in the project. Transaction categorization is proving to be challenging. All we get from the banks is an abbreviation of the transaction type. For large organizations, we can infer that WM SUPERCENTER is Walmart, and thus groceries, but there are many transactions that are not easy to categorize. On top of that, big institutions like Walmart tend to use WM, WM SORE, and other codes to refer to the same organization. When building dashboards attempting to organize transactions, you’re finding it difficult to make a compelling dashboard organizing information when businesses like walmart are several different categories. You’ve been working on this for 6 weeks, with very slow progress. You decide to get ahead of the problem. You assume the issue will get solved, so you start building dashboards to be ready when the time comes. You build the dashboard, regional comparison, and recommendation tools pages, and populate them with fake data to start with. You show these off to prospective customers, and the results are lukewarm. People can see the value in an abstract way, but they can’t quite imagine the utility because they can’t see data that’s actually relevant to them. You need to get transaction categorization figured out. It’s now week 16. You’ve given transaction categorization another two weeks of development, and progress has slowed. Any room for progress seems to have already been made, and there are some simply insoluble issues that undermine regional comparison and recommendation. Without well-classified categories of transaction types, each of these is essentially useless. They work great if categorization is done perfectly, but it simply isn’t. It’s now week 18. The boss made promises to investors about the timeline, and things are starting to heat up. It’s frustrating because the app feels like it’s almost ready, but also feels like it’s no more useful to the customer than when it was on the drawing board. The team decides to hack a solution; ask the user where the transaction came from. After 5 weeks of playing with this idea, it seems like it really just pushes the puck on the problem rather than solving it. Users might enter the name of where they got the thing from if they used the app, but the app isn’t really meant to be checked on a daily basis. Frequent users might check the app once a month to confirm their budget. Essentially, no one is willing to put in the effort to manually record this information. On top of that, there are issues categorizing what users entered. They might misspell the name of the company they purchased from, or refer to it by a nickname that some other user doesn’t use. This makes coming up with comparisons and recommendations challenging. The team will keep trudging on for the better part of a year in an effort to execute their 19-week master plan. They reimagine the app a few times, build a new waterfall, and fail each time. It’s a painful process that consumes a criminal amount of life and human spirit, but we can spare ourselves by stopping here. Let’s discuss why everything went south. Why Waterfall Fails Waterfall is not a useless strategy. When used sparingly and correctly, it can be rather useful. That said, it isn’t a good tool for managing the week-by-week progress of a project, and often leads to bitter failure. It’s an incredibly naive approach to project management, and helps to preserve that naivety far into the duration of the project. The boss was naive about how easily software could automate his job, and the technical team was naive about how easily categorization could be done. Hints of problems arose around 4–6 weeks in, but instead of changing tact the team decided to get ahead on the waterfall by implementing the front end of the app, then meeting in the middle once entity extraction was solved. Because of the exhaustive planning of waterfall, there’s a certain momentum to the plan that makes substantial changes practically difficult to even consider, let alone adopt. At around 16 weeks, they had an app that was poorly performing enough to need effort, and was also compelling enough to justify continually working on the project. This is the worst-case scenario. It is better to have a project that fails, and you can fold with confidence, than a project that “kind of works”. Projects that “kind of work” can compel a team to spend years of their lives mulling over a fundamentally failed idea. The team, and especially the boss, doesn’t want to admit this because they have already invested months of time into getting to where they are now. Waterfall fundamentally treats the process of project management like a “fire and forget” missile. You punch in a trajectory, hit launch, and the project flies to its destination. In reality, a software project is like a maze, with various roadblocks and dead ends that you’ll only find once you start walking through the maze. You can have a great definition of where you want to end up, but the process to get there is going to require you to consider some feedback and make adjustments. After 20 years of waterfall being the predominant approach to software management, its flaws were being generally recognized as serious and consistent. It was clear that a new approach to software management was required. That new approach was Agile. The Birth of Agile Agile emerged organically across many organizations. If it had a birthday, that was in 2001 with the creation of the “Agile Manifesto”. Manifesto for Agile Software Development The Agile Manifesto has a Declaration of Independence vibe, like how revolutionaries freed themselves from tyranny; these developers are freeing themselves from waterfall, except these guys are wearing khakis. the 12 principles of agile. Source Most of the core tenets of agile are in direct antithesis to waterfall. Responding to change over following a plan Deliver software with a preference of a shorter timescale At regular intervals, the team reflects on how to become more effective I was four years old at the time, so I can only imagine the emotions that caused a group of independent technical professionals to create this manifesto, then millions of others to adopt it. In my experience, waterfall is a frustratingly attractive management approach for non-technical people, who are often the people who call the shots in larger companies. CEOs do not respond well to ideas like “sometimes the best plan is no plan”, and generally don’t understand the significant sacrifices they make in exchange for the false security of a waterfall-style plan. To me, the Agile Manifesto seems like an attempt to pool credibility into a single idea so it can be better sold to non-technical managers who, otherwise, through fear of the unknown, attempt to squeeze the square peg of a software project into a round hole of end-to-end certainty. Thus, the Agile Manifesto isn’t really a guide to how a project should be run, but a core philosophy legitimized by hundreds of signatures from important technical professionals. some signatures to the agile manifesto. Source Its core tenets are continual collaboration and adjustment of the vision of the project, with a focus on rapid delivery so that unknowns can be discovered and addressed as quickly as possible. There are numerous ways to actually “do” agile project management. We’ll be discussing two: Kanban and Scrum. We’ll start with Kanban, the simpler of the two approaches. Kanban, an Agile Approach to Project Management Kanban was originally developed to help make manufacturing processes more efficient for Toyota in the 1940s. Since then, it’s transcended its roots and become a common way to manage agile and iterative teams. Part of the reason Kanban is popular is because it’s so simple. It consists of only two fundamental parts; tickets, and a board to organize those tickets. A particularly chaotic kanban board. A ticket is a unit of work that your team can do in a reasonable amount of time. In most management styles, a ticket should be accomplished by your team in a few days, or two weeks tops. The kanban board organizes these tickets into columns, which represent the state of the work. The most quintessential kanban board is “TODO”, “In Progress”, and “Done” A kanban board with TODO, In Progress, and Done But there are also other columns you can choose to include. You can add one for a “backlog”, where you can plan tickets before assigning them to developers If you find there are bugs and issues in your product, and you want to tighten up testing, you can create columns specifically for more robust QA practices Kanban boards can be very complicated. I would caution you, though, to keep them simple. A simple kanban board is a powerful thing, and you can always add columns as the necessity arises. Too many columns can cause confusion and bloat a project, inhibiting performance. Once you have a board, you need to put tickets on it, which define what people will be working on. The creation of tickets can be a complicated thing, and a perfect ticket can depend on the team and context. For small teams, this is my playbook for creating high-quality tickets: Kanban Story Tickets Recall the fundamental goal of project management: to distribute effort throughout the fragmented genius of a team to create tasks of work. These tasks then need to come back together to make a single and cohesive product. Tickets, the subject of this section, define what people are working on. Thus, they are the direct tool a manager uses to distribute work and make sure that work aligns well with the final product. Unfortunately, tickets are sometimes defined in a way that fails to move the product forward. Tickets stop making sense, developers finish a task only to find that another ticket essentially undoes their work, or as the developer finishes the ticket they find the design of the application has completely changed, rendering their efforts useless. When tickets are poorly defined and managed, there’s a lot of work, but not a lot of productivity. When I was getting started, the thing that got me out of this rut was story tickets. I learned the trick from a fantastic freelance project manager by the name of Mike Shourky. While much of what this article contains is from my own experience, I learned many of the ticketing tools in this article, as well as the soul of my convictions, from Mike. If you need a project manager, I highly recommend checking him out. Generally speaking, all work that developers do should be for someone. That might be a user, a tester, or the developers themselves. Any time a developer sits down and makes something, they should do it for the benefit of someone. The idea of a user story is to make it easy for multiple developers to understand the “so what” of a particular ticket, so that the developers can work together on the problem. Imagine a ticket on a kanban board that only has the following text: The app is hard to use, it should be easier This ticket sucks. What does this mean? Should a designer take this ticket up, or should a developer? What does a successful execution of this ticket look like? When is it done? This style of ticket causes frustration in teams and leads to a whole lot of effort, without a lot of productivity. Now imagine another ticket about the same problem. The app is hard to use, it should be easier. To do that, make the button on the third screen slightly larger, make the text 2pt larger across the board except for titles, add a confirmation page that says “are you sure you want to submit” on the submission page, and change the high contrast version of the text from a dark to a light blue. Better, but still not good. This leans too far in the other direction. It micromanages the specific things you want done, and robs your team of agency. The success of this ticket is to do the laundry list of things specified, not to actually make the app easier to use. If these various changes fail to achieve the desired effect, the developer will be put in an awkward position of implementing something counterproductive. Also, it’s ignorant of the fact that you, the manager, are talking with someone who knows more about front-end development than you do. It doesn’t allow them to use their knowledge and expertise to do better than what you had imagined. With this approach to ticket creation, everything the team does is limited by the ignorance of the manager, which defeats the entire purpose of a team. In my experience, for a small team, the best type of ticket is this: An increasing number of senior citizens are using our app. Based on conversations with those users, their main issues appear to be that they don’t understand navigation, have a hard time reading the screen, and often mis-click on buttons they didn’t mean to. Through more thorough research, it appears that their main issue with navigation is that they have a hard time with the tray, which they say “goes away too quickly” when they click things. It seems, as a senior citizen, it would be better if: - Text was bigger and easier to read - Buttons were better spaced to avoid misclicks - The collapsing of the nav tray was rethought. This ticket is head and shoulders better than the previous two. It allows a developer to understand the reason they’re solving the problem, which is critical in giving them the tools to do their job effectively with agency. Immediately, the team can work together to imagine subtasks: A front-end developer can work on the logic for tray collapse A designer can work on redesigning the UI to be better. Perhaps the same front-end developer can take the designer’s output and implement it They can then work through the ticket, discussing problems as they arise, with the goal of the end user in mind. I won’t be so bold as to say this ensures the end result will align with what the customer wants, but with a decent team, this will likely get knocked out of the park. This idea of using the customer’s perspective first is what’s generally referred to as a “user story”. Some project managers are even deadset on always creating tickets like it’s an actual story, sometimes using the phrase “as a ____” before every ticket. As a senior citizen, using the app is challenging. Buttons are close together and clicking them is difficult. The navbar collapses unexpectedly when I don’t realize my fingers brush the screen, and the text is often too small to read. Some even go so far as to imagine a prospective user Gerold is a senior citizen with mild dementia and motor control issues, and he needs to use our app. He doesn’t want to, but needs to. He finds using the app challenging; buttons are close together and clicking them is difficult, the navbar collapses unexpectedly when I don’t realize my fingers brush the screen, and the text is often too small to read. Personally, I find these to put the ball a bit too far in the developer’s court. It’s the developer’s job to implement solutions; it’s the project/product manager’s job to understand customers. I do think user stories can go a bit too far, and require developers to imagine specific people that they’ve never seen. Regardless, though, I think too much story is better than not enough. The correct balance depends on the team and the product. Often, the user story is sufficient, but there are some other things that can make your tickets even better. Acceptance and Testing Criteria You need to give your team enough freedom to act autonomously, but not so much slack that they hang themselves with it. The goal of a good ticket is to be a north star for developers. It’s important that it leaves room to do their job but without causing substantial confusion. Let’s take the issue of “The collapsing of the nav tray needs to be rethought”. This can be interpreted in a litany of different ways. Imagine if, through your conversations with seniors, you’ve found that they accidentally swipe, tap, and misclick in the navbar. You can frame this in a user story if you want, but including all these specifics can lead to a long story and misinterpretation by the developer. You might not understand this, because you’ve read to this point in the article, but there are plenty of people who simply hate reading more than a paragraph or two. Thus, for specifics, it’s sometimes a good idea to include them in “acceptance criteria”, rather than the story. User Story: An increasing number of senior citizens are using our app. Based on conversations with those users, their main issues appear to be that they don’t understand navigation, have a hard time reading the screen, and often mis-click on buttons they didn’t mean to. Through more thorough research, it appears that their main issue with navigation is that they have a hard time with the tray, which they say “goes away too quickly” when they click things. It seems, as a senior citizen, it would be better if: - Text was bigger and easier to read - Buttons were better spaced to avoid misclicks - The collapsing of the nav tray was rethought. ============================= Acceptance Criteria: AC1: Text is bigger AC2: Button design avoids misclick AC3: Nav tray less prone to frustration — AC3.1: clicking out of the navbar does not collapse the navbar — AC3.2: swiping the navbar to the right does not collapse the navbar — AC3.3: There is an explicit way to close the navbar, like a big “x” There’s a lot of text here, and developers who are used to inefficient communication might roll their eyes, but notice that most of the information is repeated. If you’re interested in quickly understanding specific tasks, you can reference particular acceptance criteria. If you want to understand the “why” of what’s being asked, you can reference the user story. When you think about how much is being communicated and the amount of work the team will be doing based on this ticket, it’s actually remarkably brief. Often this is enough, but different teams have different QA policies and capabilities. Some developers test everything they do to the 9s, and other developers are overemployed, working at a contracting agency, and are enticed to push tickets as fast as possible. Depending on these realities, it might also be useful to include “testing criteria”, like this: User Story: An increasing number of senior citizens are using our app. Based on conversations with those users, their main issues appear to be that they don’t understand navigation, have a hard time reading the screen, and often mis-click on buttons they didn’t mean to. Through more thorough research, it appears that their main issue with navigation is that they have a hard time with the tray, which they say “goes away too quickly” when they click things. It seems, as a senior citizen, it would be better if: - Text was bigger and easier to read - Buttons were better spaced to avoid misclicks - The collapsing of the nav tray was rethought. ============================= Acceptance Criteria: AC1: Text is bigger AC2: Buttons avoid misclick AC3: Nav tray less prone to frustration — AC3.1: clicking out of the navbar does not collapse the navbar — AC3.2: swiping the navbar to the right does not collapse the navbar — AC3.3: There is an explicit way to close the navbar, like a big “x” ============================= Testing Criteria: TC3.1: When the navbar is opened, and the empty area outside of the navbar is clicked, the navbar does not close. TC3.2: On a navbar, swiping up, down, left, and right does not close the navbar, when swiping on any portion of the screen. TC3.3: The navbar “x” is sufficiently out of the way so it won’t accidentally be clicked, and is big enough to be easily seen. Generally, testing criteria are assigned to specific acceptance criteria. Testing criteria can serve two very important purposes: It allows you to be crystal clear about non-negotiable aspects of the ticket, perhaps going into edge cases that the developer should consider. It allows you to keep the team honest in environments where QA is lacking. Previously, I said you should trust your team implicitly, meaning that should be your default mode. If the quality of their work is poor, you might want to work with them to build more robust QA practices and help them along with high-quality testing criteria. If, even after creating high-quality and thorough tickets like the one highlighted above, you still get buggy software, then you know you need a new team. One of the great things about taking responsibility for management is that, if you do a great job, it’s clear to see where inefficiencies or dishonesties lie in terms of effort. Blame can be uncomfortable, but it’s actually incredibly healthy. As a matter of course, I avoid making technical recommendations to my team. Sometimes I share thoughts on what I might do if I were in their shoes, but only when I deem that absolutely necessary, and I always qualify that it’s a thought, not a suggestion. It’s my team’s job, and my team’s fault, if there’s a technical failure. If I tell them how to do this or that, it’s unclear whose fault it was: mine for a bad idea, or theirs for poor implementation. By not governing granular technical decisions, it preserves technical responsibility and ownership where it belongs: with the developers. Likewise, it’s up to the managers at my company to come up with what the product should do. If the product fails due to misalignment with user need, then it’s not the technical team’s fault. The objective of clear blame is not to point the finger and get nasty when things go south, but rather to know the source of issues so they can be addressed effectively, and to make sure ownership of responsibility doesn’t change throughout the course of a project. In a healthy team, blame is a very good thing and helps to keep people happy and civil, even in the most trying situations. Both acceptance and testing criteria can help with that. Here’s my template for feature tickets, should you find it useful. Thanks to Mike for teaching me this template: Story: Acceptance Criteria: Testing Criteria: Design Assets: Kanban Bug Tickets A common point of frustration for many teams is that non-technical people find issues, but developers can’t solve them. This leads to the same bugs popping up over and over again. Non-technical teammates are frustrated that the bugs they find aren’t being resolved, and technical staff is frustrated because they don’t understand the bugs being communicated to them. In my experience, the issue is that non-technical people have a hard time understanding what information is necessary to clearly communicate bugs so that devs can address them. The solution to this problem is a bug ticket. While feature tickets contain stories, bug tickets contain the necessary information to clearly and consistently communicate a bug to a developer. Here’s my template, again, thanks to Mike. Observed Behavior: Expected Behavior: Environment: Steps to Reproduce: Assets: Tests: i.e.Test 1) Do this, then this, then this Should see this, without thatTest 2) Do this, then this, then this Should see this …> Some people are too lazy to fill out all this information, and instead expect developers to naturally know what happened when the bug occurred. Not all of the information is strictly required, but all of it is necessary. The more information, the better. A well-formed bug ticket can be the difference between a developer taking hours or minutes to solve a problem. If you find an issue, it’s your job to thoroughly document it. Here’s an example of a filled-out bug ticket for our fictitious app. Observed Behavior: Pressing the new “X” button on the navbar, the navbar does not close. After further testing, it appears this happens consistently on the home page. Did not see it happening when the app is on another page, but I did notice there was a delay in closing when navigating from the home page to another page before pressing “X” Expected Behavior: The “X” button closes the nav bar on all pages instantly, regardless of navigation. Environment: The es3952 release of the app on a Google Pixel 9a. Issue found 12:33 PM CT 2/27/2026 Steps to Reproduce: 1. Open the app 2. Navigate to the home page 3. Open the nav tray 4. attempt to press “x”, will not close 5. Navigate to another page 6. Press “x”, the navbar closes with a delay Assets: Tests: Test 1) - Open the app - Navigate to the home page - open the nav bar - press “x” - navbar should close instantly Test 2) - Open the navbar - go to the home page - go to another page - press “x” - navbar should close instantly Because you, as the reporter, don’t actually know what’s causing the issue, you don’t know which information is irrelevant. If you don’t report all of the important things, it’s very hard for a developer to tackle the more pernicious issues. In other words, if you don’t make bug tickets thoroughly, you might as well throw your product in the trash. Despite the importance of thorough communication around bugs, half-baked communication remains a consistent issue in the industry. One of the jobs of a software project manager is to make sure bugs like these are communicated effectively. In my opinion, a software project manager, or product owner, should be testing the app regularly. This might be done with formal testing sessions or simply through their usage of the app while imagining new feature improvements. I’ve found, in my career, the best bug tickets are made by the manager of the product, and bug tickets don’t end up making the project more robust if the person managing the tickets doesn’t have skin in the game. Having a testing team can be helpful, but the manager testing the app is arguably more important. The Application of Kanban in Project Management We’ve talked about the tools of Kanban, but not what managing a team with Kanban looks like on a daily basis. Let’s spend some time discussing how tickets might flow through a kanban board. Let’s start with the thing that gets tickets on the board, “ticketing meetings”. Ticketing Meetings A ticketing meeting, in essence, requires decision makers in the team to sit down together and agree on user stories and acceptance criteria. The end goal of ticketing meetings is to make high-quality tickets that developers can then work on. It’s fascinating how often people haven’t considered the implications of a product having two ideas simultaneously, or how a feature idea might lead to a substantial amount of confusion rather than an improvement in the product. These issues are especially common in the early phases of a new product, which is when ticketing meetings are of utmost importance. It’s vital for the major stakeholders to sit down and agree on the specifics of all tickets before the technical team implements anything. The best way to do that is to create and think about various tickets as a group. Depending on the decision makers in the team, and their availability and interest, you might adopt slightly different approaches on how tickets get created. You might draft tickets based on discussions you’ve had with the team, then the drafted tickets you created are approved/modified/withdrawn in ticketing meetings. You might decide to take a more passive role and mediate the creation of tickets based on the input of various stakeholders in the team. A hustler/hacker team of two is going to have very different ticketing meetings than a team of three subject matter experts in a regulated industry; there’s no universal approach. The only formal requirement, I think, is that the major stakeholders agree on user stories. Achieving agreement is a very important step towards making sure the ticket is actually worth working on, and won’t be undermined by future initiatives pushed by a stakeholder. Ticketing meetings are generally required when the backlog of tickets is low, which will be the case when a new project starts off. Thus, at the beginning of a new project, it’s common to have numerous back-to-back ticketing meetings to build out a healthy backlog of tickets. Once you feel like you’ve created a healthy backlog that describes the team’s ideas of the product without major missing pieces or logical inconsistencies between tickets, then development can begin. Ticketing meetings are often stressful for a new team. They can be long, especially at the beginning of a project. They’re often required when it seems like there’s a lot of other things to do, and they can raise disagreements that boil tempers. The most important job of the project manager is to keep the team healthy, so I have a few rules I’ve learned based on my experience running these: The main reason non-technical stakeholders avoid ticketing meetings is that they think they’re a waste of time. They think this because they don’t understand how specific software development is, and how hard it is to do without clear direction. Communicating that developers will be spending days or weeks working on each ticket can help people realize how small they are relative to the amount of work they govern. Be flexible. The end goal is that stakeholders agree on tickets. Different situations might require different approaches. You might want to start with a single ticket that requests different ideas from a designer, like: “As a person managing this product, I don’t know which approach to take, and mockups would be helpful. Currently, there are a few ideas that are compelling…”. The stakeholders in the ticketing meeting can then make tickets with these designs in hand, allowing the team to reason about tickets based on something concrete. You can also just go ahead and draft tickets yourself, even if you don’t know what you’re talking about. Sometimes, communication is smoother when people are referencing something concrete rather than coming up with it themselves. Having a bad ticket that people disagree with for specific reasons can be more productive than having no ticket at all. Communicate that early ticketing meetings are an up-front investment. Once the project is up and running, they will quickly reduce in duration and frequency. As previously mentioned, ticketing meetings are triggered when the backlog is low. They can be stopped when there is a healthy list of items in the backlog. The goal is to have ticketing meetings before developers run out of tickets, so you can constantly supply your team with high-quality tickets that have been approved by stakeholders. Once a project is rolling, I like to have ticketing meetings when the developers are around a week or two away from running out of tickets to work on. Pull-Based Development One of the key ideas of kanban is that, from the developers’ perspective, there’s not a concept of priority. All of the tickets in TODO need to get done, the earlier the better. It’s up to the developers to choose which tickets are the most accomplishable right now. In Kanban-style projects, developers aren’t assigned which tickets they should work on. Rather, they “pull” new tickets from the backlog when they have bandwidth to do so. This results in a continuous flow of tickets. In essence, the team runs like a wood chipper; the manager feeds in tickets, and the team chews them up as fast as possible. If a ticket is introduced that grinds things to a halt (like our transaction classification problem in the waterfall example), it’s removed from circulation and the management team rethinks the ticket, with feedback from the developers, before re-introducing a revision. Technically, ticketing meetings and pull-based development are all that’s necessary to implement Kanban, but there’s another important step that I typically adopt when employing Kanban in a project. Pointing Sessions Non-technical stakeholders might have ideas for a product, but they might think twice about asking the technical team to implement a ticket if they knew how long that ticket would take. Once tickets are created, it’s often a good idea for the technical team to review the tickets and assign points to them. These points serve as an estimate of implementation cost, so the management team can choose what’s worthwhile to put in the TODO column. At the beginning of a project, the team assigns points based on how many days a ticket will take to complete. However, as the project progresses, points have a tendency to become a form of currency in themselves, loosely combining the ideas of implementation time, uncertainty, testing, and other important concerns. Inevitably, pointing sessions result in technical questions, discussion, and clarification. The developers read the ticket, then may have questions that need to be answered before they can come to an estimation. Developers, I’ve found, are good at productively having these discussions within the context of a ticket, so long as that ticket is well defined. Non-technical stakeholders often get confused, and thus distract and derail the conversation with questions and clarifications that are not relevant. Often, I’ve had pointing meetings go an hour over because of unproductive input from a well-meaning but ignorant and anxious stakeholder. I’ve also had important meetings be cut short when a non-technical stakeholder becomes bored with the technical conversation they don’t understand, which was, in reality, very important and had to be continued afterwards in Slack. I generally avoid information siloing. I don’t think it’s productive, and it inhibits cooperation. That said, I’ve found that having non-technical decision makers in pointing meetings is counter-productive, and I tend to avoid it. As a project manager of software projects, a big part of the job is to be a liaison between the technical team and the business team. Both of these teams speak very different languages and operate based on very different rules. Sometimes it’s good for everyone to gather round the fire and share in one big ticketing kombaya, sometimes it’s best to keep these meetings tight and lean. There’s no right answer. In my experience, things work best when it’s just me and the management team working on ticketing sessions, and it’s just me and the technical team working on pointing sessions. An occasional get-together with the whole team is sometimes productive, but as a rule, having a single voice (the project manager) serving as a messenger between both teams, paired with high-quality tickets, maximizes information availability while minimizing confusion. There are frequent exceptions, but this is a healthy approach most of the time. I should qualify this by saying I’m a software developer with years of experience running businesses, so I’m uniquely qualified to be that “man in the middle”. Depending on your capabilities, the team, and the various personalities and communication styles, you might find that a more integrated team is better. A Review of Kanban, and My Thoughts on Its Application In Kanban, you often establish what is worth doing by having ticketing sessions and filling out a robust backlog. You then talk to your development team, have them point tickets based on their user stories and acceptance criteria, and move the tickets you think are worth doing into the TODO column. The development team then works through the TODO column, working on them in the WIP column, then moving them to Done or some other QA process. An example of a kanban board I think it’s useful to reflect on how different this is from a waterfall style of management. There is a high level “plan”, but it’s encoded in the backlog. As developers are working on tickets, the backlog is being worked on at the same time based on constant communication and feedback between the technical and management teams. The inevitable result is that the plan changes based on what is happening with development, and development reacts as the plan shifts. This fundamentally iterative approach is what makes Kanban “agile”; it doesn’t treat the project as a fire-and-forget missile, but encourages managers and developers to modify the scope and priorities of the project based on the progress of the project. Kanban is a fabulous approach to project management, especially in small startups working on an unsure product and in unstable financial situations. Kanban, in its simplicity, is very robust to shifting on a dime, stopping abruptly, and restarting again. The lack of management overhead makes it particularly agile, which is why I adopt Kanban as my primary management style when working on brand new projects. Once a project has some legs, though, the lack of formality of Kanban can become a hindrance. Don’t get me wrong, it’s perfectly servicable even at larger scales, but when things get complicated some more structure can make things a bit better. In these situations, “Scrum” is a great option. Scrum, Another Agile Approach to Project Management Scrum can largely be seen as an extension of Kanban, enforcing certain roles, ceremonies, and core tenets on top of the structure of Kanban. Scrum’s roots are debatable, but by many accounts they go back to 1986, when Hirotaka Takeuchi and Ikujiro Nonaka published a Harvard Business Review article titled “The New New Product Development Game.” source The New New Product Development Game discusses a general trend in the cutting-edge companies of the 1980s. They described high-performing teams (Honda, Canon, Fuji-Xerox) working like a rugby team, passing the ball back and forth dynamically as it moves down the field. If you’re interested in project management, I highly recommend giving this document a read. It has some old school ideas (like intentionally putting teams in do-or-die situations) that I think are a bit out of vogue, but the majority of it, I completely agree with, and I think its insights remain relevant to this day. Scrum (which also gets its name from rugby) took the general ideas from The New New Product Development Game, as well as other points of inspiration I’m sure, and defines a single and prescriptive framework for project management. The central document of Scrum is “The Scrum Guide.” the Scrum Guide. Source. This document describes the core theory of Scrum, how Scrum defines a team, the core things the team does (called Scrum events), and what tools and record-keeping systems the team uses to organize themselves (called Scrum artifacts). The Scrum Guide isn’t particularly long, I recommend reading it if you consider adopting Scrum. I won’t be covering every idea, but we can go through it from a high level. Core Values of Scrum Scrum adds to Agile the idea of introspection. At regular intervals, the team re-examines past work in order to find flaws in their processes and to make improvements to how the team works. This essentially turns the process into a part of the project itself, expecting it to be adjusted continually to meet the demands of the project and team. To make sure appropriate agility and introspection are met, Scrum enforces key roles. Roles of the Scrum Team As we’ve discussed with Kanban, Agile approaches to project management require certain people to take initiative in terms of how tickets are created, who does work, who manages the team, etc. In Kanban, these are amorphous ideas that get picked up organically. In Scrum, there are three rigidly defined roles that defines who does what within a team. Developers: These are people in the team responsible for creating “usable increments of work”. An increment of work, in Scrum, is a task that moves the project towards its end goal. These are the people working on tickets. The Product Owner: This person is responsible for maintaining a healthy product backlog. They do that by talking with stakeholders and potential customers to clearly define the goal of the product, and working with developers to ensure that backlog items are clearly defined. The Scrum Master: The person responsible for the teams efectiveness. This person is, essentially, responsible for the application of Scrum in general; they help team members understand how to self-manage themselves and work cross-functionally, help the scrum team in general to define high-quality tickets that move the project forward, and help to improve the team to ensure they’re generally operating maximally in all core steps of the Scrum process. One person can wear many of these hats, and the hats can change from person-to-person as the project progresses, but the hats themselves are well defined. One of the most fundamental ideas of Scrum is that there is no hierarchy of roles. The product owner, scrum master, and developers all exist on an equal footing. They have different jobs, but all are vital, and thus no job is more important than another. These different roles typically work together through key events throughout the project lifecycle. Scrum Events The main thing in Scrum is a “sprint”. A sprint is a time-boxed unit of work. The exact timebox can change from team to team; a sprint can be one week or one month. Two-week sprints are a common standard. Pretty much everything in Scrum happens within a sprint. It starts with something called a “Planning session” where the entire scrum team gets together and discusses tickets in the backlog which the product owner created, and how they relate to the goal of the sprint. The official Scrum Guide lists three goals of a sprint planning session: Define why the sprint is valuable (how will this move the product forward) Define how much work can get done, realistically, within the sprint timeline, that satisfies the goal Define how the work will get done. This is up to the developers alone to decide, based on the sprint goal and the definition of each ticket. Each ticket is divided into sub-tasks which take about a day to accomplish Regardless of the specifics, Sprint planning gets done, then work can begin. Once work is progressing, the Daily Scrum (another event) is used to keep a pulse on the project and make sure it’s progressing well towards the final goal. This is a 15-minute developer meeting, same time every day, where developers talk about what they will be working on the following day. As I previously mentioned, developers are naturally good at these types of conversations based on their closeness to the work. Thus, Scrum doesn’t assign many rules around the event… “as long as their Daily Scrum focuses on progress toward the Sprint Goal and produces an actionable plan for the next day of work” — The Scrum Guide As the sprint draws to a close (in formal Scrum), a “sprint review” is called. The scrum team and stakeholders review what has been completed in the sprint, and discuss what should be done next. The Scrum guide says one should avoid this from becoming a “presentation” and should instead be a “working session” where the scrum team and stakeholders review the backlog and make adjustments. Finally, a “Sprint Retrospective” is called. The team is encouraged to review their own work, spot inefficiencies, and attempt to alleviate those so that the next sprint is more productive. You might notice that… this is a lot of meetings. For bigger, well-funded teams working on big problems, a one-month sprint with all the formal meetings can be vital. For smaller teams, these rules are often modified to make the meetings shorter or less frequent. When managing a well-funded team, it can be beneficial if the entire scrum process is exposed to the entire team. When working with a smaller team, I often organize the team in a leaner kanban style, and do Scrum-inspired reviews and retrospectives by myself. This keeps me accountable as a manager, and preserves the introspective spirit of Scrum, without forcing the entire team to sit in for hours of meetings every two weeks. I’ll talk more about my approach later; for now, let’s move on to another important Scrum concept. Scrum Artifacts Formally, in Scrum, there are three key artifacts: The Product Backlog: This is a backlog of well-defined tickets, made by the product owner, which can be done within one sprint. The Sprint Backlog: The tickets selected from the product backlog to be eligible for development in the current sprint. This is decided on in the sprint planning session An “Increment”: “Increments” are basically just tickets. It’s a unit of work that a developer works on. You can also define an increment as a task, or a sub-task. Really, it’s a matter of semantics. In formal Scrum, each of these comes with a “commitment” associated with it. A commitment is the overarching “why” of each of these artifacts. The “Product Backlog” is paired with a “Product Goal”, the “Sprint Backlog” is paired with a “Sprint Goal”, and each “increment” is paired with a “definition of done”. My Thoughts on Formal Scrum I think Scrum has a lot of great ideas which, in their totality, are a bit overkill in most situations. I think Scrum has a lot to say about how a team reviews itself, which I find is practically easy, and not a lot about how tickets should be defined so that they result in substantive improvements to the product, which I find to be practically challenging. If you have a team of 5–10 devs, all working together in a single office on a complicated system, the formal scrum approach can offer certain advantages that keep the team honest. Practically, though, I think the spirit of Scrum, and its focus on planning and review to bookend a sprint, are the true benefits of the approach. The Scrum Guide has some things to say about this, which I find a bit melodramatic. Changing the core design or ideas of Scrum, leaving out elements, or not following the rules of Scrum, covers up problems and limits the benefits of Scrum, potentially even rendering it useless….While implementing only parts of Scrum is possible, the result is not Scrum. Scrum exists only in its entirety and functions well as a container for other techniques, methodologies, and practices. I think Scrum, as it’s defined, suffers from a bit of a lack of trust. It assumes the scrum master and product owner won’t work together when they realize that the process is not as efficient as one might like. Instead, Scrum forces the whole team to sit down every sprint and have an obligatory meeting. If you’re not having that meeting, then you’re not doing Scrum. I like pretty much all of the ideas of Scrum, besides its focus on rigid adherence. I think different environments, and the different managers and teams in those environments, require slightly different tactics to be successful. Personally, I rarely do Scrum in its entirety (I often work with stakeholders who do not have the patience to do all of the Scrum events, even if I wanted to). I want to share my approach, which is a hybrid of Scrum and Kanban with a few other tricks built in. I have two core strategies for software management, one for new products and one for mature and well-defined products. We’ll start with my approach to managing a team working on a new product. A Modern Approach to Agile Project Management For New Products This is my approach to project management, but it’s not mine alone. Mike taught me a lot of these tricks, and the general approach seems to be a common settling point for many teams that I’ve talked with. It’s a hybrid of Kanban, Scrum, and a few other ideas from lean entrepreneurship. I essentially function as the product owner and project manager in most of the teams I manage. Typically, I don’t know the most about the product (that’s usually a stakeholder, like a CEO, COO, or CPO). Instead, my role as “product owner” is to own the backlog of tickets. I work with people knowledgeable about the product to do that. I then work with developers to manage how those tickets are worked on and when. A new project starts with a lot of ticketing meetings. I, and several stakeholders, sit down and hammer out what we think the product should be like. This requires settling on who the customer is, what they want, and how we’ll help them. Practically, this touches on other skills besides project management, like customer discovery and go-to-market strategy. For more information on those topics, I recommend reading The Lean Startup by Eric Ries and Crossing the Chasm by Geoffrey Moore. At this early phase of ticketing meetings, my main job is to keep the team focused. We’re not worried about technical specifics, or even designs. We’re interested in broad strokes functionality of the product, how synergistic different functions are with each other, and how we can provide value to the customer. Frankly, the goal isn’t even to figure out what the end product will look like, but to create a trajectory for the team. Dreaming up a product is a means, rather than an end, to establishing a trajectory. Inevitably, throughout the development process, the vision of the product will change as we get feedback from customers. After we establish a laundry list of features, these broad strokes are filtered through the lens of what’s referred to as the “minimal viable product”. This is the minimum set of functionality that’s required to create something that the user will actually want and pay for. This, again, won’t actually be realized. As the product gets worked on, the team will realize customers “actually need ____” or “don’t need ____”. The minimal viable product is what the team imagines is the minimum set of requirements needed to be successful on the outset. It creates a lean and actionable list of requirements to get the team in the right direction. As the project progresses, this will become less relevant, and that’s ok. It’s useful in the beginning. Now that you have a vision of the minimum viable product, it’s time to sit down and make tickets. If you’re lucky, the stakeholders will be willing to sit down with you and patiently define story tickets. recall the template: Story: Acceptance Criteria: Testing Criteria: Design Assets: But you may be forced to build these out solo. Thats fine, because you have a strong idea of what the minimal viable product (MVP) should be. Essentially, your job is to turn the idea of the MVP into a list of story tickets. I do recommend at least having one meeting where stakeholders can confirm key user stories and acceptance criteria. Disagreement at this phase is productive. Disagreements after tickets have been completed and the product is taking shape are much messier to deal with. In terms of practical application, I opt to use Trello for most of these tasks. Some people like fancy integrated systems, but for working with a frothy startup on a vague product, simplicity and clarity are best. The actual Trello board template I use looks like this: My Trello Template It’s essentially a normal Kanban style, with some columns designed to accommodate a few key Scrum-like events that I’ve found to be impactful when working with small teams. There’s a README column, which I use to store a ticket that explains how the board works, and a template ticket for bugs and features. Because I have templates saved, I can conveniently make new bug or feature tickets by clicking this little button in one of the columns. Creating new tickets with templates in Trello The README describes how the board works. Noone reads it but me, but it’s nice to have anyway. It describes the general process. This is a lightweight implementation of a SCRUM style Kanban Board designed for planning and executing on a new major release. This relies on the following workflow (aka Ceremonies, loosely): -New ideas go in the backlog -They are fleshed out with the requisite information to be fully designed. When fully baked out they end up in staged. -During a pointing meeting with developers, the developers point staged tickets. The resulting tickets end up in pointed. -A sprint is planned, where we plan out what will be done over a 2 week period. The tickets to be worked on go into sprint TODO -A sprint is executed, with regular meetings as necessary -A review is done at the end of the sprint These are some critical concepts: -Feature tickets contain all the information one would plausibly need to understand from a users prospective and a technical prospective before implementing. -Bug tickets contain all information that can realistically be provided -Tickets change minimally throughout a sprint -One of the goals of sprints is to find how quickly we work, so we can plan ahead. Thus tickets are pointed so we can understand how long a ticket will take, and look back at how long tickets have taken previously The ticketing team (me and a few stakeholders) flesh out some tickets in the Backlog Within the ticket, I have my template for a fully defined ticket, which should be filled out in order to call the ticket done. There’s also room for comments, so that developers and stakeholders can converse about the ticket as necessary. The inside of a ticket Once a ticket is fully baked out (which can take a few ticketing sessions to complete sometimes), the ticket ends up in Staged, meaning the ticket is ready for pointing. During pointing sessions, the tickets are assigned point values, which get placed inside the placeholder . Fully pointed tickets are placed in the Pointed column, and are thus ready to be introduced into a sprint. Recall the previous discussion we had on pointing sessions. Points are loosely understood as a number of days it will take to implement something. I use two-week sprints, which is 10 working days. Thus, a 10-point ticket is rather large. Anything longer than 10 points I definitely subdivide, but I prefer to keep my tickets under 8 points if possible. This threshold might change based on your team and their capabilities, though. Now that tickets have been pointed, the ticketing team might decide to reconsider A fully defined and pointed feature <10>, and instead try to create two or more tickets that subdivide the effort. They might be surprised that the dev team assigned a value of 10, perhaps have a brief conversation as to why, and shelve the idea as too expensive. Regardless, the tickets that are good can go into Sprint TODO, and the team can begin working on them. Depending on the team, you might decide to do a Kanban style of project (allowing developers to pull as quickly as possible) or a more formal Scrum style (thinking of a particular batch of tickets as a sprint, and having a defined start and end date for the sprint). In more chaotic environments the simplicity of Kanban can be benificial. In more mature environments, where more is going on, Scrum can have significant advantages. The developers then manage how the tickets move from Sprint TODO to Sprint WIP and Sprint To Review. They use these columns to keep track of what they should work on, what they are working on, and what they consider done. Once the developers have a ticket in Sprint To Review , that means it’s done and ready for review. You don’t have to review tickets in every situation, often dev teams have their own QA practices to minimize bugs, but it’s often a good idea for the product owner to at least check the product of the tickets from a high level. I have a QA column to help me keep track of the tickets I’m currently evaluating. For big teams, a larger QA team could pull tickets from Sprint To Review, and thus QA would be like the “work in progress” for the QA team. You can play around with the specifics depending on your team’s composition. At the end of the day, though, after a ticket is done it ends up in the done column. 99% of the time, when a ticket is done by the development team it makes it through QA and to Done. Occasionally, though, an implementation is not faithful to the story, or some fundamental bugs are introduced, which undermine the ticket’s quality. In either of these cases, the product owner can decide to either call the ticket done anyway, and create new bug or feature tickets to communicate the new issues, or a comment can be left in the ticket itself, the ticket definition adjusted based on the result, and the ticket can be sent back to the backlog rather than progressing to done. Often, in progressing through this process, a large number of tickets will end up in the backlog. As the weeks go by, old tickets might become “stale”, a term referring to a ticket that’s no longer practically relevant. I, as the product owner and project manager, review and archive these stale tickets throughout the duration of the project. While not solely related to project management, I would be amiss not to mention environments. An environment, in software development, is an identical version of the app deployed on different URLs. You might have myproduct.dev.com , myproduct.qa.com, myproduct.com. Each of these environments has its own computers, databases, and is essentially completely isolated from one another. This is useful because it allows developers to see how their changes will impact the product in the dev environment, it gives the QA team a playground to test changes in the qa environment, and the team can decide on only the functional output to be pushed to the production environment. Developers often use strategies, like CICD allong with code branching strategies, to organize their code such that they can control which feature ends up in which environment. With some small revisions, the Trello board can accommodate a centralized control of this type of work, simply by creating columns that signify when developers should push certain functionality. When things are going smoothly, and you have a few sprints recorded in terms of the sprint goal and what the team managed to get done in the two-week interval, you begin understanding the “cruising altitude” of the team. The “cruising altitude” is the number of points the team can get done in a single sprint. This is super useful because, after it’s been established based on the history of the team, it can be used to estimate how long certain tickets, or batches of tickets, will take. Instead of a fly-by-night guess of how long a new core feature will take, you can draft the requisite tickets, have the devs point the tickets, and thus have a reasonable estimation of how many sprints are required to make it happen. So, that’s it. That’s my approach to managing small projects. It’s relatively simple, but I hope you can see the depth that lies underneath that simplicity. Good project management should feel simple, but to settle on something that works, you need to understand a lot. In larger teams, and larger projects, a slightly different tactic is required. Let’s discuss my approach in that context. A Modern Approach to Agile Project Management For Existing or Large Products I have a core ethos when dealing with a large project; it’s actually just many small projects. I’m not the only one who thinks this way. If you can’t feed a team with two pizzas, it’s too large. — Jeff Bezos The best large organizations operate as a network of entrepreneurial teams. — Reid Hoffman Thus, fundamentally, a large team is operated identically to a small one, the only difference is that you divide the large team into several smaller teams, and have them work on specific problems. Thus, the management of large projects is essentially the task of managing many small projects simultaneously, and keeping them aligned in terms of strategy and results. There are as many ways to divide a team into teams as there are companies with teams to divide. The process is essentially arbitrary, but can have serious repercussions later down the line. I recall, years ago, watching a video that compared the way Windows and MacOS handle audio. I was annoyed about how, on MacOS, it’s very difficult to record audio while doing a screen recording, and I wanted to know who in their right mind decided that MacOS should have such a convoluted approach to audio. Well, as I recall, in early development, Windows had one team for Audio and Video. At Apple, they were two separate teams (or something to that effect). This caused the two features to be more richly coupled on Windows, where there are still some quirks of video and audio that persist to this day in MacOS. The idea of the fragmented mind, I think, comes into play when defining how larger teams should be divided into smaller functional teams. You want to keep the teams able to reason about a whole problem, while not making the problem so large that you need a ton of people. Depending on the size of the company, it might be impossible to solve a problem with a single small team. Thus, you might need to think in terms of layers of teams so that you can keep teams small but sufficiently coupled to solve those specific, higher-order problems. An example of an organizational chart. These arise in large organizations as a result of needing to keep teams small, but also the fact that some problems are so large they need multiple teams. Source One division I think is immediately required, even in small software teams, is the division of Development, Research, and Marketing. Each of these worlds need to work together to create a successful company, but each of these also speaks such different languages that they need to have their own space to operate independently. You might be curious why I include marketing in my list, considering this is an article on software project management. There’s a tongue-in-cheek expression in entrepreneurship that describes the inclusion: “First time product-focused, second time distribution-focused”. Essentially, in your first go-around building a product, it’s common to be excessively focused on the product itself. Quickly, though, it becomes apparent that you have no idea how to make the customers aware that the product exists, or how to get it to them. Thus, an entrepreneur’s second venture often focuses on distribution moreso than the product itself. To mitigate the problem of distribution, it’s common in software to attempt to sell a product before it’s been built. You might have designs or a proof of concept, and it’s the marketing/business team’s job to see if that product has traction. The feedback they get can and should inform the priorities of the development and research team. Thus, in a healthy organization, there should be a healthy interplay between marketing and technical teams. Research and Development are often lumped into one idea, but from a management perspective, I believe these should be isolated. Practically, development requires coming up with user stories, acceptance criteria, and a list of sub-tasks necessary to make the user story a reality. For development to go smoothly, you want nice, neat, timeboxed items that push the project forward in predictable ways. If one of the features requires, I don’t know, the creation of an energy-efficient nuclear fusion reactor, you’re going to have a hard time wrangling that into tickets. Experimental fusion reactors exist, but none of them have yet to be energy efficient. You’re going to need to figure out a lot of unknowns, and figure out the core principles of fusion reactors, before you can even begin considering adding the technology to your product. Thus, Research and Development are two complementary but distinct worlds. If the development team doesn’t know how to solve a problem, creating a ticket and having them work on it will only serve to gum up the well-oiled machine that is an agile project. For things that need to be explored, but have unknown timelines and approaches, that’s a research job. Research makes teams uncomfortable, largely because they usually lack the foresight to invest in research upfront. Often, development teams foresee a technical hurdle in the future, but non-technical management does not understand the challenge and are thus not willing to write a blank check to dedicate time to understanding the problem before it becomes an actual problem. This is understandable, but it results in research initiatives being started when they’re needed, rather than being finished before they’re needed. In order to do research effectively, you need to look at the project, project it into the future, predict the biggest unknowns based on the product right now, and have the research team pre-emptively explore those problems before they inhibit the development team. It’s up to management to understand that this is vital, foster sufficient resources, and trust that things will go smoothly. Frankly, research is one of the hardest but most vital ideas in software project management. As a result, I usually do most of the research in projects I manage by myself. Good research requires both a business and technical knowledge that exceeds the norm, and requires a level of authority to act autonomously outside the scope of proscriptive budgets and timelines. The personnel required is often expensive, and the ability to estimate timelines can be nonexistent. If you’re not in the situation where one person can just “do all the research”, or perhaps there are too many unknowns in your large project, there are some management approaches you can use to allocate research time within an agile project. Often, the issue of research isn’t that there’s no one on the team that’s capable, but that the capable people are busy doing other tasks. If you find that unknowns are looming over a project, you can define “Spike” tickets. Spike tickets are time-boxed tickets dedicated to exploring a particular research topic. These can be very high-level and minimal tickets that developers can use to minimize unknowns as much as possible. Spike: Evaluate WebRTC streaming latency under load Timebox: 3 days Deliverable: report + prototype Sometimes, in cutting-edge projects where unknowns amount to a fair chunk of the overall work, a dedicated research team is required. This is common in projects that require a lot of data science/AI, for instance. Data science (as you might know from the rest of my articles) is complicated and generally non-deterministic, making it very hard for researchers to work within rigid requirements or time constraints. In situations like these, “time horizons” are usually better than deadlines. These are time goals that the researcher works towards, and makes as much progress as they can within that time. They then write a report, or make a deliverable. Thus, the goal isn’t “solve the problem”, it’s “figure out as much as you can in this time frame, then create something to share with the rest of the team”. The manager can then see the relative progress on each thing being researched, and adjust research priorities, without relying on rigid constraints that stifle deep research work. The general game of management in a big company is to keep all of these teams aligned, as best as possible, to the same goal. You might have one product, but there are several versions of the product the company plans on releasing to different markets. You might have several competing philosophies on future feature rollouts that require comparison and consideration. This “meta management” requires more abstraction than Trello, the kanban tool we used previously, provides. This article is already rather long, and I think I’ve covered everything I wanted to in-terms of managing a team. At this point, we’re starting to approach product and organizational management, which is where more comprehensive tools like Jira really shine. I think it’s time to wrap up, we can save that for another installment. Conclusion Teams are powerful, but they’re hard to manage correctly such that that power is realised. Proper management is the gateway to realizing that power. Agile emerged to deal with the flaws of more traditional approaches to project management. It focuses on iteration and updating dynamically as new information arises in a project. There are several ways to actually “do” agile project management. Two common approaches are Kanban, a pull style approach which is incredibly simple, and more prospective approaches like Scrum, which include dedicated roles and events to keep a team honest. Each of these has costs and benefits; I use a mix of both. As previously mentioned, we’ll cover higher-level product and organizational management ideas in future articles. Cheers! Combinatorics in Probability — Intuitively and Exhaustively Explained Daniel Warfield — Thu, 05 Feb 2026 15:03:02 GMT “Counting” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained. Probability is a fundamental topic in science, technology, and understanding in general. To a large extent, the modern era is a direct result of humans unlocking the ability to understand and reason with uncertainty. Considering its importance, it seems like something worth learning about. I set off to describe probability just like any other topic, but found I couldn’t be properly intuitive nor exhaustive. With probability, there’s just too much to cover. As a result, I decided to break up my exploration of probability into multiple parts. This article is the first in that effort. The end goal of the series isn’t to learn probability for fun, but to use a thorough understanding of probability to cover advanced topics in AI and machine learning. This article concerns the beginning of that journey, and the most fundamental idea of probability; “combinatorics” or, in another word, “counting”. You might think you know how to count. Unless you’ve studied combinatorics, you probably don’t. Subscribe To Intuitively and Exhaustively Explained. Share this article with friends and colleagues Share Who is this useful for? Anyone interested in understanding probability or developing a more thorough understanding of how AI models deal with uncertainty. This article doesn’t discuss AI, but will serve as a fundamental reference as I explore the subject in future installments. How advanced is this post? This article is designed for those who know nothing about probability; we’re starting from home plate here. That said, combinatorics can be complicated and counterintuitive. You may need to reflect on this article and read it several times before completely understanding it. Prerequisites: A basic high school level understanding of mathematics. Probability Before we talk about combinatorics, I want to talk about our end goal. Defining probability is what got me started on this series. People throw around the term “probability and statistics” like they’re two neat and well-defined concepts. I was curious about the exact difference between “probability” and “statistics”, but, to my chagrin, found there isn’t a single agreed-upon difference between the two. It’s kind of like the difference between “math” and “physics”; it’s a simpler distinction the less you know about either subject. So, setting that to the side, I tried to tackle a simpler question “What is probability?” This is also a murky question, that gets murkier the more I study it. There’s frequentist philosophy, Bayesian philosophy, the controversial history of causality, etc. I don’t think there is a comprehensive and fully encompassing definition of what “probability” is that some very smart person won’t disagree with. Before writing on the subject, I thought I had a reasonable grasp of probability. In my education of mechanical engineering, I took a few classes concerning signals, experimentation, and other probability adjacent topics. I thought myself reasonably confident in the subject. I imagine there are data scientists, statisticians, medical researchers, and economists who also think they’re reasonably proficient in the general subject of probability. If you gathered them all up in a room, though, few of them would probably know what the others know, implying each of them might have a narrower grasp on the subject than they expect. For most people, probability is studied piecemeal as an aside to some other education. You learn the formula and concepts that are relevant to your particular domain of expertise, and often don’t learn the underlying principles that make it tick. Personally, I’ve found this mode of learning to be frustrating. It’s hard to understand how to construct a house when you don’t understand the core principles of wood, nails, and paint. Similarly, it’s hard to understand probability without understanding its constituent parts. I think this is why I heard the following phrase so often when I was in college: “I hate probability” — Most students of most disciplines We will be building our knowledge from the bottom up, rather than the top down. As we explore probability in depth, our definition of what probability is will become more nuanced. However, we can start with the most superficial definition for now. As a numeric quantity, probability describes how likely something is to happen. As a field of study, Probability is the study of how to calculate probability for different events and in different situations. If you stick your hand in a bag and pull out a rock, how likely is the rock to be black? If you have a certain hand in poker, how likely are you to win the game? Who is most likely to be the next president? What is the likelihood that a nuclear reactor will melt down? These, and many other questions, are questions of probability. Because it’s our north star, I’d like to take some time to define probability in mathematical terms before we explore counting. A Naive Definition of Probability The term “naive definition” is used in STEM to denote a simple and intuitive definition of something. Often, it’s the definition that comes to mind. When we call something a “naive definition”, it usually means useful, intuitive, but incomplete in some subtle ways. As we explore probability we’ll revise this definition, but for now “naive” is more than sufficient. A naive definition of probability. Source: Introduction to Probability by Jessica Hwang and Joseph K. Blitzstein I’d like to describe this expression by way of an example. Imagine you have 10 pebbles in a bag. Two of them are white, the rest are black. If you randomly select a pebble from this bag, what are the odds it will be a white pebble? There are 10 possible selections, and only two of them are white. So the odds are 2/10 = 0.2 = 20%. We can come to the same conclusion using the definition highlighted above, but to talk about it we need to define a few key words. In probability, each pebble is referred to as a possible “outcome”. An outcome is a single result that is possible (i.e. choosing a particular pebble). A subset of outcomes is called an “Event”. The “event that we choose a white pebble” would be the subset of outcomes that correspond to a white pebble. The “space” is all possible outcomes. No matter which pebble we choose, we will choose one of the pebbles in the bag. Looking back at our definition, we can see |A|, and |S|. A naive definition of probability. Source: Introduction to Probability by Jessica Hwang and Joseph K. Blitzstein In our pebble world, S and A represents “sets”, a set being a group of possible outcomes. S is the set consisting of all pebbles in the bag, a.k.a. the sample space. and A is the set consisting of just the white pebbles in the bag, a.k.a. the event that the pebble is white. The vertical bars around each of the sets is a math symbol for “the size of the set”. Here, the size of the set S is 10, and the size of the set A is 2. P_naive is a function that we’re making that is being applied to a set. So, you would read this expression A naive definition of probability. Source: Introduction to Probability by Jessica Hwang and Joseph K. Blitzstein as “We’re defining what it means to calculate the naive probability of an outcome in event A actually happening. We’re defining it as the number of outcomes in set A, divided by all of the outcomes in the sample space”. The issue, and the reason for this article, is that it’s not always easy to count how many possible outcomes there are in the space S, nor how many outcomes are in the event A. For instance, imagine we wanted to calculate the probability that we had a winning hand in poker. To do that, we would need to count how many possible outcomes there were to our poker game, and how many of those games resulted in us winning. Give it a shot, it’s not so easy. Combinatorics is the study of counting; not counting sheep on a hillside, but the number of combinations, permutations, and arrangements there are in complicated systems. If we count things properly, it’s often easy to calculate probability. The counting is often the hard part. There are a lot of ways to count things, but in this article we’ll discuss some of the most important ideas as they relate to probability. Sampling With Replacement Back to our pebble example. Let’s ask ourselves two questions. “If you take a pebble out of the bag, put it back in, shake it, then take another pebble out, what are the odds that it will be white both times”? “If you take a pebble out of the bag, then take another pebble out of the bag (so you’ve taken two pebbles out of the bag), what are the odds that both of them will be white”? These are two subtly different problems. The first is called “sampling with replacement”, the second is called “sampling without replacement”. The distinction concerns whether you put the previous pebble back in the bag before choosing a new one. Let’s answer each of these questions by counting the possible number of outcomes. We’ll start with sampling with replacement: Question 1 If you take a pebble out of the bag, put it back in, shake it, then take another pebble out, what are the odds that it will be white both times? There are 2 white pebbles, and 8 black pebbles in the bag. There are several famous examples of many brilliant minds throughout history messing up problems that are similar to this one. To prevent these types of mistakes, it’s often best to give each possible outcome in the space a label. The end goal is to calculate P_naive(A), where A represents the set of all winning outcomes, and S represents the set of all possible outcomes. A naive definition of probability. Source: Introduction to Probability by Jessica Hwang and Joseph K. Blitzstein To ground this in our example, which asks how likely it would be if both the pebbles we took out of the bag were white, A would represent all the scenarios where both pebbles could be white, and S represents all possible selections of two pebbles. We can start with counting the set of all winning scenarios “A”. There are four possible winning outcomes. We choose pebble a, then pebble b We choose pebble b, then pebble a We choose pebble b, then pebble b again We choose pebble a, then pebble a again. Recall, we can choose a pebble twice because we are replacing the pebble into the bag before making our next selection. So, the number of winning combinations is 4. More formally, the “size of the event A is 4”, and in equivalent mathematical shorthand; |A| = 4. Now we need to calculate the number of possible outcomes, more formally, “the size of the sample space”, or in mathematical shorthand |S| This is going to be hard to do element by element. Instead, we can use “the multiplication rule”. There are some unintuitive math definitions, but I find a picture speaks 1,000 words. If we have 10 pebbles, we can put all the pebbles in a row, and all the pebbles again in a column. The number of results possible in the scenario of choosing two pebbles out of a bag, after replacing the first pebble before selecting the second, is equal to how many times each pebble can be paired with each other pebble (including itself), which is 10x10. So, there are 100 possible results in the space S. The multiplication rule describes how many ways one set of things can be combined with another set of things, which we’ll be exploring throughout a few of the example problems in this section. Earlier, we calculated how many winning possibilities there were, |A| = 4 , and we just counted how many total possibilities there are, |S| = 100, so the probability of choosing two white pebbles (event A) is P(A) = |A|/|S| = 4/100 = 0.04 = 4% In a way, you can conceptualize this as simplifying the problem. Instead of choosing two pebbles out of a bag of 10 twice, we changed the problem to choosing one pebble out of a bag of 100, where each of the pebbles represents one of the possible pairs. Re framing the problem, as choosing a pair out of all of the possible pairs. Here, the winning event A is not a single pebble, but a pair of pebbles selected which are both white. By counting the number of winning solutions, and the total number of solutions, it’s like we abstracted the problem to choosing a single pebble. Often, many probability problems turn into “choose k things within a space of N possible choices, with replacement”. Assuming we’re taking k samples from a space of N things with replacement, then there are N^k possible outcomes in the space. We just keep multiplying N for each of the k selections we’re making. If there are 10 things and we’re sampling 3 times, there are 10 x 10 x 10 = 10³ possible outcomes. This was a pretty simple problem because counting the size of the event A was trivially easy. As previously mentioned, actually counting this can be tricky. To get a hang of this, we can go through another example. Question 2 Let’s say we have 4 white pebbles, 6 black pebbles, and we’re selecting 5 times with replacement. What are the odds that we get alternating white and black pebbles? The alternation can start with black, or white. One might think this problem is simple. There are only two valid resulting sequences. Here, b represents black and w represents white. [b,w,b,w,b] and [w,b,w,b,w] So, one might think |A| = 2. However, there are 4 white pebbles and 6 black pebbles, so there are actually numerous selections that can lead to each of these two sequences. Considering each selected object as having a label is important for avoiding these types of issues. For instance, if pebbles 1, 2, 3, 4 are white, and 5, 6, 7, 8, 9, 10 are black, here is a subset of the sequences which would be valid alternations of black and white. [5, 1, 5, 1, 5] [5, 1, 5, 1, 6] [5, 1, 6, 3, 9] [4, 9, 3, 5, 2] ... The trick of this task is counting how many of such valid combinations there are. The easiest way to solve this problem is to consider each winning sequence separately. We can calculate the number of ways to create a [b,w,b,w,b] sequence, and the number of ways to make a [w,b,w,b,w] sequence. If we add those together, that will be the total number of winning sequences. Let’s start with sequences starting with black. In the first spot there are 6 valid pebbles, in the second there are 4 possible pebbles, in the third there are 6, fourth there are 4, and fifth there are 6. A breakdown of the problem. We have 4 white pebbles and 6 black pebbles in a bag, and we’re sampling them with replacement. The sub-question we’re trying to answer is how many ways can we construct the sequence [b,w,b,w,b] based on the pebbles we have. We can use the multiplication rule to calculate how many results are possible in this subset. First, we find how many combinations of the first two selections there are We can use the multiplication rule to figure out how many combinations there are of the first two selections. then how many ways that can combine with the second selection We can further leverage the multiplication rule by combining the number of combinations of the first and second selection with the number of possible third selections to count the number of possibilities for the first three selections. etc, until we’ve calculated all possible sequences. At the end of the day, we calculate that by just multiplying everything together. 6 * 4 * 6 * 4 * 6 = 3456 We can do the same things with sequences that start with white squares, using a similar logic. 4 * 6 * 4 * 6 * 4 = 2304 If we add those numbers together, we get 3456 + 2304 = 5760possible winning sequences that have alternating colors. Therefore, |A|=5760 Next, we need to count all sequences possible. We can use the same logic, but now there are 10 selections in each spot (all of the pebbles are possible to be selected in each spot). So, 10 x 10 x 10 x 10 x 10 = 100000, so |S| = 100,000 Now that we’ve counted the number of possible winning sequences, and the number of sequences possible, we can use those in our formula to calculate the probability of event A happening (event A being a sequence of 5 rocks randomly pulled from the bag with replacement, being in alternating order of white and black). P_naive(A) = |A|/|S| = 5760/100,000 = 0.057 = 5.7% There are some nasty questions you could ask about this situation that require more complicated approaches to counting. We’ll cover some more tools around that later, but for now I think we sufficiently understand the idea of sampling with replacement. Let’s move onto an alternative approach to sampling stones out of our bag. Sampling Without Replacement Sampling without replacement is a very similar problem to sampling with replacement, save one key difference; in sampling without replacement, when you take a pebble out of the bag, you don’t put it back in before selecting the next one. A good first question to ask is “how many possible results are there in sampling without replacement?” If we had, let’s say, 10 pebbles, how many possible sequences could you make by randomly taking pebbles out of the bag, and placing them in a row one after another? In the first selection there would be 10 possible pebbles to choose from. In the next one, there would be one less, so 9. In the next selection there would be one less, so 8. etc. We can use the multiplication rule we used previously to combine those together and calculate the total number of possible sequences. If there are 10 possible first stones, and 9 possible next stones, then there are 10 * 9 = 90 possibilities for our first two stones. There are 8 possible third stones, which we can combine with our 90 possibilities for the first two stones, to get 90 * 8 = 720 possibilities for the first three stones. You might be able to read between the lines, but there are 10 * 9 * 8 * 7 * 6 * 5 * 4 * 3 * 2 * 1 = 3,628,800 possible outcomes when sampling 10 stones out of a bag without replacement. This idea of multiplying the subtraction of one less than a number until you hit 1 is pretty common, which is why it’s often denoted with the symbol !, which stands for “factorial”. 3! = 3*2*1 = 6 , or 5! = 5*4*3*2*1 = 120 , for instance. 3! counts how many ways you can choose a sequence out of three things without replacement, and 5! counts how many ways you can choose a sequence out of five things with replacement. These both assume you’re selecting all of the elements from the set. If you’re selecting a specific number of things, say 4 things out of a bag of 10, you would do 10 * 9 * 8 * 7, stopping after you got to the 4th selection. Just like sampling with replacement, calculating how many possible results exist within event A is sometimes easy, and sometimes hard. It takes some practice to get good at it, so let’s try another problem Question 3 Six students line up randomly; Alice, Bob, Joyce, Tom, Nancy, and Daniel. What is the probability that Alice and Bob stand next to each other? It’s not possible for Bob to be standing next to Bob, there’s only one Bob. Thus, this is a classic sampling without replacement problem. There are six people, which means there are 6! = 6 * 5 * 4 * 3 * 2 * 1 = 720 possible ways we could line these people up in a line, meaning |S|=720 in this example. To calculate the probability of Bob standing next to Alice, we need to count how many of those possible sequences include Bob and Alice sitting next to each other. To do that, we can use an approach called the “blocking” method. There are two characteristics of a sequence that can make it valid; if the sequence has Alice, Bob in the sequence, or if the sequence contains Bob, Alice. It doesn’t matter where in the sequence these are, just as long as one of these two sub-sequences exists within the sequence, it’s a winning sequence. Let’s consider one of those sub-sequences for now. We need to count how many possible ways there are for Alice, Bob to exist within the greater sequence. In other words, how many ways can we stick (Alice, Bob) into a sequence that also contains Joyce, Tom, Nancy, and Daniel. To do this , we can treat the sub-sequence (Alice, Bob) as a single block which gets selected just like any of the other names. after we combine Alice and Bob into a single unit, there are 5 things, meaning there are 5! = 5 * 4 * 3 * 2 * 1 = 120 ways to arrange (Alice, Bob), Joyce, Tom, Nancy, and Daniel. The act of treating multiple selections as a single “block” is the namesake for the “blocking” method of counting. By the same logic, there are 120 other ways to create sequences that has (Bob, Alice), so in total there are 240 possible and equally likely sequences where Bob and Alice end up standing next to each other. Because there are 720 possible sequences, that means there’s a |A|/|S| = 240/720 = 33% chance that the two will be standing next to one another. There’s a subtlety that I think is important to address; in this example, we care about the number of ways to order the sequence. All of the sequences have the same set of people in them, but we’re counting how many ways we can arrange that set. Let’s cover a different but very common flavor of question in the sampling without replacement space, where we don’t care about the order of the result. N Choose K Let’s take our previous people, Alice, Bob, Joyce, Tom, Nancy, and Daniel. How can we make a random committee of three people? Or, in other words, how many ways are there to select three people out of our six possible people in this scenario? It’s important to note, in this scenario, we don’t care who was chosen first, only what committee was chosen. So, the selection (Alice, Bob, Joyce) is treated as the same result as (Joyce, Bob, Alice). This problem, and problems like it, are solved using “N choose K”, also known as the “Binomial Coefficient”, a very common approach to counting that’s helpful when you need to count how many ways some number of element (K) among a larger number of elements (N) can be selected in a “sampling without replacement” manner. We can actually derive the formula for “n choose k” ourselves with a little bit of pondering and eyebrow furrowing. There are several ways to come up with a formula for “n choose k”, I’m partial to using an approach called “adjusting for overcounting”. The essential idea is that you can adjust for over-counting things if you know exactly how many times you over-counted. If you put five pennies on a table, count each of them twice, then divide by two, you would get five pennies. Obviously this is a bit silly, but sometimes it’s easier to calculate some multiple of something, then adjust by that multiple. In our committee example, a committee of Bob, Joyce, Tom should be considered the same as Tom, Bob, Joyce. Let’s put that to the side for a moment and count how many three person sequences are possible, where different orders of people are treated as different outcomes. We’ve already explored the answer to this question, it’s just like taking three stone out of a bag without replacement. There are 6 names, so there are 6 * 5 * 4 = 120 possible ways to make sequences of three names out of the possible 6. This won’t tell us how many committees there are, because that 120 number contains both Bob, Joyce, Tom and Tom, Bob, Joyce. Each committee is represented some number of times within that 120 number. The question is, how many times? Well, we know how many ways three people can be represented as a sequence. If we had three names, Tom, Bob, and Joyce for instance, We could represent those people in 3! different sequences, 3! = 3 * 2 * 1 = 6. So, we know how many sequences of three names are possible out of all of the names, 120, and we know that each three person committee is represented 6 times within that number (because there are 3!=6 ways of expressing 3 people as an ordered sequence). To calculate the total number of committees, we can simply divide 120/6 = 20, effectively adjusting for the fact that we counted our committees too often when we counted the number of possible ordered sequences of length 3. Because “n choose k” is so widely used, it has a variety of shorthand representations in mathematics. Three representations of n choose k. Source: Introduction to Probability by Jessica Hwang and Joseph K. Blitzstein The definition in the middle is the one we came up with. If you have n things, and you want to choose k sequences from that set of things, you can do that n * n-1 * n-2 * ... * n-k+1 ways. That will include counting the same group in a different order as a different result. If you want to adjust for that, and only count the number of ordered sets possible, you divide by k!. To the left of our definition is the common shorthand for “n choose k”. When you see a number over another number between parentheses like that, in a probability and statistics context, then that means “n choose k”. It’s so common it gets it’s own fun math representation. Over to the right is an algebraically equivalent, but slightly more convenient definition of “n choose k”, which you can plug right into a calculator. The one on the left is convenient shorthand, the one in the middle is the most conceptually simple, and the one on the right is the most mathematically convenient. They all represent the same thing, n choose k, also occasionally referred to as the “Binomial Coefficient”. Now that we understand the general idea, we can use concepts of “n choose k” to solve all manner of problems. We’ll start with a softball question, and then ramp up our understanding. Question 3 You and your 10 colleagues are on the board of directors. A committee of three people is being randomly formed. What is the probability that you will be on the committee? First of all, to figure out the probability of some event A happening ( A representing all the results which include you on the board of directors), we can use the equation P(A) = |A|/|S| (The probability of A happening is the size of event A divided by the size of the entire sample space), which we’ve used a few times now. To do that, we’ll need to calculate the size of event A, and the size of the sample space S. S, in this context, represents all of the possible committees that can be created. We can calculate the size of this space easily using our newfound knowledge of “n choose k”. In a group of 10 people, a committee of 3 can be chosen 120 different ways. 10 choose 3, the total possible number of committees. Now we need to calculate the size of space A, which represents all possible ways a committee can be created where you are a member. We can calculate this easily with a bit of mental tai chi. If we’re a member, then the committee will always include us. So, another way to ask “how many possible committees contain me” would be “how many ways could we complete a committee, assuming I was already selected”. If we’re already a known member, then we would need to find how many committees of 2 can be created from the remaining 9 people. To do that, we can just use “n choose k” again. 9 choose 2, the total number of committees where you are a member. So, 10 choose 3 represents the total number of possible committees, 120 , and 9 choose 2 represents the total number of committees where you are a member, 36. P(A) = |A|/|S| = 36/120 = 0.3 = 30% Thus, the probability that you will be a member of the committee is 30%. Repetition brings mastery; let’s do another. Question 4 You have a bag with 10 pebbles. 4 are red, 6 are blue. You randomly select 3 pebbles in one handful. What is the probability that exactly two of the pebbles are red? Based on what we know so far, this is a bit of a trick question. We’ve done sampling with and without replacement, but never conceptualized what it means to grab a handful of rocks out of a bag. As humans, it’s common for us to give order to things, and to conceptualize things based on that order. I’m going to clean the house, which means I’m going to clean the living room, bathroom, kitchen, bedroom, my shrine to Rihanna, and the dining room. Simply by listing those things in that order, we naturally decide that the plan must be to clean the things in that order, but cleaning in any other order would be the same result. In some sampling without replacement problems, we care about order. If we take black and white rocks out of a bag, we might decide that the sequence white, black, white, black should be counted as a difference result than the sequence white, white, black, black. Then again, we might not care about the distinction. It depends on the context of the question. When we grab a handful of pebbles out of the bag, we are sampling without replacement, without regard to the order of the sequence. When we grab three stones, we’re not sure which stones we’ll get, and there’s no concept of order, but we do know we won’t get the same stone twice. “n choose k” Also does not care about the order of the sequence, only that the elements in the sequence are mutually exclusive. That is, you can’t pick the same stone twice. This can be conceptualized as choosing the first person then the next two people (like in our committee example), or it can be conceptualized from back to front (like if you were counting how many ways the winners could be arranged in a race when you know some number already lost), or you can disregard the concept of order entirely and just grab a handful of pebbles out of the bag. The concept of order can be helpful in rationalizing a problem, but because “n choose k” is adjusted to account for how many orderings there are of k things, it doesn’t care about order at all. Recall the whole point of n choose k. We want to know how many committees of k people can be chosen out of n people. The same committee, chosen in a different order, is still the same committee. Recall that we derived the middle definition of the function by representing the top as how many possible k sequences could be constructed out of the n options, then we divided by the number of ways a sequence of k things can be written. Thus, we adjusted our count to compensate for the fact that we had several versions of each committee, but in different orders. Source: Introduction to Probability by Jessica Hwang and Joseph K. Blitzstein So, we can happily apply “n choose k” to grabbing three pebbles out of a bag, or we could re-frame the problem conceptually as choosing three stones one by one, then employ “n choose k” in that context, or we could get weird and imagine choosing the last stone, the middle stone, then the first stone. The math would be the same regardless of how you conceptualize the problem. There are examples where you can’t disregard order, like in environments where choices are not mutually independent. Choosing to buy a drink, then a straw, might not be mutually independent choices (you probably bought the straw because you bought the drink), so we would probably want to consider their order in some way. We’ll discuss those types of problems later. In this problem, choosing one pebble doesn’t make you any more or less likely to choose some other pebble in particular, so we can disregard order. Anywho, back to the task at hand. To calculate the possibility of choosing exactly two red marbles and one blue one, which we’ll call event A, we can use the naive definition of probability, p_naive(A) = |A|/|S|. So, like usual, we need to count out how many results are in event A, and in the space S. |S| is straightforward, provided you understand the idea that n choose k doesn’t care about order. If you want to select 3 pebbles out of the possible 10, there are 10 choose 3 possible ways to do it. So, |S| = 120 . Now we need to count how many ways we could choose two red pebbles and one blue pebble. In other words, we need to figure out how many ways we could choose two red pebbles out of the four red pebbles in the bag, and combine that with the number of ways we could select one blue pebble out of the possible six blue pebbles. We can calculate each of those independently fairly easily. There are 4 choose 2 ways to choose the two red pebbles we need out of the four in the bag, and there are 6 choose 1 ways to choose the remaining blue pebble. 4 choose 2, and 6 choose 1, respectively. To calculate |A|, we need to calculate the total number of ways two red pebbles could be chosen in our handful of three. To do that, we can use the multiplication rule we discussed previously. There are 6 ways to choose the two red pebbles, and 6 ways to choose one blue pebble, so there are 6 * 6 = 36 possible ways to choose exactly two red pebbles in a handful of three, in this scenario. Thus, |A| = 36. If you’re like me, this might make logical sense, but not feel intuitive. We’re going for a thorough understanding here, so “I’m following you” isn’t sufficient. If you’re still confused, here’s a description that may be slightly more intuitive. Keep in mind, in figuring out the size of the event A, we’re counting all the possible ways we could have a set of three pebbles with two red stones and one blue stone. We don’t care about order, there isn’t even a concept of order implicit to this problem, but there are a certain number of red and blue stones in the bag. If we had 10,000 blue stones, and 2 red stones, it would be very unlikely for us to get our two red one blue combo, because pulling out the 2 red stones would be very unlikely. Also, if we had 10,000 red stones and 1 blue stone, it would also be unlikely for us to get a handful consisting of two red and one blue. Regardless of the approach, intuitively, we need to account for how many stones of each type there are. Recall, a good way to do that is by assigning each stone in the bag with a label. If we had red pebble 1, 2, 3, … 10,000, then there are 10,000 different ways we can pull out a red pebble. Regardless of the sequence we’re going for, this is also useful to keep in mind; no one stone is more likely to be pulled than any other stone. If we had 10,000 red stones and 1 blue stone, you would be 10,000x more likely to to pull out a red stone. This isn’t because one of the red stones is very likely to be pulled, but because there are more red stones than blue stones. It is equally likely that you would pull the blue stone as it would be for you to pull red stone #6, for instance. Every stone is just as likely as any other. The event of pulling a red stone is more likely than the event of pulling out a blue stone because the event of pulling out a red stone consists of 10,000 possible results, and the event of pulling out a blue stone only consists of the one. By the same logic, if we go back to pulling out a handful of three stones, any handful is just as likely as any other. You are just as likely to pull stones #1, #3, #5 as you are to pull stones #1, #2, #3, for instance. The reason you are more likely to pull some collection of red and blue stones is because there are multiple red and blue stones. Because any collection of individual stones is equally likely, we can calculate the probability of pulling two red stones and one blue stone by finding the percentage of winning handfuls there are, which is what |A| / |S| represents. This idea of equally likely things is super important, and we’ll revisit it later in the article. Counting |S| as 10 choose 3 hopefully feels fairly intuitive. We care about unordered sets, not ordered sequences, and n choose k allows us to count how many ordered sequences of k things there are, divided by how many ways there are to arrange k things, thus giving us how many unique unordered sets of 3 can be created from the 10 pebbles. Counting |A| is more intuitively complicated, but keep in mind what it represents; the total number of possible sets of pebbles #1, #2, #3, #4, #5, #6, #7, #8, #9, #10 that would have two red and one blue pebble. It stands to reason that the total number of possible 2 red and 1 blue sets would be equal to the number of 2 red sets possible times the number of 1 blue sets possible. Imagine if we could lay out all the ways we could get 2 red pebbles vertically, and all the ways we could get 1 blue pebble horizontally. The height of that square, times the width, would be the number of possible ways to get 2 red squares and 1 blue square. That’s why the multiplication rule works in this scenario. Recall, in a previous example, we combined the number of ways two things could be combined, with the number of selections for another thing, to get the total number of possible selected sequences. We can use a similar intuition to combine the number of ways to get two red stones, times the number of ways to get one blue stone, to get us the number of ways we can get two red and one blue stone. So, the question then becomes, how many ways are there to get 2 red pebbles, and how many ways are there to get 1 blue pebble. The second question is trivial to answer; there are 6 blue pebbles, so there are 6 ways to select a blue pebble from the bag. I think the only conceptual hang-up here might be something along the lines of “but wait, there are other pebbles, red ones!” But we don’t care about those in this part of the answer. We’re counting how many ways there are to create a winning hand, which means calculating how many ways there are to get a blue pebble. Similarly, in counting how many ways there are to get two red pebbles, we don’t care that there are blue pebbles in the bag. We account for that in other parts of the answer. To help us count the number of winning hands, we need to count how many ways we can get two red pebbles. When the context is properly framed, the answer is obvious; to get 2 red pebbles out of the 4 that exist in the bag, there are 4 choose 2 ways to do it, which is equal to 6. Recall, 4 choose 2 is equal to 6 So, we’ve counted how many ways there are to get 1 blue pebble, 6 different ways, and we’ve counted how many ways there are to get the two red pebbles, 6 different ways. We can multiply those together, and conclude that there are 6 x 6 = 36 possible winning handfuls of pebbles. That’s our |A|, the size of the event A in which we have a winning hand. In probability, it’s easy to make mistakes. Not because the math is hard, but because the concepts can be slippery, and misconceptions are common. It’s worth the time to try to really think through a problem. Regardless, we’ve established that |A| = 36, and earlier we established that |S| = 120. So, the probability of a handful of pebbles having two red and one blue in this scenario is P_naive(A) = |A|/|S| = 36/120 = 0.3 = 30%. n choose k is used in a huge number of disciplines; mastering all its subtle uses is outside of the scope of this particular article. Rest assured, though, I will be reviewing it often in future articles on probability. For now, I want to cover a few more topics before we wrap up. Complete and Partial Permutations We’ve already covered the likely suspects of combinatorics from a probability perspective, but I wanted to cover a few more dangling concepts. A “permutation” is a way you can organize things. If we have the set of numbers {1, 2, 3}, they could be represented as the ordered sequence (1,2,3), or (3,2,1), for instance. Each of these would be called a “permutation” of the set {1, 2, 3}. Permutation can also be used as a verb, “Permute” or “permuted”, which means to generate all the permutations possible from a set of numbers. The numbers 1, 2, and 3 can be “permuted ” in the following ways: (1, 2, 3) (1, 3, 2) (2, 1, 3) (2, 3, 1) (3, 1, 2) (3, 2, 1) We’ve encountered this under another name: sampling without replacement. When you take pebbles out of a bag, one by one, without replacement, you are making an ordered sequence out of a set of possible outcomes. Finding the number of possibilities in a sampling without replacement problem is the same as finding the number of permutations of that set. As we previously discussed, the number of results you can get by sampling all pebbles out of a bag without replacement is n! (n x (n-1) x (n-2) x … x 1). Counting the number of permutations is the same thing, and thus has the same mathematical definition. There’s another type of permutation, called a partial permutation, which we’ve also discussed; It’s how to make some number of ordered sequences that is smaller than the total number of selections available. For instance, we can find the number of partial permutations of length 2 which we can generate out of our set of choices {1, 2, 3}: (1,2) (1,3) (2,1) (2,3) (3,1) (3,2) Interestingly, there are six of them; the same number as the number of full permutations. That’s a complete coincidence, usually the number is different. There are numerous ways partial permutations are notated in math several different and valid ways partial permutations are notated. The most common is P(n,k), where P denotes “permutation”. Naturally, this can get confusing as we already have a very important “P”, which denotes probability. Thus, the other notations are sometimes used. We’ve discussed how to calculate a partial permutation. If you have 20 things, and you want to calculate the number of ways you could select ordered sequences of length 3 from that set, it would be 20 x 19 x 18 = 6,640 possible ways. In other words, there are 6,640 partial permutations of length 3 that are possible when selecting from a space of 20 possible events. If you look at math textbooks you might see the following formula used to describe partial permutations: The formula for partial permutation of n things into ordered sequences of length k. This is calculating the number of partial permutations, but by way of a slightly different conceptual path than we’ve discussed. It uses the idea of adjusting for overcounting. Let’s imagine we had the set {1,2,3,4,5}. Let’s generate a single permutation, and a single partial permutation of this sequence. Let’s use a partial permutation of length 2 for this example. permutation: (5,1,2,4,3) length-2 partial permutation: (5,1) As you can see, in this example, the partial permutation of length 2 is a subset of the full permutation. You can re-frame this question. Instead of asking “how many partial permutations are there of length 2 ”, you could say “how many full permutations exist which start with (5,1)”. If you took 5 and 1 out of the set, you would have these remaining numbers: {2,3,4} And there are 3! ways of organizing these three numbers into a sequence (a.k.a, 3! permutations of this set). Thus, there are 3! ways these numbers can be tacked onto (5,1) that would result in a unique sequence. That’s not just the case for the sequence of (5,1); for any sub-sequence of length 2, there are 3! permutations that will start with that sequence in the space of all possible full-length permutations. Thus, to answer the question of “how many partial-permutations of length 2 are possible”, we can count the total number of full permutations possible (5!) and correct that by dividing it for the fact that there are 3! (n-k)!) full length permutations for each partial permutation. Thus, the mathematical definition of the number of partial permutations: Let’s cement our understanding of the idea with a question. Question 5 You have 5 distinct books labeled A, B, C, D, and E. You randomly arrange 3 of the books on a shelf, in order. What is the probability that book A is first and book B is second? We actually have the ability to solve this problem without the concept of a permutation. We solved a very similar problem in Question 3, where we tried to find the number of ways Alice and Bob might sit next to each other. Partial permutations are simply a named idea which we can use to help us solve this type of problem with less brainpower. Recall the naive definition of probability; P_naive(A) = |A|/|S|. We simply need to calculate the number of ways the event A can happen (book A is first and book B is second in the three spots), divided by the size of the space of total possibilities S. Counting |A| is super simple; we know A and B are in spots 1 and 2, so we just need to figure out which books can be in spot 3. There are three remaining books, C, D, and E, so there are three possible sequences where A and B are in spots 1 and 2. thus |A| = 3. Counting |S| is a bit more complicated, but with a knowledge of partial permutations, we don’t need to think through the logic from bedrock. We have books A, B, C, D, and E; 5 possible choices. We need to figure out how many unique ordered sequences can be created by putting them into 3 spots. That’s the textbook definition of a partial permutation where n=5 and k=3. Thus, we can count |S| by using the partial permutation formula. So, |A|=3 and |S|=60 . That means P_naive(A) = 3/60 = 0.05 = 5%. Complements and Derangements Another important concept I want to cover is the idea of a complement. In math speak, a complement of an event is all of the possible results that do not fall within that event, but are in the sample space. If you have a rock-solid intuition of an “event” as a set of possible outcomes this should be intuitive, but the verbiage of probability has a way of making simple concepts confusing. Using an example to hopefully cement the idea of a complement; if we had a bag of pebbles, we could define event A as if we selected pebbles #1, #2, or #3. The “complement” of A would be all of the pebbles that are not in event A, so pebbles #4, #5, and #6. If A is a set, then the complement (denoted as A superscript c) is all other possible results in the sample space. Complements are incredibly useful in both combinatorics and probability. It’s sometimes easier to calculate the complement of some event and the size of the possible space, rather than the event itself. For example: Question 6 You take five distinct books off a shelf, shuffle them, then put them back on the shelf. What is the probability that at least one book is in its original position? You could try to count how many ways one book could be in it’s correct spot, then count how many ways two books could be in the same spot, then three, then four, etc. This is possible, but difficult. Instead, we can solve this problem much more easily by leveraging a simple fact: The number of ways the books can be arranged on the shelf is equal to the number of ways they can be arranged where at least one is in the correct spot, plus the number of ways they can be arranged where no books are in the correct spot. Doing some linguistic algebra here, you could also say the number of possibilities where one book is in the correct spot is equal to the number of possibilities in total, minus the number of possibilities where no books are in the correct spot. If we can calculate how many arrangements there are, and how many arrangements do not satisfy the criteria, we can use that information to solve the problem by way of complements. The size of the space is simply 5!, classic permutation/sampling without replacement problem. Calculating the number of ways of organizing the books so that none of them are in the same spot… Well… I said it’s easier, but it’s not easy. The idea of shuffling things so that no items are in the same spot is called a “derangement”. A derangement, by definition, is a permutation where no items end up in the same spot. The number of derangements for n things is written as !n, notice how the exclamation point is in front of, rather than behind the n. n! represents the number of permutations, !n represents the number of derangements. There’s a formula for the number of derangements, which we could use without any understanding of how it worked if we really wanted to. The formula for the number of deranagements of a sequence of `n` things. But, this is IAEE, and derangements are a common topic in combinatorics and probability, so we need to build an intuitive understanding of why this works. The whole idea of a derangement !n is to count all the ways to arrange a sequence of n things so none of the things end up in the same spot. We can start with the number of permutations, n!, which is all of the ways we can list out the sequence. We can then think of this as an overcounting problem. The number of permutations n! includes examples where one of the books is in the right spot, so we want to then subtract from that the number of examples where one of the books is in the right spot. If we want to count the number of ways book 1 can be in spot 1, that would be equal to the number of ways we could arrange all the other books, (n-1)!. One might think this applies to all n books, which means there are n*(n-1)! ways that at least one book could be in the same spot it started. If we subtract that from the total number of permutations, we would account for all of the ways one book being in the right spot is represented in n! n choose 1 is equal to n, so this expression is n!-n(n-1)! but this is slightly incorrect. See, imagine a case where there are two books in the same spot they started in. That case would be counted in both of the books (n-1)!‘s, meaning we over subtracted and accounted for that case twice. We can figure out how many groups of two people there are using n choose 2, and multiply that by how many sequences there are to add them back in. The number of permutations, minus the ways there can be an overlap, plus a correction for over-subtracting instances where there are two books in the correct spot. But wait, what about permutations where there are three books in the correct spot? Let’s imagine one, where books 1, 2, and 3 are all in the correct spot. We’re trying to count the number of derangements, so a case where books 1, 2 and 3 are in the right place should not exist in the count. Well, it’s counted once in n!, which it shouldn’t be. It’s then subtracted three times in the next term; once for the (n-1)! of book 1, of book 2, and of book 3. Keeping in mind it was over counted by 1 previously, means this sequence has now been excessively compensated for by -2. books 1, 2, and 3 would be counted in situations where two books are in the right spot. In fact, it would be included three times; for books 1 and 2 being in the right spot for books 2 and 3 being in the right spot for books 1 and 3 being in the right spot So we add +3 to the count for this sequence. Because we under counted by -2 from the previous step, that means in total, across all terms in this expression, sequences of three are over counted by 1, so we need to compensate for that by adding a term which subtracts each sequence of three books in the right spot. You may be able to read between the lines. This would over compensate correct sequences of 4, then over compensate the other direction for sequences of length 5, etc. etc. Thus the general form of the expression is defined as or, with some factoring out of n choose k, we get the simpler expression If you’re not familiar with summation notation, I won’t bore you with the details; our original representation of the equation is more intuitive, which is the same thing just rolled out. The formula for the number of deranagements of a sequence of `n` things. You can see we’re starting with n! (which is multiplied by the 1 in the parentheses), then subtracting n!/1!, then adding n!/2!, subtracting n!/3!, and on and so forth until we’ve done that n times. This is the general expression which allows you to do the same intuition we’ve been discussing, but for any sequence of length n. You might not have a complete understanding of derangements, but that’s ok. These things take practice, and you can review the concept as it comes up. I think what’s most important is that you know what a derangement is, and understand the vague idea of how they’re counted. So, back to the task at hand, we can now calculate the number of derangements of our five books, it’s So the number of ways none of the books will end up in their original position is 44. Recall that the point of the problem is to count how many ways at least one book would end up in its original position. Well, we know how many sequences are possible, |S|=5!=120, and we know how many sequences there are which don’t have any books in the original spot, !5=44, so the number of ways at least one book will be in the original spot is simply |A| = 5! — !5 = 120-44 = 76. Now that we know the number of winning sequences possible, and the size of the whole sample space, we can calculate the probability as P_naive(A) = |A|/|S| = 76/120 = 0.63 = 63%. Distinguishable and Indistinguishable Things I’ve mentioned a few times that it’s a good policy to assign labels. If you have 10 people, and you need to make a committee of 3, it’s a good idea to assign names or numbers to each of the people so you correctly conceptualize the problem. In some situations, identifying what is distinguishable is easy. We have 10 people, give them a name. Sometimes assigning labels is a bit more counterintuitive. Take the following example: Question 7 You have 5 balls and 3 boxes. In how many different ways can you distribute the balls among the boxes? You can decide to label the balls and sample the boxes which the ball is in, you can decide to label the boxes and sample the balls that land in them. There are various ways you can wrap your head around this problem. We can look at a slightly different wording of the problem, which has a big impact: Question 8 You have 5 identical balls and 3 boxes. In how many different ways can you distribute the balls among the boxes? The word “identical” means that results should be treated as the same if ball A ended up in box 1 , or if ball B ended up in box 1. By saying the balls are identical, we’re saying we don’t care which ball ends up in a box, so long as a ball ends up in the box. This changes how we would count. This type of problem comes up occasionally in probability, and requires a slightly different mental model to count effectively. When you have buckets of indistinguishable things, often the “stars and bars” approach is used. We can model our problem as a series of stars and bars. The bars represent where we transition from bin 1 to bin 2, or bin 2 to bin 3. Here’s an example: * * | * | * * In this example of the notation, we’re saying two balls ended up in the first bin, one ended up in the second, and two ended up in the third. In another example: | | * * * * * We’re saying no balls ended up in the first and second buckets, and they all ended up in the third bucket. Using the stars and bars as a modeling tool is useful because it allows you to visualize how these indistinguishable things might end up in distinguishable places. When using the stars and bars, the next intuition becomes slightly more obvious; To count how many different ways our identical balls can be distributed across bins, we just need to count how many ways we can position our bars within the stars. In this example, there are 7 total positions the bars can be, which is the total number of stars plus the number of bars (the number of bars being the number of buckets minus 1, so 5 + 3–1). * * | * | * * 1 2 3 4 5 6 7 We need to choose which of these 7 locations one of our two bars falls into. Here, the bars are treated identically; there is no bar “a” and bar “b”, the bar on the left always means the transition from the first bin to the second, and the second bar always means the transition from the second to the third bin. So, to count how many distributions of balls we can have within boxes, we need to choose how many unordered sets of two numbers can be chosen from the set {1, 2, 3, 4, 5, 6, 7}. As you may recall, that’s n choose k. So, the number of possible distributions of five identical balls in three boxes is 21. I saved this for the end of the article because the way it relates with probability is kind of sneaky. In this problem, we’re not coming up with any probabilistic claims about the sequences, we’re simply counting how many possible distributions of balls within boxes are possible. Just because we counted 21 things doesn’t mean each of those 21 things is equally likely. One could say that, if there are 21 possible distributions, each of those possibilities would have a 1/21 chance of happening. One could also say that, if there are eight planets in the solar system, and one of them has life, then each of them has a 1/8 chance of having life. Just because you have some number of things doesn’t mean they have equal probability. Recall the naive definition of probability we’ve been using A naive definition of probability. Source: Introduction to Probability by Jessica Hwang and Joseph K. Blitzstein We’ll discuss this at more length in future installments, but a big reason this is naive is because it assumes that all of the possible outcomes are equally probable, which isn’t always the case. We were told we have 5 identical balls, which was important for counting how many distributions of those identical balls are possible in the three boxes. However, if we tried to calculate the probability of one distribution or another, assuming the balls might land in each box with equal probability, it’s obvious that a distribution like this: | | * * * * * is less likely than a distribution like this * * | * | * * There’s only one way to create the first distribution, but there are 30 ways to create the second For the second sequences, two of the five balls land in the first box, and one of the remaining balls lands in the second box. Multiplying these together yields the total number of possible ways this distribution could be created. any specific distribution of labeled balls in boxes is just as probable as any other, but there are 30 of them that can represent * * | * | * *, while only one can represent | | * * * * *. Sometimes the verbiage of probability can make understanding it difficult. The stars and bars approach is very uncommon in probability, because it’s very uncommon that each of the distributions is actually equally likely. Typically, when trying to make probabilistic claims, it’s best to treat each thing you’re selecting as unique, even if it isn’t. For the sake of thoroughness, there is another flavor of this type of problem, where both the balls and boxes are considered identical. Question 9 You have 5 identical balls and 3 identical boxes. In how many different ways can you distribute the balls among the boxes? The math for this is a bit more complex, and it’s application is even less common than stars and bars, so we won’t go over this in depth. To be brief, though, this is called a “partitioning” problem. We want to figure out how many ways there are to divide up the 5 balls. In this example, there are 5 ways 5 4 + 1 3 + 2 3 + 1 + 1 2 + 2 + 1 Again, if you’re chucking balls into boxes, then each of these partitions is not equally likely. While this falls within “combinatorics”, the study of counting things, it’s not often relevant to probability. I think there’s a general lesson we can take away from this section. Just because you can count things doesn’t mean each thing is equally probable. When making probabilistic claims, it’s useful to label things, and to spend some time making sure the things you’re counting are equally likely. That will make coming to terms about probabilistic claims easier. We’ll explore that general theme in future articles. For now, I’d like to cover one more topic before we close out our discussion of combinatorics for probability. Counting by Category We’ve already discussed this topic, but I want to take some time to cement it as a concept. Recall the idea of the multiplication rule. If you have some number of possibilities of one thing, and some number of possibilities of another thing, you can multiply those numbers together to get the total number of possibilities. Recall the multiplication rule Based on the previous section, you might have some suspicions about if each of those combinations is equally likely, which is a great intuition. When making probabilistic claims in the real world, that’s very important to keep in consideration. Assuming certain combinations aren’t especially privileged, there’s a useful way of conceptualizing these problems in complex situations, which is to use “counting by category”. Essentially, you can divide the problem in to categories of things, then combine the counts of the categories together. Rather than be theoretical, let’s use an example. Question 10 You draw two cards in order from a standard 52-card deck without replacement. What is the probability that the first card is a heart, and the second card is a face card (Jack, Queen, or King of any suit)? This is a sampling without replacement problem, where we care about the order of the outputs. Because every pair is equally likely, we can use the naive definition of probability P(A)=|A|/|S|. Counting the size of the possible space is easy, there are 52 possible first choices, and 51 possible second choices, so |S| = 52*51 = 2652. Now we need to count the size of space |A|, which is a bit more complicated. We can start to make ground on this problem by breaking it into two parts: Counting the number of possibilities for the first card, and counting the number of possibilities for the second card. For those of you who don’t play a lot of cards, there are 13 heart cards in a standard deck (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, K, Q), so there are 13 choices for the first card. What’s tricky, though, is that some of those cards are face cards (the Jack, King, and Queen). If we select those cards in the first spot, then we can’t also select those in the second spot as well. We need to account for that when counting. To do that, we can divide our count into two subclasses: one where the category of the first card is a numbered heart card, and one where the category of the first card is a face heart card. If the first card is a numbered card, then there are 12 possible face cards to pick (a Jack, King, or Queen for each of the four suits, clubs, diamonds, spades, and hearts). So, the number of possibilities of choosing a winning hand, assuming the first card was a numbered card, is number_of_numbered_heart_cards * number_of_remaining_face_cards = 10 * 12 = 120 If the first card selected is a face card (the Jack, king, or queen of hearts), then there is one less face card we can select for our second card, so 11 cards. That means, the number of possibilities of choosing a winning hand, assuming the first card was a face card, is number_of_face_heart_cards * number_of_remaining_face_cards = 3 * 11 = 33 So, we’ve counted all the possibilities of winning if the first selection was and was not of the class “face card”. Because each of those classes ultimately considers all possibilities, we can add them together to get our final result. number of winning posabilites = number of wining possibilities if the first card is not a face card + number of winning possibilities if the first card is a face card = 120 + 33 = 153 Thus, the size of the solution space |A| = 153 , and therefore P(A) = |A|/|S| = 153/2652 = ~0.06 = ~6%. This type of problem can come up a lot. For instance, if you have a talent pool where some people are engineers, some people are designers, and some people are both, and you’re trying to make probabilistic claims about creating a team, you’ll need to account for the fact that there may be people that occupy both the class “designer” and “engineer”, like how we accounted for cards that were both face cards and hearts. You might also be dealing with scheduling problems, where certain types of meetings need to occupy certain time slots, but there’s overlap between the classes of meeting types. It’s also used a lot in medicine, where different classes of people (those that do or do not have a disease, for instance) are used to assign probabilistic claims to the rate of effectiveness of a drug. Getting good at counting by category takes experience and practice, but I think a valuable first step is knowing that the approach exists in the first place. With that, I think I’ve covered everything I wanted to in this article. Conclusion Hopefully you have a stronger intuition of probability just by reading this article. We have yet to dive into the subtleties of probability itself, but for many problems simply being good at counting, and counting the right things, is more than sufficient. Now that we have a thorough understanding of counting, we can explore probability itself to a greater degree of depth in future installments. Stay tuned, and consider subscribing to intuitively and exhaustively explained. If this type of content is particularly resonant with you, consider becoming a paid subscriber if you’re not already. The generous patronage of subscribers is what allows me to invest the time to create such comprehensive material. Don’t Be Distracted In 2026 Daniel Warfield — Thu, 01 Jan 2026 15:00:47 GMT “Freedom In Constraint” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained. The content of this article consists of tools and tricks I learned to make my life happier and healthier. You won’t find sponsored links in this article, you won’t find erroneous multipliers like “10x your productivity”, you won’t find AI tools, you won’t find ways to hustle harder, or grind faster. The goal is to empower you to strike a balance between using the internet to enrich your career and life, while simultaneously minimizing the risk of addiction, distraction, and negative influence. The goal of this article is to create a substantive change in your life, and it centers around three key goals: Minimizing Distraction Having More Productive Downtime (not just measured in terms of career productivity, but in fulfillment as well) Maintaining Long-Term Goals We’ll start with minimizing distraction. Subscribe To Intuitively and Exhaustively Explained. Share this article with friends and colleagues. Share Fewer Distractions This is the hardest and most important part to get right. It doesn’t matter if you’re smart, dedicated, have good connections, have great friends, if you’re rich, whatever. If a large part of your time is spent mindlessly scrolling on your phone, you will never achieve your personal or professional goals. “Lost time is never found again.” — Benjamin Franklin The issue is, there are a lot of distractions in life and, frustratingly, those distractions are increasingly interwoven into the necessities of life. I can’t just leave my phone outside my office; I need it for two-factor authentication. I can’t just avoid the infinite reel; I need to watch YouTube videos to learn new stuff. The goal of this section is to get what we want out of technology, while not letting the tendrils of attention-seeking design control our minds. There are both tools and habits we can use to help us in that endeavour. We’ll kick it off with one of my favorite tools. Fewer Distractions — Minimalist Phone Your phone is your greatest source of distraction. Americans average around 1/4 of their waking life on their phones. Don’t let this fact pass you by; this is an existentially concerning waste of human life. That’s 20 years of your life slipping away into the void. The first trick on this list is to install Minimalist Phone, which helps you reclaim some of that lost time. That might sound like a sales pitch, but I earnestly mean that. I’m not affiliated with Minimalist Phone in any way other than as an avid user. Apparently available on iOS. For a long time it was only available on Android. source I swung by the T-Mobile store to get a new phone last week; I installed Minimalist Phone on it before I walked out the door. For the last four years, Minimalist Phone has been one of the most important investments I have ever made to myself. The most obvious thing it does is replace your home screen. No more buttons, just the names of your favorite apps, the time, date, and a ring that indicates your battery level. The Minimalist Phone Home Screen. Source. If you swipe to the right, you can see a list of all of your apps installed on the device, in alphabetical order. You can also type the name of an app to search for it. The Minimalist Phone App Browser. Source. I remember the moment I installed Minimalist Phone. I was stressed out in engineering school. I had too many classes to study for, and was co-founding an AI company at the same time. I didn’t have time to waste. Still, I would find myself scrolling on Instagram when I didn’t even remember clicking the app. I was using my phone as a form of escapism, getting short-term comfort, but making my stress worse by shortening my days. Each time I would unconsciously click an app, it would result in 10 minutes, 15 minutes, an hour, gone. Each time I clicked an app, all of my deadlines got that much closer. “How did it get so late so soon?” — Dr. Seuss When you need to search for apps by word, rather than impulse click their shiny little icons, it drastically changes how you interact with your phone. Minimalist Phone has a variety of additional tools besides the interface. You can set certain apps on timers, you can set reminders to keep you honest about how long you’re on an app, and you can straight-up block certain apps based on a schedule. Different functions of Minimalist Phone. Source Perhaps more importantly, Minimalist Phone allows you to engage in a new philosophy around technology use; one of intentionality. This app was the start of my journey in being intentional about my device use, and I highly recommend it as a start for you as well. Fewer Distractions — Content Filters I’m a young man, but old in internet years. I’m old enough to remember a time before I had a smartphone, when high-quality answers were easier to find on Google, and when YouTube was in its prime. I can’t explain how much I used to love the recommendation system of YouTube. in the 2010s, the “YouTube rabbit hole” was something positive people would obsess over. This was the era of educational YouTube, and it felt like you could get a master’s degree by just clicking on the top recommended videos. I love YouTube; it’s important not to be too pessimistic about a good thing. However, I have noticed the platform has gotten more distracting for me. Maybe it’s because of shorts, maybe it’s because the recommendation system has changed, maybe it’s because I have a degree in mechanical engineering, minors in computer science and math, a year of writing my own technical blog, and 8 years of experience under my belt. A random YouTube video simply has less to offer me than it did a decade ago. Regardless of the reason, I started having a palpable problem as my career progressed. YouTube stopped being productive. I would need to research something, perhaps a dry topic that I wanted to learn but wasn’t exactly gripping. I would find a YouTube video, then I would be taken off track by recommendations of more engaging content. I could will myself into keeping on track, but it was hard. The recommendations were always there, every time I opened the home screen, every time a video ended. So, I downloaded Unhook. Again, unsponsored. Unhook is a tool that allows you to edit the elements that appear within YouTube. You can turn your home screen from this: My home screen. I Love Any Austin, which is part of the problem. to this: A blank screen with no recommendations. Perfect, if you want to be in control of your time. When you log into YouTube, you decide what you’re watching, not the recommendation system. After you search for a topic, a list of high-quality responses comes up, as you would expect. the result of searching “Differential Calculus” While I’m watching a series of videos, it’s easy to be distracted. Avoiding a picture of Will Ferrell with a bird on his shoulder can be easier said than done. Remember, there is a cost to watching a random video on Will Ferrell. I’m all for entertainment, but if I click that picture of Will Ferrell, then click something else, then click something else, an hour can go by. That’s an hour I didn’t spend with my girlfriend, an hour I didn’t spend working, an hour I didn’t spend writing a new article, an hour I didn’t spend working out, an hour I didn’t spend enjoying nature. An hour I didn’t spend doing the things that give my life purpose and meaning. With Unhook, you can just turn the recommendations off. Youtube, with sidebar recommendations off In fact, you can turn off pretty much everything. No comments, no notifications, nothing but the video. Unhook allows you to granularly control what you want to see, and what you don’t, to a fantastic degree. This simple ability has allowed me to reclaim my love of YouTube, and all of the new and fantastic content people are putting out, while avoiding some of the more modern and addictive aspects I don’t like. After discovering Unhook, I decided to look for similar options for other websites. News Feed Eradicator is another good example. the News Feed Eradicator extension It removes feeds on a few popular websites. Sometimes I hop on LinkedIn to check if I’ve gotten a message from a colleague, and get sucked into watching videos of robots. With News Feed Eradicator, I get a nifty little quote instead. I could go on, but I don’t think it’s necessary. Once you know that content filters exist, and you start using them, you’ll find that they’re an indispensable tool in keeping your life in your control. Well, provided you do one little tiny thing… Fewer Distractions — Stop Using All Forms of Social Media Time for some tough love. If you don’t do this, you’re not serious about having a healthier or more productive life. A very small number of people actually get something out of social media. The vast majority of people are wasting their time. I’m on Reddit to find answers to specific questions, I’m on Medium to write content, I’m on Slack to talk with colleagues, I’m on LinkedIn to talk with colleagues, and I’m on YouTube to watch tutorials on specific things. I use these for a purpose. Frankly, if you think you’re getting something out of social media, and you can’t name that specific purpose, you’re probably being delusional If you don’t have an actual, real purpose, there is no reason to be on any website that even vaguely resembles social media or an infinite content reel. Unless you have a business that requires it, there is no reason to be on Instagram, TikTok, or Facebook. You might think it’s to keep up with your family. It’s not. You’re keeping up with a visage of your family which Facebook dangles in front of you to keep you addicted. You might think it’s about learning cooking recipes. It’s not. Instagram is using your passions to give you a sufficiently plausible excuse to avoid the fact that you’re wasting a sizable percentage of your life on their platform. You can not do anything until you earnestly decide to take mastery of your own time. Sometimes that means making hard decisions. Tough love over. I included it because someone has to hear it. Now that, hopefully, you have earnestly decided to work on mastering your own time (which is a skill one hones, not a milestone one reaches), you can consider the remainder of the article. More Productive Downtime Productivity is a funny word. I don’t mean more productive strictly from a career sense, but rather more substantive. I think a lot of people think of their lives as things they’re paid to think about, and things they are at liberty to think about when not paid. I work, then I go home and watch Netflix. It becomes a reflex. I know, it happens to me all the time. You have a show you’ve been watching for a while, and it becomes a ritual. Every night you watch just one or two episodes. An hour or two. An hour or two, every night. You need to and deserve to relax, enjoying a TV show is good in moderation, and that moderation is a function of your life and goals. It might be fine, or it might not be fine. Let’s answer that with a simple question. Is there anything you’ve been wanting to do in your free time, but you just can’t seem to find the time or energy? If the answer is yes, which it probably is if you’re a normal modern person, then you’re letting your habits of leisure dictate too much of your free time. You would have the time, if you were more in control of your time. I can’t speak to anyone’s experiences but my own, but I can confidently say this: I answer no to the question above. I do pretty much everything I want to do, and I think it started from transitioning to reading Audiobooks More Productive Downtime — Audible, Libby, and Audiobooks in General I can’t stress this enough: the goal of downtime is not constant career growth. Pick up a fantasy book, a historical book, a thriller. I’ve been obsessed with Ursula K. LeGuin for the last few years now. When I read with my eyeballs, it’s slow and tiring. I do so much writing in my daily life (between writing and software development), I’m not particularly interested in sitting around and reading some more for fun. A decent audiobook totally fixes that problem. When I suggest audiobooks to a person, I often get the response that they’re “bad” at listening to audiobooks. I think listening to audiobooks is a skill, ljust ike reading text is a skill. The more you do it, the better you get at it. What’s great is that you can get better at it while driving, while doing laundry, while cleaning the dishes. I love cleaning dishes, because it’s a consistent time I have to listen to my audiobook. I’m not trying to be an elitist, but I find that listening to incredible books, which have taken years of one’s life to curate, is more fulfilling than listening to one-off podcasts or TV shows in the background. I learn something from listening to audiobooks. I become a better person. It really is enjoyable, and engaging, and relaxing, and a good use of my time. It also gets a bit tiring after a while, which prompts me to take a nap or just sit around and think about stuff. This is good, and a natural aspect of being a person, which I have not found to be sufficiently stimulated by other forms of entertainment. Also, it does require some engagement. Sometimes I avoid my audiobook and watch a mindless TV show, or play video games. What’s important is, that’s not a habit for me, but a reflection of what I really want. If your default is a more productive form of leisure, more relaxing forms become ever the more sweet. Here are a few of my favorite audiobooks, in no particular order. I like to bounce around genres The Lathe of Heaven — scifi/philosophy — Ursuala K Le Guin Off to Be the Wizard — scifi/comedy- Scott Meyer A Princess of Mars—scifi/adventure — Edgar Rice Burroughs Airport — Adventure Fiction — Arthur Hailey On Writing — Self Help — Stephen King Never Split the Difference — Self Help — Chris Voss, Tahl Raz Positioning — Marketing — Al Ries, Jack Trout The Roald Dahl Audio Collection — Fantasy — Roald Dahl Blott on the Landscape — Comedy — Tom Sharpe The Dark Tower Series — Fantasy — Stephen King Lawrence In Arabia — History — Scott Anderson I’ve learned a lot from each of these books, and enjoyed them to boot. I think there’s an abstraction to this general advice. Don’t be so quick to think leisure has to be all or nothing. I speak to Audiobooks because, if you find a good one, it will completely flip your perception of leisure in a heartbeat. Of course, there are other forms of leisure which might be more to your liking. I also really enjoy partner dancing (honky-tonk two step, specifically), and do that regularly. Having a few options of leisure in your back pocket that satisfy you in different ways, and require varying degrees of effort is good. Try something. Speechify I write a lot of very long and technically complex articles. 8,000 words, 10,000 words, 15,000 words. I need to research each and every one of these topics. There’s not always a YouTube video or an Audiobook. Plus, I do this on top of my day job. Do you think I sit around reading textbooks for fun all day? No. I use Speechify. Ok that sounded like an ad. I love and hate Speechify. It’s the best on the market, but it’s also kind of buggy and overpriced. I’ve thought a lot about making my own inexpensive alternative. For now I have to be honest; it’s a game-changer despite its problems. Sometimes I need to read an entire textbook to prepare for an article, or at least a handful of academic papers. Because I’ve spent so much of my life studying technology and listening to audiobooks, I can actually get a lot out of listening to an academic paper while, for instance, doing the dishes. This ability is how, at my peak, I was putting several articles out each week. I’m telling you, start listening to Audiobooks. The skill is directly applicable to being able to comprehend large amounts of information rapidly. If you buy that, I’ll share another trick: The Microsoft Edge browser has a text-to-speech function built into it. It’s not as sophisticated as Speechify, but it does allow you to do something really cool: You can listen to what you write, as you write it. I listen to everything I write on Microsoft Edge (including this article), and it makes proofreading and editing significantly faster and better. Minimize Screen Use an Hour and a Half Before Bed For me, the only exception is stuff I can fall asleep to. I love the Great British Bake Off. Still, I try to keep that to a minimum. I also write, take baths, and do skincare things. My significant other has helped me a lot with this (I struggle with unwinding), and it’s greatly improved my life. Relaxation is a skill, and if you give yourself leeway and experiment with it, you can find rituals that allow you to unwind better at the end of the day, resulting in a happier and healthier life. You need empty space to do that, though, which means putting the phone down. Continual Alignment Towards a Goal If you’re mastering control of your time, and are happy and healthy on account of relaxing more productively, you might find yourself with a bit of extra fuel in the tank, rather than less. Before you were demoralized at the end of the day with an insurmountable stack of responsibilities. Now you have more time to get some extra things done. I don’t just say that as utopian hyperbole, that really is my lived experience as a result of investing in controlling my own time. The next step is to use your newfound time of resource to do other things. New things. Move forward in your life. I think there are a billion tools to help with this, but I like really simple ones. These two are my favorites. Google Keep/ Todoist Google Keep is a little-known Google application that allows you to keep track of TODO items and notes. google keep It’s elegant, easy to use, devoid of distractions, and syncs across devices. I used to have a “5-minute list” where I would keep track of chores I wanted to get done that would take less than 5 minutes. When waiting for a meeting, for instance, I could knock it out real quick. I’ve also used Google Keep to keep track of what I want to get done this month and a few months out. I usually try to keep it to a few big goals that I want to keep myself accountable for. When the month ends, I can look back on the previous month and reflect on what I got done. It’s useful to be able to create honest projections into the future, and to be able to see your life in terms of a few months at a time, rather than a few days. Another alternative I like is Todoist Todoist Todoist is kind of like Google Keep in that it’s a list of things, but it’s designed to keep you on the ball. You can set deadlines, and if something is overdo it will go to the top of the stack. I don’t always like this; sometimes I like having a vague plan and adjusting on the fly, but when things have to get done, I often find myself making a list on Todoist and checking it every morning. Conclusion I’m sure these tricks won’t work for everyone. Honestly, they don’t even work for me sometimes. Being productive consistently is impossible. I find my life has seasons where I’m more and less productive, and I think that’s ok. That said, I do think these are some of the simplest tools and tricks I’ve incorporated in my life that have seen a significant return on investment. I hope this helps you to be more productive and happier as a result. Cheers, and Happy New Year! Docker, Kubernetes, and Helm — Intuitively and Exhaustively Explained Daniel Warfield — Thu, 11 Dec 2025 15:02:46 GMT “Helmsman” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained. In this article, we’ll explore how sophisticated backend environments for complex applications can be efficiently managed and scaled using Docker, Kubernetes, and Helm. From the highest level, Docker allows you to package your code into something called a “container”, a fully self-contained and portable unit of software that can be duplicated and run pretty much anywhere. Kubernetes allows you to manage how these containers are applied to resources at scale, coordinating them into complex, interrelated applications. Helm adds an additional layer of abstraction on top of Kubernetes by bundling these interconnected components into reusable, shareable packages called charts, making it far easier to develop and re-use complete systems. When used together, these three technologies transform backend infrastructure into a modular system of building blocks that you can mix, match, scale, and rearrange with remarkable flexibility. This power of abstraction is why virtually every major organization uses the three technologies to manage their backend infrastructure. There’s a ton of depth to this topic, which isn’t realistic to explore “exhaustively” in a single tutorial. If you build an application with Kubernetes, you’re going to have to do some googling along the way. That said, we will form a solid conceptual understanding of the three technologies by building two demo applications: one that can estimate digits of pi at scale, leveraging numerous machines in a cluster to do it, and another that exposes a chess application with a simple database and workers which run a chess engine. Both of these are simplifications of very real-world use cases that large companies use Kubernetes for on a daily basis. [init] Waiting for 10 worker pods to complete... No resources found in default namespace. [init] Workers finished: 0/10 No resources found in default namespace. [init] Workers finished: 0/10 No resources found in default namespace. [init] Workers finished: 0/10 [init] Workers finished: 2/10 [init] Workers finished: 10/10 [init] All workers completed, starting reducer. [reducer] Collecting results from: /results [reducer] inside_total = 863939569 [reducer] points_total = 1100000000 Distributed π estimate: 3.141598432727273 In this article we’ll make a system that estimates digits of pi, and a chess application that allows us to play chess against a computer, both in a manner that’s scalable and easily deployable in Kubernetes via Helm. Subscribe To Intuitively and Exhaustively Explained. Share this article with friends and colleagues. Share Who is this useful for? Anyone interested in building an application that serves large numbers of users, or processing large amounts of data, with multiple computers working together in a robust and scalable manner. How advanced is this post? This is designed to be a first exposure to containerization with Docker, orchestration with Kubernetes, and deployment with Helm. That said, the technology is complex, and I’m skipping through a lot of the application logic and focusing on the orchestration and deployment side. Thus, this article may feel a bit abstract for more junior developers. Prerequisites: Anyone can read this article and get a general understanding of the topic. Realistically, though, you should probably be somewhat comfortable with building at least simple full-stack applications to get the most out of it. The Case For Learning on a Virtual Machine While the power of these technologies exists on the cloud, I want to avoid setting up a cloud account and racking up a thousand-dollar AWS bill because we accidentally overallocated resources. We’ll be experimenting with the technology locally so we can get comfortable. As a result, following along with this tutorial might require a bit of configuration. A big reason I avoid local configuration in tutorials is because of discrepancies about how your computer is set up, vs how my computer is set up. To help mitigate those issues, I tend to use an emulator that I can use to spool up new operating systems, then tear them down again, making it very easy to start from scratch. This isn’t sponsored, but I use Parallels Desktop on Mac to do this type of stuff. When I open Parallels Desktop, I can create or access various virtual computers within my computer. I can open up Ubuntu, for instance, and I have a little Ubuntu Linux machine on my computer. If I make some changes to this computer, for instance by installing packages, and I want to try again from scratch, I don’t have to do anything complicated. I can simply go back into Parallels and create a new Ubuntu system from scratch. This is really powerful for me as a writer, because it helps me make sure I’m not forgetting to mention some setup process because I already did it on my computer a few months ago. It can be powerful for you, as a learner, because it minimizes the likelihood of something going wrong because you didn’t realize you made a change to your computer a few months ago. There are numerous emulators available on most major operating systems. Some of them are free, and some of them are paid. Whether you use an emulator or not, you should be able to follow along, though the steps to install might differ slightly based on your operating system. Setup To get started, we’re going to need to install a few things. I’ll be assuming you’re in a Linux environment like Ubuntu. If not, you might need to do some googling to find the equivalent approach for you. Setup 1) Installing Docker Engine On Mac and Windows, it’s common to install Docker Desktop, but on linux it’s a bit quirky, and it’s more common to install the Docker engine directly via the command line. Open up a terminal. First, it’s recommended to run this: sudo apt update sudo apt install ca-certificates curl gnupg This updates the list of packages your Linux machine can install, and installs a few packages that Linux will use to securely install Docker. Then run this: sudo install -m 0755 -d /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg sudo chmod a+r /etc/apt/keyrings/docker.gpg In essence, this downloads a trusted key from Docker, which can be used to secure Docker installation on your machine. This isn’t that important for us right now, but Docker being secure is very important for the big business that use it regularly, thus there’s a significant layer of protection even within docker installation. Now run this: echo \ “deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo “$VERSION_CODENAME”) stable” | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null This adds Docker’s official software repository to the package manager on your Linux computer. Once you do that, you can run this: sudo apt update sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin which actually installs Docker. You can now verify that Docker is installed successfully by running sudo docker run hello-world Which will return the following Unable to find image ‘hello-world:latest’ locally latest: Pulling from library/hello-world 198f93fd5094: Pull complete Digest: sha256:f7931603f70e13dbd844253370742c4fc4202d290c80442b2e68706d8f33ce26 Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the “hello-world” image from the Docker Hub. (arm64v8) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/ We’ll dive further into Docker throughout this tutorial. If you want a sneak peek, you can get an idea of what’s going on by reading this output. Setup 2) Installing Kubectl Kubectl is a command-line tool for communicating with and managing “Kubernetes clusters”. We’ll explore what a “cluster” in Kubernetes is later. For now, it’s enough to know that Kubectl is necessary to do stuff with Kubernetes, so we need it on our computer. This installs kubectl: curl -LO “https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl” chmod +x kubectl sudo mv kubectl /usr/local/bin/ And this verifies that it’s been installed: kubectl version --client You should get something like this: Client Version: v1.34.2 Kustomize Version: v5.7.1 If you don’t, you might need to install for a different CPU architecture, for instance, by running this: curl -LO “https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl” chmod +x kubectl sudo mv kubectl /usr/local/bin/ You can find more instructions for installing kubectl here. Setup 3) Installing MiniKube MiniKube is designed to allow you to run Kubernetes on a single machine. The whole point of Kubernetes is to run an application that’s distributed across many computers working together, but local deployment is convenient for senior devs who want to test out changes they’ve made on their computer. It’s really useful to us because it means we can play around with Kubernetes without needing to set up any cloud subscriptions or buy a bunch of hardware. Just like the last command, you might need a slightly different install based on the architecture of your CPU. For me (running on Apple silicon, which is ARM-based), this does the trick sudo apt-get install -y conntrack curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-arm64 sudo install minikube-linux-arm64 /usr/local/bin/minikube We can make sure it’s set up by running minikube start --driver=docker Which should get us an output that looks something like this, emojis and all. 😄 minikube v1.37.0 on Ubuntu 24.04 (arm64) ✨ Using the docker driver based on user configuration 💣 Exiting due to PROVIDER_DOCKER_NEWGRP: “docker version --format -:” exit status 1: permission denied while trying to connect to the docker API at unix:///var/run/docker.sock 💡 Suggestion: Add your user to the ‘docker’ group: ‘sudo usermod -aG docker $USER && newgrp docker’ 📘 Documentation: https://docs.docker.com/engine/install/linux-postinstall/ Setup 4) Installing Helm It’s hard to appreciate why Helm is necessary, given the fact that we haven’t dived into any of the other technologies yet. Basically, it will make it easier for us to manage and reuse existing Kubernetes. Installation is super easy: curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash Running this afterward helm version should result in something like this version.BuildInfo{Version:”v3.19.2”, GitCommit:”8766e718a0119851f10ddbe4577593a45fadf544”, GitTreeState:”clean”, GoVersion:”go1.24.9”} Now that we installed everything, it’s probably a good idea to get a sense of what it all does. We’ll start with Docker. Why Containerization, and Containers vs Virtual Machines Docker allows you to package your application, and everything it needs to run, into a portable and repeatable unit called a container. This general process is called “Containerization” and is important for building robust and scalable backend applications. Imagine you wanted to build a web service that converts videos to gifs in Python. You might find some library called super_video_to_gif , install it on your computer, and then build a server around that library. Great. You make it into a little GitHub repo that has all your dependencies listed out in a requierments.txt file. All you need to run your application on a computer is to do something like the following: git clone https://...myrepo cd myrepo pip install -r requierments.txt python run main.py You hop onto AWS, run those commands, and oh no. Your pip install failed. Turns out the super_video_to_gif requires Python 3.10, and the AWS machines have Python 3.8 installed. So you change your install script to update Python. That works for a while until AWS updates some defaults on their computers, and it turns out they upgraded some package that super_video_to_gif relied on, and it isn’t compatible with the new version. Etc. Hopefully you get my point. Any time you make any change, you need to debug a bunch of incompatibilities. Even worse, if AWS updates something under your nose, you might find your application randomly stops working. This general fragility is what Docker chiefly attempts to avoid. By bundling everything together in a neat little package, it ensures that if a Docker container did run, it will run again, and will do so reliably across many machines. A big concept that empowers this is “base images”. When you go to build a fancy Docker container, you do it by building on something that’s already predefined. Each of these base images has a file system with things like system libraries and runtimes already installed. exploring base images on docker hub. If we were developing our fancy video-to-GIF backend server, we might choose to use the official Python image, so that we know we’re always starting from the same place. Then, if we cloned our repo, installed our dependencies, and ran our application, we could do it with the confidence that we know exactly which version of Python we have, because it’s whatever Python version comes with the image. You might be thinking this sounds like the virtual machine concept I introduced at the beginning of this article. Recall I’m spooling up virtual machines to help me explore this concept, and I can reproduce steps from scratch by installing a new virtual machine. Virtual machines and containerized images share a lot of conceptual ties, but they’re different in a few key ways that I’d like to briefly explore. A Virtual Machine is very much like a virtual computer within your computer: It has a kernel, which is the core of whatever operating system you’re using. It has its own drivers, which can interact with the hardware on your computer: keyboard, mouse, speakers, screen, network adapters, etc. It has all your files and installed applications. It has its own virtualized representation of resources, like RAM, storage, and CPUs. As a result, VMs are very, very isolated from one another. This is extremely useful if you want to run very different software stacks, experiment with risky tools, or keep workloads completely separated. But with this benefit, they come with two major drawbacks: They’re heavy: You need to download an entire operating system image, which can be several gigabytes in size. They’re slow: You need to boot and run that operating system inside your existing operating system, meaning you’re essentially doubling up on everything; two kernels, two sets of drivers, two full environments running simultaneously. Even with good virtualization support, this adds noticeable overhead. Containers address these drawbacks by taking a different approach. Instead of virtualizing an entire computer, containers reuse the host machine’s kernel. They only package the application and requirements needed to run the application; things like the runtime, libraries, configuration, and filesystem. This makes containers dramatically lighter and faster than VMs; they can start in milliseconds, take up a fraction of the space of a VM, and scale far more easily. Their environment is still isolated and reproducible, but not as completely independent as a full VM because they rely on the host’s kernel and drivers. This does have the tradeoff that containers are not perfectly isolated from the host machine. Containers abstract a lot, but if you need a certain GPU driver to run your application, that won’t be helped by using a specific Docker image, because Docker images don’t handle GPU drivers. Cloud providers make this problem easier by having standard definitions for hardware that you can build off of. Depending on your application, you may need to think outside of the container and consider the actual machine that is hosting your docker image. On AWS, the are different pre-configured machines that you can deploy. These contain lower level OS and driver specific definitions. This is pretty advanced, and out of scope of this article, but you could find yourself making these considerations if you were, for instance, deploying GPU accelerated AI workloads within your Docker container. For our purposes, Docker images can be thought of as consistent enough. In many applications, that is indeed the case. What is a Docker Image, and How Do I Make One? Great question. A Docker image is, essentially, a specific way to containerize an application. There are a few different approaches to containerization, but Docker is the most popular. First, you develop your applications like you normally would. For instance, you can imagine a simple Python application called my-hello-app, which consists of a single Python script called hello.py within the src source code directory. my-hello-app/ └─ src/ └─ hello.py Instead of running hello.py based on whatever version of Python is on our machine, we can define something called a Dockerfile within our application. Typically, this is done outside the source code directory, because it works on the source code, and thus doesn’t belong in the source code. my-hello-app/ ├─ Dockerfile └─ src/ └─ hello.py This Dockerfile will include instructions on how the docker image, which is wrapped around our application, should be built. Here’s an example of what our Dockerfile might look like: FROM python:3.10-slim # Set a working directory inside the container WORKDIR /app # Copy your entire src/ folder into the container COPY src/ ./src/ # Run the main script CMD [”python”, “src/hello.py”] Basically, what this Docker file says is that it will be based on the python:3.10-slim base image, A working directory within that image called /app will be created, and then the src/ directory of our application will be copied to the ./src/ directory of the image. We just created a working directory called /app, and . means the current working directory, so we’ll be copying our code into the /app/src/ directory within our image. When we say CMD [“python”, “src/hello.py”], we’re specifying the default command that will execute when we actually run our Docker image, which is to run src/hello.py via python, assuming we’re within the /app current working directory. There are a few ways to define the default thing our container actually does when it gets started. This is one of the approaches. Now that we have a Dockerfile defining how a Docker image ought to be built around our application, we can actually build the image by opening a terminal to our my-hello-app directory, and running. sudo docker build -t hello-python . This means we’ll build a Docker image based on the Docker file in ., which is in my-hello-app. We’ll assign that image a tag, which is hello-python. You might notice that, after running, nothing gets created. That’s because, by default, images are stored across various folders in Docker’s managed file system. If you wanted to save it to a single file, you could do so by running sudo docker images Which will return something like this IMAGE ID DISK USAGE CONTENT SIZE EXTRA hello-python:latest 0e6f976ab46f 208MB 45.2MB These are all the Docker images that have been created. You might notice that hello-python is followed by :latest. This is a tag. We didn’t specify a tag, so Docker, by default, assigned a tag of latest to the build. You can assign other tags, for instance dev, qa , and prod to align docker containers with a CICD pipeline, but for now we’ll just keep things basic with the defaults. If we want to actually run our built container, we can do so by running sudo docker run --rm hello-python This runs our image (with the default tag, latest). Also, --rm automatically deletes the container after it stops running. The Python script in our Docker container simply prints out the current Python version and spits out all the dependencies. Running it results in: Hello From Python in Docker! Python version: 3.10.19 (main, Nov 18 2025, 04:44:01) [GCC 14.2.0] Installed packages: pip==23.0.1 setuptools==79.0.1 wheel==0.45.1 autocommand==2.2.2 backports.tarfile==1.2.0 importlib-metadata==8.0.0 inflect==7.3.1 jaraco.collections==5.1.0 jaraco.context==5.3.0 jaraco.functools==4.0.1 jaraco.text==3.12.1 more-itertools==10.3.0 packaging==24.2 platformdirs==4.2.2 tomli==2.0.1 typeguard==4.3.0 typing-extensions==4.12.2 zipp==3.19.2 We just built and ran a docker container! That’s cool and all, but what’s really cool is that we can move this to a completely different operating system, and we’ll get the same result. I can export the Docker image into something called a tar file, via sudo docker save -o hello-python.tar hello-python this specifies the output file with -o , and saves our hello-python (default :latest) to that file. Now our directory contains the following my-hello-app/ ├─ hello-python.tar ├─ Dockerfile └─ src/ └─ hello.py I’ve been doing all this in my Linux VM. But we can copy this hello-python.tar file onto a Mac, for instance, and load it by running docker load -i hello-python.tar Which will load the image onto docker on my mac. I can then run the Docker image on my mac with docker run --rm hello-python And, lo and behold, I get the same exact output I got on my linux machine. Which is pretty cool. No installation, no configuration, same exact output. We’re just scratching the surface with Docker, but often the surface is all you need. It handles containerization, so you don’t have to. We’ll explore more sophisticated Docker tricks as necessary later in the article. For now, though, we can move on to exploring Kubernetes. Jumping into Kubernetes (K8s) First of all, Kubernetes is a long word, and is often abbreviated into “k8s”, because there are 8 words between the “K” and “s”. Is Kubernetes plural? Is it’s singular form “Kubernete”? Does it matter? I guess not. What matters is that Kubernetes is often abbreviated as either K8 or K8s. I have a tendency to flip-flop between the two. Kubernetes (K8s) is a system for automating the deployment, scaling, and management of containerized applications. Basically, if you have a bunch of hardware, you can use K8s to manage which containers are running and how they work together, allowing you to build sophisticated applications that operate across multiple computers. Instead of speaking theoretically, we’re going to jump right into an example application. We’ll describe a problem, build a containerized solution with Docker, and deploy it with k8s. Once we have some understanding of k8s, we’ll then explore Helm. All code for all examples in this article can be found here Sample Problem Definition, Calculating Digits of Pi. My boy Archimedes just hit me up, apparently he got a slick new MacBook Air, and he wants to use it to help him calculate an approximation of pi. He’s not very tech savvy, and he asked for our help. We don’t need to use Kubernetes for this, but what the heck. “Archimedes on Mac” by Daniel Warfield using Midjourney. To do this, we’re going to use Monte Carlo estimation. Basically, we’ll stick a circle in a square and randomly place points within the square. We can calculate if the point is inside the circle with the following expression. I won’t get into the math, but the ratio of points in the circle should be approximately equal to pi over 4. If we had infinite points, this would be infinitely accurate. We can do some algebra to move this expression around, to create an approximation for pi. Obviously I’m doing this on my macbook, but you might imagine doing this on a very large set of computers that use a very large number of points to predict pi. Because we’re using kubernetes, we could take what we’re making in this tutorial and apply it to many machines in a server with minimal effort. It’s crazy that Apple figured out those curved bezels before we calculated a decent approximation for pi, but I won’t try to think about it too hard. Let’s plan out how we might tackle this in K8s. The Plan If you read my article on Apache Spark, you might be familiar with the concept of “map-reduce,” which we’ll be using in this example. We’re going to break this problem into two parts. First, we’ll define a worker that generates some number of random points in a cube, calculates if they are or are not within the square, and writes to a file how many of the points were in the circle, and how many points it tried. We’ll then build a simple “reducer” script, which takes in a few of those files and calculates digits of pi. This separation is useful because we can employ more than one worker, who can work on the problem in parallel. Once the workers generate a bunch of points, the reducer can run when they finish, and reduce the results down into a single output. We’ll be using k8s to manage how we’re orchestrating containers. To get started, let’s build the container itself. Implementing the Worker and Reducer in Docker To make things easy, we’re only going to specify one container, which holds both the worker and reducer logic. This will be the project’s file structure to start. pi-estimator/ └─ src/ ├─ worker.py └─ reducer.py Our worker.py file looks like this: import os import json import random import uuid def estimate_chunk(num_points: int) -> tuple[int, int]: “”“Simulate num_points Monte Carlo trials. Returns (inside, total).”“” inside = 0 for _ in range(num_points): x = random.random() * 2 - 1 # Uniform in [-1, 1] y = random.random() * 2 - 1 if x*x + y*y <= 1: inside += 1 return inside, num_points def main(): # How many points this worker should simulate points_per_worker = int(os.environ.get(”POINTS_PER_WORKER”, “100000”)) # Where to write the result file result_dir = os.environ.get(”RESULT_DIR”, “/results”) # Make sure the directory exists os.makedirs(result_dir, exist_ok=True) # Perform the Monte Carlo trials inside, total = estimate_chunk(points_per_worker) # Generate a unique filename so workers don’t clash result_file = os.path.join(result_dir, f”result-{uuid.uuid4()}.json”) # Write out the result with open(result_file, “w”) as f: json.dump({”inside”: inside, “total”: total}, f) print(f”[worker] Completed {total} points → inside={inside}”) print(f”[worker] Wrote result to {result_file}”) if __name__ == “__main__”: main() We have a function estimate_chunk that does the actual work to generate random points and count which ones landed within the circle, and a main function that serves as an entry point and handles some key configuration details. K8, which we’ll be using later, likes to communicate with containers via environment variables. As a result, we’re configuring how many points the script should test and where it should place the results, via environment variables. reducer.py should look something like this: import os import json def load_results(result_dir: str): “”“Load all JSON result files from the directory.”“” inside_total = 0 points_total = 0 if not os.path.exists(result_dir): print(f”[reducer] Results directory not found: {result_dir}”) return None, None files = [f for f in os.listdir(result_dir) if f.endswith(”.json”)] if not files: print(”[reducer] No result files found. Did any workers run?”) return None, None for name in files: file_path = os.path.join(result_dir, name) try: with open(file_path, “r”) as f: data = json.load(f) inside_total += data.get(”inside”, 0) points_total += data.get(”total”, 0) except Exception as e: print(f”[reducer] Failed to read {file_path}: {e}”) return inside_total, points_total def compute_pi(inside: int, total: int) -> float: “”“Compute the Monte Carlo estimate of pi.”“” if total == 0: return float(”nan”) return 4 * inside / total def main(): result_dir = os.environ.get(”RESULT_DIR”, “/results”) print(f”[reducer] Collecting results from: {result_dir}”) inside, total = load_results(result_dir) if inside is None: print(”[reducer] No valid data found. Exiting.”) return pi_estimate = compute_pi(inside, total) print(f”[reducer] inside_total = {inside}”) print(f”[reducer] points_total = {total}”) print() print(f”Distributed π estimate: {pi_estimate}”) if __name__ == “__main__”: main() It gets all the files in the output path, generated by all workers that have run, and computes pi. It also uses environment variables so the result directory can be configured externally. We can make sure this works before we try to set up k8s around it. I’m going to define a script called local_test.py which runs this code using whatever version of Python comes in default in my Ubuntu virtual machine. pi-estimator/ ├─ local_test.py └─ src/ ├─ worker.py └─ reducer.py And here’s the actual implementation for local_test.py import os import json import shutil import subprocess RESULT_DIR = “local_results” def run_worker(points=100000): “”“Run the worker.py script locally.”“” print(f”[test] Running worker with {points} points...”) env = os.environ.copy() env[”POINTS_PER_WORKER”] = str(points) env[”RESULT_DIR”] = RESULT_DIR subprocess.run( [”python”, “src/worker.py”], env=env, check=True ) def run_reducer(): “”“Run the reducer.py script locally.”“” print(f”[test] Running reducer...”) env = os.environ.copy() env[”RESULT_DIR”] = RESULT_DIR subprocess.run( [”python”, “src/reducer.py”], env=env, check=True ) def reset_results(): “”“Clear out the results directory.”“” if os.path.exists(RESULT_DIR): shutil.rmtree(RESULT_DIR) os.makedirs(RESULT_DIR, exist_ok=True) def main(): print(”[test] Starting local Monte Carlo π estimation”) reset_results() # Run multiple workers for i in range(3): print(f”[test] Worker {i+1}/3”) run_worker(points=200000) print() print(”[test] Running reducer to aggregate results:”) print(”--------------------------------------------”) run_reducer() print(”--------------------------------------------”) print(”[test] Done.”) if __name__ == “__main__”: main() Essentially, this script specifies environment variables and runs the worker a few times, then it runs the reducer. It’s not parallelized, but it does allow us to test the core logic and make sure everything is working correctly. When we run it, we get this output: [test] Starting local Monte Carlo π estimation [test] Worker 1/3 [test] Running worker with 200000 points... [worker] Completed 200000 points → inside=157035 [worker] Wrote result to local_results/result-b6a69810-fe60-4f3d-bd2c-f4a29c80b249.json [test] Worker 2/3 [test] Running worker with 200000 points... [worker] Completed 200000 points → inside=156804 [worker] Wrote result to local_results/result-255f2b1b-01ce-467b-bc9d-135c06dd7758.json [test] Worker 3/3 [test] Running worker with 200000 points... [worker] Completed 200000 points → inside=156995 [worker] Wrote result to local_results/result-ba2466b4-04ec-4bc0-a866-b064b4b126d5.json [test] Running reducer to aggregate results: -------------------------------------------- [test] Running reducer... [reducer] Collecting results from: local_results [reducer] inside_total = 470834 [reducer] points_total = 600000 Distributed π estimate: 3.1388933333333333 -------------------------------------------- [test] Done. This runs great for a relatively small number of points, but if we increase the number of points past around a million, we quickly bump into significant slowdowns. Let’s Dockerize this and run it with K8s so we can get a few of these workers running in parallel. Dockerizing Our Application We’re going to create a Dockerfile outside of our source so we can dockerize the application. pi-estimator/ ├─ local_test.py ├─ Dockerfile └─ src/ ├─ worker.py └─ reducer.py └─ local_results/ <- automatically created by running local_test.py ├─ result-...json ├─ result-...json └─ result-...json We could implement two Docker containers, but to make things easier, we’ll be implementing one Docker container that contains both the worker and reducer code. Here’s the Dockerfile that will make that work. FROM python:3.10-slim WORKDIR /app COPY src/worker.py src/reducer.py . ENTRYPOINT [”python”] This is essentially the same as our Dockerfile from our previous example. It uses a base image, sets a working directory, and copies worker.py and reducer.py into that directory. The only difference is that, instead of specifying a command to run when the image is started up with CMD, we specify an ENTRYPOINT. If we use CMD our docker container will run the same command every time. With ENTRYPOINT we can specify arguments when we actually run our docker image, which can modify how it’s run. When we run docker run my_image , it will be appended to whatever as specified in the ENTRYPOINT. In this case, that would be python . Thus, when we start up an image, we can specify if it’s a worker or reducer simply by changing the arguments we set when we run the image, via something like docker run my_image worker.py or docker run my_image reducer.py. We can build this Docker container by navigating into the pi-estimator directory and running sudo docker build -t pi-estimator . We can make sure everything works by running the Docker file all by itself. Recall that the worker requires some environment variables to be properly configured, and then will output a JSON file to a directory. We’re going to run the image, specify that it’s a worker, and set up the configuration and output path all within one command: sudo docker run --rm \ -e POINTS_PER_WORKER=200000 \ -e RESULT_DIR=/results \ -v $(pwd)/local_results:/results \ pi-estimator worker.py There’s four core things going on. First, this means we’re running a Docker image and killing it when it’s done. sudo docker run --rm -e allows us to set environment variables. We’re setting two, one is how many points the worker generates, and the other is the output directory. These are compatible with what we set up in worker.py -e POINTS_PER_WORKER=200000 \ -e RESULT_DIR=/results \ This is kind of a tricky one. Recall that docker containers have their own file system, which isolates them from the host system they’re running on. That’s great, but it would mean we can’t actually see our results, because they’ll be deleted when the image stops executing. To alleviate that, we’re going to attach a volume ( -v ) which allows us to specify a file from our computer’s file system that will be mounted to Docker. When Docker writes to that folder, it will be written to the folder on the host machine. -v $(pwd)/local_results:/results The syntax is -v local_path:path_in_docker, so we’re mounting local_results in our file system into our container as the results directory. Recall that we set the environment variable -e RESULT_DIR=/results in the previous step, so the Docker container will write its results to this folder, which means it will write its results to our computer. The final line is the actual Docker container we’ll be running, and the argument that will be tacked onto the ENTRYPOINT pi-estimator worker.py So, within our container, we’ll be running python worker.py. Running that whole command results in this output [worker] Completed 200000 points → inside=157024 [worker] Wrote result to /results/result-89f2d5d5-3b62-441b-a075-cc62baad0a10.json and, if we look into our local_results directory, we’ll see a new json file was created. We can run this command a few more times if we want, then run this to execute our reducer sudo docker run --rm \ -e RESULT_DIR=/results \ -v $(pwd)/local_results:/results \ pi-estimator reducer.py which, for me, outputs this: reducer] Collecting results from: /results [reducer] inside_total = 785219 [reducer] points_total = 1000000 Distributed π estimate: 3.140876 Alright, we’ve containerized our application. It’ll work pretty much the same on any operating system, and we can spool it up with a single command to Docker. Let’s automate the orchestration and parallelization of these containers with Kubernetes. A Brief Interlude, The Anatomy of Kubernetes I typically avoid jargon as much as possible, but with K8s it’s hard to avoid. Before we move on I’d like to cover some basic Kubernetes ideas. These are the core elements that make up most K8s clusters. Feel free to refer back to this section if you’re confused about specific names. We’ll be using some of these words later in the article. Clusters: The whole idea of K8s is that it allows you to run a job across multiple computers. Those computers, together, form a cluster. Node: Each individual computer within the cluster Pod: The smallest runnable unit in K8s. Kubernetes manages the creation and deletion of pods, which contain containers and run within a node. Usually, a Pod only contains a single container, but in some more advanced applications, a few containers can exist within a single pod. You can think of a “pod” as a scheduled unit in K8s, usually corresponding to a single container. Container: We’ve covered this in depth. It’s a container, like Docker. There are more containerization technologies, like Podman, but Docker is the most common. Volume: This is persistent storage mounted onto a Pod, which that Pod’s containers can access and use. If there’s a failure and a pod needs to restart, this allows the data to persist. Also, if more than one Pod in a container needs to talk with one another, we can do that by having them share files. Persistent Volume: Unlike a normal volume (which lives only as long as its Pod), a PersistentVolume exists independently from any Pod that uses it. It’s used for databases, queues, things like that. Service: Basically, networking for a pod. Different services expose Pods in different ways. For instance, you can assign an IP address to your Pod, so that Pods within a cluster can communicate with each other over HTTP requests. You can also employ load balancers, which can balance incoming requests from external sources and distribute them to various nodes in the cluster. Ingress: This allows you to route incoming requests to different services, and thus to different pods. For instance, you might have a pod that manages login, and a pod that manages core application logic. When your K8s cluster gets a request for example.com/login, it can route that request to the service that corresponds to the pod for login. In other words, Ingress allows you to set rules as to how requests are routed throughout the pods in your cluster. Controller: A controller is a background process inside Kubernetes that constantly compares the actual state of the cluster with the desired state, and takes action to make them match. In K8s you declare what you want with something called a “manifest”, then k8s uses controllers to try to make what you want. Replica Set: Ensures that a certain number of identical Pods are always running. If a Pod crashes or a node dies, the ReplicaSet creates a new one, keeping your pods alive through failure. This is usually managed by the controller, not the user. The Control Plane: The control plane is the brains of kubernetes. It contains the controller manager (which actually runs the controller), the API server (which manages all traffic into and out of the cluster, including communication to manage the cluster itself), the scheduler (which observes resource usage in pods, obeys specified constraints, and allocates pods to certain nodes as necessary), and etcd (which serves as memory for the control plane). Kubelet: Runs on each node, receives instructions from the control plane, and manages the running of pods. This is managed by K8s. Namespace: A way to organize and isolate resources inside a cluster by assigning them by name. Resources in one namespace don’t conflict with resources in another, allowing you to spool up several parallel versions of an application within a single cluster. This is commonly used to separate dev/staging/prod or to give teams their own space. ConfigMap and Secret: We won’t be touching on these a lot in this tutorial, but they’re super important when building actual applications in the real world. Many K8s use cases involve complex environment variables that might be sensitive (like API keys) or require sophisticated workflows around setting them (like CICD). ConfigMaps and Secrets are special ways of dealing with environment variables that make them modular and more secure. Orchestrating with Kubernetes We’re going to make a new folder in our folder structure called k8s , which will contain a file called pi-job.yaml. This will specify how we configure kubernetes to run our application. We’re also creating a helper file called pi-image-structure.yaml, which we’ll discuss later. pi-estimator/ ├─ local_test.py ├─ Dockerfile └─ k8s/ ├─ pi-job.yaml └─ pi-image-structure.yaml └─ src/ ├─ worker.py └─ reducer.py └─ local_results/ └─ ....json pi-job.yaml is the major file that tells K8s what to do. We’ll go through line by line, but here’s the whole file: # ------------------------------------------------------- # Persistent Volume Claim (shared across workers & reducer) # ------------------------------------------------------- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pi-results-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi --- # ======================================================= # RBAC so reducer can check worker pod status # ======================================================= apiVersion: v1 kind: ServiceAccount metadata: name: pi-reducer-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pi-reducer-role rules: - apiGroups: [”“] resources: [”pods”] verbs: [”get”, “list”] - apiGroups: [”batch”] resources: [”jobs”] verbs: [”get”, “list”] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pi-reducer-binding subjects: - kind: ServiceAccount name: pi-reducer-sa roleRef: kind: Role name: pi-reducer-role apiGroup: rbac.authorization.k8s.io --- # ------------------------------------------------------- # Worker Job (4 Pods in parallel) # ------------------------------------------------------- apiVersion: batch/v1 kind: Job metadata: name: pi-workers spec: completions: 4 parallelism: 4 backoffLimit: 0 template: spec: restartPolicy: Never containers: - name: worker image: pi-estimator imagePullPolicy: Never command: [”python”, “/app/worker.py”] env: - name: POINTS_PER_WORKER value: “200000” - name: RESULT_DIR value: “/results” volumeMounts: - name: results mountPath: /results volumes: - name: results persistentVolumeClaim: claimName: pi-results-pvc --- # ------------------------------------------------------- # Reducer Job (waits for all 4 workers to finish) # ------------------------------------------------------- apiVersion: batch/v1 kind: Job metadata: name: pi-reducer spec: completions: 1 parallelism: 1 backoffLimit: 0 template: spec: restartPolicy: Never serviceAccountName: pi-reducer-sa initContainers: - name: wait-for-workers image: bitnami/kubectl:latest command: - sh - -c - | echo “[init] Waiting for 4 worker pods to complete...” while true; do succ=$(kubectl get pods -l job-name=pi-workers \ --field-selector=status.phase=Succeeded \ --no-headers | wc -l) echo “[init] Workers finished: ${succ}/4” if [ “$succ” -ge 4 ]; then echo “[init] All workers completed, starting reducer.” break fi sleep 2 done containers: - name: reducer image: pi-estimator imagePullPolicy: Never command: [”python”, “/app/reducer.py”] env: - name: RESULT_DIR value: “/results” volumeMounts: - name: results mountPath: /results volumes: - name: results persistentVolumeClaim: claimName: pi-results-pvc This is called a Kubernetes manifest file, which declares what we want Kubernetes to create and manage for us. It’s organized into six resource manifests: One for a persistent volume (so different processes can send data to one another) Three to set the correct permissions (we’ll talk about these later) One to kick off four of the workers, which run in parallel One to kick off the reducer, which aggregates the output from each of the workers and returns a prediction for pi. Each of these manifests starts with an apiVersion, something like this: apiVersion: batch/v1 Originally, Kubernetes only had one apiVersion, which was v1. This contains the core elements of Kubernetes. As Kubernetes evolved and became more popular, different types of use cases became more prevalent. People were using Kubernetes to run large computational jobs, and people were using Kubernetes to support long-lasting applications. To support these types of workflows, the batch/v1 and apps/v1 APIs were created to run ephemeral jobs and long-lived applications, respectively. v1, apps/v1, and batch/v1 cover the vast majority of Kubernetes, though there are many other APIs that are important for some specific functionality. on top of the API version, each resource also has a kind, which specifies the kind of resource we want to make within the apiVersion we’re using. Exactly what a kind represents can change depending on the apiVersion we’re using. For instance, it might represent what type of batch operation we’re making apiVersion: batch/v1 kind: Job ... it might represent that we want a persistent volume apiVersion: v1 kind: PersistentVolumeClaim ... Or that we want to create roles and bind those roles to resources. apiVersion: v1 kind: ServiceAccount ... apiVersion: rbac.authorization.k8s.io/v1 kind: Role ... apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding There’s a lot going on in this manifest file, chiefly because we need a few different resources to get everything working. We can pick this apart by understanding each resource individually and what it’s doing. The simplest one is probably the persistent volume. # ------------------------------------------------------- # Persistent Volume Claim (shared across workers & reducer) # ------------------------------------------------------- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pi-results-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi Each resource manifest consists of four key parts; the apiVersion and kind, which we discussed and will continue to explore, and the metadata and spec. The metadata field allows you to assign information that’s useful to both humans and Kubernetes in terms of organization and grouping of resources. For this example we’re keeping it simple and just assigning a name which is pi-results-pvc (the persistent volume claim for the results of our pi-finding workers). Setting a name is important because it’s common to control and interact with resources by name in K8s. This will be super important in both the examples in this article. The spec is where all the magic happens; it’s where you define what you actually want out of the resource. The most important for a persistent volume claim ( pvc ) is the request for some volume of storage. resources: requests: storage: 1Gi When you create a persistent volume claim and pass it to K8s, K8s then passes that request to the control plane in your cluster. Depending on whether your Kubernetes is running locally using Minikube or on a cloud provider like AWS, GCP, or Azure, different storage provisioners are used to fulfill that request. It’s important to note here that manifests are “declarative”, meaning they declare the state we want to achieve. If an existing PersistentVolume already satisfies the claim, the control-plane controllers will bind the PVC to it. On top of the amount of storage, we’re also defining the access modes. spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi This says that we want to allow many different pods (essentially, different instances of containers) to be able to read and write to this volume simultaneously. There are a few different approaches that can be useful in different use cases. ReadWriteOnce: One machine can write to it, and any pods on that machine can use it. ReadWriteOncePod: Only a single pod is allowed to write to it, even if multiple pods are on the same machine. ReadOnlyMany: Many pods on many machines can use it, but only for reading. ReadWriteMany: Many pods on many machines can use it and write to it at the same time. ReadWriteMany can result in race conditions and conflicts, so it’s not good for every use case, but for this application it should work fine; our worker nodes run in isolation, and our reducer will only read once all workers are done running. Now that we have a claim for volume, let’s discuss the worker job, which uses that volume. # ------------------------------------------------------- # Worker Job (4 Pods in parallel) # ------------------------------------------------------- apiVersion: batch/v1 kind: Job metadata: name: pi-workers spec: completions: 4 parallelism: 4 backoffLimit: 0 template: spec: restartPolicy: Never containers: - name: worker image: pi-estimator imagePullPolicy: Never command: [”python”, “/app/worker.py”] env: - name: POINTS_PER_WORKER value: “200000” - name: RESULT_DIR value: “/results” volumeMounts: - name: results mountPath: /results volumes: - name: results persistentVolumeClaim: claimName: pi-results-pvc Recall that, in our original Docker container, we defined two scripts; worker.py could be used to generate a bunch of points and say if they’re inside or outside of a circle, then reducer.py could take those results (written as a json file) to calculate pi. Here, we’re defining a Job which spawns four workers and runs them in parallel. Recall, there are three important core APIs in K8s, the v1 , app/v1, and batch/v1 apis. From a high-level view, the batch api has controllers for managing finite and scheduled tasks, while the app api has controllers for managing long-running applications. These are the most critical ones: batch/v1: Job — A controller for executing on a specific task, then tears everything down when done. Good if you want to press play, execute a job, then free up resources when it’s done. batch/v1: CronJob — Similar to jobs, tares down on completion, but the controller sticks around so it can restart the task on a schedule. Good if you want to run a job every hour/day/week/etc. app/v1: Deployment — Runs long-lived applications and stateless applications. Probably the most important type of controller in K8s. Think the resources necessary to run a standard website. app/v1: StatefulSet — Similar to Deployment, but with extra rules imposed to enforce statefulness. app/v1: DaemonSet — Runs something on every machine (node) in the cluster. Useful for things like logging and monitoring. A batch/v1: Job, then, is perfect for our application. We want the job to run, then stop running. Within this Job we’re assigning a name of pi-workers in the metadata field then defining a spec. The spec has a few high-level parameters spec: completions: 4 parallelism: 4 backoffLimit: 0 This says we want to spawn pods (containers) until four of them complete successfully, and we want to run four of them in parallel. K8s is designed to be resilient to error, so it will automatically retry running containers if they fail. This allows us to stop once we’ve run a certain amount successfully. By default, K8s will keep trying indefinitely, but we also set backoffLimit to zero, meaning K8s won’t retry running a pod that’s failed. This isn’t strictly required, but it means if we accidentally introduce a bug K8s won’t try to keep launching the same container over and over again. After those simple configurations, we have the template for the worker itself spec: completions: 4 parallelism: 4 backoffLimit: 0 template: spec: restartPolicy: Never containers: - name: worker image: pi-estimator imagePullPolicy: Never command: [”python”, “/app/worker.py”] env: - name: POINTS_PER_WORKER value: “200000” - name: RESULT_DIR value: “/results” volumeMounts: - name: results mountPath: /results volumes: - name: results persistentVolumeClaim: claimName: pi-results-pvc Now that we’ve covered some definitions and core ideas, this is pretty straightforward to understand. Each of the four pods this Job is spawning consists of a single container with an image named pi-estimator, which is the name of the Docker image we made. We’re starting that container with the command python /app/worker.py, which runs our worker script once the container starts. We’re also specifying a few environment variables within the env clause, allowing us to specify how many points each worker simulates, and where the results end up. We’re setting the RESULT_DIR environment variable to the same path we’re mounting the persistent volume to, meaning after our script runs, each of our workers will write its own output file to the persistent volume, allowing us to aggregate our results in a single place. There’s two other fields that aren’t strictly necessary, but made their way in through the debugging process. imagePullPolicy: Never means we’ll never pull the image from an external registry, like Docker Hub, and restartPolicy: Never means if the pod fails, never restart it. This is common for a Job because it needs pods to conclude to be able to manage the pods correctly. If the pod keeps restarting itself every time it finishes, then the pod never concludes, and the Job can’t be managed properly. The astute among you might notice that we’re calling /app/worker.py. Our Dockerfile which defines our image looks like this FROM python:3.10-slim WORKDIR /app COPY src/worker.py src/reducer.py . ENTRYPOINT [”python”] It can be kind of hard to be 100% sure where all our files are within the docker container. Instead of guessing, I made a simple manifest file that lets us explore the folder structure of our image, called pi-image-structure.yaml. apiVersion: batch/v1 kind: Job metadata: name: pi-image-structure spec: backoffLimit: 0 template: spec: restartPolicy: Never containers: - name: explorer image: pi-estimator:latest imagePullPolicy: Never command: [”/bin/sh”, “-c”] args: - | echo “=== DIRECTORY LISTING OF /app (detailed) ===”; ls -alh /app; echo; echo “=== RECURSIVE LISTING OF /app ===”; ls -R /app; echo; echo “=== TREE VIEW OF /app (manual) ===”; find /app -printf “%p\n”; echo “=== DONE ===”; It’s just like the Job we were just talking about, except it only spawns the image once, and instead of running a command to run our script, it runs some commands to view the structure of the files within the image. I don’t want this tangent to take too long, we haven’t finished discussing the big manifest file running our job. It’s just a little hack I used to make sure all of my paths were set up correctly. Anyway, back to our manifest file for our pi estimation. We discussed two of the resources so far; the persistent volume claim and the parallel job for the workers. We also need another Job which runs the reducer after the workers are done. This resource manifest makes that happen. # ------------------------------------------------------- # Reducer Job (waits for all 4 workers to finish) # ------------------------------------------------------- apiVersion: batch/v1 kind: Job metadata: name: pi-reducer spec: completions: 1 parallelism: 1 backoffLimit: 0 template: spec: restartPolicy: Never serviceAccountName: pi-reducer-sa initContainers: - name: wait-for-workers image: bitnami/kubectl:latest command: - sh - -c - | echo “[init] Waiting for 4 worker pods to complete...” while true; do succ=$(kubectl get pods -l job-name=pi-workers \ --field-selector=status.phase=Succeeded \ --no-headers | wc -l) echo “[init] Workers finished: ${succ}/4” if [ “$succ” -ge 4 ]; then echo “[init] All workers completed, starting reducer.” break fi sleep 2 done containers: - name: reducer image: pi-estimator imagePullPolicy: Never command: [”python”, “/app/reducer.py”] env: - name: RESULT_DIR value: “/results” volumeMounts: - name: results mountPath: /results volumes: - name: results persistentVolumeClaim: claimName: pi-results-pvc This is virtually the same as the worker manifest, save two key differences. First and most trivially, it calls the reducer.py script, rather than the worker.py script. It also uses an initContainer. When you run a manifest file in K8s, it creates all resources at the same time asynchronously. That means K8s will create our worker job with 4 workers (which take a while to run because they have a lot of work to do) and our reducer at the same time. If we run our reducer as soon as it’s spooled up, it will look in the shared volume and see that there’s nothing in there, as the workers haven’t finished running yet. There are a few ways of dealing with this, which have their costs and benefits. I opted to use something called an initContainer. initContainers run before your actual container, in order if you have more than one init container. Your actual container in your pod only runs after all of your init containers have finished running. Thus, if we make our initContainer run as long as we have workers running, then it will effectively delay our reducer from starting until after all of our workers have finished. This is the definition for the initContainer initContainers: - name: wait-for-workers image: bitnami/kubectl:latest command: - sh - -c - | echo “[init] Waiting for 4 worker pods to complete...” while true; do succ=$(kubectl get pods -l job-name=pi-workers \ --field-selector=status.phase=Succeeded \ --no-headers | wc -l) echo “[init] Workers finished: ${succ}/4” if [ “$succ” -ge 4 ]; then echo “[init] All workers completed, starting reducer.” break fi sleep 2 done It uses a base image called bitnami/kubectl:latest which is a very popular kubernetes image designed to run kubectl commands from within a container. This lets it spool up an image that can talk with the cluster it’s in. The command we’re sending to this image is formatted a bit weirdly, simply because it’s a Yaml file. The YAML file expects a list of commands, so a YAML file like this: command: - thing1 - thing2 - thing3 would look like a command like thing1 thing2 thing3 The symbol | is a convention in YAML which means “treat the following thing as a single multi-lined string, line breaks and all. So we’re essentially running the command: sh -c “ echo \”[init] Waiting for 4 worker pods to complete...\” while true; do succ=\$(kubectl get pods -l job-name=pi-workers \ --field-selector=status.phase=Succeeded \ --no-headers | wc -l) echo \”[init] Workers finished: \${succ}/4\” if [ \”\$succ\” -ge 4 ]; then echo \”[init] All workers completed, starting reducer.\” break fi sleep 2 done “ the command sh specifies that we’re simply running a command in the Unix POSIX shell, which every Linux machine has. the -c argument says “run the following string as a command”, then we’re passing the string into it, which is written in POSIX shell script. If you’re not familiar with POSIX shell scripts this might look a bit daunting. In essence: It prints the text [init] Waiting for 4 worker pods to complete… while true; do starts an infinite loop which will always run until broken out of succ=… counts the number of completed worker pods. It does that by getting all the pods with kubectl get pods with a filter that only selects pods of the correct name with -l job-name=pi-workers and only keeps pods that have successfully finished with — field-selector=status.phase=Succeeded. It makes sure there are no headers with — no-headers, and | wc -l counts how many lines exist within the resulting text from the previous commands. Basically, we’re making a line of text for all of the workers that have finished successfully, and are counting the number of rows. If the number of successful pods is greater than or equal to four if [ \”\$succ\” -ge 4 ]; then , then print and break out. Otherwise sleep for two seconds. I think it’s kind of cool how we didn’t need to implement any code. By using a base image that already had kubectl set up, we could just use a kubectl and some fancy shell scripting to whip up some sophisticated functionality. The end result is that our init container doesn’t stop running until all workers have stopped running. It also consumes practically zero resources, as the Docker image is lightweight and the majority of time the image is sleeping (thus not consuming resources). By default, a pod can’t run kubectl commands to be able to communicate with the cluster it exists within, as a matter of security. To enable these permissions, we specify serviceAccountName: pi-reducer-sa, which is another set of resources we need to define in our manifest. apiVersion: v1 kind: ServiceAccount metadata: name: pi-reducer-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pi-reducer-role rules: - apiGroups: [”“] resources: [”pods”] verbs: [”get”, “list”] - apiGroups: [”batch”] resources: [”jobs”] verbs: [”get”, “list”] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pi-reducer-binding subjects: - kind: ServiceAccount name: pi-reducer-sa roleRef: kind: Role name: pi-reducer-role apiGroup: rbac.authorization.k8s.io First, we create a service account, which pods can use as an identity when talking with the Kubernetes API in the control plane. apiVersion: v1 kind: ServiceAccount metadata: name: pi-reducer-sa Next, we define a role, which defines a set of actions that an account would be able to perform. This allows for getting (reading one) and listing (reading all) pods and jobs. apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pi-reducer-role rules: - apiGroups: [”“] resources: [”pods”] verbs: [”get”, “list”] - apiGroups: [”batch”] resources: [”jobs”] verbs: [”get”, “list”] We then bind that role to the service account, by specifying the service account and role by name within a RoleBinding. apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pi-reducer-binding subjects: - kind: ServiceAccount name: pi-reducer-sa roleRef: kind: Role name: pi-reducer-role apiGroup: rbac.authorization.k8s.io A bit verbose, and some boilerplate, but conceptually intuitive I think. And that’s actually all we need to code up to get our container working in kubernetes. Lets spool up minikube and run it. Running in Kubernetes First of all, we defined a lot of stuff in the last section, and it’s probably a good idea to do a simple smoke test. We have two manifest files in our k8s folder; pi-job.yaml which is super complicated, and pi-image-structure.yaml which simply prints out the directory structure within our image. Let’s try running pi-image-structure.yaml first, to make sure everything is set up nicely. pi-estimator/ ├─ local_test.py ├─ Dockerfile └─ k8s/ ├─ pi-job.yaml └─ pi-image-structure.yaml └─ src/ ├─ worker.py └─ reducer.py └─ local_results/ └─ ....json First of all, recall I’m running all this in a virtual machine. If you’re doing something similar, I recommend allocating plenty of RAM in the VM so Kubernetes doesn’t have any issues. On Parallels, you can do that by making sure the VM is shut down, opening it by double-clicking the name, and clicking the gear on the top right. You can then give it a good amount of processors and RAM to play with. Once you do that, go ahead and start up the VM. Open up a terminal, and CD into wherever your pi-estimator directory is then run minikube start You’ll get a bunch of emojis and a message saying minikube started. Then run minikube image build -t pi-estimator:latest . This will build the docker image for pi-estimator, based on our Dockerfile, and load it into minikube. if we then run minikube image list we’ll be able to see our image in minikube. Let’s go ahead and cd into our k8s folder, and run kubectl apply -f pi-image-strcture.yaml which should return something like this. job.batch/pi-image-structure created Recall that pi-image-structure.yaml defines a simple command that prints out the directory structure of our pi-estimator docker image. Running kubectl apply applies this manifest to the cluster. We can then run the following command to see that our job ran and completed. kubectl get jobs NAME STATUS COMPLETIONS DURATION AGE pi-image-structure Complete 1/1 4s 59s We can also run the following commands to see the individual pod which was created during the job, ran successfully, then destroyed. kubectl get pods NAME READY STATUS RESTARTS AGE pi-image-structure-rmgh2 0/1 Completed 0 2m52s We can run the following command to check out the logs from all pods that ran in this job (only the one) kubectl logs -l job-name=pi-image-structure --prefix [pod/pi-image-structure-rmgh2/explorer] === RECURSIVE LISTING OF /app === [pod/pi-image-structure-rmgh2/explorer] /app: [pod/pi-image-structure-rmgh2/explorer] reducer.py [pod/pi-image-structure-rmgh2/explorer] worker.py [pod/pi-image-structure-rmgh2/explorer] [pod/pi-image-structure-rmgh2/explorer] === TREE VIEW OF /app (manual) === [pod/pi-image-structure-rmgh2/explorer] /app [pod/pi-image-structure-rmgh2/explorer] /app/worker.py [pod/pi-image-structure-rmgh2/explorer] /app/reducer.py [pod/pi-image-structure-rmgh2/explorer] === DONE === And we can see that the /app directory contains our worker.py and our reducer.py. More importantly, though, we just successfully ran our first Kubernetes manifest! Of course, our actual pi-job.yaml is way more complicated, but running it is pretty much the same thing. First, though, let’s run this command to delete all traces of our pi-image-structure so it’s not polluting our output. kubectl delete job pi-image-structure These two commands should both result in an empty output kubectl get jobs kubectl get pods We can now run kubectl apply -f pi-job.yaml and see that a bunch of resources were created (or unchanged, because I ran this previously) persistentvolumeclaim/pi-results-pvc created serviceaccount/pi-reducer-sa unchanged role.rbac.authorization.k8s.io/pi-reducer-role unchanged rolebinding.rbac.authorization.k8s.io/pi-reducer-binding unchanged job.batch/pi-workers created job.batch/pi-reducer created And we have a bunch of pods that all finished pretty quickly. kubectl get pods NAME READY STATUS RESTARTS AGE pi-reducer-zwxnb 0/1 Completed 0 5s pi-workers-784w4 0/1 Completed 0 5s pi-workers-sq2q5 0/1 Completed 0 5s pi-workers-x9nnl 0/1 Completed 0 5s pi-workers-z6znv 0/1 Completed 0 5s We can view the logs across all our workers kubectl logs -l job-name=pi-workers --prefix [pod/pi-workers-784w4/worker] [worker] Completed 200000 points → inside=157207 [pod/pi-workers-784w4/worker] [worker] Wrote result to /results/result-3ae62882-d551-4e5d-b5df-5f7c3b2124a6.json [pod/pi-workers-sq2q5/worker] [worker] Completed 200000 points → inside=157099 [pod/pi-workers-sq2q5/worker] [worker] Wrote result to /results/result-e4094053-03b7-477d-a80d-d0f97b8e7cbc.json [pod/pi-workers-x9nnl/worker] [worker] Completed 200000 points → inside=157433 [pod/pi-workers-x9nnl/worker] [worker] Wrote result to /results/result-b715483c-5845-4f28-9258-ff09a20d030e.json [pod/pi-workers-z6znv/worker] [worker] Completed 200000 points → inside=157704 [pod/pi-workers-z6znv/worker] [worker] Wrote result to /results/result-d31f9b4a-daf4-43e0-a792-f62b7eb5e916.json and the logs in our reducer kubectl logs -f job/pi-reducer Defaulted container “reducer” out of: reducer, wait-for-workers (init) [reducer] Collecting results from: /results [reducer] inside_total = 314786221 [reducer] points_total = 400800000 Distributed π estimate: 3.1415790518962075 And see that it successfully aggregated all of the results from our workers! We could scale up the number of points, scale up the number of workers, and waste a whole lot of money on AWS if we really wanted to. This is pretty nifty, but you might notice a key issue. Everything is hard-coded. If we wanted to increase the number of workers or how many points each worker was processing, we’d need to manually adjust our manifest file. This is possible, and it might even be acceptable in some applications, but it feels like this particular problem needs some ease of configuration. We can do that with Helm. Helm Helm is often described as “the package manager for Kubernetes”; it abstracts Kubernetes into something called a chart, which allows you to connect different Kubernetes applications together like Legos. We’ll play around with that later. For now, we’ll be using a functionality of Helm called templating to be able to configure our Job before running it. We can kick off working in helm by navigating to pi-estimator and running helm create chart This creates a directory in pi-estimator called chart, which will contain a bunch of boilerplate we don’t need. We can clean it up with the following: rm -rf chart/templates/* rm -rf chart/templates/tests rm -f chart/values.yaml touch chart/values.yaml This will clean up a bunch of the default stuff we don’t need, and make a barebones basis for us to work off of. Something like this: pi-estimator/ chart/ Chart.yaml values.yaml templates/ ... k8s/ ... src/ ... ... Helm relies on two core ideas to do it’s magic, the chart and templates. A chart is essentially a package that contains everything necessary to define a K8s application. It has metadata about it, which is defined in Chart.yaml, default configuration values defined in values.yaml and configurable templates defined in the templates directory. Instead of defining our K8s application in one big manifest, we break up all the resources into different templates. Each of these templates can have variables, which can have default values defined in values.yaml or can be manually changed. This is what will allow us to run our pi estimation job with a varying number of workers, a varying number of points per worker, and other fun stuff like that. To get started, we can go into Chart.yaml and define it as apiVersion: v2 name: pi-estimator description: A Helm chart for running a parallel Pi estimation job in Kubernetes. type: application version: 0.1.0 appVersion: “1.0” This defines the v2 API for Helm, which is the current standard, and the name of the application is pi-estimator. We give it a description, and assign it as type: application (as opposed to the alternative type: library, which isn’t designed to be run on its own but instead consists of re-usable logic). There’s also some versioning information, which can be useful if you’re trying to keep track of logic around the helm chart or application as a whole. When we later call helm install pi-estimator ./chart, helm will look for our Chart.yaml file in the directory, and will look for a template directory, that contains our resource definitions. Within our template directory we need one template for each of our resources. Behind the scenes, Helm will compile these into a single manifest. This will look a lot like our pi-job.yaml from Kubernetes, with one minor change: variables. Here’s an example of a template for our persistent volume claim. apiVersion: v1 kind: PersistentVolumeClaim metadata: name: {{ .Values.pvc.name }} spec: accessModes: - {{ .Values.pvc.accessMode }} resources: requests: storage: {{ .Values.pvc.size }} If you scroll back through our k8s manifest file, you’ll see this is exactly the same, except some of the fields are abstracted away into variables surrounded by double curly brackets. This would be saved in templates/pvc.yaml, and all of the other resources would get their own templace, like so. templates/pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: {{ .Values.pvc.name }} spec: accessModes: - {{ .Values.pvc.accessMode }} resources: requests: storage: {{ .Values.pvc.size }} templates/job-workers.yaml apiVersion: batch/v1 kind: Job metadata: name: {{ .Values.workers.jobName }} spec: completions: {{ .Values.workers.count }} parallelism: {{ .Values.workers.count }} backoffLimit: 0 template: spec: restartPolicy: Never containers: - name: worker image: {{ .Values.image }} imagePullPolicy: {{ .Values.imagePullPolicy }} command: [”python”, “/app/worker.py”] env: - name: POINTS_PER_WORKER value: “{{ .Values.workers.pointsPerWorker }}” - name: RESULT_DIR value: “{{ .Values.resultDir }}” volumeMounts: - name: results mountPath: /results volumes: - name: results persistentVolumeClaim: claimName: {{ .Values.pvc.name }} templates/job-reducer.yaml apiVersion: batch/v1 kind: Job metadata: name: {{ .Values.reducer.jobName }} spec: completions: 1 parallelism: 1 backoffLimit: 0 template: spec: restartPolicy: Never serviceAccountName: {{ .Values.rbac.serviceAccount }} initContainers: - name: wait-for-workers image: bitnami/kubectl:latest command: - sh - -c - | echo “[init] Waiting for {{ .Values.workers.count }} worker pods to complete...” while true; do succ=$(kubectl get pods -l job-name={{ .Values.workers.jobName }} \ --field-selector=status.phase=Succeeded \ --no-headers | wc -l) echo “[init] Workers finished: ${succ}/{{ .Values.workers.count }}” if [ “$succ” -ge {{ .Values.workers.count }} ]; then echo “[init] All workers completed, starting reducer.” break fi sleep 2 done containers: - name: reducer image: {{ .Values.image }} imagePullPolicy: {{ .Values.imagePullPolicy }} command: [”python”, “/app/reducer.py”] env: - name: RESULT_DIR value: “{{ .Values.resultDir }}” volumeMounts: - name: results mountPath: /results volumes: - name: results persistentVolumeClaim: claimName: {{ .Values.pvc.name }} templates/rbac-role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: {{ .Values.rbac.role }} rules: - apiGroups: [”“] resources: [”pods”] verbs: [”get”, “list”] - apiGroups: [”batch”] resources: [”jobs”] verbs: [”get”, “list”] templates/rbac-rolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: {{ .Values.rbac.roleBinding }} subjects: - kind: ServiceAccount name: {{ .Values.rbac.serviceAccount }} roleRef: kind: Role name: {{ .Values.rbac.role }} apiGroup: rbac.authorization.k8s.io templates/rbac-serviceaccount.yaml apiVersion: v1 kind: ServiceAccount metadata: name: {{ .Values.rbac.serviceAccount }} For those playing at home, this is the full folder structure of our project so far. pi-estimator/ ├─ local_test.py ├─ Dockerfile └─ chart └─ charts (empty directory, we’ll explore this later) └─ templates ├─ job-reducer.yaml ├─ job-workers.yaml ├─ pvc.yaml ├─ rbac-role.yaml ├─ rbac-rolebinding.yaml └─ rbac-rserviceaccount.yaml └─.helmignore (auto created, when we called helm create) └─ Chart.yaml └─ values.yaml └─ k8s/ ├─ pi-job.yaml └─ pi-image-structure.yaml └─ src/ ├─ worker.py └─ reducer.py └─ local_results/ └─ ....json Hopefully, by now you have some idea of what most of these files represent. Now, we can specify what’s in our values.yaml, thus defining default values for our templates image: “pi-estimator” imagePullPolicy: Never resultDir: “/results” pvc: name: pi-results-pvc size: 1Gi accessMode: ReadWriteMany rbac: serviceAccount: pi-reducer-sa role: pi-reducer-role roleBinding: pi-reducer-binding workers: jobName: pi-workers count: 4 pointsPerWorker: “200000” reducer: jobName: pi-reducer We referenced {{ .Values.workers.count }} within a few of our templates. The Values file has a workers field that has a count, which defaults to 4. We can run our cluster via this chart by running this command from our pi-estimator folder. helm install pi-estimator ./chart You might get some errors if you try to run this though. Recall that Kubernetes is declarative, meaning if you re-run the same script over and over again, it won’t create multiple persistent volumes, for instance. It will just create the one because it already exists. The issue is that helm didn’t create these resources, but they already exist. Helm is squeamish about managing resources that it didn’t create. You can alleviate that by simply deleting all the resources we made from previous runs. kubectl delete jobs --all kubectl delete pods --all kubectl delete serviceaccount pi-reducer-sa kubectl delete role pi-reducer-role kubectl delete rolebinding pi-reducer-binding kubectl delete pvc pi-results-pvc If you re-run helm install pi-estimator ./chart, it should work now. We run this as many times as we want, simply by running helm uninstall pi-estimator then re-running helm install pi-estimator ./chart which is pretty nifty within itself. Instead of having a bunch of resources floating around, they all exist within the same chart, and can be spooled up, taken down, and otherwise managed based on that chart. We can also override the default values right in our helm call. For instance, this will spawn 10 workers which each processes 10 million points for estimating pi. helm install pi-estimator ./chart \ --set workers.count=10 \ --set workers.pointsPerWorker=10000000 10 million points takes a while to get through, which is cool because we can watch our job process real time. If we run the previous helm install command, we can then run kubectl get pods -w which shows all the pods in “watch mode”, meaning it’ll update as pods update their state. At the start, we have a bunch of workers working through their tasks, and the reducer is stuck on initialization. NAME READY STATUS RESTARTS AGE pi-reducer-bjsr5 0/1 Init:0/1 0 44s pi-workers-48qxm 1/1 Running 0 44s pi-workers-7d9cw 1/1 Running 0 44s pi-workers-8k7dn 1/1 Running 0 44s pi-workers-c6kcj 1/1 Running 0 44s pi-workers-dtgl9 1/1 Running 0 44s pi-workers-fkqkd 1/1 Running 0 44s pi-workers-h2522 1/1 Running 0 44s pi-workers-jxhrm 1/1 Running 0 44s pi-workers-mh6z7 1/1 Running 0 44s pi-workers-strsl 1/1 Running 0 44s Then, once the workers finish up, we’ll see the reducer kick into gear soon after and then quickly complete. NAME READY STATUS RESTARTS AGE pi-reducer-bjsr5 0/1 Completed 0 91s pi-workers-48qxm 0/1 Completed 0 91s pi-workers-7d9cw 0/1 Completed 0 91s pi-workers-8k7dn 0/1 Completed 0 91s pi-workers-c6kcj 0/1 Completed 0 91s pi-workers-dtgl9 0/1 Completed 0 91s pi-workers-fkqkd 0/1 Completed 0 91s pi-workers-h2522 0/1 Completed 0 91s pi-workers-jxhrm 0/1 Completed 0 91s pi-workers-mh6z7 0/1 Completed 0 91s pi-workers-strsl 0/1 Completed 0 91s Helm manages the deployment of Kubernetes, but we can still use kubectl to look into the logs from these workers to see what’s up. We can inspect the logs from the workers kubectl logs -l job-name=pi-workers --prefix ... [pod/pi-workers-8k7dn/worker] [worker] Wrote result to /results/result-c8a590c3-3aa6-480c-a0c0-2756e3c11e28.json [pod/pi-workers-h2522/worker] [worker] Completed 10000000 points → inside=7855214 [pod/pi-workers-h2522/worker] [worker] Wrote result to /results/result-7c5332b8-34ed-4672-b4da-27b79908ac7f.json [pod/pi-workers-jxhrm/worker] [worker] Completed 10000000 points → inside=7853418 [pod/pi-workers-jxhrm/worker] [worker] Wrote result to /results/result-80b63c07-b5a4-4035-807c-2ae38e504d1e.json The actual pod id for our reducer was pi-reducer-bjsr5, and we can look into both the initContainer logs kubectl logs pod/pi-reducer-bjsr5 -c wait-for-workers -f [init] Waiting for 10 worker pods to complete... No resources found in default namespace. [init] Workers finished: 0/10 No resources found in default namespace. [init] Workers finished: 0/10 No resources found in default namespace. [init] Workers finished: 0/10 [init] Workers finished: 2/10 [init] Workers finished: 10/10 [init] All workers completed, starting reducer. And we can check out the actual output of the reducer, which yields our prediction. kubectl logs -f job/pi-reducer Defaulted container “reducer” out of: reducer, wait-for-workers (init) [reducer] Collecting results from: /results [reducer] inside_total = 863939569 [reducer] points_total = 1100000000 Distributed π estimate: 3.141598432727273 Here I have a few extra points, because my persistent volume persisted from a previous run, but you get the idea. We could probably stop here if we wanted to, but I don’t. Let’s build an application using some more cool Helm stuff! Using Helm For Real What we just described will get you far, now I want to describe how to get far fast. Templates are a powerful part of Helm, but arguably more powerful part of Helm is the modularity of charts. Recall that, in our folder structure for our Helm chart, there was an empty directory called charts. pi-estimator/ ├─ local_test.py ├─ Dockerfile └─ chart └─ charts <- this guy ... ... That folder is for recording charts that our chart is dependent on. Kind of like how in programming languages like Python, where you can write code that is dependent on certain libraries, in Helm you can co-opt and re-use complete charts that are already created. There’s a whole lot of them, which can do all sorts of cool things. exploring pre-made charts. Source We’re going to make another application that uses some of the more popular pre-built components. I’ve been enjoying chess lately. Let’s make a chess website where we can play against a computer. The Plan If you’ve made it this far in the article, you’re the type of nerd who’s at least heard of chess, and maybe you’ve played a game or two. We’re going to build a chess application that allows us to play chess against a computer opponent. It will consist of the following: A server pod, which manages session data, like active games and moves played, and serves a website, allowing us to play chess on a chessboard A redis database, allowing us to store data and coordinate actions between pods. We’ll talk about this in-depth later. A worker pod, which sees when a player has moved and does computer processing to come up with a retaliatory move We’ll also have monitoring and logging tools, like what a proper production application would have. We’ll cover that later. Ultimately, we’ll be deploying this in Kubernetes with helm, but this is a fair amount of work to do in one big bite. We’re going to explore a subset of this by setting up some Docker files which run our server, redis, and worker pods. We’ll build an application by spooling these up manually, then we’ll work to orchestrate their deployment onto k8s within a single helm chart. Throughout this process, I found that starting with containers then switching to Helm is more trouble than it’s worth. If you’re building a new project, I recommend just starting in Helm. The transition to “these containers work” to “it runs in Helm” is not always as trivial as one might expect. That said, I think exploring the containers themselves is a good place to start from a learning perspective, we’ll first explore our chess application as a set of containers, then make a few modifications to make it work in Helm. The Application The actual logic of the application isn’t the point of this article, so I’m going to blitz through it. The structure of the directory looks like this: chess-app ├── backend/ │ ├── app.py │ ├── Dockerfile │ └── requirements.txt │ ├── chart/ │ └── we’ll talk about this later │ ├── chess-client/ │ ├── Dockerfile │ ├── play.py │ └── requirements.txt │ └── worker/ ├── Dockerfile ├── requirements.txt └── worker.py getting the frontend (which allows us to interact with a board online) to actually work when just running containers ended up being kind of a pain, so I made a simple console-based application called chess-client, which is designed to let us play chess against the computer and see if it’s working. The idea is that we can hopefully run the Worker image in one terminal, the Backend image in another terminal, redis in another terminal, then run our chess-cleint in another terminal. Our chess-client will talk with our backend, which in turn will talk with redis. The worker will observe jobs queued up on redis, and write move results which the Backend will pass back to the chess-cleint. Once we get everything working in K8s via helm, we’ll re-incorporate the frontend and ditch the chess-client. The Backend The following is backend/app.py , which is like the glue of our application that ties all the components together. from fastapi import FastAPI from pydantic import BaseModel import uuid import redis import json from prometheus_client import Counter, Histogram, generate_latest from fastapi.responses import Response import time # Simple logger helper def log(*args): print(”[BACKEND]”, *args, flush=True) app = FastAPI() # -------------------------------------------------------- # Redis initialization # -------------------------------------------------------- REDIS_HOST = “redis-master” log(”Connecting to Redis at:”, REDIS_HOST) try: r = redis.Redis(host=REDIS_HOST, port=6379, decode_responses=True) r.ping() log(”Connected to Redis successfully.”) except Exception as e: log(”ERROR connecting to Redis:”, e) raise REQUEST_COUNT = Counter(”api_requests_total”, “Total API requests”) JOB_LATENCY = Histogram(”job_latency_seconds”, “Time waiting for Stockfish”) class MoveRequest(BaseModel): game_id: str move: str fen: str # -------------------------------------------------------- # Routes # -------------------------------------------------------- @app.get(”/start”) def start_game(): REQUEST_COUNT.inc() game_id = str(uuid.uuid4()) log(f”/start → new game_id generated: {game_id}”) return {”game_id”: game_id} @app.post(”/move”) def make_move(req: MoveRequest): REQUEST_COUNT.inc() job_id = str(uuid.uuid4()) log(f”/move received: game_id={req.game_id}, move={req.move}, fen={req.fen}”) log(f”Generated job_id={job_id}”) # Push job to Redis queue job_payload = { “job_id”: job_id, “game_id”: req.game_id, “move”: req.move, “fen”: req.fen } r.rpush(”jobs”, json.dumps(job_payload)) log(”Job pushed to Redis:”, job_payload) # Wait for worker response with JOB_LATENCY.time(): log(”Waiting for worker to compute result...”) while True: result_raw = r.get(f”result:{job_id}”) if result_raw: log(”Result received from worker:”, result_raw) # Cleanup Redis key r.delete(f”result:{job_id}”) log(”Deleted Redis key:”, f”result:{job_id}”) return json.loads(result_raw) # Avoid spinning too hot — also logs periodically to prevent total silence time.sleep(0.05) @app.get(”/metrics”) def metrics(): log(”/metrics scraped”) return Response(generate_latest(), media_type=”text/plain”) This is the Backend of the application. It has some details we’ll talk about when we get to the Helm part (namely Prometheus), but the majority of it is pretty simple. It’s a fastAPI api, which is a lightweight and convenient way to make an API in Python. We’re defining a start, move, and metrics endpoint which will allow us to start a new game of chess, make a move in that game of chess, and get some key metrics (which uses Prometheus. Again, we’ll talk about that later). The start endpoint simply identifies a new UUID. A UUID is a “universal unique identifier”, which is essentially a long string of random numbers. Technically, it’s possible to create a UUID that’s the same as another one, but they’re so long, it’s practically impossible. the annual risk of a given person being hit by a meteorite is estimated to be one chance in 17 billion, which means the probability is about 0.00000000006 (6 × 10−11), equivalent to the odds of creating a few tens of trillions of UUIDs in a year and having one duplicate. In other words, only after generating 1 billion UUIDs every second for the next 100 years, the probability of creating just one duplicate would be about 50%. — Source Thus, practically, we can create a new, completely unique identifier, which will identify a new game. After our client calls the start endpoint, they’ll use the UUID to tell the server which game they want to make a move in. Thus, multiple parallel games can be going on simultaneously, with different users making different games with different UUIDs. When a player does make a move, they use the move endpoint. arguments to the move endpoint consists of the UUID of the game you’re playing in, the move you want to make, and something called a “fen”, which stands for “Forsyth-Edwards Notation”. It’s essentially a string of text that represents a chessboard position. For instance, this represents the starting position of chess: FEN: rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR These details don’t matter for understanding Kubernetes, nor do they matter for Helm. However, naturally, it’s important to make a system that can play chess. We also create a job_id, which is yet another UUID. The whole idea of this application is that, after you make a move, a computer will compute and make a retaliatory move. The process of calculating that move can be computationally expensive. Imagine we had a few different people playing games of chess at the same time, we might need to process many different games simultaneously. Thus, we might want a few computers working in parallel to solve positions as quickly as possible. To solve this, our app uses something called a “redis queue”. Redis is a very efficient and very popular in-memory database that can be used to store small amounts of high-speed data. It’s great for keeping track of a queue of jobs that many pods might be interested in within a cluster. At the beginning of this script, we ran r = redis.Redis(host=REDIS_HOST, port=6379, decode_responses=True) which connects to a container running Redis, and assigns that connection to a variable r. When we make a move, we register the data for a job to redis with r.rpush(”jobs”, json.dumps(job_payload)) This is a queue of jobs that the pods running our worker nodes need to process. Once they get around to processing the job (which creates a new move), the worker will add an element in the queue with an id of result:{job_id} Once we find that, we delete it from the queue, and send the computer-evaluated move to the client to update the board. This general architecture is what’s called “producer-consumer”. Our backend “produces” jobs, and our workers “consume” them. Similarly, our workers “produce” results, and our backend “consumes” them. Thus we consistently have workers trying to solve positions as fast as they can, and the backend updates the players board position as soon as a result is available. The Backend is served with uvicorn, just like any standard fastAPI app, so our Dockerfile looks like this FROM python:3.10-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY app.py . CMD [”uvicorn”, “app:app”, “--host”, “0.0.0.0”, “--port”, “80”] We also have a requierments.txt, which the Dockerfile uses to install python dependencies fastapi uvicorn[standard] redis prometheus-client pydantic The Worker Here’s worker/worker.py, which consumes jobs in Redis and spits out predicted moves back into Redis. import redis import json import time import traceback from prometheus_client import start_http_server, Counter from stockfish import Stockfish import os # Simple logging helper def log(*args): print(”[WORKER]”, *args, flush=True) # -------------------------------------------------------- # 1. Load Stockfish # -------------------------------------------------------- STOCKFISH_PATH = os.getenv(”STOCKFISH_EXECUTABLE”, “/usr/games/stockfish”) log(”Using Stockfish path:”, STOCKFISH_PATH) try: ENGINE = Stockfish( path=STOCKFISH_PATH, parameters={”Threads”: 1, “Skill Level”: 10} ) log(”Stockfish initialized successfully.”) except Exception as e: log(”Stockfish FAILED to start:”, e) log(traceback.format_exc()) raise # -------------------------------------------------------- # 2. Redis connection # -------------------------------------------------------- REDIS_HOST = os.getenv(”REDIS_HOST”, “redis-master”) log(”Connecting to Redis at:”, REDIS_HOST) try: r = redis.Redis(host=REDIS_HOST, port=6379, decode_responses=True) r.ping() log(”Connected to Redis.”) except Exception as e: log(”FAILED to connect to Redis:”, e) log(traceback.format_exc()) raise REQUESTS = Counter(”engine_requests_total”, “How many Stockfish requests”) # -------------------------------------------------------- # 3. Worker loop # -------------------------------------------------------- def main(): log(”Starting Prometheus metrics on port 9000...”) start_http_server(9000) log(”Worker READY. Waiting for jobs...”) while True: try: job = r.lpop(”jobs”) if not job: time.sleep(0.5) continue log(”Got job:”, job) data = json.loads(job) REQUESTS.inc() # --------------------------------------------- # FIX: translate ‘startpos’ into full FEN # --------------------------------------------- fen = data[”fen”] if fen == “startpos”: fen = “rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1” log(”Translated ‘startpos’ to full FEN:”, fen) log(”Setting FEN:”, fen) ENGINE.set_fen_position(fen) best_move = ENGINE.get_best_move() log(”Best move computed:”, best_move) result_key = f”result:{data[’job_id’]}” r.set(result_key, json.dumps({”best_move”: best_move})) log(”Wrote result to Redis:”, result_key) except Exception as e: log(”ERROR in worker loop:”, e) log(traceback.format_exc()) time.sleep(0.05) if __name__ == “__main__”: log(”>>> Worker starting up...”) main() We’re using Stockfish to do analysis. I want to make an article on Stockfish and how it works, but basically it’s a very small, efficient, and powerful chess engine. Once we set up Stockfish and connect to Redis, the workers are really simple. Every so often, our worker checks Redis for jobs job = r.lpop(”jobs”) it then passes the position to Stockfish to get a move, then publishes the result to Redis in a way our backend will understand. The Dockerfile for this has the usual suspects, paired with some code for installing Stockfish into the Docker image. FROM python:3.10-slim # Install Stockfish engine binary RUN apt-get update && \ apt-get install -y --no-install-recommends stockfish && \ rm -rf /var/lib/apt/lists/* WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY worker.py . # documenting where stockfish lives (on Debian it’s usually /usr/games/stockfish) ENV STOCKFISH_EXECUTABLE=/usr/games/stockfish CMD [”python”, “worker.py”] and it also has a requierments.txt redis prometheus-client stockfish The Chess Client Later, we’ll pack this up into a website, but for now, I wanted to make a simple container that we could interface with via the command line to play chess. Just something simple we could play around with to make sure everything works properly. chess-client/play.py import requests from chess import Board import os API_URL = os.getenv(”API_URL”, “http://backend-test:80”) # default if running inside minikube def print_board(board): print(board) print() def main(): print(”Starting a new game...”) try: game_id = requests.get(f”{API_URL}/start”).json()[”game_id”] except Exception as e: print(”ERROR contacting backend:”, API_URL) print(e) return board = Board() print_board(board) while not board.is_game_over(): move_uci = input(”Your move (e.g., e2e4): “).strip() try: board.push_uci(move_uci) except Exception: print(”Invalid move, try again.”) continue print(”\nYou played:”, move_uci) print_board(board) print(”Waiting for engine response...”) res = requests.post( f”{API_URL}/move”, json={ “game_id”: game_id, “move”: move_uci, “fen”: board.fen() } ) if not res.ok: print(”Backend error:”, res.text) break data = res.json() engine_move = data[”best_move”] print(”Engine plays:”, engine_move) board.push_uci(engine_move) print_board(board) print(”Game over!”) print(board.result()) if __name__ == “__main__”: main() This is really chess-specific, so I don’t want to spend too long getting into it. Basically, it connects to our backend image API_URL = os.getenv(”API_URL”, “http://backend-test:80”) Starts a new game game_id = requests.get(f”{API_URL}/start”).json()[”game_id”] requests you to make a move move_uci = input(”Your move (e.g., e2e4): “).strip() then sends that move to the backend, which will in turn make a job and thus get a new computer-generated move response res = requests.post( f”{API_URL}/move”, json={ “game_id”: game_id, “move”: move_uci, “fen”: board.fen() } ) and updates the board with that move board.push_uci(engine_move) It does this until the game is over, allowing us to play a game of chess against a computer! Running Our Chess Game directly in Minikube We don’t need to set up all the Helm stuff to play around with this. Again, I think I probably should have because I essentially needed to make this app twice and do a bunch of work, but you can benefit from my suffering. Grab some popcorn, I’m gonna spin up a few terminals. We’ll need four; one for redis, one for the backend, one for the worker, and one for our client. These will be like pods in a Kubernetes cluster talking with one another, except I’m managing all the images manually. Before that, though, we need to actually build our images. If we cd into our chess-app directory, we can run the following commands docker image build -t chess-backend ./backend docker image build -t chess-worker ./worker docker image build -t chess-client ./chess-client We’ll then need to build something called a network docker network create chess-net By default, Docker containers can’t communicate with each other, which obviously they’ll have to to make this whole thing work. If we create a network, then create our images within that network, it will allow those images to talk with one another. Redis is kind of the center of all of this, and both our backend and worker need Redis to be running to work properly. So, we’ll start up Redis first. docker run -d --network chess-net --name redis-master redis This runs redis in our network with the name redis-master, which is required because we connect to it by name in our code. We don’t need to keep this terminal open, the Redis pod is running in the background. Now we can open up a terminal and run docker run --network chess-net -p 8080:80 --name backend-test chess-backend you should see this [BACKEND] Connecting to Redis at: redis-master [BACKEND] Connected to Redis successfully. INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit) Now our backend is working. We can open up yet another terminal and run our worker docker run --network chess-net --name worker-test chess-worker you should see something like this [WORKER] Using Stockfish path: /usr/games/stockfish [WORKER] Stockfish initialized successfully. [WORKER] Connecting to Redis at: redis-master [WORKER] Connected to Redis. [WORKER] >>> Worker starting up... [WORKER] Starting Prometheus metrics on port 9000... [WORKER] Worker READY. Waiting for jobs... And, finally, we can open up yet another terminal and launch our chess client, which will talk with our backend. docker run -it --network chess-net chess-client And, with that, we can play chess Playing chess against our computer This is pretty rad, and we might be able to deploy this on a computer and have a little chess website. But what if we get more people? What if a server goes down? To make this a proper application that could stand up to real world usage, we’re going to deploy this to K8s via helm. The Chess App in K8s via Helm As I previously mentioned, I had to do a bunch of tiny little tweaks to the application, and I don’t want to go through everything again. In the GitHub repo attached to this article I included both versions of this application, which you can take a gander at yourself if you really want to. We’re going to shift over to the helm-deployed version of this, and talk over some of the highlights. Another caveat is that this app isn’t a perfect production application, realistically, you would want to do more research and think more closely about configuration if you were actually rolling out a product. Technical maturity comes with time. One major change actually has nothing to do with Helm. I wanted to have this run in a proper website, so within the backend I added a few files chess-app-helm ├── backend/ │ ├── app.py │ ├── Dockerfile │ ├── requirements.txt │ └── static │ ├── app.js │ ├── index.html │ └── style.css ... other things the new static folder in backend represents all of the assets needed to serve to a website to get it to work correctly. and, in our backend’s app.py, we serve this to the user ... @app.get(”/”, response_class=HTMLResponse) def index(): with open(”static/index.html”, “r”) as f: return f.read() ... Originally, I thought it would be cute to have two servers, one that does all the “backend-y” stuff, and another that serves just the website. There’s not a really compelling reason for this, though, and I kept bumping into CORS issues, so I decided to just serve it from the backend. Besides that, the only significant change is that we have a chart directory with templates. And this time we’re actually using the charts/charts/ directory, so we can actually talk about that. CHESS-APP-HELM ├── backend/ │ └──... │ ├── chart/ │ ├── charts/ │ ├── templates/ │ │ ├── _helpers.tpl │ │ ├── backend-deployment.yaml │ │ ├── backend-service.yaml │ │ ├── ingress.yaml │ │ ├── worker-deployment.yaml │ │ └── worker-service.yaml │ ├── Chart.lock │ ├── Chart.yaml │ └── values.yaml │ └── worker/ └──... As usual we need to define some templates, but first I want to take a look at Chart.yaml apiVersion: v2 name: chess description: Chess engine demo with backend, worker, and monitoring type: application version: 0.1.0 appVersion: “1.0” dependencies: - name: kube-prometheus-stack version: “55.8.0” repository: “https://prometheus-community.github.io/helm-charts” - name: redis version: “19.6.0” repository: “https://charts.bitnami.com/bitnami” It’s got some dependencies! About fifteen thousand words into the article, we have arrived at the most important use of Helm, arguably. Helm lets you use pre-made charts within your chart, allowing you to connect different powerful pre-made components together. Obviously, we’re using redis. When we launch our Helm chart, we’ll automatically spawn a redis cluster based on the dependency we specified. I wanted to get fancy, so I also included prometheus as a dependency as well. This is a logging tool used by big, serious companies to keep track of key metrics within their cluster. We won’t be digging too hard into prometheus, but I wanted to include it to show how you can just, kinda, tack on whole complex charts with a single dependency call. When we actually get this thing running, Helm will download those dependencies, and they’ll end up living in the charts/charts directory. This is part of the reason it’s nice to just start with Helm, rather than build things out yourself and then stick them into Helm later. It defeats a big part of the purpose of the helm in the first place. I spent a long time re-configuring stupid things with Redis because of differences in how I deploy it manually vs how I set up deployment in Helm. Also, Helm makes it really easy to uninstall, reinstall, and update a whole chart, which is useful when you’re building stuff. Just start with Helm. Anywho, within our templates we have a new file type; _helpers.tpl, which is a template file. It lets us define a few handy snippets which we can use across templates, and looks like this: {{- define “chess.backendName” -}} {{ include “chess.fullname” . }}-backend {{- end }} {{- define “chess.workerName” -}} {{ include “chess.fullname” . }}-worker {{- end }} {{- define “chess.frontendName” -}} {{ include “chess.fullname” . }}-frontend {{- end }} {{- define “chess.fullname” -}} {{ .Chart.Name }} {{- end }} This lets us define a few standard naming conventions we can use across different templates to keep things straight. It’s a little pedantic because it’s AI generated, but whatever. It defines: chess.fullname: The name of the chart chess.frontendName: The name of the chart, plus “-frontend” Same deal for the worker and backend. Because resources communicate with one another by name, including connecting to each other in the code, it’s super useful to define these as consistent variables that can be injected throughout various templates. Also, see how there’s a definition for the frontend, but previously I said I ditched the frontend? I got things working and didn’t want to go through all the files. There might be some dangling references to the frontend, but they generally don’t matter (probably, or if you delete it it will cause some random error somewhere). I also have a Values.yaml, with some defaults backend: image: chess-backend imagePullPolicy: IfNotPresent port: 80 replicas: 1 service: type: NodePort nodePort: 30081 worker: image: chess-worker imagePullPolicy: IfNotPresent replicas: 1 redis: image: tag: latest master: persistence: enabled: false replica: persistence: enabled: false auth: enabled: false ingress: enabled: false host: chess.local The charts we’re calling in dependencies can be configured within the values.yaml, which is another reason you should start with Helm. I had some issues where I was expecting to assign a name to redis, but that broke Redis because it expected a name that it had defined itself. I had to go through the worker and backend and adjust their naming convention to be able to find Redis based on the name it wanted to have, which was chess-redis-master. I’m assigning ports and doing generally boring stuff in this, which I landed on after boring research and looking up examples with defaults. I’m sure you could dig into each of these decisions, but they’re not very conceptually interesting, just the kind of stuff you need to play around with when actually making something. One exception to that, though, is ingress within values.yaml. ingress: enabled: false host: chess.local One of my templates, ingress.yaml uses this as an if statement. {{- if .Values.ingress.enabled }} apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: chess-app spec: rules: - host: {{ .Values.ingress.host }} http: paths: - path: / pathType: Prefix backend: service: name: {{ include “chess.frontendName” . }} port: number: 80 {{- end }} Recall that, when we download with Helm, it essentially combines all templates into a single manifest. By wrapping our configuration in this if statement, we can use our values.yaml to completely disable certain components. Again, this doesn’t matter because our frontend doesn’t even exist anymore, but I still thought it was a cool note. Let’s look at some of the templates for the actual resources we’re using. backend-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: {{ include “chess.backendName” . }} spec: replicas: {{ .Values.backend.replicas }} selector: matchLabels: app: {{ include “chess.backendName” . }} template: metadata: labels: app: {{ include “chess.backendName” . }} annotations: prometheus.io/scrape: “true” prometheus.io/port: “{{ .Values.backend.port }}” prometheus.io/path: “/metrics” spec: containers: - name: backend image: “{{ .Values.backend.image }}” imagePullPolicy: “{{ .Values.backend.imagePullPolicy }}” ports: - containerPort: {{ .Values.backend.port }} env: - name: REDIS_HOST value: “{{ .Release.Name }}-redis-master” First of all, unlike the jobs from the previous example, this is a deployment from the app/v1 api, meaning Helm will try to keep this thing running indefinitely. It has a name , which it gets from _helpers.tpl and a spec with a few different things going on. First of all, we define some number of replicas for the backend deployment. The backend has a service associated with it (which we’ll look at next), which automatically load balances to some number of replica instances of a pod. This allows k8s to be more robust to traffic, and more resilient to outages. Weirdly, deployments keep track of managing pods not by keeping track of which pods it created, but only based on labels. This code snippet basically says “these are the pods I control” selector: matchLabels: app: {{ include “chess.backendName” . }} And this other one, in the template, says this is the label I will assign to pods I create. template: metadata: labels: app: {{ include “chess.backendName” . }} Generally, these should always agree with each other, for obvious reasons. Otherwise, the deployment would make pods that it then immediately forgets about, and I don’t even know what would happen. We also have some annotations that tell Prometheus that we want it to record data from the pods we’re creating. annotations: prometheus.io/scrape: “true” prometheus.io/port: “{{ .Values.backend.port }}” prometheus.io/path: “/metrics” You might recall, in our app.py in our backend, there was an endpoint for metrics @app.get(”/metrics”) def metrics(): return Response(generate_latest(), media_type=”text/plain”) generate_latest is imported from the Prometheus library, and returns metrics we manually created through the applications, and also other metrics that Prometheus records all on its own. Because Prometheus exists as a dependency, it’s sitting around waiting to collect data within our cluster. By setting these annotations in the template, we’re making it so Prometheus actually collects that data, and knows where to grab that data. The rest is a lot of the typical stuff we’ve discussed previously spec: containers: - name: backend image: “{{ .Values.backend.image }}” imagePullPolicy: “{{ .Values.backend.imagePullPolicy }}” ports: - containerPort: {{ .Values.backend.port }} env: - name: REDIS_HOST value: “{{ .Release.Name }}-redis-master” We’re deploying our Docker container, exposing some ports, and saving the name of Redis as an environment variable so our backend can connect to it. These backend pods, one if we only have one replica, more if we have more, are exposed behind one networking endpoint due to backend-service.yaml. apiVersion: v1 kind: Service metadata: name: {{ include “chess.backendName” . }} spec: type: {{ .Values.backend.service.type }} selector: app: {{ include “chess.backendName” . }} ports: - name: http port: {{ .Values.backend.port }} targetPort: {{ .Values.backend.port }} nodePort: {{ .Values.backend.service.nodePort }} The worker deployment is pretty much the same as the backend deployment worker-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: {{ include “chess.workerName” . }} spec: replicas: {{ .Values.worker.replicas }} selector: matchLabels: app: {{ include “chess.workerName” . }} template: metadata: labels: app: {{ include “chess.workerName” . }} annotations: prometheus.io/scrape: “true” prometheus.io/port: “9000” prometheus.io/path: “/metrics” spec: containers: - name: worker image: “{{ .Values.worker.image }}” imagePullPolicy: “{{ .Values.worker.imagePullPolicy }}” env: - name: REDIS_HOST value: “{{ .Release.Name }}-redis-master” ports: - containerPort: 9000 The worker also has a service, worker-service.yaml. It’s not really necessary because each of the pods are just getting jobs from redis, and writing results. It is necessary for Prometheus to be able to scrape performance data, though. apiVersion: v1 kind: Service metadata: name: {{ include “chess.workerName” . }} spec: type: ClusterIP selector: app: {{ include “chess.workerName” . }} ports: - name: metrics port: 9000 targetPort: 9000 If it seems like this is a lot of handwavy descriptions, it is. I think that’s the nature of this type of work; you shouldn’t be re-inventing the wheel, and there’s no reason for most developers to understand each parameter in depth. Follow templates, look up best practices, and dig in when you encounter an issue. I think the most important part is to understand how these resources relate from a high level to power applications, and to know enough to have a base you can branch off of as necessity demands. This isn’t designed to be a poster child of the perfect helm app, just an example to start building a conceptual understanding. I think we covered all the helm stuff. Let’s run it Running the Helm App, Using it, and Observing Metrics It’s funny. It takes forever to describe what’s going on in Helm, then running it is like three command lines. I separated the helm deployment of the chess app into its own folder (chess-app-helm), so I’m going to go ahead and build all the images required. These are being built with minikube so they’ll end up in minikube. minikube image build -t chess-backend ./backend minikube image build -t chess-worker ./worker Then we need to install the dependencies helm dependency update ./chart This will populate our chart/chart/ with our downloaded charts. Then we can go ahead and install the app. helm install chess ./chart -n chess --create-namespace If we call this to get the pods we just created kubectl get pods -n chess we’ll see we’re encountering issues NAME READY STATUS RESTARTS AGE alertmanager-chess-kube-prometheus-stac-alertmanager-0 2/2 Running 0 34s chess-backend-57dd7cf788-cht69 0/1 Error 2 (30s ago) 35s chess-grafana-6547bcdd7d-6fkqb 3/3 Running 0 35s chess-kube-prometheus-stac-operator-85b4f7c55-j86kf 1/1 Running 0 35s chess-kube-state-metrics-6d67db698b-qlnxt 1/1 Running 0 35s chess-prometheus-node-exporter-wlnzt 1/1 Running 0 35s chess-redis-master-0 1/1 Running 0 35s chess-redis-replicas-0 1/1 Running 0 35s chess-redis-replicas-1 0/1 Running 0 7s chess-worker-75b8f44456-4b79q 0/1 CrashLoopBackOff 2 (16s ago) 35s prometheus-chess-kube-prometheus-stac-prometheus-0 2/2 Running 0 34s This is because the backend and worker are trying to connect to redis, but Redis isn’t fully set up yet. After around 30 seconds it connects successfully. NAME READY STATUS RESTARTS AGE alertmanager-chess-kube-prometheus-stac-alertmanager-0 2/2 Running 0 2m14s chess-backend-57dd7cf788-cht69 1/1 Running 3 (114s ago) 2m15s chess-grafana-6547bcdd7d-6fkqb 3/3 Running 0 2m15s chess-kube-prometheus-stac-operator-85b4f7c55-j86kf 1/1 Running 0 2m15s chess-kube-state-metrics-6d67db698b-qlnxt 1/1 Running 0 2m15s chess-prometheus-node-exporter-wlnzt 1/1 Running 0 2m15s chess-redis-master-0 1/1 Running 0 2m15s chess-redis-replicas-0 1/1 Running 0 2m15s chess-redis-replicas-1 1/1 Running 0 107s chess-redis-replicas-2 1/1 Running 0 81s chess-worker-75b8f44456-4b79q 1/1 Running 3 (116s ago) 2m15s prometheus-chess-kube-prometheus-stac-prometheus-0 2/2 Running 0 2m14s We could probably make this a bit more elegant by making an initContainer on the worker and backend that waits for Redis to be set up, but whatever. We can expose the backend service as a url via minikube service chess-backend -n chess --url and, if we click the url we get, we’ll see a chess board! We can move a piece, and we’ll get an indicator that the bot is thinking. Once it’s done, the bot will move one of its pieces. Et voila, chess app in kubernetes with helm. It’s missing some core features of a chess app (like move validation, you can move the pieces around anywhere, which is pretty funny), but that’s not the point. The point is that we could deploy this on AWS, Google Cloud, or an on-prem server relatively easily. We can also scale up resources and have them load balance effectively. Of course there would probably be some bumps along the way, this is just a demo, but we could get there. We can view some of the logs in Prometheus. First lets list out our pods in the chess namespace kubectl get pods -n chess NAME READY STATUS RESTARTS AGE alertmanager-chess-kube-prometheus-stac-alertmanager-0 2/2 Running 0 34m chess-backend-57dd7cf788-cht69 1/1 Running 3 (34m ago) 34m chess-grafana-6547bcdd7d-6fkqb 3/3 Running 0 34m chess-kube-prometheus-stac-operator-85b4f7c55-j86kf 1/1 Running 0 34m chess-kube-state-metrics-6d67db698b-qlnxt 1/1 Running 0 34m chess-prometheus-node-exporter-wlnzt 1/1 Running 0 34m chess-redis-master-0 1/1 Running 0 34m chess-redis-replicas-0 1/1 Running 0 34m chess-redis-replicas-1 1/1 Running 0 34m chess-redis-replicas-2 1/1 Running 0 33m chess-worker-75b8f44456-4b79q 1/1 Running 3 (34m ago) 34m prometheus-chess-kube-prometheus-stac-prometheus-0 2/2 Running 0 34m we can then port-forward the local Prometheus port to a remote port, so we can see it in the browser. kubectl port-forward -n chess prometheus-chess-kube-prometheus-stac-prometheus-0 9090:9090 we can then open up Firefox, go to http://localhost:9090 (click the link to view mine), and you should see a cool dashboard with stuff going on. We can even navigate around and check out cool graphs of our CPU usage and stuff. Again, this isn’t a Prometheus tutorial, but it’s kind of cool how we get all this functionality just by installing a dependency graph and doing a bit of configuration. It’s a lot to set up for a tutorial, but for a large-scale product it’s pretty easy for how powerful it is. For whatever reason my manually defined metrics aren’t working in this example, but whatever. I think we’ve covered enough for one article. Conclusion Helm/Kubernetes isn’t the easiest thing to work with. There’s a ton of configuration, a bunch of gotchas, and the abstraction makes development complicated and counterintuitive. It’s not worth investing in if you’re building a weekend project, and I would avoid it in favor of more pre-made backend tooling if working in a small team. That said, if you need it, you really need it. With the added complexity comes a tremendous benefit of power and flexibility. Kubernetes is an upfront investment that pays dividends as resources, users, and complexity scale within an application. I think that’s a big reason there aren’t a ton of accessible tutorials for it online. It’s not super fun or flashy, it’s abstract, and demos look the same as what you could whip up in npm in 5 minutes. You should really only learn Kubernetes if you’re interested in working on things that matter, which is a subject many technical tutorials avoid in my experience. I lied in the title; this is not an “exhaustive” tutorial, I don’t think that’s possible, practical, nor necessary. I think my job has been a success if you have a sufficient conceptual basis to know what Kubernetes is good for, and a base of understanding you can leverage when working with it on practical projects. I know I haven’t posted in a while, I have a few big articles like this one in the works. Stay posted. Apache Spark — Intuitively and Exhaustively Explained Daniel Warfield — Tue, 07 Oct 2025 14:02:33 GMT “Spark” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained. In this article we’ll discuss Apache Spark, a popular computing system designed for modern, large scale data processing. Netflix, Uber, Facebook, JPMorgan Chase, Amazon, and many more major companies use Spark to do a huge number of critical operations on a daily basis. In this article we’ll explore why Spark is useful, and how it works. Subscribe To Intuitively and Exhaustively Explained. Share this article with friends and colleagues. Share Who is this useful for? Anyone interested in building cutting edge data processing pipelines at-scale How advanced is this post? This article is designed to be a first principles conceptual breakdown of spark, and is accessible to readers of all levels. Prerequisites: None, though Python experience is recommended in understanding the implementation section. Experience in SQL, Pandas, and Polars is also helpful. A Core Problem In the early 2000s, Google was on the verge of an absurd amount of growth. Number of googles per year. source And, on top of that, they were gearing up to release a huge number of what would later become wildly successful products. Timeline of Google releases source These required massive computational resources; think creating search indexes, sorting, data mining, machine learning, and doing statistical analysis. On the volume of data Google was working with, many of their operations couldn’t be done on a single machine. Instead, many computers had to work together to tackle these problems. At the time, getting work like this done was un-standardized. Google had a variety of technologies for distributed data management, but it was up to very experienced developers to stitch these together when building any new, major parallel operation. Enter MapReduce, the predecessor to Spark. MapReduce Engineers at Google realized that the majority of their bespoke parallelization code was doing the same two fundamental operations: Map: Transforming an input into some output given a function Reduce: Accumulate results together. For example: If you wanted to count all the word occurrences in a large document (how often each word is used), you could map all of the pages into a word count, then take all of the word counts for all pages and reduce them to a single word count. If you wanted to calculate how often certain websites are linked, you can map each website to a list of links then reduce that to a count across all websites. If you wanted to calculate the average luminosity of all videos on a social media platform, you can map each video into an average luminosity across the video, then reduce that across all videos. Because so many different types of large scale, common operations could be easily conceptualized with mapping and reduction, Google decided to make a framework specifically for these operations, called MapReduce. The MapReduce paper. Source The high level workflow of MapReduce is defined with three key things: The input data, and how it can be logically divided. A mapping function, that can work on an individual piece of data. A reduction function, that can take many different mapped values and convert them into an output representation. figure from the map reduce paper. You can use multiple workers assigned to the mapping step, and multiple workers assigned to the reduce step, and thus run very large processing jobs in parallel. Source. There are more details under the hood. For instance, MapReduce uses a concept of shuffling to randomly distribute load, and key/value pairs to control which data is allowed to be reduced, but these specifics are out of scope for this article. What’s important to know, though, is how MapReduce functions in essence: it splits data up into bite sized operations so that different computers can work on the problem in parallel, and it does that in a standardized way that allows one to leverage a large cluster of computers without needing to code up all the low level parallelization. MapReduce was a dominant paradigm for a long time. It allowed individual developers to implement solutions that previously required large and expensive multi-disciplinary teams. Also, the standardization of so many different tasks into a single framework allowed the industry to standardize on a universal set of tooling. “Apache Hadoop” was an incredibly popular framework for doing MapReduce at scale, and allowed companies to standardize their tooling around how Hadoop implemented MapReduce. source. To make MapReduce work at scale, Hadoop implemented a bunch of critical components which work in conjunction with MapReduce: YARN (Yet Another Resource Negotiator): manages cluster resources (like how CPUs and memory get allocated throughout a cluster) and task scheduling. HDFS (Hadoop Distributed File System): Designed to store large datasets across many systems, with redundant stores for fault tolerance. A bunch of tools for data management, compatability with many file formats, search tools, and other fun stuff. This was a paradigm shift for large scale computing. Before Hadoop, large scale parallel jobs were largely bespoke. Now, standardized tools could be plugged into a standardized system, allowing for unprecedented interoperability at scale. The Issue With Hadoop Hadoop is still used at a large number of companies, and for good reason; it has a mature ecosystem which is well integrated with the needs of many large scale companies that need large scale data processing. However, if you pull a data engineer to the side in 2025 and ask them what tool is worth learning, most of them will say Spark is more important than Hadoop. The main problem with Hadoop is the times it was designed in. Hadoop hit the scene in 2006 when cheap harddrives was the norm, RAM was expensive, and machines failed all the time. The name of the game was fault tolerance over speed, so Hadoop spends a lot of time saving data to and reading from disk. The anatomy of a typical computer. Most computers have a CPU, which does most of the major computation, RAM, which is like the working memory of the computer, and storage, like a hard drive or SSD. RAM is much faster, and much more expensive than storage, so Hadoop stores and reads from storage often. As hardware has progressed, RAM prices have fallen and hardware has gotten more reliable. From my article on CUDA for AI. In 2012, when spark was released, this paradigm was starting to feel dated. RAM was much more plentiful, networks were faster and more reliable, and more robust processing tasks were in higher demand. Some of the core design paradigms which made Hadoop a success were losing relevance. Enter Spark If you go to the main site for Apache Spark, it defines the project as: Apache Spark™ is a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters. The Wikipedia page says: Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance And the PySpark PyPI page describes Spark as Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing. In Hadoop, you had an engine that just did MapReduce. Spark, in contrast, is more general purpose, has more complex functionality, and is thus harder to define. Spark is inspired by Hadoop, but there are some fundamental differences in their core design principles which make them fundamentally different. Hadoop expects the developer to define two major tools; map and reduce. Once you implement those tools, Hadoop applies them to your data. The essence of turning some algorithm into something that can run on Hadoop. You have to turn your algorithm into mapping and reduction, then you can execute it at scale with Hadoop. In modern Hadoop this isn’t true, there are a ton of frameworks and abstractions built on top of Hadoop that allow for more complex operations (You can even use Spark to interface with the Hadoop resource manager), but this core paradigm was present during the creation of Spark. Spark, on the other hand, employs an “execution engine” paradigm. Spark exposes a library of tools which you can choose to employ. These are defined as “transformations” and “actions” which we’ll discuss in future sections. Spark has tools for map and reduce, which allows Spark to emulate the MapReduce functionality of Hadoop, but the Spark execution engine also has functionality for filtering, sampling, grouping, counting, and iterating, allowing Spark to execute much more complex operations. In Hadoop you define two rigidly constrained tools to be broadcast over the cluster. In Spark, you can use a range of more complex pre-defined tools to do much more flexible and sophisticated tasks. To support this more complex general functionality, Spark employs a computational graph which we’ll also explore later. Another major difference between Spark and Hadoop is that Spark employs an in-memory first design paradigm. Instead of reading and writing to storage, like what Hadoop usually does, Spark tries to keep as much data in-memory as possible, and only relies on storage to drive as a fallback. This makes Spark significantly faster than Hadoop in many applications, and enables real-time analysis of very, very large datasets. That’s the thousand yard view of Spark. It’s like Hadoop, but it’s more flexible and faster. Let’s dig into some of the core components of spark that make it tick. Resilient Distributed Datasets (RDDs) and Lazy Evaluation The “Resilient Distributed Dataset”(RDD) is one of the most fundamental ideas in Spark. RDDs represent distributed collections of data and define what operations should be computed to transform that data across a cluster. They don’t hold the data themselves — instead, they record where the data came from and what transformations should be applied. We’ll cover this more in depth later, for now you can think of RDDs as a record of how data changes throughout a data processing pipeline. There are four defining properties of RDDs that I think are fundamental to understand when getting into Spark: Immutable: Once you make an RDD, you can’t change it. If you want to change the data, you have to make a new RDD. Distributed: RDDs can be stored in a distributed manner across many nodes (computers) within a cluster. Lazily Evaluated: RDDs don’t hold data per-se, they keep track of where data comes from and what operations have been applied to that data. Partition-Oriented: RDDs aren’t designed to work on a particular type of data. Rather, they work on partitions of data. You can have text files, images, audio, whatever. RDDs don’t care, they just care if you can sub-divide the data. RDDs manage those subdivisions. Take, for example, the following PySpark code (PySpark being a way to interface with Spark via Python). rdd = spark.textFile(”hdfs:///data.txt”) words = rdd.flatMap(lambda line: line.split()) filtered = words.filter(lambda w: w != “the”) here, rdd, words, and filter are all RDDs which, when linked together, define a computational graph. A conceptual diagram of what’s happening. the “rdd” RDD is created by reading data.txt. The “words” RDD is created by doing an operation on the “rdd” RDD, and the “filtered” RDD is created by doing an operation on the “words” RDD. This is called a “lineage graph”. In Spark, relationships between RDDs and their connections is used to optimize and orchestrate computation. When applying the flatMap operation to rdd, we don’t modify rdd, but rather create a new RDD called words. rdd still exists as an immutable record. Each of these three RDDs are resilient and distributed, meaning they can be referencing data across various different computers. One of the key, defining characteristics of an RDD is that they can be distributed across various computers. For a single small text file this would hardly be necessary, but when processing large amounts of data (which might be larger than what a single machine can store) it’s critical to be able to work on that data in chunks. RDDs are fundamentally designed to work on partitionable data that can be applied across multiple computers in a cluster. This allows Spark to parallelize both memory and computation. These computations are not executed by default. Rather, they’re only executed when triggered, for instance, by calling collect on the last RDD. result = filtered.collect() This triggers Spark to actually execute the computations in the directed graph and provides the final result. This is called lazy evaluation, and it has some compelling benefits which we’ll describe later in the article. So, Spark uses data divided up into partitions to do large scale processing, and encodes operations applied to those partitions as a chain of RDDs which will be executed when triggered. Before we get into the weeds of building pipelines with Spark, let’s unpack how we can set up spark in the first place. Setting up Contexts and Sessions in PySpark Luckily for us, Google Colab already has PySpark installed with all the necessary dependencies and configuration setup. All we have to do is create a spark session. This will allow us to play around with the basics of Spark. If you want to see the full notebook, you can check it out here. import pyspark from pyspark.sql import DataFrame, SparkSession from typing import List import pyspark.sql.types as T import pyspark.sql.functions as F spark= SparkSession \ .builder \ .appName(”sparkApp”) \ .master(”local[*]”) \ .getOrCreate() spark Here, we’re creating both a “context” and a “session”. A context in spark defines the cluster we’ll be connecting to. .master(“local[*]”) defines that we want to run spark locally, so we’re not connecting to some external cluster to do massively parallelized computation, we’re just using the colab instance we’re running. [*] defines that we want to use all available CPU cores. We could also specify [1], [2], or some other number to constrain spark to a certain number of cores. Of course, the power of spark is in its ability to connect to an external cluster. You can use something like Amazon EMR, Google Dataproc, or DataBricks to spool up a cluster, then you would simply connect to it by pointing to the IP and port like so. spark = SparkSession.builder \ .appName(”ClusterApp”) \ .master(”spark://192.168.1.10:7077”) \ .getOrCreate() So, a spark context defines the compute resources you’re connecting to. In PySpark, each process can only be connected to a single context, so if you want to establish two separate contexts you’ll need to, for instance, run two seperate python scripts. Within a single context, though, you can have numerous “sessions”. A session in Spark is an isolated scope of data and functionality which is defined within our context. So, if we had a single spark context which was connected to a large spark compute cluster, we could create and manage several sessions within that context all from a single Python script. When we call getOrCreate we’re telling spark that we want to either create a new session or connect to an existing session. We can then create multiple other sessions which are isolated. Here’s an example of creating two different spark sessions with different data. from pyspark.sql import SparkSession # First, we initialize a SparkSession and SparkContext spark1 = SparkSession.builder.appName(”sparkApp”).getOrCreate() # Create a DataFrame and a temporary view in spark1 df1 = spark1.createDataFrame([(1, “Alice”), (2, “Bob”)], [”id”, “name”]) df1.createOrReplaceTempView(”people”) # Run SQL in spark1 print(”Session 1:”) spark1.sql(”SELECT * FROM people”).show() # Now create a separate session spark2 = spark1.newSession() # Session 2 does NOT see the temp view from session 1 print(”Session 2 (before creating view):”) try: spark2.sql(”SELECT * FROM people”).show() except Exception as e: print(”Error:”, e) # Create a separate view in session 2 df2 = spark2.createDataFrame([(3, “Charlie”)], [”id”, “name”]) df2.createOrReplaceTempView(”people”) print(”Session 2 (after creating its own view):”) spark2.sql(”SELECT * FROM people”).show() print(”Back to Session 1:”) spark1.sql(”SELECT * FROM people”).show() Without getting into the weeds of exactly what this code is doing, you can see that each session contains separate data, but if we print out the high level sessions we can see that they both belong to the same context (via the AppName). So, the sessions have different data which is logically isolated, but that data occupies the same physical hardware. One quirk, which I mentioned before, is that there can only be one context per python process. Since we already created a context with the name sparkApp, and it already has a session associate with it, if we try to create new spark contexts they’ll end up defaulting to our existing session. spark1 = SparkSession.builder.appName(”AppOne”).getOrCreate() spark2 = SparkSession.builder.appName(”AppTwo”).getOrCreate() print(spark1 is spark2) # True — same session print(spark2.sparkContext.appName) # “AppOne” This can be a bit counterintuitive if you don’t understand the underlying nature of contexts and sessions. You might think you’re building a unique session by assigning a different appName, but spark only allows a single context per process so it simply loads up our old sparkApp. Creating an RDD and Evaluating Computational Graphs One of the most fundamental concepts within spark is the “Resiliant Distributed Dataset” (RDD). RDDs are how Spark can take a single source of data and distribute it over many seperate computers to parallelize computation. There are many ways to create RDDs. Here, we’ll create a simple RDD from scratch using a Python List. from pyspark.sql import SparkSession # ---- Setup ---- spark = SparkSession.builder.appName(”RDD Demo”).getOrCreate() sc = spark.sparkContext # ---- Create RDDs ---- numbers = sc.parallelize([1, 2, 3, 4, 5]) This creates an RDD called numbers , which is automatically divided into some number of partitions based on the cluster we’re working on. We could do some operation to this data, like add 10 to each number. # ---- Transformation: add 10 to each number ---- added = numbers.map(lambda x: x + 10) Here the map function takes x, one of the numbers in our original list, and replaces it with x+10, effectively adding 10 to each element in the list. If we print the results of added, however, we don’t get a list of numbers but rather a reference to an RDD. print(added) This RDD is a placeholder for what we would get if we did add 10 to each value. Recall that Spark is evaluated lazily, meaning we can line up all the calculations we want it to do, but it wont actually do those calculations untill we tell it to. We can do that with the collect function. # ---- Action: collect results back to the driver ---- result = added.collect() print(”Original numbers:”, numbers.collect()) print(”After adding 10:”, result) The whole point of an RDD is that it allows us to manage how data is distributed across various different machines. We can actually inspect the partitions that get created via the following: numbers = sc.parallelize([1, 2, 3, 4, 5]) print(numbers.glom().collect()) The glom transformation allows you to view the content of each partition as a python list, then we can call collect to actually run that function and view the result. As you can see, 8 partitions were created, as there are 8 cores in my machine. The content of the list is distributed within that list of available partitions. We could also reduce the number of partitions #the 2 means we want the data to be parallelized across 2 partitions numbers = sc.parallelize([1, 2, 3, 4, 5], 2) print(numbers.glom().collect()) or add more data that could fill out our default of 8 partitions. numbers = sc.parallelize([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]) print(numbers.glom().collect()) We can do several operations on this data simultaniously. For instance, imagine we wanted to print out the following lists: a list of input numbers a list of all those numbers squared a list of all of the squares that are even numbers a sum of all of the numbers a sum of all of the even squares We can do that with the following code: from pyspark.sql import SparkSession # ---- Setup ---- spark = SparkSession.builder.appName(”RDD Demo”).getOrCreate() sc = spark.sparkContext # ---- Create RDDs ---- numbers = sc.parallelize([1, 2, 3, 4, 5]) # ---- Transformations ---- squared = numbers.map(lambda x: x * x) even_squares = squared.filter(lambda x: x % 2 == 0) # ---- Reduce ---- total = numbers.reduce(lambda a, b: a + b) total_even_squares = even_squares.reduce(lambda a, b: a + b) # ---- Printing ---- print(”Original numbers:”, numbers.collect()) print(”Squared:”, squared.collect()) print(”Even squares:”, even_squares.collect()) print(”Sum of numbers:”, total) print(”Sum of even squares:”, total_even_squares) spark.stop() There’s some new concepts going on in this code that I’d like to break down. First of all, we’re creating an RDD consisting of a list of numbers, same as before. # ---- Setup ---- spark = SparkSession.builder.appName(”RDD Demo”).getOrCreate() sc = spark.sparkContext # ---- Create RDDs ---- numbers = sc.parallelize([1, 2, 3, 4, 5]) We’re then applying transformations to the data, in the form of a mapping function. # ---- Transformations ---- squared = numbers.map(lambda x: x * x) even_squares = squared.filter(lambda x: x % 2 == 0) There are many other transformations that can be performed in spark. Sorting, inner and outer joins, sampling, and filtering to name a few. You can chain many transformations together, and Spark will construct an execution graph of how the different operations inter-relate with one another. In Spark, you can chain transformations together without actually triggering them, allowing you to construct a computational graph. If you want to actually trigger the computational graph, you need to call an “Action”. collect is one such action. It triggers the computational graph and returns the result as a list. reduce is another action that, instead of returning all results as a list, combines all results into a single value based on a function. # ---- Reduce ---- total = numbers.reduce(lambda a, b: a + b) total_even_squares = even_squares.reduce(lambda a, b: a + b) recall that spark groups data into partitions A result from a previous code block, using `glob` to view partitions as lists. Each partition might have multiple values it’s working on. the reduce function first combines all pairs of values in a single partition into a single value, then combines all pairs of values across partitions into a single value. Thus by using the reduction function (lambda a, b: a + b), we’re effectively telling spark to add up all values across all partitions. # ---- Printing ---- print(”Original numbers:”, numbers.collect()) print(”Squared:”, squared.collect()) print(”Even squares:”, even_squares.collect()) print(”Sum of numbers:”, total) print(”Sum of even squares:”, total_even_squares) To actually print out our output, some RDDs need to have collect called on them, because they were not evaluated. However, total and total_eve_squares are not RDDs, but actual values. We triggered them to be evaluated when we called reduce, which is an action and not a transformation. So, in simple words: you can chain actions together, to create a chain of RDDs. when you call an action, like collect or reduce, it actually triggers that chain of transformations and returns a result. There’s an important note in this example; it’s inefficiently implemented. We calculate squares, filter those into even squares, then calculate the sum of all of the even squares. One might expect the computation to look something like this: If we wanted these operations to be calculated efficiently, we would reuse the results of various calculations to create the final output. however, in actuality, we’re defining five different computational graphs that are executed separately for each action we undertake. Instead of re-using computations, we’re treating each action like they’re fundamentally separate, and thus re-doing a large number of calculations. This means we’re wastefully re-computing the same thing over and over again. Not a big deal if you’re processing a list of five numbers, huge deal if you’re processing petabytes of data. One way one might alleviate this issue is to use caching. from pyspark.sql import SparkSession # ---- Setup ---- spark = SparkSession.builder.appName(”RDD Caching Demo”).getOrCreate() sc = spark.sparkContext # ---- Create RDD ---- numbers = sc.parallelize(range(1, 11)) # 1 to 10 # ---- Transformations ---- squared = numbers.map(lambda x: x * x) even_squares = squared.filter(lambda x: x % 2 == 0) # ------------------------------- # Without caching: Spark recomputes each time # ------------------------------- print(”=== WITHOUT CACHING ===”) print(”Squared:”, squared.collect()) # triggers a Spark job print(”Even squares:”, even_squares.collect()) # recomputes from numbers → squared → even_squares print(”Sum of even squares:”, even_squares.reduce(lambda a, b: a + b)) # recomputes again # ------------------------------- # With caching: Spark computes once and reuses results # ------------------------------- squared_cached = numbers.map(lambda x: x * x).cache() even_squares_cached = squared_cached.filter(lambda x: x % 2 == 0).cache() print(”\n=== WITH CACHING ===”) print(”Squared (cached):”, squared_cached.collect()) # first action → computed & cached print(”Even squares (cached):”, even_squares_cached.collect()) # reused from cache, and result stored to cache print(”Sum of even squares (cached):”, even_squares_cached.reduce(lambda a, b: a + b)) # reused cache # ---- Clean up ---- spark.stop() Here, the same spark code is implemented twice. First without caching (exactly as was done previously), and second with caching. If an RDD with caching is triggered for execution, it does the following: If the RDD hadn’t been executed previously, do all proceeding computations, store the result, and return the result. If the RDD had been executed previously, and a value is stored, don’t do proceeding operations and simply return the stored value. This allows us to effectively re-use computations. Even without proceeding in this article, you’ve unlocked a massive tool for doing large scale parallelized work. If you can turn your data into a partitioned RDD, you can specify transformations, actions, and caching to do massively parallelized work across large datasets by using the power of a cluster of computers. For instance, this code snippet applies similar concepts of parallelization we previously explored, but applies it to audio processing. import matplotlib.pyplot as plt import librosa spark = SparkSession.builder.appName(”Audio FFT DAG Demo”).getOrCreate() sc = spark.sparkContext audio_files = [os.path.join(”test_audio”, f) for f in os.listdir(”test_audio”)] rdd = sc.parallelize(audio_files) # ========================================== # Step 3. Define DAG transformations # ========================================== def load_audio(path, sr=22050): import librosa # This is happening inside a worker, so we need to import y, sr = librosa.load(path, sr=sr) return (path, y, sr) def compute_fft(item): import numpy as np # This is happening inside a worker, so we need to import fname, y, sr = item fft_vals = np.fft.fft(y) return (fname, y, sr, fft_vals) def compute_ifft(item): import numpy as np # This is happening inside a worker, so we need to import fname, y, sr, fft_vals = item y_reconstructed = np.fft.ifft(fft_vals).real return (fname, y, sr, fft_vals, y_reconstructed) # DAG audio_rdd = rdd.map(load_audio).cache() fft_rdd = audio_rdd.map(compute_fft).cache() ifft_rdd = fft_rdd.map(compute_ifft).cache() # ========================================== # Step 4. Execute DAG + Collect # ========================================== results = ifft_rdd.collect() # ========================================== # Step 5. Plot results # ========================================== for fname, y, sr, fft_vals, y_reconstructed in results: # --- 1. Original waveform --- plt.figure(figsize=(3, 0.5)) librosa.display.waveshow(y, sr=sr) plt.title(f”Waveform of {os.path.basename(fname)}”) plt.xlabel(”Time (s)”) plt.ylabel(”Amplitude”) plt.show() # --- 2. FFT spectrum --- fft_freqs = np.fft.fftfreq(len(fft_vals), 1/sr) plt.figure(figsize=(3, 0.5)) plt.plot(fft_freqs[:len(fft_vals)//2], np.abs(fft_vals)[:len(fft_vals)//2]) plt.title(f”FFT Spectrum of {os.path.basename(fname)}”) plt.xlabel(”Frequency (Hz)”) plt.ylabel(”Magnitude”) plt.show() # --- 3. Reconstructed waveform --- plt.figure(figsize=(3, 0.5)) librosa.display.waveshow(y_reconstructed, sr=sr) plt.title(f”Reconstructed Waveform of {os.path.basename(fname)}”) plt.xlabel(”Time (s)”) plt.ylabel(”Amplitude”) plt.show() # ========================================== # Step 6. Cleanup # ========================================== spark.stop() If you’re not familiar with audio processing, don’t worry about it. The point of this code block is to communicate that the general paradigm of RDDs are applicable to a wide range of data types. Here, instead of thinking of an RDD as a list of numbers, I’m thinking of it as a list of file names called rdd. Then, I’m actually loading the data, resulting in audio_rdd. Then I’m doing audio processing techniques to turn audio_rdd into fft_rdd and ifft_rdd, which represent the output of fancy audio processing techniques (You can read more about audio processing here, if you’re interested). Thus, with spark, doing arbitrary, massively parallelized workloads is as simple as defining a few functions and connecting to a cluster. Let’s continue our exploration of spark by exploring some of the cool abstractions that are commonly used on top of Spark to make it even easier to use. Data Frames and SQL Most data scientists are familiar with dataframes, a la Pandas or Polars. Spark also has a dataframe concept, built on top of RDDs. We can point PySpark to a csv file using the following code: df = spark.read.csv(’/content/’+csv_file_name, header=True, inferSchema=True) This will create an RDD which I nameddf, which stands for “Data Frame”. If you’re familiar with popular libraries like Pandas or Polars, or even if you’re familiar with Microsoft Excel, you’re familiar with the concept of a data frame. It’s an object that stores data contained in rows and columns. Because tabular data is so common, Spark allows you to define a dataframe that’s built on top of RDDs, allowing for the parallelized processing of large amounts of row and column based data. We can use all of the same functions you might use in a normal dataframe library, like describe #Summary stats df.describe().show() We can do stuff like filtration and ordering #DataFrame Filtering df.filter(df.location == “United States”).orderBy(F.desc(”date”)).show() Grouping #Simple Group by Function df.groupBy(”location”).sum(”new_cases”).orderBy(F.desc(”sum(new_cases)”)).show(truncate=False) We can also convert this dataframe into an SQL equivilent view, and apply SQL commands to it. df.createOrReplaceTempView(”covid_data”) spark.sql(”“” SELECT location, SUM(new_cases) AS total_new_cases FROM covid_data GROUP BY location ORDER BY total_new_cases DESC “”“).show(truncate=False) Feel free to check out my article on SQL, if you want to learn more. Conclusion Spark has a ton of tooling, and I can’t go in depth into all of it. The main job of this article was to serve as an introduction to Spark, its core paradigms, and what it’s good for. Using these core principles, and a bit of application specific research you can use Spark to do a wide range of large scale data processing operations: Real time analytics over extremely large datasets Distributed machine learning and model training Large-scale data cleaning, feature engineering, and preprocessing Large-scale ETL pipelines And so much more. In my mind, Spark doesn’t do much on it’s own. Rather, it’s a quick way to turn your algorithm or technology into a massively parallelized system. Thus, it’s an incredibly usefull tool for developers working with large amounts of data. AI Generated In-Text Citations — Intuitively and Exhaustively Explained Daniel Warfield — Tue, 02 Sep 2025 14:02:45 GMT “Tag Out” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained. You’ve probably played around with tools like ChatGPT, Anthropic, and Perplexity. When you ask these systems a question, they generate a response with a bunch of useful information. An example of a question and corresponding response in perplexity.ai. Along with text, these systems also provide in-text citations, allowing users to view the source of data that is relevant to a particular statement. Hovering over these links results in a popup, and clicking these links allows the user to navigate to the relevant source. In-text citations are an incredibly powerful tool that allows for more robust, more useful, more trustworthy, and more testable LLM applications. In this article, we’ll break down the subject in-depth and build a system to generate in-text citations ourselves. An example of in-text citations made with proprietary data. Subscribe To Intuitively and Exhaustively Explained. Share this article with friends and colleagues. Share Who is this useful for? Anyone interested in building cutting-edge AI applications. How advanced is this post? This article contains simple explanations of fundamental approaches in AI-powered application development. Thus, this article is relevant to readers of all levels. Prerequisites: None, though some Python experience may be useful in understanding the implementation sections later in the article. Why In-Text Citations Matter To be completely honest, I rarely use in-text citations when interfacing with chat bots like ChatGPT. Personally, I use LLMs as resources for high-level descriptions and quick code revisions, and rarely use them as the beginning point for deeper research. I think that’s part of the reason more experienced developers don’t often discuss in-text citations; we treat AI like a junior assistant, not a source of knowledge. It’s often faster for us to look at the response itself and judge it as right or wrong, rather than comb through a bunch of sources. As LLM-powered applications reach new domains, ones that require domain expertise, deep research, and complex logic, I think this paradigm will change. LLM applications will start telling us things we didn’t already know, and users will need to quickly confirm that information to be accurate. I’ve been obsessed with in-text citations for the last three months, chiefly for this reason. I’ve been working on a fraud detection platform that’s designed to find new evidence of fraud within large sets of legal documents called FraudX. It does a pretty good job at detecting fraud; our slogan is “detect fraud 40x faster”, but I think that’s a bit of an understatement. Months of work can be summarized in minutes. That’s great, but issues arise when evidence of fraud leaves the auditor's desk and ends up in the courtroom; it’s not enough to throw up your hands and say “AI told me so”. Our customers need to be able to back up claims with evidence. Thus, detecting fraud is only half the battle; you also need to expose the correct sources to the user so they can build a well-supported case. In-text citations are critical because they provide the link between what the AI said and the supporting evidence. The FraudX Platform allows users to view indicators of Fraud and also observe key pieces of evidence which suggest if those indicators are high or low risk. In-text citations are critical in grounding these responses in reality. On the left you can see a high risk indicator of fraud, and on the right you can see a deposition which includes evidence to support our LLM generated analysis. Robust in-text citations are also critical for developers. Compared to our customers, I know next to nothing about law, legal procedure, and fraud. Despite this, I need to quickly be able to understand if our system is performant. In-text citations allow me to judge output from the LLM on a statement-by-statement basis. The LLM said that Joe got surgery for a broken femur in early July. Check the source, yep that’s true. The LLM said that there’s a picture of Joe in a salsa class, without a cast, in late July. Check the source, yep that’s true. The LLM said this is likely a case of insurance fraud because it’s impossible for Joe to have recovered that quickly. Makes sense to me! As developers integrate AI into increasingly sophisticated domains, testing is becoming more difficult. Robust and well-developed in-text citations are a critical tool that allows developers to better understand LLM responses. This makes testing significantly easier for developers. Before we get into generating in-text citations, I want to cover some of the foundational techniques we’ll be building on top of. RAG — The Foundation of In-Text Citation As you probably know (I’ve covered RAG about a billion times at this point) “RAG” stands for “Retrieval Augmented Generation”, and is a general workflow for injecting context into a language model. It consists of the following steps: Get a query from the user. Use that query to retrieve relevant information that might be used as context in answering that query. Construct a prompt consisting of the query and context. Pass that prompt to an LLM for generation. The core workflow, from my article on RAG. This works because language models are in-context learners, yadah yadah yadah. I don’t want to harp too much on RAG in this article; I have plenty of articles on the subject if you want to explore the technology more in depth: For the purposes of this article, we only have to understand RAG from the highest level. Basically, with RAG, we look stuff up, feed the model a bunch of information, then ask it a question. A simple example of a prompt we might pass to a language model in a RAG context. We look up a bunch of information that might be relevent to a question, then pass that as context, allong with the question, to the language model. The whole idea of in-text citation is pretty simple. Instead of passing references as a big block of text, you inject unique identifiers to each retrieved source. The first step of creating in-text citations is to add unique identifiers to each piece of information. Then, you adjust the system prompt of the model and tell the LLM to print out one of those unique identifiers whenever the LLM uses the information from a particular source. The second step of implementing in-text citations is to add a system prompt, describing how the LLM should reference a particular source. Once a response has been generated, the LLM response can be post-processed. We can look through the text and replace all of the referenced IDs with buttons, hyperlinks, or whatever else your application demands. hyperlinks, or whatever else your application demands. The company I work for, EyeLevel.ai, maintains a RAG system called GroundX (which has an open source version designed for enterprise clients called GroundX on-prem and a hosted offering called the GroundX Platform). To make GroundX easier to use, we’re releasing a set of open source tools called GroundX community, which is designed to accelerate certain workflows. The first feature GroundX community supports is in-text citations, which we’ll be using in this article. Creating In-Text Citations with GroundX Community If you install GroundX community via pip install git+https://github.com/EyeLevel-ai/groundx_community.git you can import a function called generate_cited_response from groundx_community.chat_utils.citing import generate_cited_response This function expects the following input: generate_cited_response(chunks, system_prompt, query) chunks: A list of contextual information that’s relevant to the query, with some important metadata attached to each chunk for the purposes of generating in-text citations. system_prompt: A prompt to the model describing how it should act query: The question or command from the user It also has the optional parameter llm that accepts any LangChain LLM that inherits from langchain_core.language_models.chat_models.BaseChatModel. The generate_cited_response function does all the work necessary to format a prompt to the LLM, then the LLM is used to generate the final response with in-text citations. By default, gpt-4o is used, and is authenticated via an API key in the OPENAI_API_KEY environment variable. The chunks variable itself is a list of chunks. This consists of the references passed to the LLM, with some key metadata so that in-text citations can be generated properly. chunks = [ { "text": "reference text retreived from some source" "uuid": "unique_identifier-cccc-dddd-eeee-000000000001", "render_name": "source1.txt", "source_data": { "arbitrary_key": "example value", "url": "https://example.com/source1.txt", "filename": "source1.txt", "file_type": "txt", "document_uuid": "doc-001" } }, { "text": "reference text retreived from some source" "uuid": "unique_identifier-cccc-dddd-eeee-000000000002", "render_name": "source2.txt", "source_data": { "arbitrary_key": "example value", "url": "https://example.com/source2.txt", "filename": "source2.txt", "file_type": "txt", "document_uuid": "doc-002" } }, ... ] There are three necessary fields for each chunk: The text, which contains the information passed to the LLM which will be referenced. This might be information from a website, PDF, word document, or whatever. The UUID, this will be used by the in-text citation system to reference a particular chunk. One may choose to generate a random UUID for each document, or you might have one lying around which is convenient to use. The render_name, is used for actually generating the in-text citation. This is the text which will pop up. This could be something like source1, source2, source3, the actual filenames, an abbreviation of the filenames, whatever. There is an additional field called source_data. This allows one to attach arbitrary key-value pairs, which can be used in the application. For instance, a url might be attached, which can allow the application to open a document when a source is clicked on. Note that chunks could be generated from virtually any RAG retreival; If you’re using GroundX, LangChain, LlamaIndex, or some bespoke RAG pipeline, the results of virtually any RAG pipeline could be used to construct chunks, and thus can be fed into generate_cited_response to create a response with in-text citations. We also need a system_prompt . On the tin this is pretty simple, it’s just a system prompt to describe to the model the types of responses it should generate. Here are some examples: system_prompt_1=( "You are a helpful assistant. Use the provided excerpts to fully analyze and answer the user's question. " ) system_prompt_2=( "You are a helpful assistant. Use the provided excerpts to fully analyze and answer the user's question. " "If any information is irrelevant, do not cite it or mention it. Think carefully and show clear reasoning." "provide your answer as a \"rationale: \" which includes in-text citations, and \"final answer: \" which is a concise answer to the question. " "the final answer should be separated by the rationale by a paragraph break. The rationale should not include newlines." ), system_prompt_3=( "You are a helpful assistant. Use the provided excerpts to fully analyze and answer the user's question. " "You must rely only on information from the excerpts, and cite every excerpt you used using the $REF: ID$ format. " "If any information is irrelevant, do not cite it or mention it. Think carefully and show clear reasoning." "provide your answer as a \"rationale: \" which includes in-text citations, and \"final answer: \" which is a concise answer to the question. " "the final answer should be separated by the rationale by a paragraph break. The rationale should not include newlines." ), As a general rule, it’s recommended not to include information about generating in-text citations within the system prompt, as is done in the third example. Under the hood, the generate_cited_response function contains the following prompt which is also passed to the model. I am going to ask a question in my next message. Here are some excerpts uniquely identified by an ID that may or may not be relevant. You need to perform 2 tasks: 1) Generate a response to answer the question. If these excerpts relate to my question, use them in your response. If not, ignore them, and rely on our conversation context. 2) If any excerpt is used, generate in-text citation using the excerpt ID as follows - $REF: ID$. The formatting must be strictly followed. For example, if excerpt corresponding to ID 03b994bc-2fae-4e1e-a4cd-f0f3e6db2d90 is used in generating the response, source attribution must be - $REF: 03b994bc-2fae-4e1e-a4cd-f0f3e6db2d90$ Strictly follow the instructions of the above tasks. Do not mention whether the content or previous context was used or not; respond seamlessly. Avoid phrases like "the provided content" or similar. Take into account everything we've discussed so far, without assuming everything is relevant unless it clearly supports your answer. ----- If the system prompt passed to the generate_cited_response conflicts with this instruction, it will damage in-text citation creation. Thus, it’s recommended that the system_prompt only contains application level instructions and not prompts about how in-text citations should be structured. Once chunks and a system prompt have been specified, one simply needs to define a query and we’re off to the races. In a traditional RAG context the chunks would be retrieved based on the query, but we can implement a simple example which forgoes this step and uses a static context. from groundx_community.chat_utils.citing import generate_cited_response # ------------------------------- # Defining Variables # ------------------------------- CHUNKS = [ { "text": ( "George currently possesses a variety of fruits in his market stall, including three bunches of grapes and a crate of strawberries. " "The grapes are organic, grown locally, and usually sold by the bunch. He typically sells the grapes in quantities of one or more, depending on customer demand. " "The strawberries, on the other hand, are imported and priced differently based on weight rather than quantity." ), "uuid": "11111111-aaaa-bbbb-cccc-000000000001", "render_name": "market_inventory.txt", "source_data": { "url": "https://example.com/market_inventory", "filename": "market_inventory.txt", "file_type": "txt", "document_uuid": "doc-001" } }, { "text": ( "Natalie recently exchanged €5.00 into U.S. dollars and received exactly $5.45 USD. " "She has expressed interest in buying grapes and strawberries from George. " "However, she also mentioned she wants to keep $1.00 in reserve to buy bread from a nearby bakery later in the day." ), "uuid": "22222222-bbbb-cccc-dddd-000000000002", "render_name": "wallet_and_intentions.txt", "source_data": { "url": "https://example.com/wallet", "filename": "wallet_and_intentions.txt", "file_type": "txt", "document_uuid": "doc-002" } }, { "text": ( "George stated that he is willing to sell grapes for fifty cents per bunch, but added that if someone buys three bunches, he will offer them for a flat $1.25. " "Strawberries are sold for $3.00 per crate, and George is unwilling to negotiate on that price unless multiple crates are bought. " "All prices are quoted in USD and must be paid in cash." ), "uuid": "33333333-cccc-dddd-eeee-000000000003", "render_name": "pricing_policy.txt", "source_data": { "url": "https://example.com/pricing", "filename": "pricing_policy.txt", "file_type": "txt", "document_uuid": "doc-003" } }, { "text": ( "A bystander once noted that George used to raise llamas on a small patch of land adjacent to his current fruit farm. " "While this fact has no bearing on his current business dealings, it is often brought up in conversation due to the llama’s quirky behavior." ), "uuid": "44444444-dddd-eeee-ffff-000000000004", "render_name": "llama_memoirs.txt", "source_data": { "url": "https://example.com/llamas", "filename": "llama_memoirs.txt", "file_type": "txt", "document_uuid": "doc-004" } } ] SYSTEM_PROMPT = ( "You are a helpful assistant. Use the provided excerpts to fully analyze and answer the user's question. " "If any information is irrelevant, do not cite it or mention it. Think carefully and show clear reasoning." ) QUERY = "Can Natalie afford to buy all of George's grapes and also buy strawberries?" # ------------------------------- # Call function # ------------------------------- result = await generate_cited_response( chunks=CHUNKS, system_prompt=SYSTEM_PROMPT, query=QUERY, ) And, after running, this is the result we get. Natalie has $5.45 in total but wants to keep $1.00 for bread, leaving her with $4.45 to spend on George’s fruits . George has three bunches of grapes, which he sells for $0.50 apiece or $1.25 for all three . Buying all three bunches would cost Natalie $1.25. George also has a crate of strawberries priced at $3.00 . To buy both all the grapes and the strawberries, Natalie would need $1.25 (for grapes) + $3.00 (for strawberries) = $4.25. Since $4.25 is less than the $4.45 she has available to spend, Natalie can indeed afford to buy all of George's grapes and the strawberries and still have money left over for her bread . You might notice that, in the response, in-text citations are populated as custom HTML tags of type InTextCitation. The chunkId and renderName are both added as attributes, along with any of the other fields specified in the source_data provided with each chunk. Under the hood, the LLM generates citations based on the chunk_ids associated with each chunk, then the generate_cited_response function uses regex expressions to find and replace those ids with richer HTML tags. This simple script processes the result from generate_cited_response and renders InTextCitation as clickable buttons. from IPython.display import display, HTML def render_citations_as_buttons(html: str) -> str: styled_html = f""" {html} """ return styled_html display(HTML(render_citations_as_buttons(result))) The result of generating in-text citations. Clicking these buttons will open the url associated with each button. There’s some application-level magic that makes FraudX render PDFs in a window, and produces highlighted sections that are relevent to the particular question being asked, but this system is the essence of how FraudX connects LLM inferences with source material. Recall the FraudX Platform. We built a PDF viewer with the ability to highlight key information that is relevent to a particular source. There’s a lot of application specific design decisions one can make when creating an in-text citation system. Now that we have a solid idea of how to create in-text citations, let’s set it up using an actual RAG system. Creating In-Text Citations With GroundX In this tutorial, we’ll be using GroundX as our RAG engine, but keep in mind that this general approach will work with virtually any RAG system. The only criterion is that you can generate the list of chunks correctly. Full code for this example can be found here. First, we need some documents to play around with. For this example, I’ll be downloading a few research papers from Arxiv. import arxiv import os # Define target download directory output_dir = os.path.join("temp", "arxiv_papers") os.makedirs(output_dir, exist_ok=True) # List of arXiv IDs and desired filenames papers = { "1706.03762": "attention_is_all_you_need.pdf", # Attention is All You Need "1901.02860": "transformer_xl.pdf", # Transformer-XL "2104.09864": "rope_roformer.pdf", # RoPE (RoFormer) } # Download each paper for arxiv_id, filename in papers.items(): print(f"Downloading {filename} (arXiv:{arxiv_id})...") paper = next(arxiv.Search(id_list=[arxiv_id]).results()) paper.download_pdf(dirpath=output_dir, filename=filename) print("✅ All downloads complete.") To follow this tutorial you’ll need a GroundX account, which has more than enough free tokens to follow along. You can sign up for one here. Once you have an account and have set up an API key, you can upload documents to GroundX. """Creating a bucket and uploading content to it or hard coding an existing bucket ID """ from groundx import GroundX client = GroundX( api_key=os.getenv("GROUNDX_API_KEY"), ) # Creating a bucket response = client.buckets.create( name="test_bucket", ) bucket_id = response.bucket.bucket_id print(f"Created bucket with ID: {bucket_id}") """Uploading downloaded files to the bucket """ client.ingest_directory( bucket_id=bucket_id, path="temp/arxiv_papers", ) In GroundX, you can query a bucket of information and get back a list of relevant chunks from all the documents that have been uploaded to that bucket. In the previous code block we uploaded all our documents to a bucket specified by bucket_id. Now, we can submit a query to that bucket_id. query = "Describe the main differences between the original transformer, the RoFormer, and the Transformer-XL architectures." context = client.search.content( id=bucket_id, query=query, ) Then we get back a bunch of information which we can use for various RAG approaches. One of the elements of the response is context.search.results, which is a list of all chunks that are relevent to the question. Each chunk has a field called suggested_text, which contains a textual description of each chunk. GroundX does a lot of work to turn tables, graphs, figures, and other things into LLM ready data. suggested_text contains what GroundX thinks each chunk should be textually described as. for result in context.search.results: print('-----------') print(result.suggested_text) The code block above prints all the suggested_text for all of the chunks retreived that were relevant to the query. Here’s an example of an individual chunks suggested_text. {"figure_title":"RoFormer Position Encoding","figure_number":"Figure 1","keywords":"transformer, position encoding, attention mechanism, relative position embedding, absolute position embedding, mathematical equations","summary":"This figure presents equations related to position encoding in transformer models, focusing on both absolute and relative position embeddings. It illustrates how attention scores are computed and how these scores influence the output of the model. The equations provide a mathematical foundation for understanding the RoFormer architecture and its enhancements over traditional methods.","components":["a_{m,n}", "o_m", "q_m", "k_n", "v_n", "P_i", "p_{i,z}", "p_{i,z+1}"],"relationships":[{"source":"query vector q_m","target":"key vector k_n","type":"attention computation"},{"source":"attention scores a_{m,n}","target":"output o_m","type":"weighted sum"}]} {"equation":"a_{m,n} = exp(q_m^T k_n / √d) / Σ_{j=1}^{N} exp(q_m^T k_j / √d)","relationships":[{"source":"q_m","target":"k_n","type":"dot product"},{"source":"a_{m,n}","target":"o_m","type":"output computation"}]} {"equation":"o_m = Σ_{n=1}^{N} a_{m,n} v_n","relationships":[{"source":"a_{m,n}","target":"v_n","type":"weighted sum"},{"source":"o_m","target":"output","type":"final computation"}]} {"equation":"f_t:t∈{q,k,v}(x_i, i) := W_t:t∈{q,k,v}(x_i + P_i)","relationships":[{"source":"x_i","target":"P_i","type":"addition"},{"source":"W_t","target":"context representation","type":"transformation"}]} {"equation":"P_{i,2t} = sin(k / 10000^{2t / d}), P_{i,2t+1} = cos(k / 10000^{2t / d})","relationships":[{"source":"k","target":"P_{i,2t}","type":"sinusoidal encoding"},{"source":"k","target":"P_{i,2t+1}","type":"sinusoidal encoding"}]} {"equation":"f_q(x_m) := W_q x_m, f_k(x_n, n) := W_k(x_n + p_r), f_v(x_n, n) := W_v(x_n + p_r)","relationships":[{"source":"x_m","target":"W_q","type":"query transformation"},{"source":"x_n","target":"p_r","type":"relative position addition"},{"source":"W_k","target":"key transformation"},{"source":"W_v","target":"value transformation"}]} {"equation":"q_m k_n = x_m^T W_q^T W_k x_n + x_m^T W_q^T W_k p_{m-n} + p_m^T W_q^T W_k x_n + p_m^T W_q^T W_k p_{m-n}","relationships":[{"source":"x_m","target":"W_q^T W_k x_n","type":"content-based computation"},{"source":"x_m","target":"W_q^T W_k p_{m-n}","type":"relative position computation"},{"source":"p_m","target":"W_q^T W_k x_n","type":"absolute position computation"},{"source":"p_m","target":"W_q^T W_k p_{m-n}","type":"relative position computation"}]} {"equation":"q_m k_n = x_m^T W_q^T W_k x_n + x_m^T W_q^T W_k p_{m-n} + u^T W_q^T W_k x_n + v^T W_q^T W_k p_{m-n}","relationships":[{"source":"x_m","target":"W_q^T W_k x_n","type":"content-based computation"},{"source":"x_m","target":"W_q^T W_k p_{m-n}","type":"relative position computation"},{"source":"u","target":"W_q^T W_k x_n","type":"trainable vector computation"},{"source":"v","target":"W_q^T W_k p_{m-n}","type":"trainable vector computation"}]} {"equation":"q_m k_n = x_m^T W_q^T W_k x_n + b_{i,j}","relationships":[{"source":"x_m","target":"W_q^T W_k x_n","type":"content-based computation"},{"source":"b_{i,j}","target":"attention weights","type":"bias addition"}]} {"equation":"q_m k_n = x_m^T W_q^T W_k x_n + P_m^T U^T U_k P_n + b_{i,j}","relationships":[{"source":"x_m","target":"W_q^T W_k x_n","type":"content-based computation"},{"source":"P_m","target":"U^T U_k P_n","type":"projection matrix computation"},{"source":"b_{i,j}","target":"attention weights","type":"bias addition"}]} The image contains mathematical equations related to position encoding in transformer models. The first equation computes attention scores $a_{m,n}$ as the normalized exponential of the dot product between query vector $q_m$ and key vector $k_n$, divided by the square root of the dimension $d$. The second equation calculates the output $o_m$ as the weighted sum of values $v_n$ based on the attention scores $a_{m,n}$. Subsequent equations explore absolute and relative position embeddings. Absolute position embedding adds position vectors $P_i$ to token representations, with sinusoidal functions used to generate these vectors. Relative position embedding incorporates relative distances between tokens, represented by $p_r$, into the attention mechanism. Advanced equations decompose the attention computation into content-based and position-based components, introducing trainable vectors $u$ and $v$ for greater flexibility. Later refinements include bias terms $b_{i,j}$ and projection matrices to model relationships between positions and tokens. These equations collectively describe the mathematical foundation of the RoFormer architecture, emphasizing enhancements over traditional position encoding methods. This is a textual block that contains a figures title, and a variety of equations. It also contains a narrative breakdown about what the equations represent, and where they fit in the greater context of the document. Allong with suggested_text, GroundX provides a url of the source material, and a page number where the particular reference can be found. Using this data, we can iterate over all chunks provided by GroundX and construct the chunks that we’ll be passing to the generate_cited_response function. We’re calling that citable_sources in this particular code block. import uuid citable_sources = [] for result in context.search.results: #getting the suggested text from each chunk text = result.suggested_text.strip() if not text: continue # Skip empty chunks #getting the page number and url from each chunk page_num = result.bounding_boxes[0].page_number page_url = f"{result.source_url}#page={page_num}" #adding to the citable sources. citable_sources.append({ "text": text, "uuid": str(uuid.uuid4()), "render_name": result.file_name, "source_data": { "url": page_url, "page_number": page_num } }) The documents uploaded are pdfs. Most modern browsers can open up pdf files. Furthermore, if you add #page= to the url, most browsers will open the PDF at the specified page; allowing our in-text citations to snap directly to the page of the PDF that the LLM is referencing. GroundX also returns more sophisticated data, like bounding boxes, which can be used to annotate the PDF and highlight the specific information that the LLM is referencing. Feel free to check out our RAG visualization tool XRay if you’re curious. For now, though, we’ll keep it simple and just use page numbers. So, with that, we constructed citable references based on GroundX retrievals, which were retrieved based on the query from the user. Now we can pass those results into generate_cited_response and create our LLM response with in-text citations. result = await generate_cited_response( chunks=citable_sources, system_prompt=( "You are a helpful assistant. Use the provided excerpts to fully analyze and answer the user's question. " "You must rely only on information from the excerpts, and cite every excerpt you used using the $REF: ID$ format. " "If any information is irrelevant, do not cite it or mention it. Think carefully and show clear reasoning." "the final answer should be separated by the rationale by a paragraph break. The rationale should not include newlines." ), query=query ) display(HTML(render_citations_as_buttons(result))) Those in-text citations are clickable links that will open the relevant page of a PDF. Before we conclude, let’s peek behind the curtain and explore how generate_cited_response works. A Peek Behind The Curtain Before we wrap up, let’s explore how the generate_cited_response works. It’s pretty straightforward, the entire implementation can be found here. import os import re from urllib import parse from typing import List, TypedDict, Dict, Any, Optional from langchain_openai import ChatOpenAI from langchain_core.language_models.chat_models import BaseChatModel from langchain_core.messages import SystemMessage, HumanMessage class Chunk(TypedDict): text: str uuid: str render_name: str source_data: Dict[str, Any] def get_openai_api_key() -> str: key = os.getenv("OPENAI_API_KEY", "").strip() if not key or not key.startswith("sk-"): raise ValueError( "Invalid or missing OPENAI_API_KEY environment variable. " "Ensure it exists and starts with 'sk-'." ) return key async def generate_cited_response( chunks: List[Chunk], system_prompt: str, query: str, llm: Optional[BaseChatModel] = None, ) -> str: if llm is None: llm = ChatOpenAI( model="gpt-4o", api_key=get_openai_api_key(), ) ref_mapping = {} chunk_texts = [] for ref in chunks: full_uuid = ref["uuid"] chunk = ref["text"] chunk_texts.append(f"**ID:** {full_uuid}\n**Text:** {chunk}\n") ref_mapping[full_uuid] = ref.copy() context = "\n---\n\n".join(chunk_texts) if chunk_texts else "" human_prompt = f""" I am going to ask a question in my next message. Here are some excerpts uniquely identified by an ID that may or may not be relevant. You need to perform 2 tasks: 1) Generate a response to answer the question. If these excerpts relate to my question, use them in your response. If not, ignore them, and rely on our conversation context. 2) If any excerpt is used, generate in-text citation using the excerpt ID as follows - $REF: ID$. The formatting must be strictly followed. For example, if excerpt corresponding to ID 03b994bc-2fae-4e1e-a4cd-f0f3e6db2d90 is used in generating the response, source attribution must be - $REF: 03b994bc-2fae-4e1e-a4cd-f0f3e6db2d90$ Strictly follow the instructions of the above tasks. Do not mention whether the content or previous context was used or not; respond seamlessly. Avoid phrases like "the provided content" or similar. Take into account everything we've discussed so far, without assuming everything is relevant unless it clearly supports your answer. ----- {context} """ answer = await llm.ainvoke( input=[ SystemMessage(content=system_prompt), HumanMessage(content=human_prompt), HumanMessage(content=query), ] ) answer = answer.content pattern = r"\$REF: ([a-f0-9\-]+)\$" for match in re.finditer(pattern=pattern, string=answer): to_replace = match.group() ref_id = match.groups()[0] if ref_id in ref_mapping: ref = ref_mapping[ref_id] props = { "chunkId": ref["uuid"], "renderName": ref["render_name"], **ref.get("source_data", {}), } prop_str = " ".join( f'{key}="{parse.quote(str(value))}"' for key, value in props.items() ) citation = f'\n' answer = answer.replace(to_replace, citation) else: answer = answer.replace(to_replace, "") return answer First, this function takes our chunks, along with their corresponding UUID, and constructs a textual context that ties each chunk with a corresponding id. ref_mapping = {} chunk_texts = [] for ref in chunks: full_uuid = ref["uuid"] chunk = ref["text"] chunk_texts.append(f"**ID:** {full_uuid}\n**Text:** {chunk}\n") ref_mapping[full_uuid] = ref.copy() context = "\n---\n\n".join(chunk_texts) if chunk_texts else "" This function also constructs a map between the UUID and the original chunk, ref_mapping, allowing us to more easily inject HTML tags in post-processing. This context is added to the end of another prompt, which describes how the LLM should format in-text citations. human_prompt = f""" I am going to ask a question in my next message. Here are some excerpts uniquely identified by an ID that may or may not be relevant. You need to perform 2 tasks: 1) Generate a response to answer the question. If these excerpts relate to my question, use them in your response. If not, ignore them, and rely on our conversation context. 2) If any excerpt is used, generate in-text citation using the excerpt ID as follows - $REF: ID$. The formatting must be strictly followed. For example, if excerpt corresponding to ID 03b994bc-2fae-4e1e-a4cd-f0f3e6db2d90 is used in generating the response, source attribution must be - $REF: 03b994bc-2fae-4e1e-a4cd-f0f3e6db2d90$ Strictly follow the instructions of the above tasks. Do not mention whether the content or previous context was used or not; respond seamlessly. Avoid phrases like "the provided content" or similar. Take into account everything we've discussed so far, without assuming everything is relevant unless it clearly supports your answer. ----- {context} """ Then, the user-defined system prompt, context with referencing instructions, and user query are passed to the LLM. answer = await llm.ainvoke( input=[ SystemMessage(content=system_prompt), HumanMessage(content=human_prompt), HumanMessage(content=query), ] ) answer = answer.content Finally, we iterate through all of the references within the response generated from the LLM. If we find a reference in the text, we replace the reference with an HTML tag that includes the attributes tagged to that particular reference. pattern = r"\$REF: ([a-f0-9\-]+)\$" for match in re.finditer(pattern=pattern, string=answer): to_replace = match.group() ref_id = match.groups()[0] if ref_id in ref_mapping: ref = ref_mapping[ref_id] props = { "chunkId": ref["uuid"], "renderName": ref["render_name"], **ref.get("source_data", {}), } prop_str = " ".join( f'{key}="{parse.quote(str(value))}"' for key, value in props.items() ) citation = f'\n' answer = answer.replace(to_replace, citation) else: answer = answer.replace(to_replace, "") et voila, in-text citations. Conclusion Don’t let the conceptual simplicity of in-text citations fool you. They are incredibly powerful and have a ton of creative use cases. One of the biggest problems with LLM-powered products is hallucination; we simply don’t know when we can trust AI. In-text citations are one of the best approaches to address this problem, making them a critical tool in modern AI application development. Transformer XL Positional Encoding —By Hand Daniel Warfield — Mon, 11 Aug 2025 14:03:01 GMT “Relative” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. I recently released a piece that explores various positional encoding strategies in-depth. This article is meant to be a companion piece to that article, and explores relative positional encoding with content-dependent and global bias terms from the Transformer XL paper. This is an early relative positional encoding scheme designed to allow LLMs to better and more efficiently understand arbitrarily long sequences. Like other installments in the By-Hand series, this article is designed to serve as a brief reference rather than an exhaustive breakdown. For a conceptual breakdown, see the following IAEE article: This article assumes a strong working knowledge of attention in general. I have beginner friendly IAEE articles on the subject As well as By-Hand articles on classic attention approaches. Step 1: Defining the Inputs and Parameters First, we need to convert our input sequence to a vector embedding, usually by using a lookup of vectors that are optimized through the training process. So, in this example, we have a sequence length of 4 and a modeling dimension of 4 (the length of the vector used to encode each word). The positional encoding scheme we’re discussing uses four weight matrices and two bias vectors. These would be randomly initialized then updated throughout the training process. These parameters represent the following: Wq is the weights used to project the input into the query of self attention (just like in traditional attention mechanisms). Wk is the weights used to project the input into the key of self attention (just like in traditional attention mechanisms). Wv is the weights used to project the input into the value of self attention (just like in traditional attention mechanisms). Wr are weights used to apply locational information to the attention mechanism u allows the attention mechanism to learn biases to content information v allows the model to learn biases to positional information We’ll explore exactly how these works in the following sections. In this article, we’ll imagine a simple model consisting of a single attention block. Naturally, a real model would have many decoder blocks with many layers of attention. In such a model, each attention block would have its own parameters. This means each attention layer learns positional information independently. Step 2: Calculating Relative Positional Encoding Transformer XL uses sinusoidal positional encoding, similar to the original transformer. The following function can be used to calculate the positional encoding at a particular index: here: pos is the position in the sequence a particular word is (word 1, word 2, word 3, etc). i is the position in each word embedding a value is in (value 1 in word 1, value 2 in word 1, value 3 in word 1, etc). d the modeling dimension (the length of the vectors in the model, in this case 4). One major difference between the original transformer and transformer XL is what pos represents. In the original transformer pos represents the location in the sequence 0, 1, 2, 3, 4, 5, 6, .... n Whereas, in Transformer XL, pos represents the distance relative to the current last token n ... , 6, 5, 4, 3, 2, 1, 0 This means that, no matter how long the sequence is, the positional encoding for the token three steps before the most recent token will always be the same. Because the text closest to the current token has the largest bearing on generation, this allows Transformer XL to generalize much more easily to longer sequences compared to absolute positional encoding. For our sample input, we can calculate a positional encoding for each token, relative to the final token. here, pos=0 is the position of the final token, where pos=1 , pos=2 , etc. are preceding tokens. calculating “r”, the positional information for our for our attention layer. Step 3: Projection Just like any other flavor of attention, vectors representing words are passed through neural networks (represented as weight matricies), and those representations are used in attention calculations. The only real difference is that the location information is projected independently to allow the model to learn how to reason about location. First, we can take the input embeddings and pass them through the a weight matrix to result in the query, key, and value like normal. Then we can pass the relative positional matrix through its own weight matrix. The model will use these projected values to construct a few different attention representations simultaneously. Step 4: Calculating Terms and Attention Scores Unlike earlier approaches that add positional information to the initial input to the model, Transformer XL injects positional information through each attention layer. In each head of each attention mechanism of the model, four terms are calculated: Term a: Attention based on content Term b: Attention based on a combination of content and position Term c: Attention based on a global content bias Term d: Attention based on a global position bias These are all added together to create attention scores which incorporate both information about tokens as well as where they’re located Term A Term A is, essentially, classic attention. The Query and Key are multiplied together to construct attention scores. “Term A”, in this image, represents attention scores pre soft-maxing. We’ll be softmaxing to calculate the actual attention values after we calculate all terms and add them together. Term B We can multiply the query by our projected r values to calculate term B. Recall that Projected r is our relative positional encoding information that’s been passed through a neural network, allowing the query vector to interact with a learned representation of positional information. Normally I don’t harp on the naming convention of “query” and “key”, but in this application I find an understanding of the naming convention is useful. our “projected r” matrix in term B, and the “key” matrix in term A can both be conceptualized as a database of values. When we multiply the query vector by these matrices, the query extracts information from these matrices. We have the same query vector, representing the most recent token, and we’re applying that to different “databases” (conceptually) to extract different information. Term C In the previous two terms we co-related the query vector with information about words. Terms C an d D allow the model to build its own vectors, u and v , to interact with information about words. The previous two terms are referred to as “content dependent”, because the query is derived from the input. Terms C and D are referred to as “global bias” terms because the parameters used in place of the query ( u and v ) are learned by the model and are not influenced by the input content. Term C is the “global content bias”, meaning we multiply the learned vector u by the embedded word vectors. Term D Term D is just like term C, accept a different learned vector ( v ) is applied to the learned positional information ( projected r ), instead of the content ( key ). Thus, term D represents the “global position bias” Step 4: Calculating Attention and Applying to the Value Matrix Now that we’ve calculated terms A, B, C, and D, we can add them together and softmax them to calculate aggregate attention scores. And then we can softmax them. Softmaxing is simple, but hard to draw (and I’m already way behind on publishing), so I’ll leave that as an exercise to the reader. Refer to my article on multi-headed self attention (step 7) for a reference on that. We can then apply those attention scores to the value matrix to calculate our result. Discussion The reason Transformer XL was created was to make decoder style model better at longer sequences. It did this in three key ways: By making position relative to the token being generated. This allowed the model to generate to sequences longer than it was trained on By making inferences of long sequences more efficient. One strategy to apply models to sequences which are longer than the context length they’re applied on is to treat them like a recurrent model. You essentially just trim off some of the sequence at the beginning so you have room at the end to generate data. If positional information is encoded in the vectors themselves, then you need to re-compute all of the attention layers for this entire process. With transformer XL, because locational information is calculated on the fly for each token generated, you can efficiently move the context window around. If that’s all french, check out my article on positional encoding, which approaches this topic from a higher level. Cheers! Positional Encoding — Intuitively and Exhaustively Explained Daniel Warfield — Mon, 04 Aug 2025 23:00:50 GMT “Relative Rotation” by Daniel Warfield using MidJourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider becoming a paid subscriber of Intuitively and Exhaustively Explained. In this article we’ll form a thorough understanding of positional encoding, a fundamental tool that allows AI models to understand ideas of space and time. We’ll start by exploring the transformer (the grandfather of today's modern AI models) and why it needs additional positional information to be provided along with the input. We’ll then work through increasingly sophisticated methods of introducing positional information to transformer style models to build a thorough conceptual understanding of the approach. By the end of this article you’ll understand why positional encoding plays a fundamental role in the performance of modern AI systems, and several approaches to do positional encoding. Subscribe To Intuitively and Exhaustively Explained. Share this article with friends and colleagues. Share Who is this useful for? Anyone interested in forming a deep understanding of modern AI. How advanced is this post? This article contains intuitive descriptions of complex ideas, and is thus relevant to readers of all levels. Some of the sections have implementation that may be relevant to more advanced readers. Prerequisites: I’ll do my best to provide all foundational information so that you can forge a thorough conceptual understanding without prior AI experience. That said, this is a fairly nuanced topic. I’ll have references to supplementary information throughout the article should you find yourself lost. The Transformer, How LLMs Work, and Why They Need Positional Information Before we get into the various forms of positional encoding, let’s first discuss why LLMs need positional encoding in the first place. Imagine you feed in the following sequence to an LLM: Joe is six foot. Bob is five seven. Who is taller? The relative position of certain words is important. Imagine if we shuffled some of the words around: Bob is six foot. Joe is five seven. taller who is? The meaning completely changes. Even worse, if you really shuffle the words around, all meaning is completely lost Joe taller? foot. is Bob Who is is seven. five six Thus, for any LLM to do pretty much anything with text, it has to have some understanding of the order of the input sequence. To the uninitiated this might appear to be a simple problem. If you want the sequence to be in order, you just pass the sequence to your AI model in order. In fact, with many forms of AI models this is sufficient; many older and simpler AI strategies like recurrent neural networks, convolutional networks, and classic neural networks have a strong ability to understand the order of an input implicitly. Two previously popular modeling strategies (recurrent models on the left and convolutional models on the right) had an implicit understanding of position due to the way the models were defined. From the transformer and convolution IAEE articles. Modern LLMs don’t use these older strategies. Instead, they use a fairly new modeling architecture called the “transformer”. Transformers allow any word to interact with any other word, instead of only allowing nearby information to interact. The core idea of the transformer is to create a vector that represents every word in an input (query) and another vector that represents each word in the input (key). These interact with each other, allowing every word to interact with a representation of every other word. From the IAEE article on transformers. This arbitrary interaction allows the model to create a robust understanding of the entire input, but the added complexity makes it very hard for the model to implicitly understand the order of the input. To hammer home this idea, I’d like to take a moment to explore how a model like GPT works. How Data Flows Through a Transformer, and Why That Breaks Position Let’s imagine we have some LLM, like GPT. A diagram of the original GPT model. source. I discuss GPT in depth in the IAEE article on the subject Then imagine we pass GPT the following query Translate to French: I am a manager Our input is first passed through a text embedding. This turns our sequence of words into a sequence of vectors. By turning words into vectors, we allow the LLM to reason mathematically about how words relate with other words. Text embedding maps all words in the input into some vector of numbers. From the IAEE article on transformers. Our input is also treated with a positional encoding, which we’ll talk about in a bit. For now, we’ll skip this step to explore why positional encoding is necessary. A conceptual diagram of positional encoding, which we’ll discuss later. source. From my article on GPT. Now that we have a vector that’s associated with each word, we can pass that through a system called “self-attention”. In self attention we first use a neural network to convert our input representation into three different representations. First, the input is used to construct three different representations of the input. From my article on transformers Then, the representation called the query and key are multiplied together and used to create something called an “attention matrix” The key is rotated (transposed) so that a representation of each word in the query has an opportunity to interact with a representation of each word in the key. From my article on transformers. This attention matrix is then multiplied by the value matrix, resulting in the output of self-attention The attention matrix is multiplied by the value matrix to create the output of self-attention. From my article on transformers. There’s a bunch of math we would need to cover to thoroughly understand this mechanism. It’s not incredibly complicated, I have several articles on the subject that will allow you to completely understand it if you’re interested: But, for our purposes, there’s one critical concept I’d like to discuss. When you multiply the attention matrix by the value matrix, each row in the attention matrix essentially functions as a set of weights dictating how rows should be combined to create the output. Here’s an example of multiplying an attention matrix by a value matrix from the “Multi-Headed Self Attention — By Hand” article: multiplying an attention matrix by a value matrix. Source. The first row in the output is equivalent to the first row in the value matrix because the first row in attention is 100% the first value, and 0% all of the other values. If we move onto the next row in the output, it’s a combination of 39% of the first row and 61% of the second row in the value matrix, as specified by the attention matrix. Recall that we calculated the attention matrix by multiplying the query and key together. Recall that the key is rotated (transposed) so that a representation of each word in the query has an opportunity to interact with a representation of each word in the key. The result is the attention matrix. From my article on transformers. Because of the way matrix multiplication works, the values in the attention matrix are not governed at all by the location of a word in a sequence. Rather, they’re solely governed by the values of the word vectors in the query and key. If the query and key multiplied together to create an attention matrix that looked something like this: an example of an attention matrix that, when multiplied by a value matrix, would perfectly reverse the order of the sequence. Then multiplying this matrix by the value matrix would produce a perfectly reversed output. So, in allowing word vectors to arbitrarily interact with one another (one of the fundamental strengths of the transformer), the transformer has the ability to destroy all sense of order within the sequence. The thing that governs the order of the sequence, the attention matrix, is calculated by the values of the word vectors, not their location. Thus, LLMs can very easily destroy all sense of position within an input. Let’s explore how the original transformer got around this problem. Additive Sinusoidal Positional Encoding The creators of the original transformer architecture were confronted with the issue of their new model losing the order of inputs, and had to find a way to solve it. They elected to do that with a strategy commonly referred to as “additive sinusoidal positional encoding” Essentially, the idea is to bake positional information into the values of the vectors that represent each word. Recall that one of the first steps of feeding words into a transformer style model is to convert those words into vectors. Recall that one of the first steps of passing a sequence through a transformer style model is to convert each word into a vector. In this image each word is actually represented as two vectors. One represents the word, and the other represents the location of the word. source. From my article on GPT. In the original transformer paper, they essentially generated wiggly lines for each spot in the input, where the lines got progressively more wiggly throughout locations in the input sequence. They added these wiggly lines to the word vectors, constructing vectors that represent both the values of words as well as their locations in the input sequence. An example of some of the wiggly lines that are added to word vectors in the original transformer. From my article on transformers. Now, because each vector has positional information, the model can use that information when constructing queries and keys, If the word vectors have positional information baked into them, that information can be propagated to the query and key. From my article on transformers. which in turn dictate the creation of the attention matrix From my article on transformers. Which, ultimately, dictates how words are contextualized together. The attention matrix is multiplied by the value matrix to create the output of self-attention. From my article on transformers. In this strategy, positional information is added to the word vectors that are input into the model. It’s then the models job to learn parameters which reason about that information effectively. Let’s get an idea of how this might work with a toy problem. Exploring Additive Sinusoidal Positional Encoding With a Toy Problem Imagine the following problem: Given a sequence of random numbers, can we make a transformer style model that can tell us the index of the largest number in the sequence? Naturally, this requires a model to understand position, as the task requires the model to return the location of the largest value in the input sequence. Let’s try implementing a transformer style model with additive sinusoidal positional encoding, and see if it’s up to the job. Full code can be found here. First, we’ll create a function that defines our modeling objective. This simply returns a list of random numbers, and the corresponding maximum index. # --- Data: Predict index of max token --- def generate_batch(num, seq_len): x = torch.randint(1, vocab_size, (num, seq_len)) y = torch.argmax(x, dim=1) return x.to(device), y.to(device) Notice how we generate numbers between one and vocab_size. Transformers require us to create a vector for each word in our vocabulary. Here, each word is a number between 1 and whatever vocab_size is specified to be, in this case 20. seq_len defines how long the sequence we generate will be. Before we implement the transformer, let’s implement the positional encoding which the transformer will use. # --- Sinusoidal Positional Encoding --- def sinusoidal_encoding(seq_len, d_model, device): position = torch.arange(seq_len, dtype=torch.float, device=device).unsqueeze(1) div_term = torch.exp(torch.arange(0, d_model, 2, device=device) * (-math.log(10000.0) / d_model)) pe = torch.zeros(seq_len, d_model, device=device) pe[:, 0::2] = torch.sin(position * div_term) pe[:, 1::2] = torch.cos(position * div_term) return pe # [seq_len, d_model] this function accepts seq_len, which is important because we need that to know how many squiggly lines we need to make. d_model, on the other hand, defines how long those squiggly lines need to be. That will be whatever length we use to represent the words (or, in this case, numbers) in our vocabulary. Having a squiggly line for each spot in the sequence, and having those squiggly lines represented as vectors that are the same length as the word vectors, will make applying the positional encoding as easy as adding the two vectors together. Recall that one of the first steps of passing a sequence through a transformer style model is to convert each word into a vector. In this image each word is actually represented as two vectors. One represents the word, and the other represents the location of the word. These are added together to represent each word. source. From my article on GPT. To facilitate actually creating our positional encoding, we first generate a list of indexes with position = torch.arange(seq_len, dtype=torch.float, device=device).unsqueeze(1). This gets us an array of positions like. [0, 1, 2, ..., seq_len-1] The rest is excessively complicated given how conceptually simple the approach is, and I don’t want to waste a bunch of time unpacking all of the math. The positional encoding formula for this approach looks like this: I talk about this function, and it’s implementation, in depth in my article on transformers. Which, if you’re familiar with trigonometry, you might realize that these are sin and cosine functions. Here, pos represents the location of a particular word within the sequence, and we’re modifying the frequency both by i, the location in the embedding vector, and dmodel, the size of the model. The final positional encoding vectors look something like this: Example of positional encoding. The Y axis represents subsequent words, and the x axis represents values within a particular words positional encoding. Each row in this image represents the positional encoding for an individual word. From my article on transformers. In the image above, each row represents the squiggly line that would be added to a word. with subsequent rows representing subsequent words. Here’s the same plot, but with a smaller model dimension and smaller sequence length to aid in visibility. We can use our generated positional encodings within a transformer, like so: # --- Config --- device = "cuda" if torch.cuda.is_available() else "cpu" vocab_size = 20 d_model = 16 n_heads = 4 train_seq_len = 64 num_train = 30000 num_test = 500 batch_size = 64 epochs = 10 num_layers = 2 # --- Transformer Model --- class MaxIndexTransformer(nn.Module): def __init__(self, seq_len, num_layers): super().__init__() self.token_emb = nn.Embedding(vocab_size, d_model) self.pos_enc = sinusoidal_encoding(seq_len, d_model, device) self.layers = nn.ModuleList([ nn.TransformerEncoderLayer(d_model=d_model, nhead=n_heads, batch_first=True) for _ in range(num_layers) ]) self.norm = nn.LayerNorm(d_model) self.classifier = nn.Linear(d_model, 1) def forward(self, x): tok = self.token_emb(x) pos = self.pos_enc[:x.size(1)] tok = tok + pos for layer in self.layers: tok = layer(tok) logits = self.classifier(tok).squeeze(-1) # [B, T] return logits Here we’re defining a transformer style model with PyTorch. It has a word embedding for each word as defined by self.token_emb = nn.Embedding(vocab_size, d_model) and we also pre-calculate positional embeddings self.pos_enc = sinusoidal_encoding(seq_len, d_model, device) When we go to make an inference with the model, we add the positional information to the embeddings with tok = tok + pos before passing the tokens through the model. Recall that our toy problem was to find the index of the largest value in the sequence. We can train and evaluate our model on a few examples and see that the model can performantly identify the largest value in the input sequence. # --- Train --- def train(model, optimizer, criterion): model.train() for _ in range(epochs): for _ in range(num_train // batch_size): x, y = generate_batch(batch_size, train_seq_len) logits = model(x) loss = criterion(logits, y) optimizer.zero_grad() loss.backward() optimizer.step() # --- Evaluate --- @torch.no_grad() def evaluate(model, seq_len): model.eval() x, y = generate_batch(num_test, seq_len) logits = model(x) preds = torch.argmax(logits, dim=1) acc = (preds == y).float().mean().item() print(f"Eval acc at seq_len={seq_len}: {acc:.3f}") return acc # --- Run --- model = MaxIndexTransformer(seq_len=train_seq_len, num_layers=num_layers).to(device) optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) criterion = nn.CrossEntropyLoss() train(model, optimizer, criterion) evaluate(model, train_seq_len) Thus, we’ve seen how the addition of additive sinusoidal positional encoding can impart a transformer style model with the ability to understand sequence order. Learned Positional Encodings OpenAI famously took ideas from the transformer and built on them with GPT. One of the many things they changed was how GPT handled positional encoding. Instead of using some fancy math to create fancy squiggly lines for each location in a sequence, GPT simply created a vector for each spot in the sequence and had the model learn what values make sense in each spot. Recall that we’re adding together a vector that represents each word, and a word that represents each spot. The vectors representing the vocabulary are randomly initialized and updated throughout the training process, allowing the model to learn which vector should be used to represent each word. GPT applied the same approach to the positional encoding, assigning and learning a vector for each location in the input sequence. source. From my article on GPT. Actually implementing this is super straightforward. Instead of doing a bunch of fancy math to make squiggly lines, we just use nn.Embedding to create a vector for each spot in our sequence (called pos_emb) # --- Transformer with Learned Positional Embedding --- class MaxIndexTransformer(nn.Module): def __init__(self, seq_len, num_layers): super().__init__() self.token_emb = nn.Embedding(vocab_size, d_model) self.pos_emb = nn.Embedding(seq_len, d_model) self.layers = nn.ModuleList([ nn.TransformerEncoderLayer(d_model=d_model, nhead=n_heads, batch_first=True) for _ in range(num_layers) ]) self.norm = nn.LayerNorm(d_model) self.classifier = nn.Linear(d_model, 1) def forward(self, x): batch_size, seq_len = x.size() positions = torch.arange(seq_len, device=x.device).unsqueeze(0).expand(batch_size, seq_len) tok = self.token_emb(x) + self.pos_emb(positions) for layer in self.layers: tok = layer(tok) logits = self.classifier(tok).squeeze(-1) # [B, T] return logits When we go to make an inference, we simply add the word and position embedding together with tok = self.token_emb(x) + self.pos_emb(positions) Running this model on our toy problem (finding the max index in a random sequence), we get almost 100% accuracy. I’m just playing around, and could almost certainly modify this training code to be 100% accurate, so this is functionally identical to the 100% accuracy we saw in sinusoidal positional encoding. The authors of the original GPT paper claimed that learned positional encoding resulted in superior output to additive sinusoidal approaches. It’s also super easy to implement it, which is why I used this approach when creating a transformer style model designed to solve a rubik’s cube. Relative Positional Encoding Both of the encoding strategies we’ve discussed previously are examples of “absolute” positional encoding, meaning the information added to the word vectors denotes the actual location of the vector in the sequence. One problem with this approach is that sequences that are longer than the model saw during training can significantly degrade the performance of the model, as the model has never seen information about those locations in the training set. Imagine if a model was trained on some sequence of data (highlighted in yellow). It would do well in that region because it’s seen that positional information before. However, imagine if you then gave the model a sequence which is longer than any of the data in the training set. The model performance would quickly devolve as the model tries to reason with positional information it’s never seen before. The paper Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context built on top of previous positional encoding strategies with a single objective: making transformer style models that can understand sequences than are larger than the ones they were trained on. One of the central ideas of transformer XL was to use relative positional encoding. Instead of tokens being enumerated based on how far they were to the beginning of the sequence, like in previous approaches: token 0, token 1, token 2, token 3, .... token n They, instead, enumerated the tokens based on how far they were to the last token. token n ... token 3, token 2, token 1, token 0 This is incredibly powerful because, as a model generates new tokens, the most recently generated tokens in the sequence are the most important. If I gave an LLM the following phrase, for instance: I don't know about you, but I think an elephant would easily crush a can if it stepped on it. Elephants are very _______ The information at the end of the sequence is more relevant than the information at the beginning of the sequence when filling in the blank. If a model is especially good at understanding information at the end of the sequence, it will be better at generating output, even if the sequence becomes very large. One practical issue with this approach is that, as you generate a sequence, the positional information for each token has to change. The first token generated is at position zero, then when another token is generated it’s kicked over to spot one, then two, etc. 0 1, 0 2, 1, 0 3, 2, 1, 0 ... Both of the previous approaches injected positional information about each word at the beginning of the model. Recall that both of the previous approaches discussed inject positional information into the vectors for each value at the input. This only happens at the beginning. Every transformer layer afterwards uses the information provided by previous transformer layers to reason about the sequence. From my article on GPT. Then, these vectors (with their positional information) are passed through a big (and expensive to run) LLM. To re-calculate new values with new positional information, you would have to change the positional information at the input and recalculate all of the inputs throughout the entire model. For every token generated. That would be ludicrously expensive. To deal with this problem, the authors of the Transformer XL paper shifted where positional information is injected. Instead of injecting positional information at the input, then calculating how word vectors should attend to one another A conceptual diagram of self attention in a traditional transformer. Each of word vectors have already had positional information injected into them, so the model uses the information about the words and where they’re located to reason about how words should interact with one another. They calculate scores based on content and position separately Instead of locational information being injected into the values of the vectors, information about what words are and where they’re defined are handled seperately. And combine them together to calculate how tokens should attend with one another. By separating locational information out of content information, in every instance of attention throughout the model, transformer XL can efficiently manipulate locational information on the fly. In reality, Transformer XL does a variety of more complex operations than just adding content and location info together, but this communicates the essential idea. I have a by-hand article coming out soon that will go over the transformer XL style of positional encoding in-depth. This allowed the authors of the Transformer XL paper to modify positional information after a token was already generated, meaning you can maintain relative positional information without needing to recalculate every token at every generated step. It’s worth noting, the actual positional information could be learned, or it could be sinusoidal. The authors of the transformer XL paper favored sinusoidal encoding like the original transformer, as continuous functions allow the model to extrapolate it’s understanding of position further into the past. I’ll probably cover Transformer XL, and similar approaches like DeBERTa, in isolation in other articles. The general approach is fascinating, but there’s a lot of subtlety in the approach and how it relates with sequence modeling, which is deserving of it’s own article. For now, I think it’s sufficient to understand the major contribution of Transformer XL: it encodes positional information relative to the most recent token instead of relative to the first token, and it does that by cleverly injecting positional information within the attention mechanism itself, rather than at the input of the model. We’ll discuss strategies that take this general idea and run with it. Bucket Relative Bias Because the Transformer XL paper has the goal of working with arbitrarily large sequences, the authors of the Transformer XL paper invest a lot of computational resources into modeling position. This results in a model which is very expressive about positional information, but at the cost of complexity. The authors of the T5 paper did not have this objective (The T5 paper is a metastudy about transfer learning, without an explicit focus on studying positional embedding). Thus, the T5 paper uses a positional encoding scheme which is inspired by the Transformer XL paper, but simplifies the approach significantly. There are two key deviations the T5 paper made from the original Transformer XL approach: They represent positional information as a learned bias that gets added to attention They group certain distances between queries and keys into buckets This allowed the T5 paper to very efficiently model relative position. To make it happen, you calculate attention scores like normal: calculate how far apart those two tokens are grab the bias term corresponding to the bucket containing each distance The bias for each bucket is a learnable parameter add the relevant biases to the attention scores, then softmax to calculate attention. The only real subtlety is the way buckets are defined. The T5 Paper defines: a set of buckets for small, exact differences for keys that are before the query in the sequence a set of buckets for small, exact differences for keys that are after the query in the sequence a set of buckets for large, inpercise differences for keys that are before the query in the sequence a set of buckets for large, inpercise differences for keys that are after the query in the sequence Here’s an implementation of a PyTorch module with learnable biases corresponding to where relative positions fall into certain buckets. # --- T5 Relative Positional Bias --- class T5RelativePositionalBias(nn.Module): def __init__(self, n_heads, num_buckets=32, max_distance=128): super().__init__() self.n_heads = n_heads self.num_buckets = num_buckets self.max_distance = max_distance self.relative_attention_bias = nn.Embedding(num_buckets, n_heads) def _relative_position_bucket(self, relative_position): num_buckets = self.num_buckets max_distance = self.max_distance # Bidirectional bucketing num_buckets //= 2 relative_buckets = (relative_position > 0).to(torch.long) * num_buckets relative_position = torch.abs(relative_position) # Small positions map directly to buckets max_exact = num_buckets // 2 is_small = relative_position < max_exact # Large positions get log-scaled buckets relative_position_if_large = max_exact + ( torch.log(relative_position.float() / max_exact) / math.log(max_distance / max_exact) * (num_buckets - max_exact) ).to(torch.long) relative_position_if_large = torch.min( relative_position_if_large, torch.full_like(relative_position, num_buckets - 1) ) # Combine buckets relative_buckets += torch.where(is_small, relative_position, relative_position_if_large) return relative_buckets def forward(self, seq_len): q_pos = torch.arange(seq_len, dtype=torch.long, device=self.relative_attention_bias.weight.device) rel_pos = q_pos[:, None] - q_pos[None, :] buckets = self._relative_position_bucket(rel_pos) bias = self.relative_attention_bias(buckets) # [T, T, H] return bias.permute(2, 0, 1).unsqueeze(0) # [1, H, T, T] We can then define a layer of a transformer style model that uses these relative attention biases like so: # --- T5 Transformer Layer --- class T5TransformerEncoderLayer(nn.Module): def __init__(self, d_model, n_heads, dropout=0.1): super().__init__() self.norm1 = nn.LayerNorm(d_model) self.norm2 = nn.LayerNorm(d_model) # Attention projections self.q_proj = nn.Linear(d_model, d_model) self.k_proj = nn.Linear(d_model, d_model) self.v_proj = nn.Linear(d_model, d_model) self.out_proj = nn.Linear(d_model, d_model) self.head_dim = d_model // n_heads def forward(self, x, relative_attention_bias): # Pre-LayerNorm attention residual = x x_norm = self.norm1(x) # Project to query/key/value B, T, _ = x_norm.shape q = self.q_proj(x_norm).view(B, T, n_heads, self.head_dim).transpose(1, 2) k = self.k_proj(x_norm).view(B, T, n_heads, self.head_dim).transpose(1, 2) v = self.v_proj(x_norm).view(B, T, n_heads, self.head_dim).transpose(1, 2) # Attention scores attn_scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.head_dim) attn_scores = attn_scores + relative_attention_bias # Attention weights attn_weights = F.softmax(attn_scores, dim=-1) # Context context = torch.matmul(attn_weights, v) context = context.transpose(1, 2).contiguous().view(B, T, -1) context = self.out_proj(context) # Residual connection x = residual + context # Final LayerNorm (no FFN) x = x + self.norm2(x) return x This is a minimal implementation of an encoder block. Usually you’d also have a feed forward network in here as well, but we’re keeping it simple in this example. You might notice that relative_attention_bias is accepted as an input to the forward function. If you wanted each layer of the model to learn it’s own positional parameters, you could define a seperate learnable T5RelativePositionalBias for each layer, but the authors of the T5 paper elected for all layers in the model to share the same bias information. As a result, the biases are passed to the encoder block, rather than stored within the block directly. As we previously discussed, these biases are added to the attention scores calculated by the multiplication of the query and key, and those biased attention scores are then softmaxed to calculate attention. We can then build a model that uses these relatie biases and encoder blocks. # --- Transformer Model --- class MaxIndexTransformer(nn.Module): def __init__(self, seq_len, num_layers): super().__init__() self.token_emb = nn.Embedding(vocab_size, d_model) self.relative_bias = T5RelativePositionalBias(n_heads) self.layers = nn.ModuleList([ T5TransformerEncoderLayer(d_model, n_heads) for _ in range(num_layers) ]) self.norm = nn.LayerNorm(d_model) self.classifier = nn.Linear(d_model, 1) def forward(self, x): tok = self.token_emb(x) T = x.size(1) rel_bias = self.relative_bias(T) for layer in self.layers: tok = layer(tok, rel_bias) tok = self.norm(tok) logits = self.classifier(tok).squeeze(-1) return logits We can then train this model on our task of finding the index of the largest value in the sequence. # --- Train --- def train(model, optimizer, criterion): model.train() for _ in range(epochs): for _ in range(num_train // batch_size): x, y = generate_batch(batch_size, train_seq_len) logits = model(x) loss = criterion(logits, y) optimizer.zero_grad() loss.backward() optimizer.step() # --- Evaluate --- @torch.no_grad() def evaluate(model, seq_len): model.eval() x, y = generate_batch(num_test, seq_len) logits = model(x) preds = torch.argmax(logits, dim=1) acc = (preds == y).float().mean().item() print(f"Eval acc at seq_len={seq_len}: {acc:.3f}") return acc # --- Run --- model = MaxIndexTransformer(seq_len=train_seq_len, num_layers=num_layers).to(device) optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) criterion = nn.CrossEntropyLoss() train(model, optimizer, criterion) evaluate(model, train_seq_len) And we see that our performance has dropped relative to previous approaches, which makes sense. We’re using buckets that include multiple positions to encode position; the model might know approximately where largest word in the sequence is, but if the largest number falls in a bucket corresponding to five locations then the model has to guess which of those locations contains the largest number. For this task, bucketed relative bias encoding is a horrible positional encoding strategy. However, for language modeling, this approach is very performant. The T5 paper was state of the art in 2020, despite using a positional encoding strategy that is drastically simpler than previous positional encoding strategies. I find it fascinating that, given such a complex subject as natural human language, position can be modeled so sparsely and imprecisely while still achieving incredible performance. Rotary Positional Encoding Rotary Positional Encoding (RoPE) is a significant deviation of previous approaches, and is also one of the most widely used positional encoding schemes in modern AI. It borrows some ideas from previous positional encoding schemes, but merges them together in a way that’s fundamentally unique to the approaches we’ve previously discussed. Recall that, in all of the previous encoding schemes, we calculated some positional information then somehow got that positional information to interact with information about the content of words. In the original transformer, we added squiggly lines to vectors representing words. Different squiggles represent different locations In GPT, we learned vectors and added them to words In Transformer XL we separated out positional information, used it to calculate attention scores, then added those attention scores to content based attention scores to calculate attention In T5 we calculated the delta between queries and keys, and assigned a bias to the attention scores based on that offset Rotary positional encoding is different because position is represented as a function which modifies the word vectors directly, instead of values that interact with content in some way. Recall that, in a transformer for instance, each word (or, more accurately, token) is represented as a vector in high dimensional space. Previously, we add squiggly lines to these vectors to impart positional information. A conceptual diagram of addative sinusoidal positional embedding in the transformer. Some vector of waves is added to each word vector to encode position. RoPE treats each of these vectors as a collection of small, 2D vectors. Then, RoPE rotates these sub-vectors depending on where they are in the input sequence. A conceptual diagram of how RoPE functions. Each vector is treated as a collection of sub-vectors, and each of those sub-vectors is then rotated to encode position. Different vectors in different positions in the sequence see different amounts of rotation, thus encoding position. Naturally, if you rotate a vector all the way around, you end up with the same vector. If we want to encode long sequences with a lot of tokens, we need a way to encode a lot of positions without creating duplicates. We also, though, don’t want to make rotations so miniscule as to make it difficult for the model to understand the positional information we’re trying to encode. If you rotate a vector enough, you end up with the same vector. This means, if you’re using rotation to encode position, you need to be clever about how you’re doing it. Otherwise, multiple locations will have the same positional information. RoPE deals with this by rotating different sub-vectors within each word vector by different amounts. Under the hood, actual rotation is done by multiplying a specially formed matrix by each individual sub-component. The rotation matrix. Multiplying this matrix by some vector will rotate that vector. This general idea is found all over the place, not just RoPE. This matrix has the parameter θ (theta), which specifies how much the vector is rotated. In RoPE, θ changes not only as a function of where a vector is within the sequence, but also which sub-component we’re operating on within each vector. Some sub-vectors are rotated quickly, while others are rotated slowly. This allows the model to be exposed to rotations that change quickly over time (communicating a lot of information, but repeating often) and slowly change over time (not communicating a lot of granular information, but also never repeating). the angle of rotation is a function of the index of the word vector we’re operating on, and the index of the subcomponent we’re operating on. Here, θ is fed into the rotation matrix to govern how much a particular matrix is rotated. p represents the index of a word vector within a sequence. ωk represents the speed at which the kth vector rotates, which is defined by 10000^(-2k/d). Here. k is which sub-component of the vector we’re operating on, and d is the total length of the vector. A conceptual diagram of the effect of rotary positional encoding. Imagine each word vector as a row, where each row contains sub-components that are rotated at different frequencies. This has strong conceptual ties to the additive sinusoidal positional encoding strategy we discussed previously. Recall adaptive sinusoidal embedding. Sin waves of different frequencies are added together to calculate the positional encoding at different locations. This is similar in spirit to RoPE, with the major difference that in RoPE, the word vector itself is rotated rather than being added to. RoPE is relatively straightforward conceptually, but it (and derivative approaches like XPos and YaRN) contain subtle qualities which are worth discussing in their own article. I’ll likely be doing dedicated IAEE and By-Hand articles on the subject in the near future. For now, we can cap off our understanding by studying an implementation of the approach. import torch import torch.nn as nn import math # --- Config --- device = "cuda" if torch.cuda.is_available() else "cpu" vocab_size = 20 d_model = 16 n_heads = 4 train_seq_len = 64 num_train = 30000 num_test = 500 batch_size = 256 epochs = 10 num_layers = 2 # --- Data: Predict index of max token --- def generate_batch(num, seq_len): x = torch.randint(1, vocab_size, (num, seq_len)) y = torch.argmax(x, dim=1) return x.to(device), y.to(device) # --- RoPE --- def apply_rope(x): b, seq_len, d = x.shape assert d % 2 == 0 half_d = d // 2 x = x.view(b, seq_len, half_d, 2) freqs = torch.arange(half_d, device=x.device).float() theta = 10000 ** (-freqs / half_d) pos = torch.arange(seq_len, device=x.device).float() angles = torch.einsum("s,f->sf", pos, theta) cos = torch.cos(angles).unsqueeze(0).unsqueeze(-1) sin = torch.sin(angles).unsqueeze(0).unsqueeze(-1) x1 = x[..., 0:1] * cos - x[..., 1:2] * sin x2 = x[..., 0:1] * sin + x[..., 1:2] * cos x_rot = torch.cat([x1, x2], dim=-1) return x_rot.view(b, seq_len, d) # --- RoPE-aware Multihead Attention --- class RoPEAttention(nn.Module): def __init__(self, d_model, n_heads): super().__init__() self.attn = nn.MultiheadAttention(d_model, n_heads, batch_first=True) self.norm = nn.LayerNorm(d_model) def forward(self, x): residual = x x = apply_rope(x) x, _ = self.attn(x, x, x) return self.norm(x + residual) # --- Transformer Encoder Layer --- class RoPEEncoderLayer(nn.Module): def __init__(self, d_model, n_heads): super().__init__() self.self_attn = RoPEAttention(d_model, n_heads) self.ff = nn.Sequential( nn.Linear(d_model, d_model * 4), nn.ReLU(), nn.Linear(d_model * 4, d_model) ) self.norm = nn.LayerNorm(d_model) def forward(self, x): x = self.self_attn(x) x = x + self.ff(self.norm(x)) return x # --- Transformer Model with RoPE --- class MaxIndexTransformer(nn.Module): def __init__(self, seq_len, num_layers): super().__init__() self.token_emb = nn.Embedding(vocab_size, d_model) self.layers = nn.ModuleList([ RoPEEncoderLayer(d_model, n_heads) for _ in range(num_layers) ]) self.norm = nn.LayerNorm(d_model) self.classifier = nn.Linear(d_model, 1) def forward(self, x): tok = self.token_emb(x) for layer in self.layers: tok = layer(tok) tok = self.norm(tok) logits = self.classifier(tok).squeeze(-1) # [B, T] return logits # --- Train --- def train(model, optimizer, criterion): model.train() for _ in range(epochs): for _ in range(num_train // batch_size): x, y = generate_batch(batch_size, train_seq_len) logits = model(x) loss = criterion(logits, y) optimizer.zero_grad() loss.backward() optimizer.step() # --- Evaluate --- @torch.no_grad() def evaluate(model, seq_len): model.eval() x, y = generate_batch(num_test, seq_len) logits = model(x) preds = torch.argmax(logits, dim=1) acc = (preds == y).float().mean().item() print(f"Eval acc at seq_len={seq_len}: {acc:.3f}") return acc # --- Run --- model = MaxIndexTransformer(seq_len=train_seq_len, num_layers=num_layers).to(device) optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) criterion = nn.CrossEntropyLoss() train(model, optimizer, criterion) evaluate(model, train_seq_len) Here, we see a lot of the usual suspects. We’re applying a transformer style model (this time, one that utilizes RoPE for positional encoding), to our task of finding the index of the largest number in the sequence. The main addition, here is the apply_rope function # --- RoPE --- def apply_rope(x): b, seq_len, d = x.shape assert d % 2 == 0 half_d = d // 2 x = x.view(b, seq_len, half_d, 2) freqs = torch.arange(half_d, device=x.device).float() theta = 10000 ** (-freqs / half_d) pos = torch.arange(seq_len, device=x.device).float() angles = torch.einsum("s,f->sf", pos, theta) cos = torch.cos(angles).unsqueeze(0).unsqueeze(-1) sin = torch.sin(angles).unsqueeze(0).unsqueeze(-1) x1 = x[..., 0:1] * cos - x[..., 1:2] * sin x2 = x[..., 0:1] * sin + x[..., 1:2] * cos x_rot = torch.cat([x1, x2], dim=-1) return x_rot.view(b, seq_len, d) I don’t want to get lost in the sauce in this article, but a bit of googling and GPTing can get you far if you want a thorough understanding of what every single line is doing. In essence, this function: calculates key parameters based on the input sequence creates the rotation matrix for each position applies those rotations returns the result After running this code, we can see that the approach performs well on our toy problem. Conclusion In this article we covered positional encoding. We first discussed why it’s necessary, and how the original transformer tackled the problem with additive sinusoidal positional encoding. We then discussed a variety of different positional encoding strategies; learned positional encoding, relative positional encoding, bucketed relative positional encoding, and RoPE. There are countless other positional encoding schemes out there, but I think the most important thing is to understand the general approach, why it’s necessary, and a few ways it can be done. In future article’s we’ll be exploring strategies that build on top of RoPE, so stay tuned! UV — Intuitively and Exhaustively Explained Daniel Warfield — Mon, 30 Jun 2025 14:02:49 GMT “Packed” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider subscribing to Intuitively and Exhaustively Explained. UV is the new and hot project/environment/dependency manager taking the Python world by storm. The GitHub stars are off the chats: uv, vs some other popular package and project management tools in Python But, more importantly, it’s being used all over the shop in high-profile, cutting-edge repos that are governing the way modern software is evolving. Anthropic’s Python repo for MCP uses UV, Google’s Python repo for A2A uses UV, Open-WebUI seems to use UV, and that’s just to name a few. It came out of nowhere, and it’s changing the way people make projects in Python. Let’s talk about it. Who is this useful for? Anyone developing in Python. How advanced is this post? This article is accessible to developers of all levels. Prerequisites: None, but if you’re not a developer or plan on being one, I have no idea why you would want to read any of this. go read something more interesting instead: Intuitively and Exhaustively ExplainedThe ultimate source in AI, for both beginners and experts alike. By Daniel Warfield Dependencies, Environments, and Projects For those uncut by the need, I envy you. For those who have been working in Python for a while, you know exactly what I’m about to talk about. If you’re a fledgling developer who’s starting to learn Python, you might do a bit of googling and be directed to install Python onto your computer. Python comes with a package manager called pip (which, cutely, stands for “pip installs packages”). If you’re on Windows you’ll likely need to mess around with environment variables and maybe restart your computer. If you’re on mac you might realize “oh hey, I already have Python installed”. Either way, you get to the point where you can install new packages with commands like pip install and you can run python files with python .py (or maybe python3 .py). So, you play around, and have a merry old time. You build a few projects with some different dependencies; some require pandas, some require uvicorn , some require numpy , scipy , pytorch , etc. You install a whole bunch of libraries, learn them, and take the world by storm. Then, you go back to one of your older projects to use as a reference, and it doesn’t work. You get an import error on some esoteric library you’ve never heard of, which is a dependency of a dependency of a dependency of one of the libraries used in the project you’re trying to run. After a while, you realized that the project you just made uses a version of a dependency that isn’t compatible with this project. If you’re on Mac, you might realize that messing with the pre-installed version of Python wasn’t such a good idea after all. To this day, these God-damned crates fill me with anxiety. Source. You do a bit more Googling (or, maybe ChatGPTing for my younger readers. I’m 27, this is the first time I’ve ever earnestly felt old), and learn that a lot of developers use something called virtualenv or conda to do something called “environment management”. Basically, the idea is that you can create separate environments of python, each with their own dependencies and maybe even Python versions. This allows you to have a container of python dependencies for one project, and another container for another project. This gets you pretty far. You build the habit of creating and curating different environments, and you activate different ones for different projects. That might be good enough, but for many developers, this is cumbersome. If you accidentally enable the wrong environment for a project, you can damage the dependencies, thus damaging a bunch of other projects that rely on that environment. After a bit more Googling/GPTing you might stumble upon poetry, which I honestly haven’t used very much. I know, I know, how dare I call myself a Python developer. I’m a data scientist, so I often find myself in completely isolated and ephemeral environments running on the cloud anyway. Because I haven’t used poetry a ton, I can’t speak for its cost and benefits; I can only give you the high-level touchpoints. Basically, poetry is a set of tools used to make dependency and package management easier, allowing you to do things like environment management, dependency installation, versioning, and other stuff all with a single tool. From what I’ve read, it’s great if you’re working in a team of people and standardizing building and deploying is a must. I’ve also read that it can be a bit heavy, and enforces rules that you need to follow to use it effectively. I recommend this video, which does a nice job of quickly going through some of the costs and benefits of UV vs poetry. From what I gather, poetry is great for application developers working collaboratively on Python projects, but if you’re whipping up personal projects or sharing lightweight repos it can be a bit of a headache. Anywho, you’re now a bigshot Python developer who’s working on cool projects. Maybe you start working with MCP, A2A, or some other new fancy tech, and you bump into this new thing called “UV”. What is uv? How does it work? What’s it good for? Let’s get into it. The core Commands of UV For some of you, all I need to tell you is that uv is written in Rust. For others, all I need to tell you is that uv is like the npm of Python. For the rest, here’s how to install uv, and what you can do with it. Installing uv uv install instructions are defined here. You can install uv either with curl curl -LsSf https://astral.sh/uv/install.sh | sh or even with pip pipx install uv Check out the uv docs for more information on installation. It seems like using curl -LsSf https://astral.sh/uv/install.sh | sh is recommended in Linux and Windows, and powershell -ExecutionPolicy ByPass -c “irm https://astral.sh/uv/install.ps1 | iex” is recommended on Windows (or just use the linux subsystem in Windows like a normal person). uv init uv does project, package, and environment management in python. When you run uv init in a particular folder, uv builds the necessary resources to create an isolated Python project. Upon running uv init, five files are created: a .gitignore, which prompts git to ignore certain generated and environment files. This is standard practice, so uv doing it is a nice convenience. a .pythonversion file, which is used for managing Python environments. uv can be used to specify different Python versions for different environments, which we’ll discuss later. a main.py, which is a simple sample “hello world” style script. a project.toml file, which details the name of the project, information about the Python version, a description of the project, the location of the top-level readme file for the project, and dependencies. a README.md file, which is an empty README file for defining top-level documentation for the project. a uv.lock file, which records granular information about dependencies and sub-dependencies. This allows uv to granularly record and re-build dependencies. Thus, the uv init command can be run in a folder to turn that folder into a uv managed project. uv add Once you have a uv initialized project, you can install new dependencies to that project with the uv add command. uv can install from pypi, just like pip, conda, and poetry, so you can install all your favorite open-source dependencies just like you would with pip. Here’s some examples, the last one showing that multiple dependencies can be installed in a single line. uv add pandas uv add scipy uv add numpy sklearn matplotlib When you install dependencies, these are added to the pyproject.toml and granular information is added to the uv.lock file. This allows uv to track the dependencies in a project. You can also install dependencies from a valid GitHub repo uv add git+https://github.com/encode/httpx or other sources. These are the following dependency sources that are supported by uv (source): Index: A package resolved from a specific package index. Git: A Git repository. URL: A remote wheel or source distribution. Path: A local wheel, source distribution, or project directory. Workspace: A member of the current workspace. Realistically, most users will just be installing from PyPi the vast majority (if not all) of the time. uv remove if you want to remove a dependency from the project, the uv remove command can be used. This will uninstall the dependency, and update pyproject.toml and uv.lock to respect the change. For instance, if we wanted to uninstall matplotlib from a particular project, we would run uv remove matplotlib uv sync After you’ve added dependencies to a project, then maybe committed that project to a repo, when other people clone that repo they’ll need to install those dependencies. uv sync synchronizes the dependencies with the uv.lock file. If you clone a repo that’s been managed with uv, this will download all the necessary dependencies. uv does a great job automatically synchronizing projects, and often calls uv sync automatically under the hood before executing other key commands like uv run, so you may not need to use this. It’s useful to know, though. uv run If we want to run a Python script within our environment, we can use the uv run command. This will ensure that the script is run correctly with the correct dependencies and Python version. uv run main.py You don’t have to exclusively run Python files directly either. If you’re using something like mcp inside your project, and you want to spool up the dev server to experiment with an mcp server’s functionality, you can run something like uv mcp dev server.py This runs mcp in dev mode based on the server.py file within your current uv environment. There’s a ton of other cool stuff you can do with uv run (docs), but the core workflow is pretty simple. Managing Python Versions When you create a uv initialized project, the pyproject.toml has a requires-python field which, as far as I can tell, is always specified as >=3.11. You can modify this, but it seems like this is the recommended minimum for using uv, which is reasonable considering Python 3.11 was released in 2022. By default, uv will work with whatever Python version you have lying around. However, you can specify a specific Python version with uv python pin . So, if you want to create a new uv managed Python project that uses Python 3.12, you can do that with the following command: uv init uv python pin 3.12 This will update the .python-version file. If we run the sample main.py file with uv run main.py, you’ll notice a .venv folder will be created. This will define a virtual environment containing the necessary resources for running Python 3.12. Using uv Ok great, I think we’ve covered most of the basics. Let’s explore how uv can practically be used. Example 1) Defining a Series of Projects When developing a complex project, a common approach I take is to make a bunch of little sub-projects before diving into the full thing. This helps me understand some of the subtleties of my plan, and refine my plan with more knowledge before taking the big bite out of the larger project. UV is great for this, because you can simply create new folders and run uv init in them to define a completely new version. If you want to copy dependencies from one project to another, you can just copy the dependencies in the project.toml file from an old project to a new one and run uv sync. Then, as you bounce around between different projects, you can use uv run to run scripts in different projects while using the dependencies for that particular project. I did this exact strategy when working on understanding A2A for my recent article on the subject, and it was super convenient. Example 1) Cloning a Pre-Existing Project If you clone a uv managed project from github, uv does a bunch of cool stuff automagically. If we cloned my A2A demo repo, for instance git clone https://github.com/DanielWarfield1/A2A-Demos.git and if we then entered into one of the demos cd demo10 we can then see scripts like car_agnet.py, airline_agent.py, etc. Which are all runnable. if we call uv run airline_agent.py uv will automatically detect that a virtual environment doesn’t exist yet, and that the proper dependencies haven’t been installed. It’ll set all that up real quick before running our script. And when I say “real quick”, I mean it. uv is blazingly fast at setting up things like virtual environments and packages, partially because it’s written in rust. When exploring uv vs other package and environment managers, you’re likely to see the following graph passed around I’ve seen this and similar image bouncing around in a few places. source, source, source, source This speed makes these on-the-fly installations feel effortless. uv is really a remarkable example of “I love it because I don’t have to think about it”. Conclusion I’ll be using uv for managing all of my personal and writing Python projects moving forward. If I’m not using some managed Jupyter environment like Google Colab, I’ll be using uv. It’s intuitive, fast, and effortlessly easy to use. I’m also (surprisingly to myself) excited to experiment with poetry at some point in the future. I’ve always found poetry to feel excessively heavy, but now that I have a tool for quick, light work, I can keep poetry in the back of my mind if uv feels like it’s not quite robust enough for a big new project. I’m a big fan of Jupyter, the cell execution and interwoven documentation are a godsend for deep theoretical work, which is most of what I do. I’ll be honest, though, a big reason I use Colab is because project management has been a cumbersome issue in Python for as long as I can remember, and I would rather spool up an ephemeral compute instance on the cloud than deal with it. I think, with uv, I’ll find myself doing a lot more local Python development moving forward. Share this article with friends and colleagues. Share Agent To Agent Protocol — Intuitively and Exhaustively Explained Daniel Warfield — Fri, 27 Jun 2025 14:00:41 GMT “Message Passing” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. This article is made available thanks to the generous support of IAEE subscribers. To support the creation of future work, consider subscribing to Intuitively and Exhaustively Explained. In this article we’ll discuss “Agent to Agent Protocol” (A2A), the new and hot communication protocol designed to allow LLM agents to talk to one another. The internet is bright with speculation about an AI-first internet featuring websites talking to each other with natural text. We’ll explore A2A, how it works, what it’s useful for, and how realistic that vision of the future is. We’ll also explore how A2A and MCP (model context protocol) might work together to build modern AI-powered applications. Subscribe To Intuitively and Exhaustively Explained. Share this article with friends and colleagues. Share Who is this useful for? Anyone interested in forming a complete understanding of AI How advanced is this post? This article is conceptually accessible to all readers, especially the earlier sections. The implementation sections are intended for more advanced readers and developers. Prerequisites: The earlier conceptual sections are approachable to anyone. Later sections are geared towards people with some knowledge of LLM agents and MCP servers. Supplementary material is provided throughout the article. Some Important Background Concepts To thoroughly understand this article, I think it’s important to first understand the context in which A2A is currently being developed. Feel free to skip any of the following subsections if you’re already familiar, or check out the supporting material if you’d like a more in-depth explanation. Background: LLMs You probably have an idea of what LLMs are and how they work. As a refresher, LLM stands for “large language model”, and is a large AI model designed to be able to output text based on input text. This approach lit up in popularity thanks to models like GPT The essential idea is that LLMs are next word predictors. They guess the next word in an input, then guess the next word, then repeatedly guess more words until an entire output is predicted. A conceptual diagram of an LLM outputting text. It does so one word at a time, using the previous words to output future words. From my article on GPT. Background: Chain of Thought Prompting LLMs are like big fancy next-word predictors; they use previous words to predict future words. The user's input is included in those “previous words”, so the user's query influences the output generation from the model. Thus, prompt engineering was born. One of the most important ideas of prompting is “chain of thought”, which is a method of prompting that coaxes the model into thinking through a problem before solving it. Because the LLM uses previous words to predict future words, if the LLM solved sub-components of a user's query, it can use that previous output when constructing the final output. In the example on the left, the example answer included in the prompt consists of only the answer. In the prompt on the right, the example answer includes a chain of thought. The model naturally mimics the style of answer provided in the example answer within the prompt. The example on the left failed with standard prompting, and was successful with chain of thought prompting, even though the model was exactly the same. From my article on Agents. Source Background: Retrieval Augmented Generation Retrieval Augmented Generation (RAG) is an approach to adding external information into a model's prompts in order to make the model better at answering certain types of questions. As the name implies, the approach consists of three key parts: Retrieval: When the user submits a query, we look through a knowledge base to find information that is relevent to that query. Augment: We construct an “augmented prompt”, which is just a combination of the user's question and whatever knowledge was retrieved in the retrieval step. Generate: We pass the augmented prompt to the LLM for generation. The fundamental RAG flow. From my article on RAG. This is an incredibly powerful general approach as it allows for, essentially, arbitrary information to be injected into a language model as needed. I describe the fundamentals of the subject in the following article: I also regularly film a podcast that covers more advanced RAG concepts in depth. In that podcast, I cover more advanced RAG topics: and also happened to cover the exact topic that we’ll be discussing today: Background: Agents Generally speaking, an “Agent” is what you get when wrapping an LLM inside some greater logical loop. There are a ton of different types of Agents, but the most common I see is a “ReAct” style of agent. When defining a ReAct agent, you usually start by defining some collection of “tools”. A tool, from an agentic perspective, is a piece of code that the agent can decide to run. You might have tools for looking up weather forcasts, running pieces of code, interacting with the user via a UI, whatever. Once you have some tools, you define a ReAct style agent by telling the model it will be able to do three things: Think: The model will be prompted to output a thought as to what to do next. Act: The model will be able to decide to execute a certain tool. Observe: The model will have an opportunity to observe results of tool use, and say what is useful in the output. This allows the agent to extract useful information and put it into the continuum of chat memory. An image of a ReAct style of agent, on the right, vs some other agentic designs. I cover ReAct style agents in-depth in my article on LLM Agents. Image source. Conceptually, you can think of this as a marriage between chain of thought and RAG. I cover Agents in general, including ReAct style agents, in the following article: Background: Model Context Protocol Model Context Protocol (MCP) has been the latest hotness in the AI space. Recently released by Anthropic, MCP is a communication protocol (sort of) that standardizes how LLM agents interact with tools. The idea is that you can define an MCP server, which in turn defines a selection of tools, and you can plug that server into any compatible MCP client (which has an LLM agent under the hood), and that agent can run those tools. MCP is directly inspired from LSPs, which serve as a middleware between coding languages and coding editors. Instead of every editor needing custom support for every language, if an editor understands LSP, and if a language has an LSP implementation, then both will work with one another. MCP does a similar thing, but for agents to connect with tools. From my article on MCP MCP has some abstractions on top of LSPs that allow for some approaches that don’t make sense for LSPs, but the core workflow is identical. In LSPs, the code editor spawns the server, and then talks with the server as a client. While MCP can be launched in different ways, it usually behaves similarly, with the MCP client spawning then talking to the MCP server. From my article on MCP The main difference between MCP and LSPs is in resource definitions. While LSPs have a bunch of resources that are useful to code editors, MCP is designed to have resources that are convenient for AI models. A screenshot of the MCP inspector. MCP exposes Resources (File-like data that can be read by clients), Tools (Functions that can be called by the LLM), and Prompts (Pre-written templates that help users accomplish specific tasks). Because these are standardized, the idea is that an LLM powered MCP client can use these as necessary based on the LLM agent’s decisions. From my article on MCP I cover MCP in depth in another article, if you want to take a look: MCP was originally designed to allow Claude Desktop to interact with local tools, but Anthropic open-sourced the technology, sparking a wave of excitement around arbitrarily interoperable AI agents. Agent to agent protocol is largely an extension of this excitement. Agent to Agent Protocol, In a Nutshell It’s important to note, both MCP and A2A are recently introduced protocols, and as such they are very much in the maturation state. Exactly what these protocols do, and what they’re used for, is likely to evolve as both MCP and A2A mature, and as new protocols are proposed within the greater LLM Agent communication space. As of now, if model context protocol (MCP) is designed to connect LLM agents to tools, then A2A is designed to connect LLM agents with each other. A2A is largely a complement to MCP, with MCP functioning as a hierarchical tool based exposure, and A2A functioning as a peer-to-peer exposure. A diagram of the A2A protocol, from the home page for the A2A protocol A2A, in its most essential conceptualization, has two fundamental parts: Agent Cards: These are JSON objects that provide textual descriptions of what the agent is designed to do, which tasks it can perform, etc. This is designed to allow agents to discover each other's capabilities so that they can choose when it’s necessary to talk to another agent, and what that agent might be able to help with. Tasks: These are units of work which the client (usually an AI agent) requests of a server (also usually an AI agent). Depending on how the A2A server is implemented, the response might be streamed in, or it might be received as a big lump response. The general flow diagram between the client and the Server for A2A. Source. So, let’s say some company has an agent you want to talk to. You could build an A2A client, request the agent card from the company, then an LLM agent in your A2A client could choose when it might need to talk with this foreign agent. This is super exciting because, now, companies can define their own AI agents and expose them to external users. The agent can operate on behalf of the user, but in a secure and controllable environment. Just like how websites allow humans to talk with servers using pre-defined APIs, the idea of A2A is that A2A will allow agents to talk with servers using pre-defined A2A interfaces. The Fundamental Protocols and Frameworks of A2A We’ll be using Python in this tutorial. The Python implementation of A2A is derivative of a few pre-existing protocols and frameworks that I think would be useful to understand before diving into the intricacies of A2A itself. Fundamental Protocol: ASGI ASGI stands for “Asynchronous Server Gateway Interface”, and is a Python-specific standard for implementing asynchronous servers. If you’re familiar with Django or FastAPI, then you’re familiar with ASGI. A2A is built on top of ASGI, which is why it’s relevant for this discussion. The fundamental idea of ASGI is to create a simple and universal interface for developing asynchronous web applications in Python. An ASGI application is as simple as a single function that takes three arguments: scope, receive, and send. async def app(scope, receive, send): #... These arguments are defined as follows: Scope: A dictionary containing details about the specific connection or incoming request. This information includes details like the protocol type (e.g., “http” or “websocket”), headers, path, and potentially other metadata specific to the connection. Receive: An asynchronous callable that allows the application to receive events or messages from the server. For instance, in an HTTP request, the receive callable would be used to obtain the request body. Send: An asynchronous callable that enables the application to send events or messages back to the server, ultimately relaying information to the client. For example, in an HTTP response, the send callable would be used to send back the response headers and body. The specifics of how receive and send are defined is, I’m sure, pretty complicated, but you don’t actually need to understand them to use ASGI. If you want to build a simple ASGI application, you simply invoke receive to get the content of the request, and invoke send to send a response. Here’s an example of me implementing an ASGI application that responds to any request with a Hello from ASGI! response. Full code for this example can be found here. async def app(scope, receive, send): """simple ASGI app that accepts an HTTP request, then sends a response """ assert scope["type"] == "http" # Wait for HTTP request event request = await receive() print("Received request:", request) # Send HTTP response start await send({ "type": "http.response.start", "status": 200, "headers": [ [b"content-type", b"text/plain"] ] }) # Send HTTP response body await send({ "type": "http.response.body", "body": b"Hello from ASGI!", }) Then, you can use an ASGI web server to run the application. Uvicorn is the most popular one. In this particular example I’m using the uv package manager (the name similarity between uvicorn and uv is a coincidence). So, we can run this server via the following command: uv uvicorn asgi_app:app --port 8000 where the script name is asgi_app.py, the function we want to call within that script is app, and we want to run the ASGI server on port 8000. We can then define a simple client application that sends a request to our ASGI-enabled server. import httpx import asyncio async def main(): async with httpx.AsyncClient() as client: response = await client.get("http://127.0.0.1:8000") print("Response status code:", response.status_code) print("Response text:", response.text) if __name__ == "__main__": asyncio.run(main()) here 127.0.0.1 is local host, and we defined our ASGI server on port 8000. If we go ahead and run this client via uv run asgi_client.py we get the following response: Response status code: 200 Response text: Hello from ASGI! This particular example is simple and thus doesn’t communicate the power of ASGI. The whole reason ASGI is to allow for parallel requests. We can make a simple modification to this application to simulate some load. Full code for this example can be found here. We’ll modify the server to wait for a specified time based on a query string on the request. The content of the query string can be extracted via the scope. import asyncio import time async def app(scope, receive, send): assert scope["type"] == "http" # Extract query param for duration path = scope["path"] query = scope.get("query_string", b"").decode() duration = float(query.split("=")[-1]) if "duration=" in query else 1.0 start = time.time() await asyncio.sleep(duration) # Simulate async work end = time.time() message = f"Waited {duration:.1f} seconds. Elapsed: {end - start:.2f}s" await send({ "type": "http.response.start", "status": 200, "headers": [[b"content-type", b"text/plain"]], }) await send({ "type": "http.response.body", "body": message.encode(), }) Then, we can create a new client that submits five requests in rapid succession, each of which has a delay specified of duration seconds (in this case, two seconds). import httpx import asyncio import time async def fetch(session_id, duration=2.0): url = f"http://127.0.0.1:8000/?duration={duration}" async with httpx.AsyncClient() as client: response = await client.get(url) print(f"[Session {session_id}] {response.text}") async def main(): start = time.time() tasks = [fetch(i, duration=2.0) for i in range(5)] await asyncio.gather(*tasks) end = time.time() print(f"Total elapsed time: {end - start:.2f}s") if __name__ == "__main__": asyncio.run(main()) So, we’re sending five requests, each of which takes two seconds long. If this were happening synchronously, we would expect to be done in 10 seconds. However, because requests are being executed in parallel, it all gets done in around 2 seconds. Here’s the output from the sample client: [Session 0] Waited 2.0 seconds. Elapsed: 2.00s [Session 1] Waited 2.0 seconds. Elapsed: 2.00s [Session 2] Waited 2.0 seconds. Elapsed: 2.00s [Session 3] Waited 2.0 seconds. Elapsed: 2.00s [Session 4] Waited 2.0 seconds. Elapsed: 2.00s Total elapsed time: 2.12s We don’t have to go too far in the weeds of ASGI. I wanted to cover it briefly, though, because A2A uses an ASGI-enabled framework called Starlette. Fundamental Framework: Starlette Starlette is a framework built around ASGI. It obeys the core ideas of ASGI, but with some quality of life stuff built on top of it. Given a simple ASGI application import asyncio async def app(scope, receive, send): assert scope['type'] == 'http' body = b"Hello from raw ASGI!" await send({ 'type': 'http.response.start', 'status': 200, 'headers': [ (b'content-type', b'text/plain'), (b'content-length', str(len(body)).encode()) ] }) await send({ 'type': 'http.response.body', 'body': body, }) We can build a similar thing with Starlette from starlette.applications import Starlette from starlette.responses import PlainTextResponse from starlette.routing import Route async def homepage(request): return PlainTextResponse("Hello from Starlette!") app = Starlette(routes=[ Route("/", homepage), ]) It’s basically the same thing, but Starlette has some quality of life abstractions. You can launch either application uvicorn raw_asgi:app uvicorn starlette_app:app Then run a simple client import httpx def call_server(): response = httpx.get("http://localhost:8000") print("Response:", response.text) if __name__ == "__main__": call_server() with uv run client.py which will talk to either server. You’ll get the response Hello from Starlette! or Hello from raw ASGI!, depending on which server you ran. Of course, there’s some other stuff running under the hood, but A2A in Python is essentially an abstraction on top of Starlette. You don’t need to know the ins and outs of Starlette to use A2A, but it’s useful to have a high-level understanding. We can get started with A2A by simply mimicking this simple hello world Starlette app. Getting Started with A2A In being an early-stage protocol without a convenient abstracted framework, A2A implementation can be a bit cumbersome. There are a lot of imports that aren’t super well documented. Most of the A2A code I’ll be showing you is derived from examples defined in this repo of examples. The full code for this example can be found here. Alright, here’s an A2A server, which mimics the “hello world” examples from the previous few examples. from a2a.server.agent_execution import AgentExecutor, RequestContext from a2a.server.apps import A2AStarletteApplication from a2a.server.request_handlers import DefaultRequestHandler from a2a.server.tasks import InMemoryTaskStore from a2a.server.events import EventQueue from a2a.utils import new_agent_text_message from a2a.types import AgentCard, AgentSkill, AgentCapabilities import uvicorn class HelloExecutor(AgentExecutor): async def execute(self, context: RequestContext, event_queue: EventQueue) -> None: # Respond with a static hello message event_queue.enqueue_event(new_agent_text_message("Hello from A2A!")) async def cancel(self, context: RequestContext, event_queue: EventQueue) -> None: pass # No-op def create_app(): skill = AgentSkill( id="hello", name="Hello", description="Say hello to the world.", tags=["hello", "greet"], examples=["hello", "hi"] ) agent_card = AgentCard( name="HelloWorldAgent", description="A simple A2A agent that says hello.", version="0.1.0", url="http://localhost:9000", skills=[skill], capabilities=AgentCapabilities(), authenticationSchemes=["public"], defaultInputModes=["text"], defaultOutputModes=["text"], ) handler = DefaultRequestHandler( agent_executor=HelloExecutor(), task_store=InMemoryTaskStore() ) app = A2AStarletteApplication(agent_card=agent_card, http_handler=handler) return app.build() if __name__ == "__main__": uvicorn.run(create_app(), host="127.0.0.1", port=9000) From a high level, there are some familiar suspects. In the final __main__ expression we’re running our A2A app with uvicorn.run. The app itself is an A2AStarletteApplication which, I guess, is a class that abstracts implementing the A2A protocol within a starlette application. Recall that the core of A2A exists in two key concepts: Agent Cards: JSON objects defining what the agent can do, which are used to communicate the capabilities of the agent to other agents. Tasks: An agent can be given a task, which it can then execute (asynchronously, because A2A is built on ASGI in Python). Within this application, the agent card is defined with the following: skill = AgentSkill( id="hello", name="Hello", description="Say hello to the world.", tags=["hello", "greet"], examples=["hello", "hi"] ) agent_card = AgentCard( name="HelloWorldAgent", description="A simple A2A agent that says hello.", version="0.1.0", url="http://localhost:9000", skills=[skill], capabilities=AgentCapabilities(), authenticationSchemes=["public"], defaultInputModes=["text"], defaultOutputModes=["text"], ) The agent card is chiefly a description of the agent, and a description of the tasks which the agent can perform. Based on the A2A documentation, the agent card consists of: details the agent’s identity (name, description), service endpoint URL, version, supported A2A capabilities (like streaming or push notifications), specific skills it offers, default input/output modalities, and authentication requirements. — Source On top of the agent card, we also need to build the logic that defines the agent itself. Normally, this is an LLM agent, but because we’re mimicking our simple startlette app from before, our “agent” will simply return a hello. class HelloExecutor(AgentExecutor): async def execute(self, context: RequestContext, event_queue: EventQueue) -> None: # Respond with a static hello message event_queue.enqueue_event(new_agent_text_message("Hello from A2A!")) async def cancel(self, context: RequestContext, event_queue: EventQueue) -> None: pass # No-op normally the execute function would be more complicated and include some fancy AI stuff. We can talk to this agent with an A2A-compatible client. import asyncio from uuid import uuid4 import httpx from a2a.client import A2ACardResolver, A2AClient from a2a.types import MessageSendParams, SendMessageRequest async def main(): base_url = "http://localhost:9000" async with httpx.AsyncClient() as httpx_client: # Resolve the agent card resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url) agent_card = await resolver.get_agent_card() # Create client from card client = A2AClient(httpx_client=httpx_client, agent_card=agent_card) # Build request payload message = { "role": "user", "parts": [{"kind": "text", "text": "hello"}], "messageId": uuid4().hex, } request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams(message=message), ) # Send message and print response response = await client.send_message(request) print("Response:", response.model_dump(mode='json', exclude_none=True)) if __name__ == "__main__": asyncio.run(main()) There’s some weirdness I still don’t completely understand. You might notice the agent_card has a url, but you also pass a base_url to the A2ACardResolver. I’ve noticed, in my experimentation, that there are issues when the agent card isn’t configured correctly, even when a base url is specified correctly. I think the base_url can be used to discover agent cards which point to the actual agent. I’ll cover some of the more subtle workflows like that in future articles. Anywho, we get the agent_card and then connect to the agent with A2AClient(httpx_client=httpx_client, agent_card=agent_card). We can then send messages to that agent, and get responses. A2A messages consist of the following: Role: either a “user” or “agent”. Message Id: A unique id for each message. Part: a component of the message. Parts contain the core components of the message. A2A is designed to be multimodal, so parts can consist of “Text”, but can also consist of “File” or “Data”, allowing A2A to communicate various types of data. We can turn the message into a request, get a response, and bada bing bada boom, the world's most over-complicated hello world application. if we run the server with uv run a2a_server.py and the client with uv run a2a_client.py then we get the following response from the client Response: {'id': '48d2ef86-d9f5-4382-9f53-2c926dbf1f64', 'jsonrpc': '2.0', 'result': {'kind': 'message', 'messageId': 'c802cb80-2cf8-497c-a9ad-267da60eb1aa', 'parts': [{'kind': 'text', 'text': 'Hello from A2A!'}], 'role': 'agent'}} And, in essence, that’s A2A. It really has nothing to do with LLM agents intrinsically, but is rather a standardized communication protocol designed to assist in building interoperable LLM agents. But this wouldn’t be an IAEE article if we didn’t dig into it. Let’s build an LLM-powered application with A2A to see how it ticks. The Plan The whole idea of A2A is to allow for the creation of arbitrary interoperable agents. Like countless programmers before me, I thought vacation booking was a decent test bed for this type of functionality. Imagine you’re a company like Trivago or Kayak, and your job consists of booking things like flights and car rentals. In the future, one might be able to imagine an AI agent that’s defined by a car rental company, and another agent that’s maintained by an airline. Trivago can’t (and shouldn’t) be able to modify the database in these booking companies directly, but perhaps Trivago could have an agent that can talk to the car rental and airline agents. Thus these agents could work together to help a customer book their vacation without exposing their information to each other. This application will mature as we explore it practically. When setting off, I imagined something like this: The idea. A travel agent has it’s own database, and can talk to an airline and car agent, which have their own database. These can all work with each other to coordinate tasks for booking travel. I also decided to have the Airline and Car agent to be able to talk to one another under the assumption it might be useful (for instance, if a question required the airline agent to book a flight when a car was available). In reality the end result didn’t turn out exactly this way, but this is the idea. A good first step to making this work would probably be to figure out how to get an MCP server to connect to an A2A server, so we can provide tools to each agent. Launching MCP from A2A Full code for this example can be found here. First, we can define a simple MCP server for our agent to use. from mcp.server.fastmcp import FastMCP mcp = FastMCP("server") @mcp.tool() def say_hello(name: str) -> str: return f"Hello, {name}! This is MCP." This MCP server exposes a single tool called say_hello , which simply returns a hello string from the MCP server. We can then implement an A2A server, which functions as an MCP client. from a2a.server.agent_execution import AgentExecutor, RequestContext from a2a.server.apps import A2AStarletteApplication from a2a.server.request_handlers import DefaultRequestHandler from a2a.server.tasks import InMemoryTaskStore from a2a.server.events import EventQueue from a2a.utils import new_agent_text_message from a2a.types import AgentCard, AgentSkill, AgentCapabilities import uvicorn from mcp import ClientSession, StdioServerParameters, types from mcp.client.stdio import stdio_client from contextlib import AsyncExitStack class MCPClient: def __init__(self): # Initialize session and client objects self.session = None self.exit_stack = AsyncExitStack() async def connect_to_server(self): """Connect to an MCP server Args: server_script_path: Path to the server script (.py or .js) """ server_params = StdioServerParameters( command="mcp", args=["run", "mcp_server.py"], ) stdio_transport = await self.exit_stack.enter_async_context(stdio_client(server_params)) self.stdio, self.write = stdio_transport self.session = await self.exit_stack.enter_async_context(ClientSession(self.stdio, self.write)) await self.session.initialize() # List available tools response = await self.session.list_tools() tools = response.tools print("\nConnected to server with tools:", [tool.name for tool in tools]) async def call_tool(self, tool_name, tool_args): response = await self.session.call_tool(tool_name, tool_args) return response async def cleanup(self): """Clean up resources""" await self.exit_stack.aclose() class HelloExecutor(AgentExecutor): def __init__(self, mcp_client): super().__init__() self.mcp_client = mcp_client async def execute(self, context: RequestContext, event_queue: EventQueue) -> None: #talking to MCP server mcp_response = await self.mcp_client.call_tool('say_hello', {'name': 'a2a Server'}) mcp_response = mcp_response.content[0].text response = f'Hello from A2A server! This server functions as an MCP client.\nThe MCP server told me:\n\n{mcp_response}' await event_queue.enqueue_event(new_agent_text_message(response)) async def cancel(self, context: RequestContext, event_queue: EventQueue) -> None: pass # No-op def create_app(): skill = AgentSkill( id="hello", name="Hello", description="Say hello to the world.", tags=["hello", "greet"], examples=["hello", "hi"] ) agent_card = AgentCard( name="HelloWorldAgent", description="A simple A2A agent that says hello.", version="0.1.0", url="http://localhost:9000", skills=[skill], capabilities=AgentCapabilities(), authenticationSchemes=["public"], defaultInputModes=["text"], defaultOutputModes=["text"], ) # Placeholder executor to be set during startup handler = DefaultRequestHandler( agent_executor=None, task_store=InMemoryTaskStore() ) app = A2AStarletteApplication(agent_card=agent_card, http_handler=handler) starlette_app = app.build() mcp_client = MCPClient() @starlette_app.on_event("startup") async def _startup(): print('startup procedure!') await mcp_client.connect_to_server() handler.agent_executor = HelloExecutor(mcp_client) print('startup completed!') @starlette_app.on_event("shutdown") async def _shutdown(): print('shutdown: cleaning up MCP client...') await mcp_client.cleanup() return starlette_app if __name__ == "__main__": uvicorn.run(create_app(), host="127.0.0.1", port=9000) This is pretty much the same as the previous A2A server we defined, but with a few important modifications. In the create_app function, we convert the A2AStarletteApplication into a Starlette app with app.build(). That allows us to do some starlette specific stuff, like defining functions that run when the starlette app starts and ends. These two functions allow us to launch the MCP server on startup, and tear it down when we close the A2A server. @starlette_app.on_event("startup") async def _startup(): print('startup procedure!') await mcp_client.connect_to_server() handler.agent_executor = HelloExecutor(mcp_client) print('startup completed!') @starlette_app.on_event("shutdown") async def _shutdown(): print('shutdown: cleaning up MCP client...') await mcp_client.cleanup() We connect to the MCP server with the following: async def connect_to_server(self): """Connect to an MCP server Args: server_script_path: Path to the server script (.py or .js) """ server_params = StdioServerParameters( command="mcp", args=["run", "mcp_server.py"], ) stdio_transport = await self.exit_stack.enter_async_context(stdio_client(server_params)) self.stdio, self.write = stdio_transport self.session = await self.exit_stack.enter_async_context(ClientSession(self.stdio, self.write)) await self.session.initialize() # List available tools response = await self.session.list_tools() tools = response.tools print("\nConnected to server with tools:", [tool.name for tool in tools]) I won’t get too in the weeds of how launching an MCP server works, if you want some more information on that, I have an article that covers the subject: Our agent executor takes the query and passes it over to the say_hello tool defined in the MCP server. class HelloExecutor(AgentExecutor): def __init__(self, mcp_client): super().__init__() self.mcp_client = mcp_client async def execute(self, context: RequestContext, event_queue: EventQueue) -> None: #talking to MCP server mcp_response = await self.mcp_client.call_tool('say_hello', {'name': 'a2a Server'}) mcp_response = mcp_response.content[0].text response = f'Hello from A2A server! This server functions as an MCP client.\nThe MCP server told me:\n\n{mcp_response}' await event_queue.enqueue_event(new_agent_text_message(response)) async def cancel(self, context: RequestContext, event_queue: EventQueue) -> None: pass # No-op Thus, when we query our A2A server to say hello, it will tell our MCP server to say hello and return the response to the user. If we whip up a client and ask our A2A server to say hello import asyncio from uuid import uuid4 import httpx from a2a.client import A2ACardResolver, A2AClient from a2a.types import MessageSendParams, SendMessageRequest import json async def main(): base_url = "http://localhost:9000" async with httpx.AsyncClient() as httpx_client: # Resolve the agent card resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url) agent_card = await resolver.get_agent_card() # Create client from card client = A2AClient(httpx_client=httpx_client, agent_card=agent_card) # Build request payload message = { "role": "user", "parts": [{"kind": "text", "text": "hello"}], "messageId": uuid4().hex, } request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams(message=message), ) # Send message and print response response = await client.send_message(request) print("Response:\n", json.dumps(response.model_dump(mode='json', exclude_none=True),indent=2)) if __name__ == "__main__": asyncio.run(main()) we get the following response { "id": "97aa4793-36fb-4bde-8bff-959bcbaa2894", "jsonrpc": "2.0", "result": { "kind": "message", "messageId": "950e2b38-0ce0-43e1-ace1-a76323fe2bdb", "parts": [ { "kind": "text", "text": "Hello from A2A server! This server functions as an MCP client.\nThe MCP server told me:\n\nHello, a2a Server! This is MCP." } ], "role": "agent" } } As you can see, our A2A server said hello, and also forwarded the MCP server's response, which also said hello. Alright, things are looking up. Let’s get an AI model in the mix. Defining an AI Agent In The A2A Server In the previous section, we defined an A2A server that talks to and communicates with an MCP server. We’re going to keep this same structure, but instead stick an actual LLM in the A2A server, which is the whole point of A2A to begin with. We’ll use Anthropic for this application, but you could swap it out for OpenAI or Google or whatever. Most major LLMs support MCP. Full code for this example can be found here. import os import uvicorn import anthropic import httpx from contextlib import AsyncExitStack from a2a.server.agent_execution import AgentExecutor, RequestContext from a2a.server.apps import A2AStarletteApplication from a2a.server.request_handlers import DefaultRequestHandler from a2a.server.tasks import InMemoryTaskStore from a2a.server.events import EventQueue from a2a.utils import new_agent_text_message from a2a.types import AgentCard, AgentSkill, AgentCapabilities from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client from dotenv import load_dotenv load_dotenv() ANTHROPIC_API_KEY = os.getenv("ANTHROPIC_API_KEY") MODEL_NAME = "claude-3-opus-20240229" class MCPClient: def __init__(self): self.session = None self.exit_stack = AsyncExitStack() async def connect_to_server(self): server_params = StdioServerParameters( command="mcp", args=["run", "mcp_server.py"], ) stdio_transport = await self.exit_stack.enter_async_context(stdio_client(server_params)) self.stdio, self.write = stdio_transport self.session = await self.exit_stack.enter_async_context(ClientSession(self.stdio, self.write)) await self.session.initialize() async def cleanup(self): await self.exit_stack.aclose() class ClaudeSessionManager: """Manages a full Claude session with optional MCP tool usage.""" def __init__(self, mcp_session: ClientSession, anthropic_client: anthropic.Anthropic, model: str = MODEL_NAME): self.session = mcp_session self.anthropic = anthropic_client self.model = model self.verbose = True async def run_session(self, user_input: str) -> str: tools_response = await self.session.list_tools() tools = [{ "name": tool.name, "description": tool.description, "input_schema": tool.inputSchema } for tool in tools_response.tools] if self.verbose: print('running Claude session') messages = [{"role": "user", "content": [{"type": "text", "text": user_input}]}] final_output = [] if self.verbose: print('initializing with message to Claude:') if self.verbose: print(messages) response = self.anthropic.messages.create( model=self.model, max_tokens=1000, tools=tools, messages=messages, ) if self.verbose: print('response received from Claude') if self.verbose: print(response) while True: assistant_content = [] for part in response.content: if part.type == "text": final_output.append(part.text) assistant_content.append(part) elif part.type == "tool_use": try: tool_result = await self.session.call_tool(part.name, part.input) messages.append({"role": "assistant", "content": assistant_content}) messages.append({ "role": "user", "content": [{ "type": "tool_result", "tool_use_id": part.id, "content": tool_result.content }] }) response = self.anthropic.messages.create( model=self.model, max_tokens=1000, tools=tools, messages=messages ) break except Exception as e: return f"❌ Tool call failed: {e}" else: break return "\n".join(final_output) class ClaudeAgentExecutor(AgentExecutor): """Handles A2A requests by invoking Claude for tool reasoning.""" def __init__(self, session_manager: ClaudeSessionManager): super().__init__() self.session_manager = session_manager async def execute(self, context: RequestContext, event_queue: EventQueue): try: #unpacking the message request from the a2a client request = context.message.parts[0].root.text print(f'request from A2A Client: {request}') response = await self.session_manager.run_session(request) except Exception as e: response = f"❌ Error: {e}" await event_queue.enqueue_event(new_agent_text_message(response)) async def cancel(self, context: RequestContext, event_queue: EventQueue): pass def create_app(): skill = AgentSkill( id="claude_tool_agent", name="Tool Agent", description="An agent powered by Claude with access to MCP tools.", tags=["llm", "tools", "claude"], examples=["Say hello", "What tools can you use?", "Get me a summary"] ) agent_card = AgentCard( name="ClaudeToolAgent", description="An A2A agent that uses Anthropic Claude 3 with tool calling via MCP.", version="0.1.0", url="http://localhost:9000", skills=[skill], capabilities=AgentCapabilities(), authenticationSchemes=["public"], defaultInputModes=["text"], defaultOutputModes=["text"], ) handler = DefaultRequestHandler(agent_executor=None, task_store=InMemoryTaskStore()) app = A2AStarletteApplication(agent_card=agent_card, http_handler=handler) starlette_app = app.build() mcp_client = MCPClient() anthropic_client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY) @starlette_app.on_event("startup") async def _startup(): await mcp_client.connect_to_server() session_manager = ClaudeSessionManager(mcp_client.session, anthropic_client) handler.agent_executor = ClaudeAgentExecutor(session_manager) print("✅ A2A server connected to MCP and Claude 3.") @starlette_app.on_event("shutdown") async def _shutdown(): await mcp_client.cleanup() print("🛑 Cleaned up MCP session.") return starlette_app if __name__ == "__main__": uvicorn.run(create_app(), host="127.0.0.1", port=9000) It’s a lot of code, chiefly because of boilerplate. The main difference is that the agent executor (now called ClaudeAgentExecutor, thanks to LLMs having a fun time renaming random stuff in my codebase) interfaces with a new class called ClaudeSessionManager. This is the bit that talks to the LLM. The run_session function in the ClaudeSessionManager unpacks the tools and passes them to the LLM in the initial prompt, allowing the model to know which tools it has access to. The user's prompt is passed into the initial pass to the LLM, and then we enter an evaluation loop. while True: assistant_content = [] for part in response.content: if part.type == "text": final_output.append(part.text) assistant_content.append(part) elif part.type == "tool_use": try: tool_result = await self.session.call_tool(part.name, part.input) messages.append({"role": "assistant", "content": assistant_content}) messages.append({ "role": "user", "content": [{ "type": "tool_result", "tool_use_id": part.id, "content": tool_result.content }] }) response = self.anthropic.messages.create( model=self.model, max_tokens=1000, tools=tools, messages=messages ) break except Exception as e: return f"❌ Tool call failed: {e}" else: break This is the bulk of the agent. If the model responds that a tool should be used, we call that tool and add the response to the chat continuum, then prompt the model again. We keep doing that until the model doesn’t think more tool use is necessary. Full disclosure, most of this is AI-generated, so there might be some janky stuff going on. I have some more scratch-made agentic implementation ideas in a few other articles. Agentic design isn’t really the point of this article, though, so let’s move on. In this particular example we didn’t modify our mcp_server.py at all from mcp.server.fastmcp import FastMCP mcp = FastMCP("server") @mcp.tool() def say_hello(name: str) -> str: return f"Hello, {name}! This is MCP." But we did modify the sample client. Here, we’re asking the agent to tell us which tools it has access to. import asyncio from uuid import uuid4 import httpx from a2a.client import A2ACardResolver, A2AClient from a2a.types import MessageSendParams, SendMessageRequest import json async def test_a2a_server(): base_url = "http://localhost:9000" async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as httpx_client: # Resolve the agent card resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url) agent_card = await resolver.get_agent_card() # Create A2A client from the card client = A2AClient(httpx_client=httpx_client, agent_card=agent_card) # Define the user message text_input = "What tools can you use?" # Build the A2A-compliant message payload message = { "role": "user", "parts": [ {"kind": "text", "text": text_input} ], "messageId": uuid4().hex } request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams(message=message), ) # Send message to A2A agent response = await client.send_message(request) # Print out formatted response print("Response:\n", json.dumps(response.model_dump(mode='json', exclude_none=True), indent=2)) if __name__ == "__main__": asyncio.run(test_a2a_server()) Here, the a2a server is called travel_agent.py (I thought I’d get further along in this particular example), and the client is called a2a_test_client.py. if we run uv run travel_agent.py then uv run a2a_test_cleint.py Our A2A test client gets the following output from Claude, running in the A2A server: To determine what tools I can use, I’ll look at the JSONSchema provided that lists the available tools and their parameters. It looks like there is just a single tool available called “say_hello”. This tool takes a single required parameter called “name” which is a string. Since the user just asked what tools are available and did not provide a name to say hello to, I do not have enough information to actually call the say_hello tool at this time. I’ll just inform the user about the one tool that is available. Based on the information provided, there is one tool I can use called “say_hello”. This tool takes a single required parameter “name” which should be a string value. When given a name, this tool presumably would output a hello greeting to the specified name. For example, if called with the name “Alice”, it may return something like “Hello Alice!”. Let me know if you would like me to use this tool to say hello to a specific name! So, the tool we have access to doesn’t do much, but our agent can clearly see and converse about our tool use! Let’s do one more intermediate experiment, building a simple A2A server with MCP, but this time let’s make the MCP server actually do something. Defining an A2A agent, with a Database Interfacing MCP Server Alright, I’ve copied virtually the same code with minor modifications a few times now. For the sake of saving a few electrons, let’s just focus on the MCP server. Full code for this example can be found here. First, I created a file called define_db.py, which defines an SQLite database. import os import sqlite3 import random from datetime import datetime, timedelta DB_NAME = "flights.db" # Remove existing database file if it exists if os.path.exists(DB_NAME): os.remove(DB_NAME) print(f"Removed existing database '{DB_NAME}'.") # Connect to the new SQLite database conn = sqlite3.connect(DB_NAME) cursor = conn.cursor() # Create the flights table cursor.execute(""" CREATE TABLE flights ( id INTEGER PRIMARY KEY AUTOINCREMENT, flight_number TEXT NOT NULL, departure_airport TEXT NOT NULL, arrival_airport TEXT NOT NULL, departure_time TEXT NOT NULL, arrival_time TEXT NOT NULL, available_seats INTEGER NOT NULL, status TEXT CHECK(status IN ('scheduled', 'cancelled', 'completed')) NOT NULL DEFAULT 'scheduled' ) """) # Sample airport codes airport_codes = ["JFK", "LAX", "ORD", "DFW", "DEN", "SFO", "ATL", "SEA", "MIA", "BOS", "LHR", "CDG", "FRA", "DXB", "HND"] # Generate random flights def generate_random_flight(index: int): dep_airport, arr_airport = random.sample(airport_codes, 2) base_time = datetime(2025, 6, 15) departure_time = base_time + timedelta(hours=random.randint(0, 300)) flight_duration = timedelta(hours=random.randint(1, 15), minutes=random.choice([0, 15, 30, 45])) arrival_time = departure_time + flight_duration flight_number = f"{random.choice(['AA', 'DL', 'UA', 'BA', 'LH', 'AF', 'EK'])}{100 + index}" available_seats = random.randint(0, 200) status = random.choices(["scheduled", "cancelled", "completed"], weights=[0.7, 0.1, 0.2])[0] return ( flight_number, dep_airport, arr_airport, departure_time.strftime("%Y-%m-%d %H:%M"), arrival_time.strftime("%Y-%m-%d %H:%M"), available_seats, status ) # Generate and insert many flights num_flights = 500 flights_data = [generate_random_flight(i) for i in range(num_flights)] cursor.executemany(""" INSERT INTO flights (flight_number, departure_airport, arrival_airport, departure_time, arrival_time, available_seats, status) VALUES (?, ?, ?, ?, ?, ?, ?) """, flights_data) conn.commit() conn.close() print(f"Database '{DB_NAME}' created with {num_flights} synthetic flight records.") If you’re not familiar with SQL and SQLite, I have an article on the subject. If you don’t feel like learning about it, don’t sweat it. All that’s important to know is that we now have a file called flights.db that has flight information. Our mcp_server.py will reference this database and expose access to it in the form of tools. import html import aiosqlite from mcp.server.fastmcp import FastMCP from typing import Annotated mcp = FastMCP("sqlite-fastmcp-server") DB_PATH = "flights.db" # change this or pass via env/arg if needed @mcp.tool() async def sqlite_get_catalog() -> str: """List tables and columns in the SQLite database.""" async with aiosqlite.connect(f"file:{DB_PATH}?mode=ro", uri=True) as conn: cursor = await conn.execute("SELECT name FROM sqlite_master WHERE type='table'") tables = [row[0] for row in await cursor.fetchall()] catalog = {} for table in tables: col_cursor = await conn.execute(f"PRAGMA table_info({table})") columns = [col[1] for col in await col_cursor.fetchall()] catalog[table] = columns return str(catalog) @mcp.tool() async def sqlite_execute(sql: Annotated[str, "SQL SELECT statement"]) -> str: """Execute a SQL query and return HTML table of results.""" async with aiosqlite.connect(f"file:{DB_PATH}?mode=ro", uri=True) as conn: cursor = await conn.execute(sql) if not cursor.description: return "Statement executed successfully." headers = "".join(f"{html.escape(col[0])}" for col in cursor.description) rows_html = f"{headers}" for row in await cursor.fetchall(): row_cells = "".join(f"{html.escape(str(val))}" for val in row) rows_html += f"{row_cells}" return f"{rows_html}" if __name__ == "__main__": mcp.run() In reality, if you were building an actual MCP server to be used in an actual production agent, I imagine you wouldn’t want the agent defining it’s own SQL queries and arbitrarily interacting with the database. For our purposes, though, I think this is fine for now. At this point, I started playing around with naming conventions as I slowly converged on our end goal. Here, the A2A server is called airline_agent.py and the A2A client is called travel_agent.py. The travel_agent.py isn’t anything special though, it’s just a simple client that asks the A2A server the following question: Can you tell me the shortest expected flight duration from each airport? Tell me the response of the query in a markdown table. if we run uv run airline_agent.py then run uv run travel_agent.py we get the following output: To answer this request, I need to: 1. Get a list of all the airports in the database by querying the airports table. This will give me the airport codes to use in the next step. 2. For each airport code, find the flight with the minimum expected duration by: a) Joining the flights and airports tables to get flights.origin_airport = airports.code b) Grouping by origin airport code c) Taking the MIN(scheduled_duration) 3. Format the results into a markdown table with columns for airport code and minimum expected duration. I have all the information needed to execute these steps with the given tools. It looks like the `airports` table doesn’t exist in this database. Let me check what tables are available. The database only contains a `flights` table, not an `airports` table. Let me try this again using just the `flights` table: Here is the minimum expected flight duration for each departure airport, formatted as a markdown table: | airport | min_expected_duration | | — — — — -| — — — — — — — — — — — — | | ATL | 0 | | BOS | 0 | | CDG | 0 | | DEN | 0 | | DFW | 0 | | DXB | 0 | | FRA | 0 | | HND | 0 | | JFK | 0 | | LAX | 0 | | LHR | 0 | | MIA | 0 | | ORD | 0 | | SEA | 0 | | SFO | 0 | Let me know if you need anything else! As you can see, the agent tried playing around with a few different tools and SQL queries before settling on an answer. I think the answer is wrong, but whatever; we’re moving on. Implementing The Travel Agent In previous examples, we: Implemented A2A Servers that can run mcp tools Implemented simple LLM agents in those servers Implemented tools in the MCP server that allow the agent to query the database Now we’re going to put it all together to build our travel booking demo. Before we do, though, let’s discuss a quirk of how A2A and MCP are conceptualized vs how they’re implemented. Recall that, in the beginning of the article, we discussed a diagram of how A2A and MCP interoperate with one another. Recall the original plan, where agents talk to each other with A2A directly. This is a fine conceptual breakdown, and I’m sure you could build something that respects this flow. However, as I was combing through tutorials, I found another structure that felt much more intuitive. An approach that I found to be easier to implement, where A2A communication is exposed to the agent via a tool in the MCP server. When integrating MCP with an agent, we can expose arbitrary tools. Needing to manage both tools for talking with different agents and tools for communicating with an MCP server feels overly complex. If we implement our A2A communication as an MCP tool, we only have to integrate MCP into each of our agents. For this example, I implemented a travel agent, a car agent, and an airline agent. The MCP server for the travel agent looks like the following. (Full code can be found here): import html import aiosqlite from mcp.server.fastmcp import FastMCP from typing import Annotated import httpx from a2a.client import A2ACardResolver from a2a.client import A2ACardResolver, A2AClient from a2a.types import MessageSendParams, SendMessageRequest from uuid import uuid4 import json mcp = FastMCP("sqlite-fastmcp-server") DB_PATH = "travel_agent.db" #defining info to communicate with the airline agent with open('port_config.json', 'r') as file: airline_port = json.load(file)['airline_agent_port'] with open('port_config.json', 'r') as file: car_port = json.load(file)['car_agent_port'] host = "127.0.0.1" AIRLINE_AGENT_URL = f'http://{host}:{airline_port}' CAR_AGENT_URL = f'http://{host}:{car_port}' @mcp.tool() async def sqlite_get_catalog() -> str: """List tables and columns in the SQLite database.""" async with aiosqlite.connect(f"file:{DB_PATH}?mode=ro", uri=True) as conn: cursor = await conn.execute("SELECT name FROM sqlite_master WHERE type='table'") tables = [row[0] for row in await cursor.fetchall()] catalog = {} for table in tables: col_cursor = await conn.execute(f"PRAGMA table_info({table})") columns = [col[1] for col in await col_cursor.fetchall()] catalog[table] = columns return str(catalog) @mcp.tool() async def sqlite_select(sql: Annotated[str, "SQL SELECT statement"]) -> str: """Execute a SQL query and return HTML table of results.""" async with aiosqlite.connect(f"file:{DB_PATH}?mode=ro", uri=True) as conn: cursor = await conn.execute(sql) if not cursor.description: return "Statement executed successfully." headers = "".join(f"{html.escape(col[0])}" for col in cursor.description) rows_html = f"{headers}" for row in await cursor.fetchall(): row_cells = "".join(f"{html.escape(str(val))}" for val in row) rows_html += f"{row_cells}" return f"{rows_html}" @mcp.tool() async def sqlite_modify(sql: Annotated[str, "SQL INSERT/UPDATE/DELETE/DDL statement"]) -> str: """Execute a SQL statement that modifies the database and return a success message.""" try: async with aiosqlite.connect(f"file:{DB_PATH}?mode=rwc", uri=True) as conn: await conn.execute(sql) await conn.commit() return "Statement executed successfully." except Exception as e: return f"Error: {html.escape(str(e))}" @mcp.tool() async def get_agent_cards() -> str: """Get cards of connected agents""" agent_cards = {} async with httpx.AsyncClient() as httpx_client: resolver = A2ACardResolver(httpx_client=httpx_client, base_url=AIRLINE_AGENT_URL) agent_card = await resolver.get_agent_card() agent_cards['airline_agent_card'] = agent_card async with httpx.AsyncClient() as httpx_client: resolver = A2ACardResolver(httpx_client=httpx_client, base_url=CAR_AGENT_URL) agent_card = await resolver.get_agent_card() agent_cards['car_agent_card'] = agent_card return str(agent_cards) @mcp.tool() async def query_airline_agent(query: Annotated[str, "Natural language query to send to the airline agent"]) -> str: """Send a natural language query to the airline agent via A2A protocol and return the response.""" async with httpx.AsyncClient(timeout=None) as httpx_client: # Step 1: Resolve the agent card resolver = A2ACardResolver(httpx_client=httpx_client, base_url=AIRLINE_AGENT_URL) agent_card = await resolver.get_agent_card() # Step 2: Create the A2A client client = A2AClient(httpx_client=httpx_client, agent_card=agent_card) # Step 3: Create the message message = { "role": "user", "parts": [ {"kind": "text", "text": query} ], "messageId": uuid4().hex } # Step 4: Wrap in SendMessageRequest request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams(message=message), ) print('travel mcp querying airline agent...') # Step 5: Send the message and get a response response = await client.send_message(request) return response @mcp.tool() async def query_car_agent(query: Annotated[str, "Natural language query to send to the car agent"]) -> str: """Send a natural language query to the car agent via A2A protocol and return the response.""" async with httpx.AsyncClient(timeout=None) as httpx_client: # Step 1: Resolve the agent card resolver = A2ACardResolver(httpx_client=httpx_client, base_url=CAR_AGENT_URL) agent_card = await resolver.get_agent_card() # Step 2: Create the A2A client client = A2AClient(httpx_client=httpx_client, agent_card=agent_card) # Step 3: Create the message message = { "role": "user", "parts": [ {"kind": "text", "text": query} ], "messageId": uuid4().hex } # Step 4: Wrap in SendMessageRequest request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams(message=message), ) # Step 5: Send the message and get a response response = await client.send_message(request) return response if __name__ == "__main__": mcp.run() I played around with the travel agent having it’s own database (like the database for Kayak or trivago), which the agent can interact with given the following functions @mcp.tool() async def sqlite_get_catalog() -> str: ... @mcp.tool() async def sqlite_select(sql: Annotated[str, "SQL SELECT statement"]) -> str: ... @mcp.tool() async def sqlite_modify(sql: Annotated[str, "SQL INSERT/UPDATE/DELETE/DDL statement"]) -> str: ... But there are also tools that query the airline and car agents @mcp.tool() async def query_airline_agent(query: Annotated[str, "Natural language query to send to the airline agent"]) -> str: ... @mcp.tool() async def query_car_agent(query: Annotated[str, "Natural language query to send to the car agent"]) -> str: ... If we look at the car agent tool, for example: @mcp.tool() async def query_car_agent(query: Annotated[str, "Natural language query to send to the car agent"]) -> str: """Send a natural language query to the car agent via A2A protocol and return the response.""" async with httpx.AsyncClient(timeout=None) as httpx_client: # Step 1: Resolve the agent card resolver = A2ACardResolver(httpx_client=httpx_client, base_url=CAR_AGENT_URL) agent_card = await resolver.get_agent_card() # Step 2: Create the A2A client client = A2AClient(httpx_client=httpx_client, agent_card=agent_card) # Step 3: Create the message message = { "role": "user", "parts": [ {"kind": "text", "text": query} ], "messageId": uuid4().hex } # Step 4: Wrap in SendMessageRequest request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams(message=message), ) # Step 5: Send the message and get a response response = await client.send_message(request) return response We can see that it simply calls the car agent at the CAR_AGENT_URL. The CAR_AGENT_URL is defined, for this example, as localhost , with the port specified in a file called port_config.json , which looks like this: { "travel_agent_port": 9000, "airline_agent_port": 9001, "car_agent_port": 9002 } I have this lying around so I can easily reference the ports from different agents. The code for the actual travel agent looks like this: from a2a_mcp_server_app_creator import create_app import json from a2a.types import AgentCard if __name__ == "__main__": #getting agent card with open('travel_agent_card.json', 'r') as file: agent_card = AgentCard.model_validate(json.load(file)) #defining mcp server mcp_script = 'travel_mcp.py' #getting port with open('port_config.json', 'r') as file: port = json.load(file)['travel_agent_port'] #defining host (local host) host = "127.0.0.1" print('=== creating car agent ===') print(f'port: {port}') print(f'mcp_server: {mcp_script}') print('=== Agent Card ===') print(agent_card) create_app(agent_card, mcp_script, host, port) The travel agent is, itself, an A2A application with it’s own model card. We won’t have any agents talking to our travel agent, but we certainly could! Because it’s an A2A agent, it needs an agent card which is defined in travel_agent_card.json. I kept this pretty minimal because no other agents will be talking with this agent. { "name": "Travel Agent", "description": "Travel agent, that talks to the airline and car agent", "url": "http://localhost:9000/", "provider": null, "version": "1.0.0", "documentationUrl": null, "capabilities": { "streaming": "True", "pushNotifications": "True", "stateTransitionHistory": "False" }, "authentication": { "credentials": null, "schemes": [ "public" ] }, "defaultInputModes": [ "text", "text/plain" ], "defaultOutputModes": [ "text", "text/plain" ], "skills": [ ] } The travel agent leverages the MCP script we previously discussed, meaning it can talk with other agents via A2A. We actually go about creating the agent via the create_app function. I got sick of copying the same code over and over again. Because I needed to create three agents, I made a function create_app that accepts (agent_card, mcp_script, host, port) and spools up an A2A agent, with access to that MCP server, at the specified host and port. If we take a gander at a2a_mcp_server_app_creator.py, which contains the create_app function, we’ll see a lot of familiar code. This is essentially a generalized version of the A2A agents we’ve defined previously. import os import uvicorn import anthropic import httpx from contextlib import AsyncExitStack from a2a.server.agent_execution import AgentExecutor, RequestContext from a2a.server.apps import A2AStarletteApplication from a2a.server.request_handlers import DefaultRequestHandler from a2a.server.tasks import InMemoryTaskStore from a2a.server.events import EventQueue from a2a.utils import new_agent_text_message from a2a.types import AgentCard, AgentSkill, AgentCapabilities from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client from dotenv import load_dotenv load_dotenv() ANTHROPIC_API_KEY = os.getenv("ANTHROPIC_API_KEY") MODEL_NAME = "claude-3-opus-20240229" class MCPClient: def __init__(self, script): self.session = None self.exit_stack = AsyncExitStack() self.script = script async def connect_to_server(self): server_params = StdioServerParameters( command="mcp", args=["run", self.script], ) stdio_transport = await self.exit_stack.enter_async_context(stdio_client(server_params)) self.stdio, self.write = stdio_transport self.session = await self.exit_stack.enter_async_context(ClientSession(self.stdio, self.write)) await self.session.initialize() async def cleanup(self): await self.exit_stack.aclose() class ClaudeSessionManager: """Manages a full Claude session with optional MCP tool usage.""" def __init__(self, mcp_session: ClientSession, anthropic_client: anthropic.Anthropic, model: str = MODEL_NAME): self.session = mcp_session self.anthropic = anthropic_client self.model = model self.verbose = True async def run_session(self, user_input: str) -> str: tools_response = await self.session.list_tools() tools = [{ "name": tool.name, "description": tool.description, "input_schema": tool.inputSchema } for tool in tools_response.tools] if self.verbose: print('running Claude session') messages = [{"role": "user", "content": [{"type": "text", "text": user_input}]}] final_output = [] if self.verbose: print('initializing with message to Claude:') if self.verbose: print(messages) response = self.anthropic.messages.create( model=self.model, max_tokens=1000, tools=tools, system=( "You are a helpful support agent. " "Use the tools provided to answer user questions accurately. " "If a tool is called, make sure to read and interpret the results. " "The user can not see tool output. Your final response should use the tool output to answer the users question." "If a users query requires querying a database, for instance, be sure to provide the actual data in the end response to the user." "The current date is 2025, 6, 15. The time is 5:00pm" ), messages=messages, ) if self.verbose: print('response received from Claude') if self.verbose: print(response) while True: assistant_content = [] for part in response.content: if part.type == "text": print(part.text) final_output.append(part.text) assistant_content.append(part) elif part.type == "tool_use": try: tool_result = await self.session.call_tool(part.name, part.input) # Record the assistant response including the tool_use block messages.append({ "role": "assistant", "content": [p.model_dump() for p in response.content] }) # Add the tool result message from the user messages.append({ "role": "user", "content": [{ "type": "tool_result", "tool_use_id": part.id, "content": tool_result.content }] }) # Re-query Claude with the updated conversation response = self.anthropic.messages.create( model=self.model, max_tokens=1000, tools=tools, messages=messages ) break except Exception as e: return f"❌ Tool call failed: {e}" else: break return "\n".join(final_output) class ClaudeAgentExecutor(AgentExecutor): """Handles A2A requests by invoking Claude for tool reasoning.""" def __init__(self, session_manager: ClaudeSessionManager): super().__init__() self.session_manager = session_manager async def execute(self, context: RequestContext, event_queue: EventQueue): try: #unpacking the message request from the a2a client request = context.message.parts[0].root.text print(f'request from A2A Client: {request}') response = await self.session_manager.run_session(request) except Exception as e: response = f"❌ Error: {e}" await event_queue.enqueue_event(new_agent_text_message(response)) async def cancel(self, context: RequestContext, event_queue: EventQueue): pass def create_app(agent_card, mcp_script, host, port): handler = DefaultRequestHandler(agent_executor=None, task_store=InMemoryTaskStore()) app = A2AStarletteApplication(agent_card=agent_card, http_handler=handler) starlette_app = app.build() anthropic_client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY) mcp_client = MCPClient(mcp_script) @starlette_app.on_event("startup") async def _startup(): await mcp_client.connect_to_server() session_manager = ClaudeSessionManager(mcp_client.session, anthropic_client) handler.agent_executor = ClaudeAgentExecutor(session_manager) print("✅ A2A server connected to MCP and Claude 3.") @starlette_app.on_event("shutdown") async def _shutdown(): await mcp_client.cleanup() print("🛑 Cleaned up MCP session.") uvicorn.run(starlette_app, host=host, port=port) We have logic for spawning an MCP server, handling communication with Claude, defining an agent executor that’s compatible with A2A, and creating a starlette app run with uvicorn at the specified host and port. Because we have this generalized code, it’s actually pretty easy to define our airline and car agents; we just need to define their MCP servers and agent cards. Here’s the MCP script I defined for the car agent import html import aiosqlite from mcp.server.fastmcp import FastMCP from typing import Annotated import httpx from a2a.client import A2ACardResolver from a2a.client import A2ACardResolver, A2AClient from a2a.types import MessageSendParams, SendMessageRequest from uuid import uuid4 import json mcp = FastMCP("sqlite-fastmcp-server") DB_PATH = "airline.db" # change this or pass via env/arg if needed #defining info to communicate with the car agent with open('port_config.json', 'r') as file: port = json.load(file)['car_agent_port'] host = "127.0.0.1" CAR_AGENT_URL = f'http://{host}:{port}' @mcp.tool() async def sqlite_get_catalog() -> str: """List tables and columns in the SQLite database.""" async with aiosqlite.connect(f"file:{DB_PATH}?mode=ro", uri=True) as conn: cursor = await conn.execute("SELECT name FROM sqlite_master WHERE type='table'") tables = [row[0] for row in await cursor.fetchall()] catalog = {} for table in tables: col_cursor = await conn.execute(f"PRAGMA table_info({table})") columns = [col[1] for col in await col_cursor.fetchall()] catalog[table] = columns return str(catalog) @mcp.tool() async def sqlite_select(sql: Annotated[str, "SQL SELECT statement"]) -> str: """Execute a SQL query and return HTML table of results. This queries the airline data, the structure of which is defined by the sqlite_get_catalog tool""" async with aiosqlite.connect(f"file:{DB_PATH}?mode=ro", uri=True) as conn: cursor = await conn.execute(sql) if not cursor.description: return "Statement executed successfully." headers = "".join(f"{html.escape(col[0])}" for col in cursor.description) rows_html = f"{headers}" for row in await cursor.fetchall(): row_cells = "".join(f"{html.escape(str(val))}" for val in row) rows_html += f"{row_cells}" return f"{rows_html}" @mcp.tool() async def sqlite_modify(sql: Annotated[str, "SQL INSERT/UPDATE/DELETE/DDL statement"]) -> str: """Execute a SQL statement that modifies the database and return a success message.""" try: async with aiosqlite.connect(f"file:{DB_PATH}?mode=rwc", uri=True) as conn: await conn.execute(sql) await conn.commit() return "Statement executed successfully." except Exception as e: return f"Error: {html.escape(str(e))}" @mcp.tool() async def get_car_agent_card() -> str: async with httpx.AsyncClient() as httpx_client: resolver = A2ACardResolver(httpx_client=httpx_client, base_url=CAR_AGENT_URL) agent_card = await resolver.get_agent_card() return str(agent_card) @mcp.tool() async def query_car_agent(query: Annotated[str, "Natural language query to send to the car agent"]) -> str: """Send a natural language query to the car agent via A2A protocol and return the response.""" async with httpx.AsyncClient(timeout=None) as httpx_client: # Step 1: Resolve the agent card resolver = A2ACardResolver(httpx_client=httpx_client, base_url=CAR_AGENT_URL) agent_card = await resolver.get_agent_card() # Step 2: Create the A2A client client = A2AClient(httpx_client=httpx_client, agent_card=agent_card) # Step 3: Create the message message = { "role": "user", "parts": [ {"kind": "text", "text": query} ], "messageId": uuid4().hex } # Step 4: Wrap in SendMessageRequest request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams(message=message), ) # Step 5: Send the message and get a response response = await client.send_message(request) return response if __name__ == "__main__": mcp.run() And here’s the MCP script I defined for the airline agent import html import aiosqlite from mcp.server.fastmcp import FastMCP from typing import Annotated import httpx from a2a.client import A2ACardResolver from a2a.client import A2ACardResolver, A2AClient from a2a.types import MessageSendParams, SendMessageRequest from uuid import uuid4 import json mcp = FastMCP("sqlite-fastmcp-server") DB_PATH = "airline.db" # change this or pass via env/arg if needed #defining info to communicate with the car agent with open('port_config.json', 'r') as file: port = json.load(file)['car_agent_port'] host = "127.0.0.1" CAR_AGENT_URL = f'http://{host}:{port}' @mcp.tool() async def sqlite_get_catalog() -> str: """List tables and columns in the SQLite database.""" async with aiosqlite.connect(f"file:{DB_PATH}?mode=ro", uri=True) as conn: cursor = await conn.execute("SELECT name FROM sqlite_master WHERE type='table'") tables = [row[0] for row in await cursor.fetchall()] catalog = {} for table in tables: col_cursor = await conn.execute(f"PRAGMA table_info({table})") columns = [col[1] for col in await col_cursor.fetchall()] catalog[table] = columns return str(catalog) @mcp.tool() async def sqlite_select(sql: Annotated[str, "SQL SELECT statement"]) -> str: """Execute a SQL query and return HTML table of results. This queries the airline data, the structure of which is defined by the sqlite_get_catalog tool""" async with aiosqlite.connect(f"file:{DB_PATH}?mode=ro", uri=True) as conn: cursor = await conn.execute(sql) if not cursor.description: return "Statement executed successfully." headers = "".join(f"{html.escape(col[0])}" for col in cursor.description) rows_html = f"{headers}" for row in await cursor.fetchall(): row_cells = "".join(f"{html.escape(str(val))}" for val in row) rows_html += f"{row_cells}" return f"{rows_html}" @mcp.tool() async def sqlite_modify(sql: Annotated[str, "SQL INSERT/UPDATE/DELETE/DDL statement"]) -> str: """Execute a SQL statement that modifies the database and return a success message.""" try: async with aiosqlite.connect(f"file:{DB_PATH}?mode=rwc", uri=True) as conn: await conn.execute(sql) await conn.commit() return "Statement executed successfully." except Exception as e: return f"Error: {html.escape(str(e))}" @mcp.tool() async def get_car_agent_card() -> str: async with httpx.AsyncClient() as httpx_client: resolver = A2ACardResolver(httpx_client=httpx_client, base_url=CAR_AGENT_URL) agent_card = await resolver.get_agent_card() return str(agent_card) @mcp.tool() async def query_car_agent(query: Annotated[str, "Natural language query to send to the car agent"]) -> str: """Send a natural language query to the car agent via A2A protocol and return the response.""" async with httpx.AsyncClient(timeout=None) as httpx_client: # Step 1: Resolve the agent card resolver = A2ACardResolver(httpx_client=httpx_client, base_url=CAR_AGENT_URL) agent_card = await resolver.get_agent_card() # Step 2: Create the A2A client client = A2AClient(httpx_client=httpx_client, agent_card=agent_card) # Step 3: Create the message message = { "role": "user", "parts": [ {"kind": "text", "text": query} ], "messageId": uuid4().hex } # Step 4: Wrap in SendMessageRequest request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams(message=message), ) # Step 5: Send the message and get a response response = await client.send_message(request) return response if __name__ == "__main__": mcp.run() They’re each essentially the same, except I made it so that the agents can talk to each other; so the airline agent can talk with the car agent, and the car agent can talk to the airline agent. I found this, practically, did nothing. For more complex queries though, one might be able to imagine the Airline and Car agents working together to figure some stuff out on the users behalf. I think the application I’m defining in this example is too simple to warrant such complex use cases, though. Feel free to check out the full code to peruse the agent cards and stuff. I also defined a simple script that spools up databases for the agents to talk to. Let’s run this sucker. Because the agents talk with one another, they all should be spooled up before we initiate communication. So, in three separate terminals, we can run the following three commands: uv run airline_agent.py uv run car_agent.py uv run travel_agent.py Then, in yet another terminal, we can run uv run test_client.py which is an easy way to spool up and talk to the travel_agent. You can see an example of me interacting with the client to the right. Notice how, after interacting with the client, it triggers the travel agent (third from the left) which triggers the airline agent (first from the left) and the car agent (second from the left). typing in the right makes the agents do stuff So, that’s the rub! Before we wrap up I’d like to conclude with some of my personal thoughts. Discussion I think A2A and MCP are super exciting and are a tremendous step in the right direction. The promise of arbitrarily communicating agents doing cool stuff is compelling. However, I don’t think MCP and A2A alone are enough to revolutionize software development into some “AI first” utopia. One of the consistent issues I have with this new paradigm of “interoperable AI” is prompt engineering (cue eye roll). I know prompt engineering is often used by quasi-developers who barely know what they’re talking about. However, critically, if you do prompting wrong, it’s very hard to make a good application. When integrating A2A and MCP applications, there’s a lot out of one’s control in terms of prompt engineering, which makes it very hard to control specific workflows. I don’t think this is an insurmountable issue, but I think the state of the art needs to evolve best practices to achieve the consistent performance that’s critical in production applications. Another problem is implementation. There are too many ways to do things, and there’s a ton of boilerplate. Now that MCP and A2A are a thing, high-level frameworks that define and implement best practices need to be popularized. We saw a similar thing with ASGI being wrapped around Starlette and Uvicorn; I think it’s just a matter of time before MCP and A2A see similar frameworks. All in all, I think this will be a cornerstone of technology moving forward. If I were building a product today, though, I would be reluctant to use either approach. It’s still important to learn, though. I’ll be covering more sophisticated A2A applications in upcoming articles. Stay tuned!

Intuitively and Exhaustively Explained

Real-Time Visualization of Agentic Interactions— Intuitively and Exhaustively Explained

Installing ahar-visualizer

Exploring the Visualizer

Agent Harnesses Standard Specific Visualizations

Conclusion

A Brief Correction to Previous Articles

Towards Self-Repairing and Repeatable AI Systems

The Goal of The Agent Harnesses Standard

The Standard in a Nutshell

Tools and Workflow

Conclusion

Infrastructure on Google Cloud—Intuitively and Exhaustively Explained

The Core Building Blocks of Google Cloud

Setting Up a GCP Account

Projects on GCP

Exploring Cloud Run

Hello

Hello

Our mission

Core Concepts Behind Deployment

Exploring Firestore

Exploring Identity Platform

Google Cloud Storage

The GCP CLI—gcloud

Queuing

Rate Limiting with Redis

Formalizing some Fundamental Ideas

Levels of Compute Abstraction

IAM

Pub/Sub

Brief Mentions

Idempotency

Sharding

Billing

Logging, Monitoring, and Tracing

Ingress

Security

Conclusion

The Documentation is the Code

The Fight Between Documentation and Code

The Changing Landscape of Software Development

AI, and the Role of Documentation

The Agent Harnesses Standard

Agent Harnesses with Claude — Intuitively and Exhaustively Explained

What is a Harness?

The Context in Which Harnesses are Being Defined

Context: Agentic Constraint

Context: Agent Skills

Limits of Constraints and Skills

Concepts in Agent Harnesses

Using Agent Harnesses with Claude

Exploring a Job Application Assistant using Agent Harnesses

A Discussion of Implications

Creating a Harness

Conclusion

Agent Harnesses — Intuitively and Exhaustively Explained

Building a Fundamental Understanding

Review: LLMs

Review: Agents

Review: Chain of Thought and Tool Use

Review: Skills

The Fundamental Idea of a Harness

The Agent Harnesses Standard

Defining HARNESS.md

Defining the Skills Directory

Defining the References Directory

Structuring Large Harnesses

Further Reading

Agent Skills — Intuitively and Exhaustively Explained

Core Concept Review

Review: LLMs

Review: Agents

Review: Chain of Thought and Tool Use

The Point

The Anatomy of a Skill

Making an IAEE reference skill

Conclusion

Developing Native Android Applications in Kotlin — Intuitively and Exhaustively Explained

Native vs Framework

`4) Animated Gradient Background`

`5) Animating Transitions Between Words`

`Our Second App: Flag Crank App`

`App 2, Part 1 — Setup`

`App 2, Part 2 — Planning The App`

`App 2, Part 3 — Exploring Jetpack Compose vs Traditional Views`

`App 2, Part 4 — Configuring Dependencies`

`App 2, Part 5— Implementing the Flag in Jetpack Compose`

`Conclusion`