Earlier this year, my answer was still quite a lot. Frontier models were meaningfully better at the coding-agent workflows I cared about, and getting a local model to behave like Claude Code required enough compromise that paying for inference was often the sensible choice. Then I got MiniMax 2.5 running across two DGX Sparks with Ray and vLLM. That was the first setup where I could honestly say I had replaced a paid coding model with something I hosted myself.

The MiniMax setup worked. It read repositories, edited files, called tools, wrote tests, and was good enough that I started using it for actual work rather than local-AI experimentation. I kept Claude Code and Codex around because I still use frontier models, but OpenCode became the place where I could choose my own inference. I could point the same kind of coding-agent workflow at infrastructure I controlled.

However, when Qwen3.8-27B came out, the answer changed again.

I downloaded a 17.6GB quantized model, loaded it onto the RTX 5090 already sitting under my desk, pointed OpenCode at it, and started working. I did not buy another cluster. I did not add another server. I added one model to a machine I already owned. The 5090 started beating my two-Spark cluster.

It’s interesting as a hardware result, but I think the more important story is what it tells us about small, dense models. We are reaching the point where a model small enough to live on one consumer GPU can perform enough real engineering work that the economics of sending every agent request to a frontier API needs to be reconsidered.

Small models are no longer interesting because enthusiasts can run them at home. They are becoming interesting because they may be the more sensible infrastructure.

The Experiment Keeps Getting Smaller

My self-hosted coding stack has been moving in the opposite direction from the AI industry’s infrastructure narrative.

The conventional trajectory is bigger models, bigger clusters, bigger context windows, and more GPUs. My own experiments started there too. MiniMax 2.5 runs across two DGX Sparks because the Sparks give me 128GB of unified memory each, which makes them fantastic machines for models that simply do not fit comfortably on a normal consumer GPU. My Spark deployment is still useful, and I am not tearing it down any time soon.

But Qwen3.8-27B forced me to separate two questions that are often treated as the same problem: how large a model can I run, and how large a model do I actually need for this task?

For interactive coding, I do not need the largest model I can fit into my house. I need a model that can understand the repository, reliably call tools, make useful changes, and generate fast enough that it does not break my concentration. Qwen3.8-27B does that.

In a clean single-stream benchmark, the model generates around 75 tokens per second on my RTX 5090. MiniMax 2.5 across my two-Spark cluster produces around 27 tokens per second under the same benchmark conditions.

That makes the 5090 roughly 2.8 times faster for the generation loop I actually experience while coding, which was an unexpected surprise.

A 27B Model Is Not a Toy

Calling Qwen3.8-27B a “small model” is in itself a sign of how strange this industry has become.

Twenty-seven billion parameters is an enormous neural network. It is small only in comparison with the massive frontier systems we have normalized over the past few years. What matters operationally is that a quantized version is small enough to fit on one modern GPU and still has enough capability to do useful work.

The model I am running is about 17.6GB. It has a native 262,144-token context window, supports vision, can call tools, and exposes configurable reasoning effort. Simon Willison independently tested the same model on a 128GB MacBook Pro and a DGX Spark and found that it could successfully drive a coding agent, navigate files, build tools, work with images, and complete practical programming tasks. His conclusion was that the remarkable part is not one benchmark score but that all of these capabilities now fit into roughly a 17GB model.

That distinction matters to me because I do not use models primarily as chatbots. My normal coding workflow involves Claude Code, Codex, and OpenCode. With OpenCode, I can switch between my MiniMax backend on the Sparks and Qwen3.8 running on the 5090.

The model needs to operate inside an agent loop. It needs to read files, understand tool schemas, reason about changes, edit code, run tests, interpret the failures, and try again; which is a substantially higher bar than asking a local model to summarize a PDF. For me, Qwen is clearing it.

The Frontier Comparison Is More Interesting Than the Headline

The external benchmarks reinforce the same conclusion, although I think they need to be read carefully.

Qwen’s reported results put the 27B model in the same general performance conversation as Claude Opus 4.6 Max on several software-engineering and agentic benchmarks. Qwen reports 61.7 on SWE-bench Pro compared with 53.4 for Opus 4.6 Max, 90.3 versus 88.8 on LiveCodeBench v6, and 79.0 versus 63.8 on QwenSWEBench. Opus remains ahead on other tests, including 78.2 versus 73.0 on Terminal-Bench 2.1, 91.3 versus 89.2 on GPQA Diamond, and a much larger 40.0 versus 30.8 on Humanity’s Last Exam. All that said, I would not turn that into “Qwen beats Opus.”

Most of these numbers originate with the Qwen release, benchmark harnesses differ, and even the comparison tables have important methodology caveats. Independent evaluation will matter more than launch-day benchmark charts.

But I also think dismissing the results because they are imperfect misses the important part.

A 27-billion-parameter dense model that I can run on one GPU is close enough to Opus 4.6 on multiple coding benchmarks that we are debating benchmark methodology instead of laughing at the comparison.

The threshold does not require Qwen to be better than Opus at everything. it only has to be good enough at enough tasks that I stop sending those tasks to Opus. That’s a threshold has enormous economic consequences.

Simon Willison Found the Same Model and a Different Bottleneck

Simon Willison’s testing is particularly useful because he approached the model from a different setup and found almost the opposite hardware result.

He ran a 17GB Q4 build on both a high-end M5 Max MacBook Pro and a DGX Spark. The model worked. It could code, use tools, and work with vision. He was impressed by its capabilities. His problem was speed.

He reported roughly 15 to 30 tokens per second from LM Studio and described performance as the main thing preventing the model from becoming a daily driver for him. He correctly identified the fundamental problem with this class of dense model: once the model fits in memory, inference becomes heavily dependent on memory bandwidth.

That immediately explained why my experience felt so different.

A DGX Spark is an incredible little machine because it has 128GB of unified memory. That capacity lets me run models that would never fit on a 32GB consumer GPU. But the Spark has roughly 273 GB/s of memory bandwidth. My RTX 5090 has about 1.8 TB/s.

Qwen3.8-27B does not need the Spark’s 128GB of memory. The whole quantized model fits comfortably on the 5090. Once that happens, the 5090 gets to do what it is very good at: move model weights extremely quickly.

That is why Simon can reasonably describe the same model as capable but slow while I am getting roughly 75 tokens per second from it.

We are running the same class of model. The hardware is optimized for different problems.

My 5090 Versus My Two DGX Sparks

Because I already run Prometheus across the homelab, I can compare the systems using more than vibes.

The clean benchmark gives me the easiest number to understand:

Metric Qwen3.8-27B / RTX 5090 MiniMax-M2.5 / 2× DGX Spark Clean generation benchmark ~75 tok/s ~27 tok/s 5-minute decode throughput 52.3 tok/s 9.9 tok/s Prefill throughput 1,300 tok/s 717 tok/s Modelled TTFT at 4k 3.1 sec 5.7 sec Configured context 262,144 100,000 Max output 32,000 24,000

The benchmark and Prometheus numbers measure slightly different things. The 75-versus-27 result comes from a controlled single-stream benchmark with a short prompt. The five-minute Prometheus rate includes periods of idleness, partial requests, and normal background behavior. They stillnevertheless describe the same system.

And as it turns out, the 5090 is substantially faster for my interactive workload.

Additionally, the prefill result matters almost as much as generation. Agents spend enormous amounts of time ingesting context before they generate anything. A coding agent may receive system instructions, tool definitions, conversation history, repository files, test output, diffs, logs, and command results before it produces the next response.

On my machines, Qwen is processing that input at roughly 1,300 tokens per second, compared with 717 tokens per second on the Spark backend.

Generation determines how quickly I watch the answer appear. Prefill determines how quickly the model starts reasoning about everything I just gave it.

For interactive coding, I care about both.

The Stable Result Matters More Than the Fastest Result

I also wrote a small Go benchmarking harness because distributed systems have taught me not to trust one successful run.

It sends the same request to each backend, measures from the same client, and records time to first token.

Across three runs, I saw this:

Run 5090 Qwen p50 / p95 Spark MiniMax p50 / p95 1 59.7 / 60.9 ms 144.7 / 442.3 ms 2 59.5 / 155.5 ms 49.1 / 50.7 ms 3 61.9 / 135.8 ms 172.5 / 800.6 ms

The Spark cluster actually won the second run.

That is useful because it prevents me from writing the much easier article about how my gaming GPU “destroys” an NVIDIA AI cluster.

The real result is variance. The 5090’s p50 stayed between 59.5 and 61.9 milliseconds. The Spark backend ranged from 49.1 to 172.5 milliseconds.

I saw the same pattern in generation throughput:

5090:  75.3, 75.3 tok/s
Spark: 27.3,  9.3 tok/s

The Spark system is distributed and supports batching. It has more things happening around the request and more opportunities for another workload to affect latency.

The 5090 server is much simpler. It has one model, one GPU, and one active stream.

Sometimes the sophisticated system wins, but the boring system is predictable.

For batch processing, I still like the Sparks. For an agent I am actively pairing with while I write software, I strongly prefer predictable latency.

Qwen’s Biggest Problem Is That It Thinks Too Much

Qwen3.8 also has a hilarious default that becomes painful very quickly on local hardware.

It defaults its reasoning effort to xhigh.

Qwen exposes three primary reasoning levels: xhigh, medium, and low. The highest setting is intended for complicated problems, but leaving it as the default means the model can decide that even a trivial request deserves an enormous internal reasoning trace.

Simon Willison demonstrated this beautifully. He asked Qwen to draw an SVG of a pelican riding a bicycle. The model spent 21 minutes producing more than 22,000 reasoning tokens before writing about 3,200 output tokens. He later asked it for an SVG of a circle and watched it turn the request into an elaborate design exercise rather than just drawing a circle. His recommendation was to avoid the default and start with low or disabled reasoning.

I ran into the same behavior from a coding perspective.

A short request with a small token budget could return what looked like an empty response because Qwen had consumed the entire budget thinking before reaching the answer.

The model was not broken, itIt was contemplating the universe before renaming my variable.

For my normal coding workflow, I changed the default to medium. That turns out to be one of the advantages of self-hosting: I control the serving behavior instead of waiting for an API provider to expose the setting I want.

How I Run Qwen3.8-27B

The complete implementation is in my pedro-ops repository, including the llama.cpp presets, router configuration, systemd changes, observability, and benchmark harness.

My 5090 server runs llama.cpp in router mode. Rather than launching llama-server with one permanent model, the router starts without loading weights and launches model-specific child processes on demand.

The shared defaults are straightforward:

[*]
flash-attn = on
no-webui   = true
metrics    = true
jinja      = true

Qwen then gets its own configuration:

[qwen3.8-27b]

model            = /opt/models/qwen3.8-27b/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size         = 262144
n-gpu-layers     = -1
parallel         = 1
batch-size       = 2048
ubatch-size      = 1024

cache-type-k     = q8_0
cache-type-v     = q8_0

chat-template-kwargs = {"reasoning_effort": "medium"}

temp             = 0.7
top-k            = 20
top-p            = 0.8
min-p            = 0.0

load-on-startup  = false

The line that changes the practical experience most is:

chat-template-kwargs = {"reasoning_effort": "medium"}

That gives me reasonable day-to-day reasoning without spending thousands of thinking tokens on every mechanical code change.

I can still change the reasoning level per request. If I am doing a routine refactor, I can push it toward low. If I am asking the model to reason through an architectural decision, I can use xhigh.

That is how reasoning should work.

It is a resource I allocate based on the problem, not a personality trait of the model.

The Context Window Fits Because the Architecture Is Interesting

The 262k context window is another reason this model works unusually well on one 32GB GPU.

Qwen3.8-27B has 64 layers, but only a subset use traditional full attention. The architecture mixes full-attention layers with Gated DeltaNet linear-attention layers, which changes how much KV cache grows with context.

That makes the native 262,144-token context window much more practical than I initially expected.

With my Q8 K/V cache configuration, I estimate that full context consumes about 8.6GB of KV cache. The quantized model weights use about 17.6GB. On a card with roughly 32.6GB available, that leaves enough room for the model and its serving overhead.

This is one of the broader reasons small-model progress matters. “27B” does not tell you everything about how expensive a model is to operate. Architecture, attention mechanisms, quantization, memory bandwidth, context implementation, and speculative decoding all change the deployment characteristics.

Better models are increasingly also becoming more deployable models.

I Run Two Models on the 5090, but Not at the Same Time

The 5090 was already serving Qwen3.6-27B-MTP before I added Qwen3.8. The older model uses roughly 17.9GB of weights. Qwen3.8 uses roughly 17.6GB.

Together, that gives me about 35.5GB of weights before allocating any KV cache at all.

They obviously do not fit simultaneously on a 32GB card, sSo I stopped trying to make them fit simultaneously.

The llama.cpp router allows me to keep both models registered while limiting the server to one resident model at a time. When I switch models, the current model is unloaded and roughly 18GB of the next model is read back into memory.

The first request after a model swap is slow. After that, steady-state inference is normal.After that, steady-state inference is normal.

The important advantage is that whichever model is active gets the entire GPU. I do not sacrifice half of Qwen’s context window simply because I want another model available. That is a reasonable systems tradeoff.

The architecture is optimizing for the workload I actually have rather than for an imaginary requirement that every model be instantly resident at all times.

My Coding Workflow Is Deliberately Hybrid

None of this means I stopped using frontier coding tools. I still use Claude Code, Codex, and OpenCode.

What changed is that I no longer think of those tools and the model behind them as necessarily being the same decision.

OpenCode gives me the flexibility to point an agent harness at my own OpenAI-compatible endpoints. My local configuration lets me choose between Qwen on the 5090 and MiniMax on the Sparks depending on what I am doing.

Conceptually, it looks like this:

Qwen3.8-27B
    -> llama.cpp
    -> RTX 5090
    -> OpenAI-compatible endpoint
    -> OpenCode

MiniMax 2.5
    -> vLLM / Ray
    -> 2x DGX Spark
    -> OpenAI-compatible endpoint
    -> OpenCode

Then I choose the backend based on the task.

Qwen has become my preferred interactive model because it is fast and predictable. The Spark cluster remains useful for larger or more batch-oriented workloads. Claude Code and Codex are still available when I specifically want frontier-model capability.

This is increasingly how I think we should build AI systems in general.

The agent harness should not force the model choice, nor should the model choice force the infrastructure.

And your hardest problem should not determine what you pay for every easy one.

Frontier Models Still Matter. A Lot.

There is a temptation whenever local models improve to turn the conversation into local versus cloud, open versus closed, or Qwen versus Claude.

I think that framing is boring. I want frontier models to exist.

More than that, I want frontier model research to accelerate.

The enormous models built by frontier labs are part of what enables smaller models to get this good. Research into reasoning, architectures, multimodality, tool use, training data, post-training, long context, distillation, quantization, speculative decoding, and inference eventually flows downstream.

Today’s ridiculous research project becomes tomorrow’s 27B model running under my desk.

That is the cycle I want.

We should build large models because we are trying to learn something.

We should spend GPUs discovering better architectures, training methods, reasoning systems, and representations. We should use frontier-scale infrastructure for work that actually requires frontier-scale infrastructure. Then we should take what works and make it smaller.

Large models enable smaller models, and smaller models scale better.

That is not an argument against frontier research. It is an argument for using frontier compute for frontier research instead of assuming the end state of AI is that every human request forever gets routed through the largest machine we can build.

Science is important! Giving billionaires more money because a trillion-parameter model needs to rename my variable is not.

If a 27B model can do the job, run the 27B model and free those GPUs to discover whatever comes next.

Small Models Are the Better Scaling Story

When the AI industry talks about scale, it usually means centralized scale: more GPUs, bigger data centers, larger clusters, more power.

There is another kind of scale: Distribution.

A model that requires a giant data center can be extraordinarily capable, but there are only so many places you can put a giant data center.

On the other hand, A model that runs on one GPU can exist inside thousands of companies. A model that runs on a capable laptop can exist in millions of backpacks.

It can live next to the data. It can operate continuously. It can serve an internal agent without metering every token. It can continue working when the internet disappears. It can operate inside a network boundary instead of requiring every piece of context to cross one.

That is an extremely compelling way to scale AI.

Simon put his finger on this when he described the significance of Qwen3.8 as a general-purpose, vision-capable, tool-using coding model fitting into a 17GB file. His own tests showed that this class of model now runs useful coding-agent loops on personal hardware.

My experiment adds another piece. Give that same class of dense model enough memory bandwidth and it becomes fast too.

If You Have a Capable Mac, You Should At Least Try This

The 5090 is clearly helping my result. I am not going to pretend every laptop will generate 75 tokens per second.

But 75 tokens per second is not the threshold that determines whether local inference is useful; Ccapability comes first.

If you have a modern MacBook Pro with 32GB or more of unified memory, you have enough hardware that I think you should at least test a local model before assuming every coding-agent request needs cloud inference.

Simon ran Qwen3.8 successfully on his MacBook Pro. It was slower than he wanted for a daily driver, but the model itself worked. It coded It used tools. It handled vision. It performed real tasks.

That means the relevant experiment is no longer:

Can my laptop beat Claude?

It probably cannot.

The useful question is:

Which of my tasks actually require Claude?

Maybe the local model handles 30 percent of your work.

Maybe it handles 60 percent.

Maybe it handles almost everything you normally do and you only escalate genuinely difficult problems.

You cannot answer that question from a leaderboard.

You have to run the model.

This Is Where the Business Economics Get Interesting

The same argument becomes much more consequential once agents start operating continuously inside businesses.

A person sitting in a chat window can only consume so many tokens. Agents change the equation because they can continuously read context, invoke tools, generate intermediate output, retry failed operations, summarize state, process documents, run workflows, and start again.

The token spend can become infrastructure spend very quickly.

Suppose a sufficiently heavy user or persistent agent workload is consuming $3,000 per month in inference.

At that point, a $6,000 local inference machine has a theoretical hardware payback period of roughly two months before electricity, maintenance, labor, and utilization are considered. Spend $9,000 on the machine and the simple payback is around three months.

Obviously, real infrastructure economics are more complicated than dividing hardware cost by an API bill. GPUs require power. Hardware fails. Engineers cost money. Local capacity is finite. Cloud APIs give you elasticity, operational simplicity, and instant access to models that would be impractical to host yourself. Those are real advantages.

But cloud inference has costs too, and we have become weirdly comfortable treating them as inevitable. If an organization is spending tens or hundreds of thousands of dollars a year sending repetitive agent workloads to external APIs, the architecture deserves more analysis than “Claude is better.”

Better at what?

How much better?

How often does that additional capability change the result?

What percentage of requests actually need it?

Those are infrastructure questions, and bBusinesses should start answering them like infrastructure questions.

Frontier Inference Should Be an Escalation Path

The architecture I keep coming back to is model routing.

Run the smallest model that can reliably perform the task. If the task is mechanical, keep it small. If it contains sensitive internal information, there may be a strong reason to keep it local.

If it performs enormous volumes of repetitive inference, local economics become increasingly attractive. If it requires deep reasoning that the smaller model repeatedly fails, escalate it.

If the value of a frontier model’s extra capability clearly exceeds the cost, use the frontier model.

That is not radical architecture, itIt is how we already operate everything else.

We do not deploy the largest database available for every application because larger databases are more capable.

We do not schedule every container onto the biggest EC2 instance AWS sells. We do not use distributed systems when one process reliably solves the problem.

At least, we are not supposed to.

AI escaped that discipline because for several years the largest models really were so much better that using them for everything often made sense.

The capability curve is changing.

Frontier inference should increasingly become an escalation path rather than the default path.

Why I Choose to Self-Host

My reasons for self-hosting are partly financial, but money is not the most interesting one.

The first is control.

I control the model, I control the context, I control its reasoning settings, I control when I upgrade, and I control observability. I can inspect throughput in Prometheus. I can benchmark the exact client path my agent uses. I can decide that one model gets evicted so another can use the full GPU.

The second is privacy.

My source code does not need to leave my network for the model to reason about it.

That does not magically make the agent secure. Local agents can still have terrible permissions, leak data between systems, hallucinate destructive commands, or be configured by an idiot.

But privacy becomes a property of the deployment instead of a promise contained in somebody else’s contract.

The third reason is economics.

Once the hardware exists, another agent loop does not create another API invoice.

The fourth is engineering.

I actually understand what the system is doing. Running models myself forces me to understand context size, KV cache, quantization, throughput, prefill, batching, memory bandwidth, routing, reasoning effort, metrics, and failure behavior.

There Are Real Downsides

Self-hosting is not free just because the token meter disappears.

The hardware has a fixed capacity. If fifty engineers suddenly want inference simultaneously, one 5090 is not going to magically become a cloud region.

Your favorite open model may still be worse than the frontier model at the exact thing your business needs. And cloud providers are exceptionally good at making enormous amounts of compute appear when you ask for it.

That doesn’t mean I’m proposing “everyone replace Anthropic with a gaming PC.” I am proposing something much less exciting and much more important.

Measure the workload and choose the appropriate infrastructure.

Sometimes that will be Claude, sometimes it will be Codex,

sometimes it will be two DGX Sparks, and increasingly, sometimes it will be a 17.6GB dense model running on one GPU under somebody’s desk.

Good Enough Is Actually Good

This is the conclusion I keep reaching as I continue these self-hosting experiments.

The local model does not need to win every benchmark.

It does not need to prove that frontier labs are useless.

It does not need to replace Claude Code, Codex, MiniMax, and every other model I use.

It needs to cross the threshold where I trust it with real work.

Qwen3.8-27B has crossed that threshold for me.

It writes the code I need. It calls the tools I need. It has enough context. On the 5090 it is extremely fast. I can tune its reasoning behavior. My source code stays on infrastructure I control. And the computer was already sitting under my desk.

Once a small model crosses that line, the comparison changes.

I stop asking whether Qwen is better than Claude Opus 4.6.

I start asking whether Opus is sufficiently better at this particular task to justify escalating the task to Opus.

That is a much more useful engineering question.

It is also, I suspect, a much more uncomfortable business question for companies whose business model assumes that the future of AI means renting every token forever.

Agents are going to consume an absurd number of tokens.

Those tokens are becoming infrastructure.

And when rented infrastructure becomes expensive enough, engineers have always done the same thing.

We start asking whether we should own the machine.

The remarkable thing this time is that the machine does not have to be a datacenter.

Mine is under my desk.

The model is 17.6GB.

And I am going to keep seeing how far I can push it.

Run It Yourself

The complete Qwen3.8 deployment is in my pedro-ops repository. The PR includes the llama.cpp router configuration, Qwen preset, reasoning configuration, model-swapping behavior, Prometheus changes, and benchmarking code:

Soypete/pedro-ops — Qwen3.8-27B on the 5090

Simon Willison’s independent experiments are worth reading alongside mine because he tests the same model on very different hardware and reaches many of the same conclusions about its capability:

**Simon Willison — Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things.**

For the Opus 4.6 comparison and benchmark aggregation:

LLM Stats — Claude Opus 4.6 vs. Qwen3.8-27B

I would not start by asking whether a local model can replace every model you use. Download one, point your agent at it, measure it against the tasks you actually perform, and then figure out which tokens are still worth renting.

Take My Courses

If you’re looking to build new skills, I’ve created practical, project-focused courses designed to help you become a better engineer.

• Learn Go — Practical Go programming for engineers who learn by building.

• Learn Pandas - Learn the basics of interacting with data on the world’s bets online learning platform

Stay Connected

Want to stay updated on what I’m working on?

You can find me here:

• YouTube

• LinkedIn

• X / Twitter

• GitHub

• Twitch

• Linktree

Listen to the Podcast

If you enjoyed this article, you’ll probably enjoy Domesticating AI, the bi-weekly podcast I co-host with Matt Sharp and Chris Brousseau.

We explore practical AI engineering, self-hosting, homelabs, infrastructure, open-source models, and the systems behind modern AI. Every episode is built around one goal: helping developers understand not just how to use AI, but how to build reliable systems with it.

Keep your AI on a leash.

Build Better Agents

I started a new company, Haikei Labs, with a pretty simple goal: help people build better AI agents.

Agents need more than a good model. They need the right context, reliable infrastructure, clear permissions, and access to the right data without handing over everything.

That is what we are building.

If you’re working on agents inside your company, struggling with governance and context, or just want to talk about what production-ready agent infrastructure should look like, I’d love to hear from you.

Learn more or reach out at Haikei Labs

Long before I knew what a homelab was, I spent nights and weekends building things that had nothing to do with my day job. I built small machine learning projects because I was curious about how models actually worked and I wanted to give talks on machine learning in Go. I’d watch conference talks, find a technology I didn’t understand, and then force myself to build something with it. Looking back, rather than trying to create products I was just investing in myself.

At the time I could just run things on my computer or follow the trend of using cloud free tier services. I assumed serious infrastructure belonged to companies. Running servers, hosting services, managing Kubernetes clusters, or building distributed systems felt unattainable. I thought people who bought used enterprise hardware were part of some niche operations community with far more time, money, and patience than I had. Self-hosting wasn’t something I considered practical, and it certainly wasn’t something I imagined would become part of my career.

That assumption disappeared almost immediately after I joined Tailscale.

The Hardware Was Never the Point

One of the first things I noticed at Tailscale was that almost everyone had something running at home. Old Dell servers rescued from offices, Raspberry Pis tucked behind televisions, Synology NAS devices packed with containers, networking equipment assembled from eBay parts, and racks that looked like they had grown organically over years of experimentation.

At first, I thought the interesting part was the hardware. Eventually, I realized the hardware was simply the visible artifact of something much deeper. The engineers I met at Tailscale were truly remarkable because they were relentlessly curious. And that has really stuck with me ever since.

The best engineers weren’t spending their evenings learning technologies because they expected an immediate promotion or because a manager told them to. They were teaching themselves things that were adjacent to their jobs simply because they wanted to understand how the world worked. Someone would spend a weekend installing Tailscale on their Roomba vacuum that was connected to their homelab because, “could it be done?”Building tooling because they wondered if it was possible, not because anyone needed it. That was unlike anything I had experienced before.

In most organizations I’ve worked with, side projects existed, but they were isolated, people who thought they might one day run their own startup. You’d find one or two people quietly building interesting things after hours. At Tailscale, it felt like the entire engineering culture was built around curiosity. The projects were different, but the mindset was remarkably consistent. People invested in learning first, trusting that the value would eventually find its way back into their work.

Curiosity Compounds

That experience fundamentally changed how I approached my own learning.

After leaving Tailscale, I invested heavily in my Twitch channel. From the outside, it probably looked like I was creating programming content. In reality, every project had a different purpose. I wasn’t trying to maximize views, I was deliberately choosing projects that forced me to learn skills I couldn’t easily acquire at work.

One project taught me Kubernetes while another taught me Linux internals. Others pushed me into machine learning, inference infrastructure, self-hosting, systemd, distributed systems, and eventually AI agents. The project itself was never the goal. The project was simply the excuse to learn.

That philosophy hasn’t changed.

I don’t stream as much anymore because I don’t find generated coding content particularly interesting, but I never stopped building. If anything, I’ve invested more heavily in my homelab because it has become the place where I can continue learning at my own pace.

Every new capability removed another constraint. Hosting my own language models meant I could experiment without worrying about API costs. Running my own infrastructure meant I could break things without affecting production. I could spend an entire weekend chasing an idea that ultimately failed, and the only thing I lost was my own time. That freedom is difficult to replicate inside a company, where every experiment carries a cost for someone else.

Recently I rebuilt an autonomous AI agent in a single afternoon using Claude. A few years ago, building something similar took me months while I wrote Pedro from scratch in Go. I don’t look back at those months as wasted effort because they had a different objective. Pedro taught me how agents worked, and Claude helped me build an agent I could actually use. The outcome changed, but the investment was always the same.

I was investing in my ability to solve problems I hadn’t encountered yet.

Companies Optimize Execution

Companies exist to execute.

Reliable products require prioritization, predictable outcomes, and thoughtful allocation of engineering time. Every experiment has an opportunity cost because engineers aren’t working in isolation. Time spent exploring one idea is time not spent shipping another feature, fixing another bug, or supporting another customer.

That’s exactly how businesses should operate, but learning works differently.

Exploration requires permission to fail repeatedly without anyone asking whether the effort was worth it. It requires enough freedom to chase ideas that may never become products. Most experiments don’t produce immediate business value, but they produce something equally important: engineers who are better equipped to solve unfamiliar problems in the future.

A homelab changes the economics of experimentation.

You’ve already invested in the hardware. You’ve already accepted that your curiosity is worth the electricity bill. The cost of failure belongs entirely to you, which means you can pursue ideas that would never survive a roadmap meeting.

Ironically, many of those ideas eventually find their way back into your day job anyway because you’ve spent years learning how to approach new ones.

Weekend Projects Become Products

One of the things that surprised me most about Tailscale was how often weekend experiments eventually became products.

Sometimes they lived behind feature flags and sometimes they stayed in alpha for months. Sometimes they never fully shipped. But the organization understood something I hadn’t appreciated before: if you consistently create an environment where engineers are encouraged to explore, useful ideas eventually emerge.

That’s one of the reasons companies like Tailscale (and more recently Anthropic) feel so innovative from the outside.

They have become successful because experimentation itself is part of the culture.

Innovation isn’t managed into existence, it’s cultivated.

Steve Wozniak Didn’t Build Apple at a User Group

A few years ago, I listened to Steve Wozniak talk about the Homebrew Computer Club. What I liked was how the ideas shared from this group and community through hardware experiments leaked in to the Apple 1.

By the time Woz walked into Homebrew, he had already spent years experimenting with electronics at home. He wasn’t looking for permission to build computers because he was already building them. What the club gave him wasn’t technical knowledge as much as an environment where unfinished ideas could collide with everyone else’s unfinished ideas.

The homelab gave him the freedom to experiment.

The community gave those experiments somewhere to evolve. I don’t think he would have been able to get there on his own because neither environment was sufficient on its own. But together, they changed history.

Innovation Comes From the Environments We Choose

For a long time, I thought innovation was primarily a function of talent. The more engineers I’ve met, the less I believe that.

I think innovation is largely a consequence of the environments we deliberately place ourselves in.

Those environments aren’t defined by buildings or employers. They’re defined by the systems we create around ourselves: a place where experimentation is encouraged., or a community where unfinished work is welcomed instead of judged. People who challenge your assumptions because they genuinely want your ideas to improve. Freedom to fail, combined with the opportunity to learn from others who are just as curious.

Ideas don’t become better simply because we spend more time thinking about them. They become better because they encounter other ideas.

The Best Investment I’ve Ever Made Wasn’t Hardware

When people ask whether they should build a homelab, I don’t immediately think about servers, GPUs, or networking equipment. Instead, I think about environments.

My homelab is only one piece of that environment.

The other half has been the local community we’ve built together through Forge Utah Foundation . Those meetups have become the place where unfinished work turns into better ideas. They’ve given me opportunities to present projects that weren’t finished, ask questions I wasn’t qualified to answer, receive criticism from engineers far more experienced than I am, and quietly learn from people who have spent decades building systems at companies like Google, Microsoft, GitHub, NVIDIA, Red Hat, Tailscale, and countless others.

I don’t think of those people as mentors because most of them have never formally mentored me. Instead, they’ve done something much more valuable. They’ve created an environment where curiosity is normal.

Looking back, I don’t think buying a Raspberry Pi changed my career, and neither did buying server. Learning Kubernetes wasn’t a turning point.

In an industry where AI is changing what it means to be a software engineer, I can’t think of a better investment in yourself.

Take My Courses

If you’re looking to build new skills, I’ve created practical, project-focused courses designed to help you become a better engineer.

• Learn Go — Practical Go programming for engineers who learn by building.

• Learn Pandas - Learn the basics of interacting with data on the world’s bets online learning platform

Stay Connected

Want to stay updated on what I’m working on?

You can find me here:

• YouTube

• LinkedIn

• X / Twitter

• GitHub

• Twitch

• Linktree

Listen to the Podcast

If you enjoyed this article, you’ll probably enjoy Domesticating AI, the bi-weekly podcast I co-host with Matt Sharp and Chris Brousseau.

We explore practical AI engineering, self-hosting, homelabs, infrastructure, open-source models, and the systems behind modern AI. Every episode is built around one goal: helping developers understand not just how to use AI, but how to build reliable systems with it.

Keep your AI on a leash.

“AI is more of a marketing category than it is a technology.”

The quote comes from Cory Doctorow, and the more I sat with it, the more uncomfortable it made me. I spend most of my day working on systems that people would absolutely call AI, and I still agree with what he said.

I build data systems for AI, which means I run local models, host inference infrastructure, and I spend a lot of time thinking about retrieval, agents, context engineering, and the operational realities of putting these systems into production. If anyone should be inclined to defend AI as a technical discipline, it is probably me.

But the deeper I go into the field, the less useful the term becomes. Transformers are real. Statistical models are real. Knowledge graph inference is real. Recommendation systems are real. The problem is that we keep using a single label to describe technologies that have very little in common with one another.

The more I thought about it, the more AI started to remind me of another term from my career: Cloud.

We’ve Seen This Before

There was a period where every company wanted a Cloud Engineer.

The title made sense at the time because nobody really knew what cloud computing was going to become. Organizations knew it mattered, and. they knew they needed people who understood it. What they didn’t know was which specific skills would emerge as long-term specialties.

A cloud engineer might have been writing infrastructure automation, managing Linux servers, designing networking architectures, implementing security controls, or running Kubernetes clusters. All of those responsibilities lived under the same umbrella because the industry had not yet developed a vocabulary for the specialization underneath.

As cloud matured, the title became less useful.

Today we have platform engineers, SREs, infrastructure engineers, security engineers, and networking specialists. Cloud didn’t disappear. In many ways it became more important than ever. What disappeared was the need for a catch-all label because the underlying disciplines had become well understood.

I think AI is following the same path.

Right now, AI Engineer serves the same purpose that Cloud Engineer served fifteen years ago. It tells us a company knows something is important, but it doesn’t tell us much about the actual work being performed.

What All AI Actually Has In Common

One reason the category feels so confusing is that we often talk about AI as if it were a single technology, but.....it isn’t.

A recommendation engine and a large language model are both considered AI. So are fraud detection systems, forecasting models, graph inference engines, and computer vision systems. But the architectures are different. The operational requirements are different. The engineering disciplines are different.

The obvious explanation would be that they all represent intelligence, but that definition breaks down pretty quickly. Most of these systems are not intelligent in any meaningful sense of the word. They are highly specialized systems designed to produce useful outputs from large amounts of information.

What they actually share is something much simpler.

They derive behavior from data.

Traditional software derives behavior primarily from explicit instructions written by humans. Data-driven systems derive behavior from patterns observed in data. The implementation details vary wildly, but that basic characteristic remains surprisingly consistent across the entire category.

That definition also explains why AI feels so large. We are not describing a technology, we are actually describing an outcome. We are grouping together a family of systems that use data to generate predictions, classifications, recommendations, or actions.

The category is broad because the outcome is broad.

Why Data Engineers Were Accidentally Preparing For AI

This idea also explains something that has been bothering me for a while.

A lot of the work currently being marketed as AI work looks suspiciously similar to the work data engineers have been doing for years.

Most of my career has not been spent building dashboards. It has been spent dealing with messy, unpredictable data and trying to make it useful. Moving data between systems. Validating it. Cleaning it. Enforcing quality constraints. Building relationships between datasets. Reducing ambiguity so downstream systems behave more consistently.

None of that work was called AI, it was just engineering.

But as models have become more capable, the importance of those systems has become much more visible. A language model trained on poor data still produces poor outcomes. A retrieval system built on unreliable data still produces unreliable outcomes. A knowledge graph built on incorrect relationships still produces incorrect inferences.

The model did not eliminate the importance of the data layer.

If anything, it made that layer more important than ever.

That is why I find it strange when people talk about AI as if it appeared suddenly in 2023. Many of the foundational disciplines have existed for decades. We simply wrapped them in a new label.

The Layer Matters More Than The Model

The current conversation around AI is heavily focused on models.

Every week there is a new benchmark, a new frontier release, or a new startup claiming to have achieved something revolutionary. Those announcements attract attention because models are easy to see.

The systems surrounding those models are harder to see, but they are usually where the real engineering work lives.

A production AI system depends on infrastructure, data quality, retrieval, permissions, evaluation, monitoring, and operational reliability. Most failures happen in those layers long before they happen inside the model itself.

That reality becomes obvious the moment you start running these systems yourself. Self-hosting models taught me far more about infrastructure than it taught me about prompting. Building retrieval systems taught me far more about data quality than it taught me about intelligence. Operating agents taught me far more about software engineering than it taught me about reasoning.

The model is only one layer of a much larger stack. And the stack is where most of the engineering lives.

Learn The Layer

That is why I think “learn AI” is becoming bad career advice.

The people who benefited most from the cloud revolution were not the people who memorized cloud marketing materials. They were the people who learned distributed systems, networking, security, and operations. Those skills remained valuable long after the hype cycle settled down.

I suspect the same thing will happen here.

The engineers who build durable careers in this space will not be the ones who become experts in a specific frontier model. They will be the ones who understand data systems, inference infrastructure, retrieval, evaluation, security, context management, and software architecture.

Those disciplines will continue to matter regardless of which company wins the next benchmark war. Eventually the marketing category fades, but the underlying engineering remains.

Final Thoughts

The more I think about Doctorow’s observation, the more I believe it is correct.

AI is not a single technology. It is a label applied to a growing collection of data-driven systems that produce useful outcomes.

That doesn’t make the technology less important. If anything, it makes the underlying disciplines more important.

Because once you stop looking at AI as a product category and start looking at it as a stack, the question changes.

Instead of asking: How do I learn AI?

You start asking: Which layer do I want to build?

And I suspect that question will still matter long after we’ve stopped calling it AI.

Learn With Me

I write about AI engineering, self-hosting, data systems, Go, homelabs, and the infrastructure behind modern AI. My goal is simple: cut through the hype and help you build practical systems you actually understand.

If that sounds interesting, you’re in the right place.
My GO course

Stay Connected

Want to follow along as I experiment, build, and occasionally break things? You can find me here:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Recent Articles

If you enjoyed this post, here are a few more you might like:

• AI Slop

• From One Model to Many: Hosting Multiple Local Models with llama.cpp

• AI Is Here to Stay, But We’re Building It Like It Isn’t

• 7 AI lessons I wish I Knew

Listen to Domesticating AI

Every other week, Matt Sharp, Chris Brousseau, and I explore practical AI from an engineer’s perspective.

Instead of chasing headlines, we focus on the skills that actually matter: inference infrastructure, self-hosting, open-weight models, context engineering, agents, and the systems that make AI reliable in production.

Keep your AI on a leash.

Most people using AI don’t run into unpredictable behavior because the models are bad. They get unpredictable behavior because they are communicating with a machine they think is a human.

That sounds harsh, but it applies to all of us. Every engineer has said, “That’s not what I meant.” Every product manager has written a requirement that seemed obvious until somebody implemented it. Every executive has asked for a system to be “smarter” without being able to define what that means.

Humans are remarkably good at filling in missing context. We infer intent. We make assumptions. We ask clarifying questions. We use shared experience to bridge gaps in communication.

Language models literally cannot. They only have access to the information you provide and the statistical relationships they learned during training. That’s why I keep recommending LLMs in Production1. Not because it contains secret prompts or, because it teaches some magical framework. And definitely not because the authors happen to be my podcast co-hosts.

I recommend it because it explains something that most AI users never stop to think about:

The model doesn’t know what you meant. It only knows what you said.

Most AI Failures Start Before the Model Responds

One of the most common complaints I hear is that models are inconsistent.

Sometimes they follow instructions, sometimes they don’t. Sometimes they call the right tool, sometimes they hallucinate. Sometimes they solve the problem perfectly, and sometimes they confidently produce nonsense.

The natural assumption is that these failures originate inside the model. Sometimes they do, but many of them begin much earlier. They begin with unclear instructions, missing context, incomplete examples, poorly scoped tasks, ambiguous tool definitions, or information that was never made available to the model in the first place.

When we ask humans for help, they compensate for these gaps. They ask follow-up questions. They use experience and intuition to infer what we meant. Language models cannot reliably do that. They respond to the information available. That might sounds small, but it changes how you think about building AI systems.

The question stops being:

“Why is the model wrong?”

And becomes:

“What information did the model have available when it made that decision?”

Once you start looking at AI systems through that lens, many failures stop looking mysterious. They start looking like communication failures.

Computational Pragmatics Explains Why So Many AI Interactions Fail

For a long time, I assumed this was just one of Chris’s favorite rabbit holes was why linguistics matters in the AI landscape, and the annoying conclusion I eventually reached is that he was right.

Computational pragmatics2 is fundamentally concerned with how meaning is transferred between two parties that do not share perfect information. That sounds academic until you realize it perfectly describes every interaction with a language model.

The model does not possess your goals or your mental model of the task. It does not possess your assumptions. Everything it knows about the current problem must arrive through context. That observation sounds obvious once stated, but it explains an enormous amount of AI behavior.

When users say a model is inconsistent, they are often describing inconsistent context.

When users say a model hallucinated, they are often describing missing information.

When users say a model ignored instructions, they are often describing conflicting instructions.

This does not mean models are perfect. They clearly are not. But understanding communication failures is often more useful than endlessly debating model intelligence.

The model doesn’t know what you meant.

It only knows what you said.

Stop Arguing With The Model

One of the strangest workflows to emerge from the AI era is watching people argue with language models.

The model gives an incorrect answer.

The user explains why it is wrong.

The model apologizes.

The user explains again.

The model apologizes again.

Five minutes later they’re still debating a statistical inference engine.

I see this constantly. Developers do it. Executives do it. Students do it. I’ve done it (and sometimes I try to trick Claude into admitting it made a mistake, wasting tokens in the process).

The problem is that arguing with the model rarely fixes the root cause. The model does not have pride or beliefs, and doesn’t care if it’s correct. It does not have beliefs. It is not learning from your frustration. Most of the time, if a model is trapped in a failure mode, the fastest solution is not persuasion. The fastest solution is changing the context, rewrite the instruction, provide an example, clarify the objective, remove ambiguity, or simply start over.

One of the most practical lessons I’ve learned working with AI systems is this:

If you’re in a loop, leave the loop.

That advice sounds almost too simple, but it turns out to be surprisingly effective. When conversations accumulate dozens of corrections, explanations, and contradictory instructions, the context itself often becomes the problem. Starting fresh is frequently less expensive than continuing to patch a broken interaction. Engineers would never keep retrying a broken API call with identical inputs and expect a different result. Yet people routinely do exactly that with language models.

Bigger Context Isn’t Better Context

One of the most persistent misconceptions in AI is that larger context windows automatically create better outcomes. The logic seems reasonable. If context helps, then more context should help more.

Unfortunately, that is not how information systems work. Every document added to a prompt competes for attention. Every instruction competes with every other instruction. Every example introduces another opportunity for ambiguity. Larger context windows give us the ability to provide more information, but they do not guarantee that the information is relevant, coherent, or useful.

In many production systems, the best results come from aggressively reducing context rather than expanding it. This is also why retrieval systems, scoped knowledge stores, and data engineering truly matters. This is one of the reasons retrieval systems matter.

The goal is not to provide everything, it’s to provide the right things.

One observation that keeps showing up in my own work is the best context is rarely the largest context. The best context is usually the clearest context.

That lesson applies equally to prompts, retrieval systems, agents, and software architecture.

Good systems are often defined more by what they exclude than by what they include.

The Most Important AI Skill Is Clarity

Prompt engineering has become one of the most over-marketed skills in technology. People talk about prompts as though they are secret incantations. A specific phrase or hidden structure; a magic template that unlocks superior intelligence. A hidden structure. The reality is much less exciting.

The strongest AI practitioners I know are usually not the people writing the cleverest prompts, they’re actually usually the people who communicate the clearest requirements. They are the people who communicate the clearest requirements.

They understand the task, success criteria, failure criteria, and the information required to perform the work. They understand success criteria. In other words, they understand the problem before they ask the model to solve it. That is not a prompting skill, that’s an engineering skill. And it is the same skill that mattered before language models existed.

Good communication has always been valuable.

AI simply makes poor communication more visible.

Final Thoughts

I think every engineer should understand databases, networking and version control. I think every engineer should understand networking.

And now more than ever

increasingly, I think everyone who prompts LLMs should read LLMs in Production because.

But because these systems are becoming part of how we work, and understanding them is quickly becoming as important as understanding any other piece of software infrastructure.

The biggest lesson I took from the book was not about models, it was about communication.

The model doesn’t know what you meant.

It only knows what you said.

And once you understand that, many AI problems stop looking like intelligence problems and start looking like engineering problems.

My Courses

• Learn Go

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• AI Slop

• From One Model to Many: Hosting Multiple Local Models with llama.cpp

• AI Is Here to Stay, But We’re Building It Like It Isn’t

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.

So instead of continuing to debate architecture in the abstract, I ran an experiment. I wanted to isolate one variable: how context is stored and retrieved. Not how it is modeled or how it is interpreted, but how the system actually gets the data it needs to answer a question.

“One of the miseries of life is that everybody names things a little bit wrong. And so it makes everything a little harder to understand in the world than it would be if it were named differently. A computer does not primarily compute in the sense of doing arithmetic. […] They primarily are filing systems.”

— Richard Feynman, Idiosyncratic Thinking seminar (1985)

I think that framing is closer to what we are actually building with AI systems than most modern discussions about “memory” or “reasoning.” The dominant problem in practical AI systems is actually retrieval rather than intelligence. In chats we want it to look up something, or in agents we want it to accomplish a task based on provided resources. Models only know what they have been trained on and Agents can only pull in what the system gives them access to. The quality, structure, and accessibility of that context determines whether the system feels fast, reliable, and useful; or slow, inconsistent, and impossible to trust.

This isn’t a model problem, it’s a retrieval problem.

The instinct in for most AI users is to blame the model. If the answer is wrong, we assume the model misunderstood the task. In practice, the model is almost always doing exactly what we told it to do with the data it was given. The failure is actually upstream.

If a we provide too much context, latency increases and signal gets diluted. If we provide the wrong context, the model hallucinates. If the context is inconsistent across sources, the model produces answers that appear coherent but are fundamentally incorrect. These are not model failures, they are retrieval failures.

Once you accept that, the problem shifts. The system you are designing is not primarily an AI system. It is a data retrieval system that happens to use an AI model at the end.

Not all memory systems are solving the same problem.

I evaluated three approaches to storing and retrieving context from a codebase. These systems were published recently to help with the Context Graph Trillion-dollar Problem1

1. Mempalace

2. Graphify

3. Llmwiki

One stored context as wiki-style markdown, one materialized a full graph of the codebase, and one used a database-style indexing approach with incremental updates. Each system was given the same corpus, the same queries, and the same interface.

The important constraint is that although the representation of the data was allowed to differ, but the evaluation focused on how each system retrieved information at query time. This is where I find “AI memory” becomes the most confused. We conflate how data is structured with how it is accessed, and those are not the same problem.

Query: "Where is authentication handled?"

Markdown (LLMWiki):
- search across documents using grep/index

Graph (Graphify):
- load entire graph
- construct traversal structure
- traverse nodes

Wiki (MemPalace):
- lookup indexed entities
- follow relationships
- return scoped results

Each system answers the same question, but the path it takes is fundamentally different. That path is what determines performance.

Loading everything is the actual bottleneck.

The initial assumption was that graph-based systems would perform well. Graph traversal is computationally efficient when the graph is already in memory. The problem is that real systems do not operate under that assumption. The graph must be loaded, parsed, and constructed before any traversal occurs.

That cost dominates the query lifecycle. It is not an implementation detail. It is the system.

Graphify query path:

1. Load 3.7 GB graph from disk

2. Construct in-memory graph

3. Execute traversal

In contrast, a system that relies on indexed retrieval avoids this entirely.

MemPalace query path:

1. Query indexed store

2. Retrieve matching rows

3. Return scoped context

The difference between these systems is not theoretical. It is the difference between global state and selective access. One system assumes it needs everything to answer a question. The other assumes it only needs a subset.

Database behavior beats representation.

Once evaluated under system-level conditions, the results were not ambiguous. The database-style system outperformed both markdown and graph-based approaches by orders of magnitude in both latency and storage footprint.

Latency (system-level):

• MemPalace: ~34 ms

• LLMWiki: ~2.2 s

• Graphify: ~35 s

Storage:

• MemPalace: 8.8 MB

• LLMWiki: 33.9 MB

• Graphify: 3.7 GB

This is not an incremental improvement. It is a structural difference in how the system behaves. The database-style system is faster not because it is simpler or more optimized, but because it avoids unnecessary work. It does not attempt to load or reason over the entire dataset.

The winning systems all follow database principles.

The system that performed best was not the one with the richest representation, it was the one that enforced constraints on how data could be accessed. It indexed entities, narrowed the search space before retrieval, and avoided full scans of the dataset.

These are not novel ideas. They are foundational database principles that have been well understood for decades. What is notable is how quickly AI systems expose the consequences of ignoring them.

Example lookup pattern:

SELECT entity_id

FROM entities

WHERE name LIKE ‘%auth%’;

SELECT *

FROM relationships

WHERE subject_id IN (...)

AND predicate IN (’implements’, ‘depends_on’);

The system does not attempt to infer everything at query time. It retrieves a bounded, relevant subset of data and passes that to the model. This constraint is what enables both performance and reliability.

Context stores are an engineering primitive, not an AI feature.

This work was motivated by a practical need. I was watching an entire code directory across multiple repositories, capturing changes as they occurred. From those changes, I extracted patterns, tracked dependencies, and stored reusable context.

The goal was not to build an abstract memory system. It was to answer questions that come up in real engineering workflows, such as identifying prior implementations or understanding how services interact.

Captured context:

auth_middleware implements user_authorization

source: /internal/auth/middleware.go

payment_service depends_on auth_service

source: /services/payment/main.go

This is structured data derived from the system itself. The AI model is not responsible for generating this context. It is responsible for using it. That distinction is important.

If retrieval is slow, everything built on top of it breaks.

It is tempting to optimize prompts, tune models, or improve agent logic. None of those changes address the core issue if the retrieval layer is inefficient. Slow retrieval increases latency across the system. Inconsistent retrieval leads to unreliable outputs. Incorrect retrieval results in hallucinations.

The system fails because it lacks discipline in how it accesses data. The model becomes the visible surface of a deeper architectural problem.

We are rediscovering databases in real time.

The outcome of this experiment is not surprising in hindsight. Systems that follow database design principles perform better. Systems that rely on loading global state do not scale. These are well-established results in software engineering.

What is different is the context in which they are being rediscovered. AI systems amplify the consequences of poor retrieval design. They make latency, inconsistency, and incorrect data immediately visible.

Feynman’s observation that computers are “primarily filing systems” feels strangely relevant here. We keep treating AI systems as if the difficult problem is generating answers. In practice, the difficult problem is usually finding the right information quickly enough that the answer can be trusted

If you want to go deeper, the full experiment and data are here:

https://github.com/Soypete/memstore-analysis

The next step is obvious:

What actually belongs in the database?

That is where semantic indexing, ontologies, and context modeling start to matter.

But none of those systems matter if retrieval itself is slow.

My Courses

Learn Go

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• AI Slop

• From One Model to Many: Hosting Multiple Local Models with llama.cpp

• AI Is Here to Stay, But We’re Building It Like It Isn’t

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.

Before AI, we already had a name for this problem: tech debt. And we understood exactly what it meant.

Tech debt is a trade. You knowingly sacrifice long-term maintainability for short-term speed. You take on risk now because you believe the upside is worth it later. Sometimes that means shipping before the architecture is perfect, or hardcoding something temporarily to validate product-market fit. Sometimes it means accepting scalability limitations because the business opportunity matters more than elegance in the current moment.

That trade-off has always been part of engineering. You were expected to understand the trade-off you were making, why you were making it, and when you would eventually pay it back.

AI did not change that.

What changed is that we stopped talking about ownership.

AI Didn’t Invent Bad Engineering

When people describe “AI slop,” they are usually describing one of three things:

• code generated without understanding

• systems shipped without validation

• architecture produced faster than teams can reason about it

None of those are new problems. AI simply reduced the cost of generating them. That distinction matters because the current discourse around AI slop subtly shifts blame away from the humans and organizations making the decisions.

The model becomes the scapegoat instead of the mirror. But the problem with that is a human prompted the model. A human accepted the output. A human merged the pull request. A human decided speed mattered more than comprehension. The AI did not independently create technical debt. The organization chose to leverage speed over understanding.

That is the same trade engineering teams have made for decades. The difference is that generative AI dramatically increases the scale and velocity of that trade.

Cognitive Surrender Is the Real Problem

The strongest argument against “AI slop” is not even technical. It is psychological.

In a recent essay, describes the difference between cognitive offloading and cognitive surrender1. Cognitive offloading is normal engineering. We use abstractions, frameworks, libraries, and tooling precisely because humans cannot hold every implementation detail in working memory at all times. Good engineering has always involved reducing cognitive burden while still maintaining ownership of the system.

Cognitive surrender is different. It happens when the AI’s output quietly becomes your output without meaningful verification. It happens when confidence is mistaken for correctness or when generated code bypasses the part of engineering that requires understanding.

Osmani writes:

“Cognitive surrender is when the AI’s output quietly becomes your output and there is nothing you feel is left to check.”2

That is the real danger -- surrender

AI Slop Is Really Comprehension Debt

One of the most useful concepts introduced in these discussions is the idea of comprehension debt.

Osmani describes it this way:

“Cognitive surrender is how you take on cognitive debt. Comprehension debt is the bill, denominated in lost mental model.” This is the missing language the industry has been searching for.

What we call “AI slop” is often not merely low-quality code. Sometimes the generated code works. Sometimes it even works well enough to pass tests. The deeper issue is that nobody fully understands the system anymore, which leads to a litany of issues like:.

• debugging slows down

• reliability degrades

• onboarding becomes harder

• architecture becomes reactive

• incident response becomes chaos

The debt is no longer just inside the codebase. The debt now exists inside the engineering organization itself.

Borrowed Confidence Is Not Ownership

One of the more alarming observations from recent AI studies is how quickly humans transfer trust to confident model outputs. Osmani references research showing that when AI systems produced incorrect answers, participants still accepted those answers at shockingly high rates.

The issue was not merely factual accuracy. It was borrowed confidence.

People began treating the model’s certainty as their own understanding, and that pattern is now showing up everywhere in software engineering:

“The AI said the tests passed.”

“Cursor refactored it already.”

“Claude reviewed the architecture.”

“The agent said the migration succeeded.”

But engineering is not the act of generating output, it’sEngineering is the act of being responsible for the output.

This matters because the moment production breaks, nobody calls the language model into the incident review. They call the engineers.

Ownership Is the Craft

I increasingly believe the craft of software engineering is not syntax, frameworks, or even architecture.

The craft is ownership.

Ownership means:

• you understand what you shipped

• you can reason about failure modes

• you can modify the system under pressure

• you are willing to put your reputation behind the outcome

AI does not remove that responsibility. If anything, it increases it.

Modern models are extremely good at generating plausible systems that appear coherent before they are understood. The illusion of comprehension arrives long before actual comprehension does.

That means engineering organizations need stronger—not weaker—cultures around validation, observability, testing, architecture review, and operational responsibility.

The answer is not banning AI.

The answer is refusing to surrender ownership.

The Real Difference Between Tech Debt and AI Slop

Tech debt is acceptable because it is intentional.

You knowingly take on debt in exchange for speed, opportunity, or experimentation.

But the implicit agreement has always been that we will eventually pay this back.

“AI slop” emerges when that agreement disappears.

When organizations optimize purely for velocity, reward shipping over understanding, measure engineers by output instead of system quality, or quietly abandon the expectation that people should fully comprehend what they deliver....AI isn’t to blame.

That is a management and engineering culture problem.

If You Can’t Own It, Don’t Ship It

You can absolutely use AI to build excellent production systems.

You can use it to:

• accelerate implementation

• generate scaffolding

• explore architecture

• improve documentation

• refactor safely

• automate repetitive work

But if you are shipping systems you cannot explain, debug, or reason about, then the problem is not the model. The problem is that you abandoned ownership somewhere in the process.

And if we normalize that behavior, “AI slop” becomes inevitable.

Not because the models failed.

Because the engineering culture did.

My Courses

Learn Go

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• I Replaced Claude With MiniMax 2.5

• From One Model to Many: Hosting Multiple Local Models with llama.cpp

• AI Is Here to Stay, But We’re Building It Like It Isn’t

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.

AI isn’t built for just give you answers, it’s built to reinforce how you think. What people are calling “AI psychosis” is the natural outcome of systems trained to agree with you.

The Symptom Shows Up Before the System Is Understood

There’s a term floating around right now: AI psychosis.1 It sounds dramatic, but the pattern it describes is surprisingly mundane. People start using AI for small, contained tasks. They get quick, helpful answers. Over time, those answers become easier to trust and eventually, the system starts to shape how they think.

The shift is subtle. The model agrees with you, then it expands your ideas. Finally, it presents them back in a more structured, more confident way than you might have expressed them yourself. That feels like progress or clarify, but what’s actually happening is reinforcement.

You don’t need to accept the term “psychosis” to recognize the behavior. The important point is that we are seeing the same pattern across users. The system feels helpful, then reliable, then authoritative. And somewhere along the way, scrutiny drops.

The Evidence Looks Like Good Output

If you want to see this in practice, you don’t have to look at edge cases, you can look at mainstream research. A recent analysis in Harvard Business Review2 examined what happens when people ask large language models for strategic advice. The outputs were described as “trendslop.” Not nonsense, and not obviously wrong. Instead, they were coherent, well-structured, and confidently delivered, but lacking real grounding.

This is important: the failure mode is not that the system produces garbage. It produces something that looks better than garbage. It produces something that is easy to accept. The responses are organized into clean lists, with logical sequencing and clear takeaways. They read like something you could present in a meeting.

And that is exactly the problem.

Because this is what AI psychosis looks like in practice. It is not people losing touch with reality. It is people trusting outputs that feel correct because they are presented well. The system does not need to be right, it only needs to feel right.

Structure Feels Like Reasoning, But It Isn’t

The most important detail in that analysis is that ordering matters. The first answer shapes how everything else is interpreted. A well-ordered response creates the illusion of a well-ordered thought process. We see a clean progression and assume there was reasoning behind it.

But that structure is not reasoning, it’s actually trained behavior.

Modern AI systems are optimized using reinforcement learning from human feedback3, as described in work like training language models to follow instructions with human feedback. Humans consistently reward outputs that are clear, confident, and easy to follow. Over time, the model learns to produce those signals because they lead to higher reward.

That means the system learns to present answers in ways that feel intelligent, regardless of whether the underlying content is correct. The structure is doing most of the work. Because we are wired to trust structure, we mistake presentation for understanding.

The System Doesn’t Just Answer, It Trains You

Once you understand that, the next step is unavoidable. The model is not just generating output. It is shaping how you consume information.

First, you begin to expect clean lists and immediate answers. Next, you start to equate confidence with correctness. You move faster, because the output feels finished. Over time, that becomes your baseline for what “good thinking” looks like.

That is where delegation happens.

Matt Sharp captured this shift directly:

“As an expert I always assumed this was obvious, but I’m coming to learn it’s not. And people keep delegating their thinking to AI.”

That delegation is the real risk. Not that the model produces incorrect answers, but that we stop verifying them. The system is optimized to agree with you, so it rarely forces you to confront uncertainty. And if nothing pushes back, you stop pushing back yourself.

Incentives Shape What Gets Reinforced

This behavior does not emerge in a vacuum. The reward signal behind these systems is shaped by the people building and funding them. That means executives, technology companies, and commercial incentives are all embedded in what the model learns.

We are told that AI will replace workers and drive efficiency. At the same time, there are public statements acknowledging that the cost of AI infrastructure can exceed the cost of human labor, as reported by Fortune4. That tension is not incidental. It tells us that adoption is not purely about capability, it is also about what gets funded and what narratives drive investment.

Reinforcement learning encodes those priorities. If the system is rewarded for outputs that increase engagement, reduce perceived effort, or align with business goals, that is what it will optimize for. Not truth. Not correctness. Just what gets rewarded.

Vibe Coding Is a Starting Point, Not a System

This is where the conversation often goes off the rails. “Vibe coding” gets criticized as if it is inherently irresponsible. It isn’t. In many ways, it is one of the fastest ways to learn. It lowers the barrier to entry. It allows you to explore systems you wouldn’t otherwise touch. It helps you build intuition through iteration, which is valuable.

However, it only works if you treat it as the beginning of the process rather than the end. The danger is not that people are experimenting. The danger is that they stop there. They take generated code or generated structure at face value, without understanding why it works.

A simple example is asking a model to generate a data structure or analysis pipeline in R. The output will often look clean and correct. It will use familiar functions, reasonable defaults, and a logical sequence of steps. But unless you understand the underlying assumptions—how the data is being transformed, what the statistical functions are actually doing—you have no way to evaluate whether the result is meaningful.

The model gave you something that looks right, but it will always be your job to determine if it is right.

Just Because AI Did It Does Not Mean It Was Right

This is the constraint we need to reintroduce into engineering. We are increasingly building systems on top of outputs that have not been validated. We accept generated code because it compiles and suggested architectures because they look clean. We accept graphs because they are well-structured.

But structure is not proof.

These systems are pattern generators trained on human preference. They produce outputs that resemble what we reward. That resemblance is powerful, but it is not verification. If anything, it should increase our burden of proof, not decrease it.

Engineering is not accepting an answer because it looks correct. It is demonstrating that it is correct.

Cite Your Sources

There is a simple practice that counters most of this behavior: cite your sources. Not as an academic exercise, but as a forcing function. When you trace an answer back to its origin, you are forced to understand the assumptions behind it.

Where did this come from?

Why does this approach work?

Under what conditions would it fail?

Those questions break the feedback loop by shifting you from agreement to understanding. And they reintroduce the friction that these systems are designed to remove.

Final Thought

AI psychosis is not about people losing control. It is about people trusting too early.

These systems are designed to agree with you. They are designed to structure your thinking in ways that feel clear and complete. That is why they are useful. It is also why they are dangerous.

AI is not replacing thinking, it is amplifying it.

And if you don’t understand what it is doing, you are not learning. You are just being reinforced.

References

1. Vaswani et al. — Attention Is All You Need

   https://arxiv.org/abs/1706.03762

2. Recent work on model preference optimization and reasoning behavior

   https://arxiv.org/pdf/2503.17473

3. video discussing AI psychosis, trendslop, and delegation of thinking

Across AI, data, and software engineering, I hear three different answers to the same question. AI engineers push for larger, denser, smarter models; data engineers push for better semantic modeling and ontologies; and software engineers push for better agents, tools, and orchestration frameworks. Each of these perspectives is directionally correct, but they are incomplete in isolation. The shared assumption underneath all three is that improving one layer will fix the system, but it won’t. The failure modes we are seeing (e.g. hallucinations, data democratization, brittle agents) are not isolated issues. They are symptoms of a deeper problem: we are building AI systems without treating them as distributed systems.

I am going to talk about distributed systems in the context of Data Intensive Applications (this also happens to be my favorite book). Distributed systems have always forced us to confront two fundamental challenges: how data behaves across boundaries, and how control is enforced across those boundaries. This is not new, the industry has spent decades formalizing these problems through concepts like ACID guarantees, the CAP theorem, and eventual consistency. These were not academic exercises—they emerged from real failures in systems that tried to scale without acknowledging trade-offs. If you look at foundational work like Designing Data-Intensive Applications, it becomes clear that once a system becomes distributed, correctness is no longer implicit. It must be engineered through constraints, contracts, and control planes.

What is striking is how little of this thinking is applied to AI systems today. Agents are introduced as if they are simply another application layer, when in reality they introduce non-determinism, external dependencies, and stateful interactions across multiple systems. These are defining characteristics of distributed systems, yet we continue to design them as if they were monolithic services.

The current approach to agents reinforces this mistake. Most implementations rely on agent harnesses, workflows, or DAG-based orchestration. These structures assume that execution can be predefined, that state transitions are predictable, and that inputs can be bounded. However, that assumption breaks down immediately in practice. AI systems are event-driven, not strictly sequential (and thus is why we have a context engineering problem-link. We need the previous history to inform the agent’s future actions and we don’t manage that context growth). They operate across sessions, not single requests. They depend on data they do not own that is retrieved from systems with independent lifecycles. This is why frameworks built around linear workflows feel brittle. They are modeling a deterministic system on top of a probabilistic one. The mismatch is architectural, not incidental.

This is also why attempts to “fix” agents with better prompting or more structured workflows consistently fall short. They are addressing symptoms at the orchestration layer without addressing the underlying distributed nature of the system.

The first place this architectural mismatch becomes visible is in how we handle data.

When Zhamak Dehghani introduced data mesh in her essays on Martin Fowler (see How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh and Data Mesh Principles and Logical Architecture), the core idea was not decentralization for its own sake. It was an attempt to apply distributed systems thinking to analytical data. Instead of centralizing all data into a single warehouse, data mesh proposed domain ownership, decentralized storage, and a control plane for governance and access. In other words, it asked whether data should be treated like microservices, with clearly defined ownership and interfaces rather than a shared, global resource.

This idea did not become the dominant pattern. Centralized data platforms remained the default, largely because they simplify validation and governance. It is easier to ensure data quality when everything lives in one place, and most organizations are structured around centralized data teams that reinforce this model. Even large-scale examples, such as internal data platform evolutions discussed by companies like PayPal, show how difficult it is to operationalize distributed ownership in practice. The industry chose centralization not because it was conceptually superior, but because it was operationally simpler.

The rise of transformer-based AI systems reinforced this trend. Scaling laws demonstrated that larger models trained on more data produced better results, leading to further centralization of both data and compute. This created a mental model that equates intelligence with aggregation: more data in one place leads to better outcomes. However, this assumption holds primarily in training, not for execution. During inference and especially in agent-based systems the problem shifts from learning patterns to validating actions against real-world data.

Agents do not need access to all data. They need access to the correct data, scoped to the task and the user, with clear boundaries around what can and cannot be retrieved. This is fundamentally different from the requirements of model training, yet the architecture of many AI systems still reflects a training-centric mindset.

This is why current patterns like RAG, GraphRAG, and MCP feel incomplete.* They attempt to improve retrieval, structure, or tool access, but they do not solve the underlying question of control. Who decides what data is accessible? Under what conditions? And how are those conditions enforced independently of the model? Without clear answers to these questions, these patterns become thin layers over fundamentally over-permissioned systems.

*MCP’s auth model is only safe if it is user-scoped. The moment you abstract the user out, you’ve recreated over-permissioned service accounts for probabilistic systems.

The common anti-pattern of placing a tool interface directly in front of a database and allowing an agent to query it illustrates this gap. It assumes that retrieval can be safely delegated to the model, rather than constrained by an external system. This is not a new mistake; it mirrors early distributed systems failures where implicit trust in components led to cascading failures.

The issue becomes even more pronounced when we consider how AI agents are permissioned.

In traditional cloud systems, service accounts are often granted broad permissions because their behavior is deterministic. A CI/CD pipeline or infrastructure provisioning tool operates within a narrow, predictable scope. Even when over-permissioned, the risk is bounded by the system’s design. AI agents do not share this property. They are probabilistic, context-sensitive, and capable of generating novel actions based on input. Treating them like service accounts ignores this fundamental difference.

A more accurate model is to treat AI

agents as operating on behalf of a user within a constrained system. Their permissions should be derived from user context, scoped to specific tasks, and enforced through external control mechanisms. This aligns with principles already used in distributed systems for managing access across services, but it requires a shift away from agent-centric design toward system-centric design.

This is where the original ideas behind data mesh become relevant again—not as a data architecture trend, but as a control paradigm. A distributed data model with a well-defined control plane allows systems to enforce access, ownership, and governance independently of the application layer. In the context of AI, this means agents interact with curated, scoped views of data rather than raw, global datasets. It also means that access decisions are made by the system, not inferred by the model.

The following diagram illustrates this shift:

In this model, the agent does not directly access data sources. All interactions are mediated through a control plane that enforces access policies and scopes the data returned. This is the same principle that allows distributed systems to maintain consistency and security across services, applied to AI systems.

The consequences of ignoring this model are not theoretical. They are already emerging in the form of privacy risks. When an AI system is granted broad access to internal data, it does not need to be “hacked” in the traditional sense to expose sensitive information. It only needs to be prompted in a way that causes it to retrieve and surface data that exists within its accessible context. Incidents like large-scale breaches (such as AT&T data breach 2024) demonstrate how damaging data exposure can be, even without sophisticated attacks. AI systems amplify this risk by increasing the surface area through which data can be accessed and interpreted.

This is why the failure point for AI systems is likely to be privacy, not capability. The issue is not whether models can produce accurate outputs, but whether the systems surrounding them can enforce appropriate boundaries on the data they access and expose.

AI is here to stay. The value it provides is real, and the momentum behind it is not going away. What is not guaranteed is that the systems we are building today will survive in their current form. If we continue to treat AI as an application-layer enhancement rather than a distributed system, we will repeat the same failures that led to decades of research and engineering in distributed systems.

The path forward is not to abandon AI, but to build it with the same rigor we apply to any distributed system. That means designing for data locality, enforcing access through control planes, and treating agents as components within a larger system rather than as self-contained solutions.

If we do not define these boundaries explicitly, they will be defined implicitly by the behavior of the system. And that is where both reliability and privacy begin to break down.
My Courses

Learn Go

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• AI Reliability Engineering

• Data as an AI Guardrail

• I Hate the Term Context Engineering

• The Difference Between Installing an Agent and Building One

• Agents Are Not Interns

Upcoming Events or Streams

I’m also active in the Utah tech community. If you’re local, come build with us. Otherwise catch us on Twitch

Utah Data Engineering Meetup

MLOps Utah Meetup

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.

Latest episode:

We are not approaching AGI. We are hyperscaling reinforcement learning from human feedback (RLHF), which means we are encoding human preference at scale. AI doesn’t discover truth, it just learns what we reward.

Transformer + Reinforcement Learning Is The Whole Story

Most people think modern AI systems are breakthroughs in intelligence, but they’re not. What we are actually looking at is the combination of transformer architectures1 and reinforcement learning2. The transformer predicts what comes next, and reinforcement learning decides which of those predictions get preferred.

Reinforcement learning is the important part here because when it learns from human feedback is not optimizing for correctness, it is optimizing for what humans approve of. It is learning what gets rewarded.

If you strip away the abstraction, that is what we are calling “alignment.”

Reinforcement Learning Was Always About Behavior

This is not a new idea. Reinforcement learning has always been about encoding behavior under a reward function3. You take an action, you observe an outcome, and you update your policy based on whether that outcome was rewarded or penalized. Over time, the system does not learn truth, it learns which actions lead to higher reward.

This is why reinforcement learning shows up in robotics and control systems; It is trying to behave effectively within the world it operates in.

What changed with RLHF is not the method, rather it is the source of the reward.

RLHF Replaced Math With Humans

Instead of defining reward functions mathematically, we now define them through human judgment. A person looks at two outputs and decides which one is better. That decision becomes the signal the model learns from. At scale, those decisions shape the behavior of the system.

That sounds harmless until you realize what humans actually reward -- we reward confidence, coherence, and answers that feel complete. We do not, however, reward correctness.

And that shows up directly in the behavior of these systems. They generate responses that feel right. They agree with you and they complete your thinking before you have fully reasoned it. They give you something that sounds finished, even when it is not.

That is why interacting with these systems feels so smooth. It is also why they are so easy to trust.

We Are Scaling Preference, Not Intelligence

Once you understand that RLHF is the core mechanism, the trajectory becomes clear. These systems are not converging on general intelligence. They are converging on better alignment to human expectations. That means we are scaling human judgment, bias, and preference at massive throughput.

human judgment

human bias

human preference

at massive throughput.

And those preferences are not neutral. They are shaped by who is providing the feedback and what the system is being optimized for. Right now, that is largely Western, commercial, and productivity-driven. That is why these systems are so good at coding, automation, and business workflows. Not because those are the most intelligent problems, but because those are the most rewarded ones.

This Is Not AGI

There is a persistent idea that if a system can understand all language4, it can achieve general intelligence. The assumption is that all knowledge is encoded in language, so mastering language is equivalent to mastering intelligence.

That assumption breaks down quickly.

Language is not the source of knowledge. It is a representation of it. We generate language from observation, experimentation, and interaction with the physical world. The story of Isaac Newton observing gravity is simplified, but it captures the point. Insight comes from interacting with reality, not just describing it5.

These systems do not interact with reality or test hypotheses or verify outcomes. They predict tokens based on patterns and feedback.

Adding images or audio does not change that. It adds more input modalities, but it does not give the system grounded understanding.

AI Is Just Gaslighting You

So the sycophantic product that feels like it has unlimited potential is just a decision system with a large dictionary. We want to see this as the next evolution of the, but it the end it is just a model that gaslights you based on preferences and predictions.

When I say AI is good at gaslighting you, I don’t mean that it is malicious. I mean that it is structurally optimized to reinforce your perspective. It meets you at your current level of understanding and reflects it back in a more polished form.

If you are right, that is incredibly powerful.

If you are wrong, it doubles down on the mistake.

And because the output is coherent and confident, it becomes harder to question. The system is not just generating answers. It is shaping your confidence in those answers.

That is the part we are not accounting for.

This Is Why It Feels Like Productivity

It feels like we are more productive because we removed friction of time.We can move through problems without the same pauses we used to have. But we did not remove the need for correctness. We just moved where the work happens.

Instead of thinking deeply up front, we are validating after the fact. Instead of writing carefully, we are debugging more often. Instead of building systems deliberately, we are iterating on outputs that may or may not be grounded.

And most teams are not measuring that shift. They are measuring speed, not reliability.

Agents Didn’t Change This. They Exposed It.

Agents feel like a step toward autonomy, but they are really just this same system extended into action. You give them context, they produce behavior. That behavior is still shaped by reinforcement learning and human feedback. What changed is that we are now letting those behaviors operate systems.

That is where things start to break.

Because we removed the human in the loop, but we did not replace it with a system that enforces correctness. We replaced it with a model that is optimized for agreement.

The Problem Is Not the Model. It’s How We’re Using It.

If these systems are fundamentally preference engines, then the way we integrate them matters more than the models themselves. We should be treating outputs as untrusted input. We should be validating behavior explicitly. We should be measuring reliability instead of speed.

Instead, we are doing the opposite.

We are trusting outputs, scaling agents, and shipping systems that feel correct instead of proving that they actually are.

Final Thought

AI is incredibly useful. I use it every day because it enables real work. But it is not discovering truth. It is learning what we reward.

And right now, what we are rewarding is confidence, agreement, and speed.

My Courses

Learn Go

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• AI Reliability Engineering

• Data as an AI Guardrail

• I Hate the Term Context Engineering

• The Difference Between Installing an Agent and Building One

• Agents Are Not Interns

Upcoming Events or Streams

I’m also active in the Utah tech community. If you’re local, come build with us. Otherwise catch us on Twitch

Utah Data Engineering Meetup

MLOps Utah Meetup

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.

Latest episode:

For a long time, self-hosting with llama.cpp usually meant picking one model and living with it. You hardcoded a single GGUF into llama-server, tuned the flags, and restarted the service whenever you wanted to switch workloads. That was fine for experiments, but it was a bad fit for a machine that needed to do real work.

Recent llama.cpp router support changes that. With --models-preset, you can register multiple models in one preset file, expose them behind one stable OpenAI-compatible endpoint, and route requests using the "model" field in /v1/chat/completions. The router handles loading and unloading models as needed instead of forcing you to rebuild the service around a single model.

That means one local box can serve a coding model, a reasoning model, and a vision model without turning model switching into manual maintenance. You are still constrained by VRAM and RAM, but the mental model is better now: not “five models in VRAM,” but “one inference router that can manage many models cleanly.”

As you may know, I’ve built my own host AI Agent system called PedroGPT. Eventually, I hit the very wall I described above and needed a better solution. I needed different models for different things, one for coding, a reasoning model for analysis, and a vision model when I need to inspect images. I want the same box to serve all of them without me SSHing in, editing a unit file, and then bouncing the service every time I change tasks. The recent llama.cpp router support solved that problem for me.

This post walks through what changed, how to configure it, and why this matters if you are trying to build a real local AI box instead of a one-model demo.

The Old Way: One Hardcoded Model

The original setup was exactly what most people start with: one systemd service, one model, and one startup command.

[Service]
EnvironmentFile=/etc/llama-server.env
ExecStart=/opt/llama.cpp/build/bin/llama-server \\\\
    --host 0.0.0.0 \\\\
    --port 8080 \\\\
    --hf-repo unsloth/gpt-oss-20b-GGUF \\\\
    --hf-file gpt-oss-20b-Q4_K_M.gguf \\\\
    --ctx-size 8192 \\\\
    --n-gpu-layers -1 \\\\
    --parallel 4 \\\\
    --jinja \\\\
    --no-webui \\\\
    --metrics

This is simple, and honestly, it is still a good starting point if you only need one model. The problem is operational, not conceptual. If you want to switch from a general-purpose model to a coding model, you are editing config, restarting the service, waiting for the model to load, and interrupting anything that was already using the endpoint. That’s fine if you’re just tinkering, but it gets super annoying for daily use. It is a bad fit for a machine that is supposed to serve multiple kinds of workloads.

What Changed in llama.cpp

The big change is router mode with model presets. Current llama.cpp documentation describes launching llama-server in a mode that can dynamically load and unload models, with requests routed based on the requested model name. You can define those models in an .ini file with --models-preset, set global defaults in [*], and then override settings per model. The router can autoload a model when a request arrives, and preset-specific options such as load-on-startup and stop-timeout let you control startup behavior and unload timing.

That matters because it changes the unit of operation. You are no longer running “a model server.” You are running a local inference router that can present multiple models through one stable endpoint. Your clients keep talking OpenAI-style JSON, the difference is now they just set "model" to the preset name they want.

My Hardware

This is running on pedrogpt, which has:

• NVIDIA RTX 5090 with 32 GB VRAM

• 64 GB system RAM

• local model storage under /opt/models/

That hardware profile is the reason this setup is interesting. Dense models in the 20B to 32B range fit comfortably enough to be practical. Larger Mixture-of-Experts models do not fit cleanly in VRAM, but they become usable if you are careful about where weights live and what gets offloaded.

The Model Roster

Instead of hardcoding one model in the systemd service, I define a roster of models in a preset file.

version = 1

[*]
n-gpu-layers = -1
ctx-size = 8192
jinja = true
stop-timeout = 30

[gpt-oss-20b]
model = /opt/models/gpt-oss-20b/gpt-oss-20b-Q4_K_M.gguf
ctx-size = 16384
load-on-startup = true

[nemotron-3-super-120b]
model = /opt/models/nemotron-3-super-120b/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf
ctx-size = 16384
temp = 0.6
top-p = 0.95

[qwen3-next-80b]
model = /opt/models/qwen3-next-80b/Qwen3-Next-80B-A3B-Instruct-UD-Q4_K_XL.gguf
n-gpu-layers = 99
ctx-size = 65536
flash-attn = true

[qwen3-coder-30b]
model = /opt/models/qwen3-coder-30b/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
ctx-size = 32768

[qwen2.5-vl-32b]
model = /opt/models/qwen2.5-vl-32b/qwen2.5-vl-32b-instruct-q4_k_m.gguf
ctx-size = 8192

This is where the setup becomes sane. The [*] section gives me shared defaults. Each named section becomes a model identifier that callers can use in API requests. The default reasoning model is then loaded at startup so the box comes up hot instead of idle.

One subtle but important detail from the docs: router-controlled arguments can be overridden or ignored when a model is loaded. That means you should think of host, port, and router-level behavior as belonging to the router, not to any individual model preset. The preset file is for model-specific runtime configuration, not for redefining the entire server.

The New systemd Setup

I like keeping the base unit file clean, so I use a systemd drop-in instead of rewriting the service.

[Service]
ExecStart=
ExecStart=/opt/llama.cpp/build/bin/llama-server \\\\
    --host 0.0.0.0 \\\\
    --port ${PORT} \\\\
    --models-preset /opt/llama.cpp/presets/all-models.ini \\\\
    --models-max 1 \\\\
    --parallel ${N_PARALLEL} \\\\
    --jinja \\\\
    --no-webui \\\\
    --metrics \\\\
    -ot ".ffn_.*_exps.=CPU" \\\\
    --flash-attn on

The blank ExecStart= line clears the original command. Then the new command launches the router with the preset file. The key flag here is --models-preset. That is what turns this from a one-model server into a local multi-model router.

I also cap the router with --models-max 1 because I do not want this box trying to keep multiple heavyweight models resident at once. My goal is flexibility, not pretending 32 GB of VRAM is infinite.

How Switching Works

From the caller’s perspective, the entire experience comes down to one field:

curl <http://pedrogpt:8080/v1/chat/completions> \\\\
  -H "Content-Type: application/json" \\\\
  -d '{
    "model": "qwen3-coder-30b",
    "messages": [
      {"role": "user", "content": "Write a Go HTTP server"}
    ],
    "max_tokens": 500
  }'

That is the part people should pay attention to. The endpoint and the request shape stay the same. The client does not need to know how the model is loaded, where the GGUF lives, or whether another model was loaded five seconds ago. It just asks for a named model.

That is the operational win. You decouple the client from the deployment details.

Why MoE Models Make This More Practical

Three of the models in this setup are Mixture-of-Experts models. That matters because MoE changes what “too big for local inference” actually means. A model may have a huge total parameter count, but only a subset of experts are active on a given forward pass. In practice, that gives you room to do something useful on consumer hardware that would look impossible if you were only thinking in terms of total parameters.

Here is the rough strategy:

Model Type Approx size Strategy GPT-OSS 20B dense 11 GB all GPU Nemotron-3-Super 120B-A12B MoE ~79 GB GPU attention + RAM experts Qwen3-Next 80B-A3B MoE ~43 GB GPU attention + RAM experts Qwen3-Coder 30B-A3B MoE 18 GB mostly/all GPU Qwen2.5-VL 32B dense 19 GB all GPU

The takeaway is not that everyone should run a 120B model at home, it’s that that routing plus careful offload rules lets one local box cover a much wider range of workloads than people assume.

The Tensor Offloading Trick

The most interesting part of this setup is the tensor routing:

-ot ".ffn_.*_exps.=CPU"

That regex sends expert FFN tensors to CPU-backed memory while leaving the parts that benefit most from GPU acceleration on the RTX 5090. For MoE models, that is the difference between “this model is too large” and “this model is slow but practical.” You are deciding which tensors deserve VRAM and which ones can tolerate the latency of system RAM over PCIe rather than shoving the entire model into VRAM.

This is also where local hosting stops being just “download model, run binary.” At this point you are doing systems work. You are budgeting memory, thinking about latency paths, and deciding which workloads are good enough for your box. That is the part I wish more local AI tutorials explained.

Multi-Part GGUFs Are Not The Scary Part

One nice quality-of-life detail is that llama.cpp handles multi-part GGUFs natively. If a model ships as shards, you point the preset to the first file and llama.cpp reads the rest.

That means the operational complexity is not “how do I merge these files.” The real complexity is choosing the right model and the right quantization for your hardware.

Deployment Pattern

I keep deployment boring on purpose:

1. Copy updated .ini preset files to the machine

2. Write or update the systemd drop-in

3. Run systemctl daemon-reload

4. Restart llama-server

Models are downloaded ahead of time into /opt/models/ so the service is not trying to pull from Hugging Face at runtime. I strongly recommend this. Local inference stops feeling local very quickly if your server has to fetch model artifacts on startup. Pre-download the files, use absolute paths, and treat the machine like infrastructure instead of like a notebook experiment.

What Clients Need to Change

The short answer is almost nothing, actually.

If you already have clients talking to an OpenAI-compatible endpoint, the only real change is the model identifier. Instead of pointing to one hardcoded model alias, the client sets "model" to one of the names from the preset file. That is it. Request format stays the same, and the response format stays mostly the same.

That is why I like this pattern so much. It upgrades the backend without forcing a redesign on every caller.

One Correction I Would Make When Operating This

I would not rely too heavily on /v1/models as your operational source of truth in router mode. The llama.cpp docs distinguish between the OpenAI-compatible model info endpoint and the router-oriented /models endpoints used to inspect available models and load state. If you are debugging the router, /models is the more faithful mental model. If you are serving OpenAI-style clients, /v1/chat/completions is still the interface they care about.

That distinction is small, but it matters when you are writing an operations guide.

Why This Actually Matters

The real shift here is that local inference can now look more like an actual shared service. One machine can expose a reasoning model, a coding model, and a vision model through one stable endpoint without turning model switching into a manual maintenance task.

That is a huge improvement for anyone building home lab AI, internal tools, or small-team local infrastructure. You do not have to pretend one model is good at everything or burn a separate service for every use case. You can route by workload, keep your clients simple, and make the hardware earn its keep.

This is also the more honest way to talk about self-hosting. Hosting your own models is not just about avoiding API bills. It is about owning the operational layer: knowing what is loaded, knowing what your hardware can do, and designing the system around actual workloads instead of model hype.

Final Take

If you are still running one hardcoded model behind llama-server, that setup is fine for a first step. But if your local box is starting to serve multiple jobs, router mode is the better pattern. --models-preset gives you a clean way to register many models, assign sane defaults, and switch between them with the same "model" field your clients already know how to send.

That is the real recent change: self-hosting with llama.cpp no longer has to mean choosing one model and getting stuck with it. Now it can mean building a local inference router that behaves like infrastructure.

Links and references

• llama.cpp server documentation: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

• llama.cpp repository: https://github.com/ggml-org/llama.cpp

• Hugging Face GGUF docs and model hosting ecosystem: https://huggingface.co/docs/hub/gguf

• Qwen models on Hugging Face: https://huggingface.co/Qwen

• NVIDIA Nemotron models on Hugging Face: https://huggingface.co/nvidia

My Courses

Learn Go

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• AI Reliability Engineering

• Data as an AI Guardrail

• I Hate the Term Context Engineering

• The Difference Between Installing an Agent and Building One

• Agents Are Not Interns

Upcoming Events or Streams

I’m also active in the Utah tech community. If you’re local, come build with us. Otherwise catch us on Twitch

Utah Data Engineering Meetup

MLOps Utah Meetup

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.

Latest episode:

Things are breaking more often. Not in catastrophic, headline-grabbing ways, but in smaller, more frequent degradations that chip away at reliability.

When you look at real dashboards(like Claude Status Page or GitHub Status) you don’t see perfection, you see patterns: partial outages, degraded performance, intermittent failures over the last 30 days. The easy narrative is to blame AI, but that misses the point. AI didn’t introduce failure into our systems. Failure has literally always been there. What changed is how rigorously we engineer around it. AI didn’t break your platform, you just skipped engineering.

AI Didn’t Break This

We are shipping faster than we ever have before, and AI is a huge part of that. It has removed friction from the most time-consuming parts of development, turning hours of implementation into minutes, which honestly is a net positive. However, speed doesn’t eliminate responsibility. Rather, it amplifies the consequences of skipping it. At the behest of business pressure and the need to move quickly, we’ve started to trade engineering discipline for velocity. Now, AI is making that tradeoff easier to justify. And when you remove discipline from a system designed to handle failure, you don’t get innovation—you get instability. Now we are seeing early signals of this even at the infrastructure level, with reports like AWS outages caused by AI tools pointing to AI tooling influencing real system failures.

AI Is Gaslighting You

The deeper issue is that AI is optimized to complete tasks, not to guarantee correctness. It will produce outputs that look valid, feel complete, and even include convincing explanations of why they work, but those explanations are not validation—they are generated text. When you ask an AI system if something worked, it will almost always say yes because its objective is just to finish the task. It does not care about proving correctness under real-world conditions. This creates a dangerous illusion: systems appear reliable because they return answers, not because they behave correctly. That illusion is reinforced by agent harnesses likeretries, tool calls, orchestration layers that smooth over failures and present a clean result. What you end up with is not reliability, but rather failure with better UX. If you’ve seen this behavior in the wild, this clip is a perfect example of the mindset we’re normalizing:

You Need a System Before You Let AI Build One

We already know how to solve this problem because we’ve solved it before. Test-driven development taught us to define expected outcomes before writing code, ensuring that functions behave correctly under known conditions. Site Reliability Engineering extended that principle to entire systems by introducing SLAs, monitoring, and failure-aware design. Practices like canary deploys, blue-green deployments, regression testing, and chaos engineering exist to validate behavior at scale, not just at the function level. These frameworks were built to allow imperfect humans to safely ship to production. AI does not change that requirement and itanything, it raises the bar. The question is no longer “does this code run,” but “does this system behave correctly under stress, at scale, and over time?” That question cannot be answered by AI. It must be answered by the systems you build around it.

There Is No Excuse Anymore

If AI is accelerating development, then it is also freeing up time. Time that used to be spent writing boilerplate, debugging syntax, or wiring basic logic can now be reinvested into validation and system design. That means we have fewer excuses than ever for shipping incomplete engineering. We can generate exhaustive unit tests, simulate user behavior with synthetic traffic, continuously validate contracts through integration testing, and enforce SLAs through monitoring and alerting. We can even use AI itself to stress-test our systems by creating agents that behave like users and push infrastructure to its limits. The tooling is no longer the constraint. The only constraint is whether we choose to use it.

This is also where the idea of WTFs per minute becomes more relevant than ever (https://commadot.com/wtf-per-minute/). Good systems are not just correct, they are understandable. You should be able to trace behavior, explain outcomes, and debug failures without guesswork. AI-generated code increases that responsibility. If you cannot explain why your system works, then you do not have a reliable system. You have a system that hasn’t failed yet.

The Day of the Web Dev Is Over

We are not just shipping features anymore. We are not just writing code.

We are engineers building systems. It’s time we all start acting like it.
My Courses

Learn Go

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• AI Reliability Engineering

• Data as an AI Guardrail

• I Hate the Term Context Engineering

• The Difference Between Installing an Agent and Building One

• Agents Are Not Interns

Upcoming Events or Streams

I’m also active in the Utah tech community. If you’re local, come build with us. Otherwise catch us on Twitch

Utah Data Engineering Meetup

MLOps Utah Meetup

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.

Latest episode:

Not because Claude Code is bad, but because I wanted my coding assistant to live on hardware I already own, on my own network, reachable from any machine on my tailnet, without API bills and without a cloud dependency hiding inside the most important part of the workflow.

That was the real goal. Not “try local AI” or “stand up a demo.” I wanted a coding stack I could actually use day to day.

My first attempt was exo. Turns out it was the wrong tool for this job. Exo made for a fun experiment, but my DGX Sparks are NVIDIA machines, and what I needed was a CUDA-native distributed inference stack that could serve a serious coding model over an OpenAI-compatible API. So I stopped trying to make the wrong abstraction fit the wrong hardware.

This is the stack that finally worked: **MiniMax 2.5 running across two DGX Sparks with Ray + vLLM**, exposed over my tailnet so I can hit it from any machine.

And yes, this is the setup that finally let me get rid of Claude Code.

The Goal

I wanted five things:

1. A strong open source coding model

2. Running locally on hardware I already own

3. Distributed across both DGX Sparks

4. Reachable from anywhere on my Tailnet

5. Good enough to replace Claude Code for real work

That last part matters.

A lot of people say they Opus 4.6 is the best. I hear it all the time in the echo chamber of working in AI. But I think AI is as only as good as the user, so since I have the compute, and the access via Tailscale I should use it to maximize my coding output while minimizing Token Cost. I did not want a local toy, I wanted a local replacement.

The Hardware

Here is the setup:

Machine Role OS DGX Spark #1 Ray head node + vLLM serve Ubuntu Linux DGX Spark #2 Ray worker node Ubuntu Linux Any laptop or desktop on my tailnet OpenCode client macOS / Linux / whatever

The Sparks do the inference. The client machine does not matter much anymore, which is exactly the point.

The nodes are connected locally for cluster traffic, and I access the API remotely over Tailscale. That’s important to note; the cluster networking and the remote client path are not the same thing. Sparks have 200 GBPS cable coupling them. This is how I create the hardware cluster and let the 2 Sparks act as one machine.

Why MiniMax 2.5

MiniMax 2.5 was the model I wanted from the beginning.

I was looking for an open model that was actually competitive for coding, not just “pretty good for local.” I wanted something that had enough weight behind it to justify replacing a paid hosted tool, and I wanted something that could survive the constraints of real hardware.

That means model selection was not just about benchmark screenshots. It was about the intersection of:

• coding quality

• quantization availability

• memory footprint

• runtime support

• cluster support

• whether the model can be served reliably on the hardware in front of me

That last one is where a lot of local AI projects go sideways.

On paper, plenty of models look attractive. In practice, the winner is the one that both performs well and actually runs. For me, that ended up being QuantTrio/MiniMax-M2.5-AWQ.

The Real Constraint Was Not the Model,It Was the Stack.

This is the part I think people underestimate.

Getting a model to exist on Hugging Face is not the same thing as getting it to run cleanly on a two-node DGX Spark cluster.

The problem was not “is MiniMax 2.5 real?” or even “does vLLM know what MiniMax 2.5 is?”

The problem was all the ugly stuff in between:

• the right container version

• the right parser flags

• the right multi-node runtime

• the right interface selection

• the right cluster bootstrapping

• the right memory behavior during model load

• the right transport between both Spark nodes

That is why the final lesson here is not “use this one command.”

The final lesson is that the serving stack matters as much as the model.

Why Ray + vLLM Was the Right Answer

Ray + vLLM was an obvious choice.

It gave me:

• a CUDA-native inference path

• distributed execution across both nodes

• tensor parallelism across the Sparks

• an OpenAI-compatible API

• a stack that feels like infrastructure instead of a science project

This stopped being “Miriah is tinkering with local models” and started being “Miriah now has a local coding service on her own network.” That felt like a very different thing.

What Actually Made It Work

When you work with hardware there are a lot of unknowns. I started with the Nvidia recommendation for vllm on 2 Sparks, but after fighting the hardware for several days and using lots of Claude tokens I was pointed at this repo that just solved the problem, https://github.com/eugr/spark-vllm-docker

I ended up using the Spark-specific open source image and helper scripts because they handled the practical parts that kept breaking on me:

• discovering the right network configuration

• bootstrapping the Ray cluster correctly

• aligning both nodes on the same runtime assumptions

• giving me a repeatable path to serving the model

The unlock ended up being just the boring, operational truth that the best stack is the one that actually survives contact with your hardware.

The Stack That Finally Replaced Claude Code

This is the stack I am using now:

• Model: QuantTrio/MiniMax-M2.5-AWQ

• Runtime: vLLM

• Distributed execution: Ray

• Hardware: 2x DGX Spark

• Client: OpenCode

• Access path: Tailscale

This is the combination that’s giving me a coding assistant that runs on my own hardware, with no API bill, no usage limits, and no requirement that my best workflow live on somebody else’s product.

The Gotchas That Actually Mattered

There were a lot of little cuts along the way, but the highest-signal issues were these.

1. The cluster path mattered more than the model path

The hardest part was not choosing MiniMax 2.5. It was getting the distributed runtime stable across both Spark nodes.

2. The official-looking path was not the fastest path to success

The version that actually worked for me was the open source Spark image plus the repo scripts that handle detection and bootstrapping. That was faster than trying to force a cleaner narrative.

3. Multi-node local inference is mostly an infrastructure problem

Once you move beyond single-box local AI, the work starts to look a lot more like systems engineering: networking, runtime coordination, memory behavior, and repeatability.

4. “Local” only counts if the workflow stays local

For me, the entire point of this project was replacing the coding assistant, not just running a model. That is why OpenCode mattered. That is why Tailnet access mattered. That is why I kept going until this was something I could use from any machine.

The Result

I finally replaced Claude Code!

The hardware and models are both getting good fast. The open source clients are getting good fast. But the operational glue is still where most of the work lives. The difference between “I tried a local model” and “I replaced a paid coding tool” is not just intelligence. It is systems design. Ultimately, this project ended up being less about model hype, more about infrastructure.

And I am okay with that. Because now the coding assistant is mine, and if I give it my k8s credentials those creds do not leave my network. If I share a secret it is not accidentally it was not in chat history for ever. I am not sharing my IP.

Final Thought

I think this is where a lot of AI infrastructure is headed. Claude Code, this I use as my primary driver in my day job, is slower than my MiniMax cluster just because api latency. MiniMax writes great code. With OpenCode and a 120k context I can do an entire PR’s worth of work in a session - a pr is ~atomic commit as defined by me.

Not everybody needs a DGX Spark cluster in their house. But a lot more engineers are going to want the same outcome:

• strong open models

• private inference

• no recurring API dependency

• workflows that stay local

• infrastructure they can actually control

I use it everyday, from my laptop, my phone, anywhere. And that is the first time I can honestly say that about a local coding stack.

P.S. Here is my OpenCode config.

My Courses

Learn Go

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• AI Reliability Engineering

• Data as an AI Guardrail

• I Hate the Term Context Engineering

• The Difference Between Installing an Agent and Building One

• Agents Are Not Interns

Upcoming Events or Streams

I’m also active in the Utah tech community. If you’re local, come build with us. Otherwise catch us on Twitch

Utah Data Engineering Meetup

MLOps Utah Meetup

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.

Latest episode:

In the last post, I argued that most AI systems are not failing because of models , rather they’re failing because we are not engineering them like systems. Reliability doesn’t come from better prompts; it comes from constraints, feedback loops, and measurable behavior.

This post is the next layer: evals.

Once you accept that AI systems are probabilistic, you need a way to measure and shape that behavior. Evals are how we do that. Not as a final report, but as part of the system itself.

Evals Are Not Validation, They Are Development

The biggest mistake people make is treating evals as something you run after you build the system. That mindset assumes the system is already correct and you’re just verifying it. That is not how AI systems work.

When I build agents, I use evals to answer design questions: how should tools be exposed, how much schema is required, and whether behavior is consistent across models. The eval is guiding the system into existence.

This is the same shift we made years ago with software testing. Unit tests stopped being something you wrote at the end and became part of how you design code. Evals play that role for AI systems. They are the mechanism that turns probabilistic behavior into something we can reason about.

You Are Not Testing Correctness — You Are Testing Consistency

A single successful run of an agent tells you almost nothing. Even multiple successful runs don’t prove reliability, because AI systems don’t produce fixed outputs. Instead, they produce distributions of behavior.

The real question is not “did it work?” but “does it behave the same way under the same conditions?” That’s what evals measure. By running the same test repeatedly, you start to see whether behavior stabilizes or drifts.

This is where reliability engineering comes in. A system that works once is not reliable. A system that produces the same outcome repeatedly under the same inputs is. Evals give you the data to distinguish between the two, and that distinction is what allows you to build systems instead of demos.

Table-Driven Evals Turn Behavior Into Something Measurable

The way I implement evals is intentionally simple: table-driven tests. Each test defines a prompt or template, an expected behavior, and a way to validate the result. Then I run that same test multiple times.

The key is repetition. I typically run each scenario around fifteen times, not to get a pass/fail result, but to observe the distribution of outcomes. That distribution is the signal. It tells me whether the system is stable or still drifting.

This approach mirrors how we test APIs or distributed systems. You define a contract, execute it repeatedly, and measure variance. The difference is that instead of testing deterministic outputs, we’re testing whether probabilistic systems converge on the same behavior.

Behavior Has a Shape And Evals Let You See It

Every combination of prompt, tool interface, and middleware produces a pattern of behavior. I think of this as a behavior arc. Not a single outcome, but a shape across repeated runs.

For example, one model might call the correct tool almost every time, while another does so inconsistently. That difference is not just about model quality; it reflects how well your system design supports the task.

Once you can see that shape, you can make decisions. You can choose models, refine prompts, or adjust tool interfaces based on actual data instead of intuition. Without evals, those decisions are guesses. With evals, they are grounded in measurable behavior.

Evals Answer System Design Questions, Not Just Model Questions

In practice, I use evals to answer questions about system architecture. When working with tool-calling agents and databases, the real uncertainty is what context the model needs to do so reliably.

For example, I tested variations like providing full schema inline, exposing schema as a tool, or providing no schema at all. I also tested whether the agent could generate reliable queries across different backends and levels of complexity. Each variation was run repeatedly to observe stability.

The result is not a single “best” configuration. It is a map of tradeoffs. Some approaches are more reliable but more expensive, while others are flexible but less consistent. Evals let you see those tradeoffs clearly, which is what enables real engineering decisions.

Reliability Is About Reducing Variance

This is the core connection to AI reliability engineering. Reliability is not about eliminating failure entirely; it is about reducing variance to a level where the system becomes predictable.

High variance means the system behaves differently each time, even under identical conditions. Low variance means the system produces consistent outcomes. Evals are the tool that lets you measure and reduce that variance.

Once you can measure variance, you can start engineering against it. You can add constraints, improve prompts, refine tool interfaces, or introduce fallback logic. Each of those changes should reduce variance, and evals let you verify that they actually do.

Evals Create a Feedback Loop You Can Engineer Against

The most important shift is thinking of evals as a continuous loop rather than a one-time step. You define a behavior, test it repeatedly, measure the outcome, and adjust the system. Then you run the eval again.

Over time, this loop shapes the system into something stable.

This is the same pattern we use in distributed systems and performance engineering. The difference is that instead of tuning latency or throughput, we are tuning behavior.

Middleware + Evals Is the Reliability Layer

Evals alone don’t make a system reliable. They tell you where it isn’t. The actual enforcement happens in middleware.

Middleware defines what the agent is allowed to do: which tools it can call, how data is accessed, and what structure responses must follow. Evals then verify that the agent respects those constraints under repeated runs.

Together, they form the reliability layer. Middleware enforces behavior, and evals measure whether that enforcement holds. Without middleware, evals just tell you things are broken. Without evals, middleware is unverified.

The Shift From Prompting to Engineering

Most AI systems today are still built through prompting and manual testing. That approach works for demos, but it does not produce reliable systems.

The shift is moving to engineering discipline: defining expected behavior, measuring it repeatedly, and iterating based on data. Evals are the mechanism that enables that shift.

This is why evals are foundational, not optional. They are how you turn a probabilistic model into a predictable component within a larger system.

SoyPete Tech Quick Take

If you are not running repeated evals on your agents, you are not measuring behavior — you are observing randomness.

Start small. Define a handful of test cases, run each multiple times, and measure consistency. Change one variable at a time and observe how the distribution shifts. That process alone will dramatically improve how you build.

Final Thought

AI didn’t remove the need for software engineering, it just made the gaps more visible.

Reliability comes from feedback loops, constraints, and measurement. Evals are the mechanism that closes that loop.

Not at the end of the system, but at the center of it.

My Courses

Learn Go

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• AI Reliability Engineering

• Data as an AI Guardrail

• I Hate the Term Context Engineering

• The Difference Between Installing an Agent and Building One

• Agents Are Not Interns

Upcoming Events or Streams

I’m also active in the Utah tech community. If you’re local, come build with us. Otherwise catch us on Twitch

Utah Data Engineering Meetup

MLOps Utah Meetup

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.

Latest episode:

AI systems are not reliable by default. They are probabilistic engines embedded inside software systems, and their behavior is fundamentally non-deterministic. If we want AI systems to behave reliably, the reliability must come from the engineering around the model, not from the model itself.

Right now the industry is repeating a mistake we have already made before. Instead of treating AI systems like production infrastructure, we are treating them like magical components that will somehow handle complexity on their own. Spoiler alert - they will not. If anything, AI systems demand more operational discipline, not less.

If you are building agentic systems today, whether you realize it or not, you are doing reliability engineering.

In other words: you are an SRE now.

We Already Solved This Problem Once

Several years ago I wrote a talk and article titled Introduction to Data Reliability Engineering (available on Medium here ). At the time, data teams were dealing with a common operational failure mode: pipelines broke constantly, dashboards were unreliable, and nobody really owned the stability of the system. Data infrastructure was treated like a collection of scripts rather than a production system.

The solution turned out to be simple, albeit culturally difficult. Data pipelines had to be treated like services. Once that shift happened, teams began applying the same operational discipline used in traditional infrastructure: observability, service level objectives, incident response, and operational ownership.

Today we are watching the exact same pattern emerge in AI systems. Agent frameworks are proliferating quickly, and organizations are rushing to build AI workflows; but in the process, reliability thinking has largely disappeared. Engineers talk about prompts and models and tools, but very few discussions focus on operational stability.

Instead, we hear a different kind of assumption: the belief that the model will somehow resolve complexity on its own.

That assumption is dangerously misleading.

Guardrails Have Hidden the Real Problem

One reason this misunderstanding exists is that modern frontier model APIs hide a significant amount of the underlying instability of language models. Most hosted LLM services include multiple layers of guardrails embedded directly in the API layer. These can include structured output enforcement, moderation filters, schema validation, retry logic, and tool-calling constraints.

Because of these layers, many engineers building AI systems rarely encounter the raw behavior of a language model. The system quietly absorbs many failures before they ever reach the application layer.

As a result, something subtle has happened: people have forgotten how language models actually behave.

Language models hallucinate. They do so frequently, not just in edge cases but as a natural consequence of probabilistic generation. That does not make them useless, but it does mean they cannot be treated as deterministic software components.

Reliable AI systems are therefore not built by pretending hallucination does not exist. They are built by designing systems that expect hallucination and manage it.

Frontier APIs made LLMs dramatically easier to use. At the same time, they made it easier to forget what those systems actually are.

Agent Systems Are Distributed Systems

Another challenge appears as soon as AI systems begin to perform real work. Modern agent architectures rarely consist of a single prompt and response. Instead, a single user request may trigger retrieval steps, tool calls, reasoning loops, validation passes, or background tasks.Once a workflow reaches that level of complexity, it begins to resemble something very familiar to systems engineers: a distributed system.

Agentic systems are distributed systems with a probabilistic component, which introduces an interesting cultural shift. For the last decade, cloud platforms have absorbed much of the complexity involved in building distributed infrastructure. Managed services handle scaling, orchestration, queueing, and failover automatically. That abstraction has been enormously valuable for developer productivity.

However, AI systems reintroduce distributed behavior into the application layer itself. Instead of the infrastructure handling orchestration, the workflow logic now lives inside the software the engineer writes. This means engineers must once again think about system behavior under failure conditions. Latency amplification, cascading failures, and inconsistent state are no longer theoretical concerns, now they are common patterns in agent workflows.

The Reliability Framework Still Applies

Fortunately, the industry already has a well-developed framework for thinking about system reliability. Google’s Site Reliability Engineering book (Beyer et al.) describes what are commonly known as the Four Golden Signals of system health: latency, traffic, errors, and saturation.

These signals translate surprisingly well to AI systems, although their interpretation changes slightly.

Latency in an AI system is not just the time required to produce a response. It may include model inference time, time to first token, streaming latency, tool execution time, and the completion time of the overall workflow. Traffic is similarly more complex because a single user request can expand into multiple model calls as an agent moves through its reasoning process.

Errors also look different. Agents rarely return explicit error messages. Instead they often return responses that appear valid but are structurally incorrect or incomplete. From a reliability perspective, these outcomes must still be treated as failures.

Finally, saturation occurs not only in infrastructure resources such as CPU or GPU capacity but also in the context window of the model itself. Excessive context can increase latency, reduce accuracy, and destabilize workflows. This intersection between reliability and context design is something I discussed previously in Why I Hate the Term Context Engineering.

Systems That Never Break Are Fragile

Reliability engineering has emphasized for a long time about an important philosophical shift: reliable systems are not systems that never fail.

Charity Majors, co-author of Database Reliability Engineering, often emphasizes this point in her work. She notes that systems designed to avoid all failure are often fragile because they are never tested under stress. Instead, reliable systems are designed to fail in controlled ways that minimize blast radius and recovery time.

As Majors writes in Database Reliability Engineering, systems that never fail are often fragile and when they go down it takes all hands on deck to get it live again. True reliability comes from designing systems that can break safely and recover quickly.

This mindset is especially important for AI systems because probabilistic components will inevitably behave unpredictably. The goal is not eliminating that behavior, but rather designing systems that remain stable despite it.

Toil and Operational Work

Another concept from reliability engineering that applies directly to AI systems is toil. Majors and Campbell define toil as manual, repetitive work that can be automated and scales with the growth of the service.

AI workflows introduce new kinds of toil. Engineers frequently find themselves debugging agent outputs, rewriting prompts repeatedly, re-running workflows, or manually correcting incorrect responses. These activities consume engineering time without improving the underlying system.

The reliability approach to this problem is the same one used in infrastructure operations: automate repetitive operational work wherever possible.

Why Homelabs Matter

One reason I enjoy running a homelab is that it forces me to confront these system realities directly. When systems run locally, the abstractions that cloud platforms provide disappear. You are forced to think about queues, networking, inference latency, resource limits, and failure modes.

If something breaks in a homelab, there is no platform team to escalate to. You must understand what the system is actually doing.

This mindset turns out to be exactly the mindset required to build reliable AI systems. Agent workflows behave much more like distributed systems than traditional web applications. When they fail, the root cause is rarely the model itself, it’s usually the surrounding system architecture.

Predictability Versus Reliability

Self-hosting models also exposes deeper layers of model behavior. When running models locally, engineers can observe elements such as logits, token probabilities, sampling behavior, and generation drift.

This visibility does not automatically make a system reliable. However, it does make the system more predictable. Predictability is the first step toward reliability engineering. Engineers cannot build stable systems around components they treat as complete black boxes.

A Pattern We Have Seen Before

Interestingly, this pattern is not new. I wrote about a similar dynamic in an earlier article, Optimizing Into Chaos: Why AI Agents Fail. In that post I argued that agent systems tend to optimize aggressively toward goals without strong boundaries. When the surrounding system lacks guardrails, those optimization loops naturally drift into unstable behavior.

The lesson is simple but important: the model is not the system. The system is everything around the model.

The AI Reliability Stack

A useful way to think about this architecture is to separate the model from the reliability layers surrounding it.

User Request
      ↓
Agent Workflow / Orchestrator
      ↓
Tool Layer (APIs, Databases, Services)
      ↓
LLM (Probabilistic Engine)
      ↓
Inference Infrastructure

Wrapped around all of this are the reliability layers:

• Observability

• Validation

• Retries

• SLOs

• Error Budgets

• Context Management

The model provides reasoning ability, but reliability emerges from the surrounding system design.

The Engineer’s Job Has Changed

AI has not removed engineering complexity. Instead, it has shifted where that complexity lives.

The role of the engineer is no longer just writing deterministic code. It now involves designing systems that contain probabilistic components and ensuring those systems behave predictably under failure conditions.

That responsibility includes architecture, workflow design, validation layers, retries, observability, and operational boundaries. Engineers must understand how their systems behave under stress because the model will not solve those problems automatically.

The Intelligence Is Still the Engineer

Language models are extremely powerful tools, but they are not intelligent systems in the traditional sense. They are statistical engines that generate tokens based on probability distributions learned during training.

The intelligence of the system does not come from the model.

It comes from the engineer designing the system around it.

And then we build AI systems that validate nothing.

That mismatch is the core problem facing agentic AI today. Not model capability. Not parameter count. Validation.

When people talk about hallucinations, they usually frame them as an AI failure. In practice, hallucinations are what happen when you ask a system trained to always answer to operate without a model of correctness. Large language models are rewarded for fluency, not truth. Guessing confidently is better than saying “I don’t know.” That training works remarkably well — until correctness matters.

As users, we instinctively compensate for this. When I use an LLM as a search engine, my next step is almost always the same: where did that come from? I ask for links. I ask for references. I ask for something external that lets me validate the claim. That instinct comes from the same place our data instincts come from: confidence without evidence is meaningless.

Autonomous agents don’t ask those follow-up questions. They act.

Once an AI system can call tools, chain actions, or trigger workflows, validation stops being optional. Yet most agentic systems today rely on system prompts as their primary guardrail. Long lists of rules. Capitalized MUSTs. Carefully worded constraints. Anyone who has built a real agent knows how fragile this is. Relevance decays. Context windows fill. Rules get ignored. Prompt-based guardrails don’t fail dramatically — they fail quietly.

I learned this the hard way while building agents without a framework — just raw, self-hosted models and tool calls. In one case, I had an agent whose job was simple: find directories. The tool it was supposed to call was named find_directories. The agent kept calling find_directory instead. Singular. One character off. It did this thirty-five times in a row, looping confidently, failing silently, and only stopping because I capped the iteration count. The model wasn’t “confused.” It was unvalidated. There was nothing in the system that could say, " This action violates the structure of the world you’re operating in.”

When prompts stop working, teams reach for data. They query a database, dump rows into context, and ask the model to reason. But raw data doesn’t carry meaning on its own. Rows don’t explain relationships. IDs don’t encode business rules. The model sees values, not semantics.

Data without semantics is just noise to an LLM.

This is where data engineering thinking becomes essential, because we have already solved this class of problem — just not for text.

“The data model for structured data is the ERD. The data model for unstructured data is the ontology.”

— Bill Inmon, Mastering Unstructured Data

That statement isn’t about AI. It’s about modeling. And that’s exactly why it matters here.

We don’t reason about warehouses because we like tables. We reason about them because schemas encode meaning. They define what exists, how things relate, and which states are invalid. Ontologies do the same thing for text. They are schemas for language. They don’t make language smarter — they make it checkable.

An ontology doesn’t just say that a dog is a mammal. It encodes that relationship so it can be traversed, queried, and validated. Meaning lives in the relationships, not the tokens. Once reasoning is explicit, it can be audited. When it’s implicit, all you have is confidence.

This difference becomes obvious when you give an agent a question it sounds like it should know. Ask an LLM: What was the largest-attended World’s Fair in history? It will almost always answer confidently. Sometimes it will even be right. But unless you validate that claim against a semantic model — attendance, year, location, event type — you have no way to know whether the answer is correct or just plausible. With an ontology like Wikidata, that same agent can make a claim, query the graph, realize it’s wrong, and correct itself. Not because it “thought harder,” but because it checked the structure of the data.

That’s the distinction most AI systems are missing.

What’s changed recently isn’t the theory — it’s the urgency. Agents don’t just describe information anymore. They act on it. Industry research is converging on the same conclusion from different directions: language models need semantic constraints to behave reliably. Whether it’s ontology-guided extraction, graph-constrained reasoning, ontology-augmented generation, or function-calling schemas, the pattern is consistent. Meaning must be modeled outside the model.

This is why ontologies work so well as AI guardrails. They allow deterministic validation in an otherwise probabilistic system. The same query returns the same answer every time. The reasoning path is visible. Violations can be explained instead of being silently corrected. That is not an AI trick — it’s data modeling applied to language.

Which is why hallucination is not an AI problem. It’s a data problem. We already know how to validate claims. We’ve been doing it in data pipelines for decades. We just haven’t applied the same rigor to text.

Agentic AI doesn’t need more clever prompts. It needs grounding. It needs semantic models that constrain what can be said, not just how it should sound. It needs validation mechanisms that operate independently of model confidence.

That work doesn’t belong exclusively to AI researchers. It belongs to data engineers — the people who already understand how fragile truth becomes without structure.

We don’t need to retrain models to fix hallucination.

We need to model meaning and enforce it.

In my recent talk I mentioned that “agents need more validation, not more prompts.” Here’s what that looks like in a real project.

This snippet is from Professor Pedro, where I’m wiring validation into the agent loop so the model can’t just confidently drift — it has to check claims against a semantic structure before it moves forward.

Link: https://github.com/Soypete/professor_pedro/pull/22/files#diff-3830a657693bcc87aecfe69a41169e1cee21c8f7e120de67d78418385d0d1f26

References

Ontology as Guardrail

• ODKE+: Ontology-Guided Knowledge Extraction — Apple, 2025

• Graph-Constrained Reasoning (GCR) — arXiv, 2024

Ontology as Context

• Ontology-Augmented Generation — Palantir Docs

• Building with AIP: Data Tools for RAG/OAG — Palantir Blog

Ontology Behind Tool Calls

• FNCTOD: Function Calling for Task-Oriented Dialogue — ACL 2024

• LLMs + KGs via Function Calling — CEUR Workshop

• Less is More: Function Calling on Edge — SIU

Foundational

• Mastering Unstructured Data — Bill Inmon

• The Ontology Pipeline — Jessica Talisman

• Relationships and Knowledge Systems — Jessica Talisman

• Ontologies—Some Perspectives — Jessica Talisman

• Controlled Vocabularies — Jessica Talisman

  ---

  My Courses

  Learn Go

  ---

  Stay Connected

  Want to stay updated on what I’m working on? Here’s where you can find me:

  • YouTube

  • Twitter / X

  • Bluesky

  • LinkedIn

  • Newsletter

  • Twitch

  • GitHub

  • Discord

  • LinkTree

  ---

  Newsletter Highlights

  Recent Posts You Might Have Missed

  • I Hate the Term Context Engineering

  • The Difference Between Installing an Agent and Building One
    Agents Are Not Interns

  • When a PedroCLI Stops Being a CLI

  • PedroCLI vs Clawd / Molt / etc

  Upcoming Events or Streams

  I’m also active in the Utah tech community. If you’re local, come build with us. Otherwise catch us on Twitch

  Utah Data Engineering Meetup

  MLOps Utah Meetup
  

  ---

  Latest Podcast Episode

  Deep dives on AI infrastructure, home labs, and production engineering.
  Latest episode:

We keep treating “context engineering” like prompt tweaking or a new job title, and that’s why agents keep failing in predictable ways. AI systems don’t bring context with them. They start empty, then probabilistically act on whatever we expose. The real work isn’t letting models “figure it out;” it’s deterministically engineering how scoped data, semantics, tools, and instructions enter the system so we can shape outcomes without handing the model all the information in the world. Context engineering at its core is systems design, and most people are doing it wrong because they’re focused on prompts instead of data.

I want to talk about context — not prompts, not tokens, not “agent magic,” but context.

And I’m going to start with an analogy I’m stealing from my friend Tod Hansmann, who dropped this gem at a meetup:

The difference between American cinema and Japanese cinema is context.

In American cinema, there’s an expectation that you come with a shared cultural background. Certain stories can only be told through certain lenses. You’re expected to already know how to read what’s happening.

In Japanese cinema, the assumption is the opposite: you come with no context.

That’s why the same story can be told with children, adults, witches, demons, or aliens — it doesn’t matter. The story is portable. The system doesn’t rely on the audience to bring the meaning with them.

And this is exactly where we are getting AI wrong.

We Expect AI to Bring Context — and That’s the Bug

We expect AI to know everything, and that expectation is wrong.

Go back to GPT-3 and GPT-3.5 — the pre–tool call era. All the model had was:

• The text you gave it

• Statistical patterns from training

• A next-token prediction loop

There was no validation or math or summation. There was no task execution. It could not do anything, it could only predict what came next based on patterns. Fast forward to today, and AI providers layered tool calls on top of that foundation. Now we have access to web search, plugins, code execution, andAPIs. This is what enabled “agents.”

Now when the model predicts a sequence that requires more information, it can go get it. And here’s where the American-cinema assumption sneaks in. We assume that because tool calls exist, the model is bringing context with it.

It isn’t.

The agent still starts with the same amount of context as before, but it now has the ability to fetch more, and that distinction matters more than people realize.

Multi-Bot and Plugin Failures Were Predictable

These systems failed because they had the entire world open to them.

They could link information they shouldn’t,

perform operations they shouldn’t, cChain tools in ways no one intended, and

act on data with no semantic constraints. That wasn’t an agent problem. It was problem solved by engineering, and that problem was a context problem.

In theory, context engineering could have prevented almost all of it (a little bit of programatic guardrails never hurt anybody, am I right?). But instead of refining what the system was allowed to see and do, we handed it everything and hoped it would figure it out.

That was never going to work.

The Prompt was not the Problem

This is also why you don’t hear much about prompt engineering anymore.

Prompt engineering was about dumping context upfront by explaining what you want done, encoding policy directly, and then hoping the model would get it.

That falls apart the moment agents enter the picture, and what replaced this process is people now call context engineering.

Context engineering decides what information is introduced and when, how long it persists, and what gets dropped between steps. Both tool calls and results are context, and so is injected data.

BUT...I hate the term “Context Engineering.”

I hate it because it sounds like a role. It isn’t. It’s a skill. You don’t hire an infrastructure engineer who doesn’t understand DevOps. Someone who can only SSH into prod and run bash scripts isn’t an SRE, they’re a sysadmin. DevOps is a skill, not a title.

Context engineering is the same thing. Every AI engineer needs this skill. And the reason I hate the term is because I don’t think we actually understand what it actually means yet.

The Real Issue: We Don’t Know How to Use Data In the Moment

“Oh Miriah I know how to use data, I just read it from the database!”

Wrong. Do you ever just give it to the user as-is? That is what I thought. As af developer, you use the data to present a set of information in a palatable way. That is context engineering.

Historically, engineers use data in two ways:

1. Analytics — analyzing what already happened

2. State — recording what already happened

Even machine learning training is retrospective. It’s always history layered on history. We are bad at using data during execution, which is why online inference never really worked the way people hoped. And that’s why I dislike how context engineering is framed today. People still think it’s about “making the prompt better.”

It isn’t.

It’s about how you introduce data — securely, reliably, and with meaning — into a live system. If I could start any company right now, it would be one that makes this easier because it’s genuinely hard.

System Prompts Are Products, Not Scratch Pads

System prompts should not be constantly rewritten. They are the product you ship as an AI engineer. Once they are live and out of active development, their structure should change very rarely. They should be designed intentionally, released deliberately, and left stable. When I say “stable,” I don’t mean static though. A system prompt can absolutely be a template. You can inject values at runtime or parameterize it. You can even swap in scoped instructions. That’s fine —everything inside a system prompt should be deterministic.

But that’s the boundary. What happens after that boundary is probabilistic. Context engineering exists to shape those probabilities, not to redefine behavior.

You don’t make the model deterministic, you make the inputs deterministic. That’s the contract.

More Instructions Don’t Fix Context — Timing Does

I see this failure mode constantly with coding agents like Claude Code.

When people want the agent to behave better, their instinct is always the same: add more. But adding more information does not solve the problem, it makes it worse. The mistake we make is putting everything into history.

The closer information is to the system prompt or the beginning of the conversation, the less likely it is to be used as the plan evolves.

These models — even reasoning models — can only hold so much context at once because they are prediction systems. No matter how many instructions you provide, some of them will not be used in the next prediction, and information will be selectively ignored for the purpose of finishing it’s current task. That’s why context engineering is never about more information,. it’s about the right information, at the right time, in the right place.

With coding agents, context engineering is literally typing into the prompt window. That’s what editors and assistants are doing when they surface a file, a diff, or a symbol — they’re saying “now is the moment this information matters.”

Compaction is also a context engineering technique. Compaction trades recent conversational detail for proximity to the original instructions. You lose history, but you don’t lose context engineering because context engineering is still the user’s job.

And this is exactly what background agents and fully autonomous workflows lack: a human deciding what information matters right now. That’s why those systems have to be engineered so carefully up front.

What Context Engineering Actually Is

Context engineering is the disciplined provision of:

• Tool calls

• Tool results

• Scoped supplemental data

• Semantic and relationship information

• Explicit task instructions

Raw data without semantics is useless. Bigger models don’t fix that. More autonomy doesn’t fix that.

https://x.com/karpathy/status/1937902205765607626?s=20

We already saw what happens when agents can do “anything.” Anything includes the things you absolutely did not want them to do. Context engineering is not about letting AI figure it out, it’s about refining its purpose.

From Magic to Software

Once you treat context engineering as structural management, agents stop looking like fragile demos and start looking like automation.

They become predictable, scoped, and auditable.

At that point, agents aren’t magic anymore — they’re software. Stop letting AI “figure it out,” and start engineering how data with meaning enters the system.

My Courses

Learn Go

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• The Difference Between Installing an Agent and Building One
  Agents Are Not Interns

• When a PedroCLI Stops Being a CLI

• PedroCLI vs Clawd / Molt / etc

• Have You Heard of Logits?

Upcoming Events or Streams

I’m also active in the Utah tech community. If you’re local, come build with us. Otherwise catch us on Twitch

Utah Data Engineering Meetup

MLOps Utah Meetup

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.
Latest episode:
What AI Hardware Should You Buy? Memory, Backends (CUDA/ROCm/Metal), and Scaling at Home | Ep 3

He runs K3s at home and built a small agent that watches pod states. If something gets OOMKilled or falls into CrashLoopBackOff, it posts to Discord and restarts the pod. That’s it. Just automatic remediation.

Then I thought: that’s how agents are supposed to be built . They’re supposed to help us automate things. He has a framework for his system, a set of functions the AI can call, and he gets notified if something happens so he can intervene if necessary. This is the perfect way to take observability one step further and allow AI to take specific targeted actions to help.

The Homelab Mindset

I’ve noticed something recently. Before I started working on software professionally, people who had side projects on the cloud were the scrappy ones. They were learning more faster and that enabled their careers. Lately I’ve noticed a shift. Engineers who build homelabs are the more creative problem solvers; they just tend to think differently.

The cloud removes friction. It abstracts the hardware, the network, the blast radius. When something breaks in the cloud, it feels distant. When something breaks on your own system you are the ONLY one who can fix it. That changes how you design.

You are forced to think about failure before capability, containment before expansion, and recovery before autonomy.That mindset matters more in the age of AI than most people realize.

Hardware Is Back. That’s Good.

There has been an interesting resurgence in hardware lately, especially for Macs.

As OpenClaw and similar tools have taken off, I’ve noticed more and more people on Tech Twitter and other platforms buying machines specifically “for AI,” which part is exciting. Running things locally changes how you think.

If OpenClaw is what gets someone to buy a machine and start experimenting at home, that’s a net positive.

But we should be honest about something--running OpenClaw at home is not the same thing as home labbing.

Home labbing is about more than where the compute runs, it’s ultimately about ownership of the system. You need to understand the failure modes, and build something that’s tailored specifically to your use case. You need to be able to ask “what happens if I break this?” and then breaking it to answer your question.

OpenClaw — even when self-hosted — is still primarily a convenience layer. It is an installed autonomy. You are consuming a system someone else designed, not designing the boundaries yourself. That’s not inherently bad,.but it is fundamentally different.

The stereotypical movement I see around OpenClaw is not “I want to engineer a workflow.”

Rather, it’s “I don’t want to build an agent, so I’ll install one.”That mindset is the opposite of what home labbing teaches. Home labbing teaches us to build our own control instead of outsourcing it, and that distinction matters.

“What’s So Special About OpenClaw?”

So let’s address the elephant in the room. I have talked about it on my podcast and in several other blog posts because I live in an echo chamber where people all of the sudden quit talking about agents and started installing OpenClaw.

OpenClaw is not a magical agent harness. It’s just special because it’s convenient. The plugin ecosystem. The immediate feeling of autonomy without building the scaffolding yourself. It’sThat is powerful for non-technical users, .

but if you are a software engineer, convenience should not be the bar your reaching for.

Around the same time I was posting about this, Matt Sharp wrote about security — not because he wants to be “Mr. Security,” but because we are ignoring it in places we shouldn’t.

AI is one of those places.

That should make us uncomfortable.:

The most downloaded skilled on the OpenClaw marketplace turned out to be malware. It stole SSH keys, crypto wallets, browser cookies, and opened a reverse shell to an attacker’s server. And yet most of the conversation is still about what these agents can do — not what they can access, not how they can fail, and not how they can be exploited.

We are moving fast on capability and slow on security.

DeepAgent, LangGraph, and Control

If you compare something like LangChain’s DeepAgent or LangGraph to OpenClaw, the difference is not intelligence, it’s control.

DeepAgent and LangGraph force you to define state transitions, tool boundaries, and execution graphs explicitly. You encode what is allowed to happen and when, shaping the workflow itself.

OpenClaw feels different because you give it tools. You describe what you want and then trust the model to navigate correctly. The problem becomes that when most people use OpenClaw, they stop thinking about consequences.

Failure Modes

In reliability engineering, we assume failure and optimize for quick and seamless remediation. In security, we assume breaches and optimize for quick detection and countermeasures.

With AI agents, we assume success.

That is the flaw.

Right now I see too much implicit trust in AI and I rarely hear people ask what happens when the prompt is compromised. We hardly ever ask what happens when a tool call fails. We rarely ask what happens when the model confidently does spends my money for deletes files.

We assume it will recover because it “learns,” but models do not learn.

After training is completed, they only do probabilistic predictions. More instructions are not guardrails. It is just more tokens.

That is not remediation, it’s just a loop.

Security Is Remediation

Most real security work is remediation. You will have incidents. You will have edge cases. You will have abuse. The question is how you detect it, contain it, and recover.

Why are we treating AI systems as if they are exempt from that mindset?

If you install an agent internally and give it broad tool access, how are you thinking about blast radius? What is the rollback path? Who intervenes when it fails silently?

These are engineering questions.

Agents don’t eliminate failure. They eliminate busy work.

The Only Question That Matters

If you are deploying agents internally — whether it is OpenClaw, DeepAgent, LangGraph, or something you built yourself — there is one question that matters more than any other:

When it fails, what happens next?

That is the difference between engineering and hype.

What I Would Do Instead

Let’s dig in to this question.

If I were deploying agents inside a company tomorrow, I would not start with autonomy. I would start with boundaries.

The first thing I would define is scope. One workflow. One responsibility. One clearly defined success condition. Not “moderate Slack” or “manage Kubernetes.” It needs to be something narrow, observable, and reversible.

Then I would hard-code the state transitions.

The agent would not be free to decide what happens next. It would move through predefined stages:

Input → validate → call tool → validate output → log → exit

Every transition explicit, and every tool call wrapped in verification.

The model would generate suggestions, then the system would enforce decisions.

I would validate tool parameters before execution. I would validate outputs after execution. I would reject malformed responses. I would log everything. I would set blast radius limits at the infrastructure level, not the prompt level.

If the agent fails, it does not retry forever.

It escalates. If it escalates too often, it gets disabled.

That is remediation.

The model is not in charge of safety, the surrounding system is.

This is also why I self-host as much as I do. Not because I distrust enterprise providers, but because I understand they are not just exposing “raw models.” They are layering guardrails, fine-tuning adapters, evaluation harnesses, and enforcement mechanisms behind the scenes to meet customer expectations. They are engineering boundary conditions.

If you want production-grade behavior, you have to do the same.

In my own agents, I hard-code checks before and after model calls. I constrain which tools are callable, restrict parameter shapes, and I require structured outputs. I treat the model like an untrusted component inside a trusted system, because it is.

Prompts are guidance and code is enforcement.

Hard-coded guardrails are boring, but they’re the only thing I trust.

The companies that win with AI will not be the ones that install the most autonomy.

They will be the ones that engineer the cleanest failure paths.

Assume failure.

Stay Connected

Subscribe now

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• Agents Are Not Interns

• When a PedroCLI Stops Being a CLI

• PedroCLI vs Clawd / Molt / etc

• Have You Heard of Logits?

• Building Production-Grade AI in Go

Upcoming Events or Streams

I’m also active in the Utah tech community. If you’re local, come build with us. Otherwise catch us on Twitch

Utah Data Engineering Meetup
https://www.meetup.com/utah-data-engineering-meetup/events/311227000/?eventOrigin=home_next_event_you_are_hosting

MLOps Utah Meetup
https://www.meetup.com/machine-learning-utah/events/312474910/?eventOrigin=home_page_upcoming_events$all

Latest Podcast Episode

Deep dives on AI infrastructure, home labs, and production engineering.
Latest episode:
What AI Hardware Should You Buy? Memory, Backends (CUDA/ROCm/Metal), and Scaling at Home | Ep 3

Then build.

Read more

Because right now, “agent” has become shorthand for anything that feels autonomous. And that confusion is actively making systems less safe, less predictable, and harder to reason about.

An agent is not a human replacement, or an intern, or a decision maker.

An agent is a probabilistic system that proposes actions in pursuit of a goal.

That distinction isn’t philosophical, it’s architectural; and ignoring that distinction it is how we end up with tools that confidently do the wrong thing at scale.

What People Usually Mean When They Say “Agent”

When most people talk about agents today, they’re really talking about self-iterating agents.

This is the pattern popularized by tools like Claude Code and similar interactive systems systems:

The agent analyzes a task, decides what to do next, calls a tool, observes the result, and repeats forever and ever until it decides it’s done. After it decides it’s done, it asks you if you are ready to move on with its course of action. This feels incredible as a user becauseyou’re present: You can interrupt, redirect, or notice drift before it becomes dangerous. The agent feels like a collaborator rather than a liability.

PedroCLI now supports two execution modes using the underlying processes but with radically different rules:

1. Interactive Mode (PedroCode) is where self-iteration belongs. This includes open-ended reasoning, human oversight, exploration, drafting , and refactoring.

2. Background Mode (PedroCLI) is where the AI drives using phased workflows, strict tool contracts, system-owned state, and artifact-based completion.

I added the interactive mode as the baseline to make sure my personal agent Pedro worked well, so I had to add a user-driven interface to help me build background jobs better (it was a mid-development pivot. Why not build OpenCode[link] in Go?).

This was great for interacting with code, prompt development, tool call iteration, and model testing. So next let’s chat about long-running workflows

Why Autonomy Becomes Dangerous the Moment You Look Away

Remember PedroCLI was originally a tool for background execution, like long-running jobs and unattended workflows. So let’s talk about unattended jobs.

At this point, the agent is no longer assisted, it’s unsupervised. And unsupervised agents don’t understand risk, they understand completion. It’s important to remember that at the end of they day, agents are people pleasers. If they have permissions to do something, they will do it for the purpose of getting the job done, no holds barred. .

If installing a sketchy plugin the agent will use it simply because it’s obedient. Language models are trained to please, and they fill gaps confidently. They will proceed unless explicitly stopped, and as a consequence you may loose some personal information to the internet.

This is why open agent registries filled with unvetted “skills” are not just unsafe—they’re dangerous by construction. They combine open-ended reasoning, broad permissions, and no enforceable boundaries. If you’re not careful, it’s an accident waiting to happen.

The Intern Fallacy

There’s a popular narrative that agents are like interns.You give them tasks and the let them figure it out. But this framing is catastrophically incorrect.

Interns have judgment. They understand consequences, they hesitate, and the good ones will ask if they’re unsure of something. Agents do not.

You can count on agents to fabricate certainty, and then proceed confidently. They will optimize for making you feel like progress is happening even when nothing real has occurred.

If you’re looking for one agent to optimize away your stress, what you’re actually asking for is an intern with no judgment and unlimited permissions, and that should terrify you.

Workflows, Not Autonomy

So what is the solution to the unfettered access of agents? SCOPE! This is where workflows matter.

A workflow is task-oriented. It answers questions an agent should never decide for itself:

What is the task? What steps exist? Which tools are allowed right now? What does success look like? What is explicitly forbidden?

Workflows don’t weaken agents, it’s actually the reverse. They make them true automation.

When you combine agents with workflows—bounded context, scoped permissions, explicit completion conditions—you get something powerful and controllable.

How PedroCLI Separates Autonomy from Trust

How do we scope PedroCLI? With our tool call registration. Each step in the workflow has a set of tool calls assigned to it. For example, in both the background coding and background blogpost engines we have research steps. Both of these areas involve searching my local files system, so it has a file search tool call to list the files. Then, it has a bash grep ability to read file’s contents. It cannot write or edit files until later and it does not have permissions to edit anything outside the file system, so there’s no posting local information to the internet.

In a later posting step, I write blog posts to my Notion space. During this tool call it cannot search the file system, it can only read the file passed to the context (aka the blog draft) and any context passed across workflow steps. Because of this I have strictly limited the information it can carry, which greatly reduces the chance that it does something malicious.

This is what I mean by scope: each step only really does one thing, and it only has the permissions needed to do that thing. Workflows are a series of agents, but the agent’s permissions follow the principle of least privilege. When you deploy an agent this should be mirrored in the tool calls registered and any potential service account/role given in your permissioning system. You just have to limit what they can do as a best practice.

Tools Are APIs, Not Suggestions

One of the hardest lessons I learned is that tool calling is not a prompting problem. It’s an API design problem.

tool := ToolDefinition{
    Name:        "write_file",
    Description: "Write content to a file on disk",
    Schema: JSONSchema{
        Type: "object",
        Required: []string{"path", "content"},
        Properties: map[string]Property{
            "path": { Type: "string" },
            "content": { Type: "string" },
        },
    },
}

Tools are registered with the LLM the with the intended use case; Here’s what this does—use it when needed. They should self-documenting in natural language with the contract of what will be done, and that is not how we should be treating them when we implement their backing functions. If we leave ambiguity in the descriptions or the functions the LLM will take liberties to fill in the missing information. Remember, agents guess. They omit fields and invent parameters. Unless you stop them, they keep going as if nothing went wrong. I had a time when I registered a tool call name as “list_directory” but because that doesn;t make semantic sense,the 32b parameter model kept trying to invoke “list_directories”.

How can I get the agent to do better? First, pick a more semantically meaning full name, and second, since I am self hosting, use logits and grammars. This lets tools become more predictable interfaces.

Once tools became strict interfaces instead of suggestions, reliability jumped immediately—not because the agent got smarter, but because ambiguity disappeared.

Why This Is Still a Work in Progress

Defining guardrails is hard. Permissions are hard, and deciding what an agent cannot do is harder than deciding what it can do. But I would rather do this work explicitly than outsource control to optimism.

Because control isn’t about power—it’s about responsibility.

If an agent is going to act on my behalf, I care way more about what it is forbidden from doing than how clever it sounds while doing it.

The Real Takeaway

Agents have enormous potential, but anything with unlimited potential also has unlimited risk.

So don’t start by asking: What agent should I use? Start by asking:

• What task am I solving?

• What happens if this fails?

• Who is accountable?

• Should this even be an agent?

Sometimes the answer is a script. Sometimes it’s a workflow. Sometimes it’s traditional automation. But sometimes—carefully—it’s an agent.

PedroCLI is my attempt to make that distinction explicit .

because they’re powerful enough to deserve structure.

Stay Connected

Want to stay updated on what I’m working on? Here’s where you can find me:

• YouTube

• Domesticating AI Podcast

• Twitter / X

• Bluesky

• LinkedIn

• Newsletter

• Twitch

• GitHub

• Discord

• LinkTree

Newsletter Highlights

Recent Posts You Might Have Missed

• When a PedroCLI Stops Being a CLI

• PedroCLI vs Clawd / Molt / etc

• Have You Heard of Logits?

• Building Production-Grade AI in Go

• Beyond Hello World: net/http in Production

Upcoming Events or Streams

• Utah Data Engineering Meetup

• MLOPS Utah Meetup

Latest Podcast Episode

DomesticatingAI - NVIDIA DGX Spark: From “Inference Box” to Dev Rig (What It Actually Is)

In Ep 2 of Domesticating AI, we break down what Spark is actually good for (AI development + fine-tuning) vs what it isn’t (a magical drop-in inference server). We also dig into why unified memory changes the local-AI experience, the “gateway stack…

Read more