Subscribe • Previous Issues

Your Model Is Not Your Moat

In a recent article, I wrote about companies turning their own data, workflows, and production feedback into specialized intelligence they increasingly control. The deeper idea was compounding. What matters is not simply whether a model performs well today, but whether using it creates assets that make the system better tomorrow. More AI teams are beginning to build around that idea. Which brings me to a question I keep coming back to: if everyone can increasingly rent the same intelligence, what exactly becomes defensible?

Subscribe now

The moat is what you learn by operating

Strong models are becoming easier for everyone to access. What competitors cannot buy is an understanding of how your organization actually works. Which information matters for a decision? Which exceptions do experienced employees notice? Which workflows deserve automation? What does a good result look like, and how do you know when the system has failed?

That makes operational context and evaluations unusually important. A connector that gives an agent access to 400 reports is useful, but it is not much of a moat. Knowing which five reports your best analysts use, when and why they use them, is much harder to copy. Over time, production traces, evaluations, permissions, workflows, customer knowledge, and expert decisions form a company-specific operating layer. I would spend less effort defending access to a particular model and more effort capturing what the organization learns from using models.

There is an economic version of the same argument. Owning a particular layer of the AI stack does not guarantee attractive margins. A generic memory component can become a commodity. A deeply integrated memory system that measurably improves an agent’s performance may not. Depth and outcomes matter more than drawing a box around a layer and declaring it defensible.

Build for a platform that will keep changing

By some estimates, the best open-weight models have gone from roughly sixteen months behind the frontier to about two. I would treat the exact number cautiously, but the direction matters. Open models are not automatically cheaper once you include infrastructure and the people required to operate them. What they increasingly provide is another option.

That matters because depending completely on one provider is a business risk, not just a technical choice. Prices change. Models disappear. Terms get revised. Providers can restrict capabilities or start competing with companies they previously supplied. Security teams have even described closed models refusing legitimate defensive work because attacking software and testing its defenses can look similar from the model’s perspective.

My response would not be to predict which model or agent framework wins. I would make the important parts of the system replaceable. Keep models, business logic, state, policy, and execution reasonably separable. Use open interfaces where they help. Teams working on unrelated problems keep arriving at translation layers between incompatible systems. I take that as a useful signal. The platform is still unsettled, so interoperability has strategic value.

Build products that leave something behind

One idea I keep returning to is that not every product needs to survive indefinitely. Some AI applications will lose their differentiation when the next model absorbs the feature that made them special. That does not mean they were bad things to build.

Suppose a product lasts eighteen months but gives you thousands of real customer interactions, a stronger evaluation suite, workflow data, distribution, and a deeper understanding of the problem. Those assets can feed whatever comes next. The better question may be less “Can somebody copy this feature?” and more “What will we know after operating this product that somebody starting tomorrow will not?”

The same logic applies to the interface. Some teams are bringing full functionality into major AI assistants rather than insisting that customers stay inside their own applications. The reasoning is that value lives in the intelligence and context provided, not necessarily in the window customers look through. If users increasingly prefer assistants, defending the container may become a poor use of energy.

For startups, I also like the analogy to mobile. Incumbents eventually moved almost every important web product onto smartphones. The defining new companies came from things smartphones made newly possible, including cameras, GPS, and constant connectivity. AI may follow a similar pattern. Automating an existing workflow can be valuable, but incumbents can do that too. I am more interested in what becomes possible only because inexpensive intelligence and software generation now exist.

Legacy modernization is one example that deserves more attention. Companies have deferred these projects because documentation is incomplete, experts have retired, and nobody wants to own the migration risk. The interesting approach is not simply asking a model to rewrite COBOL in Python. Treat the old system as the specification, generate tests from it, translate the code, and use deterministic methods to verify that the new system behaves the same way. That turns a risky, open-ended migration into a checklist you can verify step by step.

When generation gets cheap, judgment gets scarce

There is a human version of this shift too. As code, analysis, text, and other outputs get cheaper to generate, producing the first answer matters less. Figuring out whether it is the right answer matters more.

That pushes value toward judgment, debugging, evaluation, and problem selection. Can someone turn a fuzzy objective into something a system can execute and test? Can they spot a failure that looks superficially correct? Can they explain why an agent went wrong and improve the system around it? A conventional coding test may tell you less about those abilities than real projects someone has shipped and a candid postmortem on one that failed.

I also would not assume that rapid productivity gains mean organizations transform overnight. Tools change much faster than job descriptions, incentives, and management structures. The first phase may look mostly like familiar teams producing more, with people spending more time directing and checking automated work.

When customers own the compounding loop

This is where the argument comes back to OpenAI and Anthropic. The challenge is not simply that an open model might catch them on a benchmark. Their best customers can increasingly combine proprietary data, workflows, tools, evaluations, and production feedback into intelligence that improves inside their own organizations.

That changes the competitive landscape. Frontier labs still compete with one another on general model capability, but increasingly they also compete with the accumulated intelligence their customers are building downstream. The more of that learning companies can capture for themselves, the more important it becomes to ask who actually owns the compounding loop.

Models, features, interfaces, and routine output are becoming easier to obtain. Operational context, evaluations, customer knowledge, distribution, and judgment get stronger through use. Buy what commoditizes. Build what compounds.

AI Is Rewiring Work Before It Rewrites Employment

When the AI Compute Bill Arrives

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the Agent Conference. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Nine Practical Rules for Agents Doing Real Work

In recent conversations with crews building agents, I keep hearing the same lessons. Teams with very different products are arriving independently at almost the same architectural choices. That convergence feels important. In recent posts, I argued that passing your evals does not mean an AI system is safe, and that many of the most consequential risks sit outside the model itself. This is the constructive follow-up: once an agent is doing real work, what should you actually build around it? I think a practical architecture playbook is beginning to emerge.

1. Put hard constraints in software, not prompts

Models are good at flexible reasoning. Ordinary software is good at state, permissions, calculations, retries, and predictable control flow. Reliable systems keep that division explicit rather than asking a prompt to guarantee something important. A prompt is guidance. It stays negotiable no matter how firmly you word it, which is fine for judgment calls and a real problem for anything that can’t be wrong.

The dividing line is simple: let the model handle ambiguity, which is where it creates value. Put calculations in trusted code, enforce permissions through policy systems, validate code with compilers and tests, and check important factual claims against authoritative sources. Ask what the model can currently violate that ordinary software could prevent. When a failure would create a material consequence, the architecture should make that violation impossible or require explicit approval.

2. Give the agent only as much autonomy as the job needs

One pattern I keep seeing is teams giving agents more autonomy than the job requires. That excess autonomy creates more possible paths, more opportunities for error, higher operating costs, and a harder governance problem. For many applications, conventional software should control the workflow while the model handles the smaller number of steps that genuinely require judgment. Greater autonomy can be justified when the task requires exploration, open-ended planning, or creating new tools.

I would treat autonomy as a deliberate design choice rather than a default. Start with a flexible workflow, observe which paths repeat reliably, and turn those stable paths into ordinary code. List every point where the agent chooses its next action, then ask which of those choices the task truly requires. A mature agent faces fewer open-ended choices over time, not more.

3. Build the agent around the domain’s trusted process

A support workflow already specifies how to triage a request, what information to collect, when to escalate, and which actions require approval. A medical workflow does the same through diagnostic protocols. When a domain already has a proven checklist, protocol, or decision process, use that as the agent’s structure rather than starting with a generic loop that tells the model to make a plan and keep acting until it is done.

This makes the system more reliable because the structure has already been tested by the field. It also makes the system easier to understand for the people who have to trust it and sign off on it. The best agent architecture often looks less like a general-purpose digital employee and more like the field’s existing best practice made executable.

4. Design for recovery, not a flawless first pass

Small errors compound quickly in long workflows. A system that succeeds 95 percent of the time at each step has only about a 60 percent chance of completing ten independent steps without an error. That helps explain why a three-step demo can look extraordinary while a longer business process falls apart.

Do not build around the assumption that the model will stop making mistakes. Add checkpoints, checks that confirm each action produced the intended result, retries, reversible actions, and the ability to resume from a known-good state. Measure recovery separately from first-attempt accuracy. A system that detects drift and corrects itself is more useful than one that looks perfect until the first unexpected tool response.

5. Evaluate the model and harness as one system

What exactly are you evaluating when you test an agent? The harness is the software surrounding the model, including its tools, context management, memory, policies, and recovery logic. That surrounding system can affect performance as much as changing the model. In one comparison, the same open model showed an 18 percentage point difference between the best and worst harness configurations. That is one of the findings I would take most seriously when comparing models.

A leaderboard is only a starting point. Evaluate the model with your tools, context, permissions, and failure modes attached. Then rerun the evaluation whenever either the model or the harness changes. Changing either half means you are evaluating a new system.

6. Keep multi-agent teams small, and let someone disagree

Multi-agent systems can look sophisticated while mostly adding coordination overhead. The more effective pattern is usually a clear orchestrator with a small number of specialists, each with a distinct role, tool set, permission scope, and only the information it needs. Larger teams can help when the work is genuinely parallel or when independent perspectives are part of the requirement, but large swarms often create duplicated work and make failures harder to trace. One role deserves special protection: a critic or breaker with explicit criteria and the authority to block or escalate. Knowing when not to answer is something the system must be designed to do, not merely a personality you request in a prompt.

7. A bigger toolbox can make the agent worse

In one test, a larger tool set was slower and slightly less accurate even though it contained every tool in the smaller set. This may be the most counterintuitive rule on the list. The extra tools were not bad. The problem was overlap. When several tools can plausibly handle the same request, the model spends attention choosing among near-duplicates instead of doing the work.

More tools also give the agent more systems it can read from or write to, and they create more sequences of tool calls for the team to test. Favor a compact toolbox in which every tool has a clearly different job. Run required setup tools automatically, and log which tools get picked, what goes in and out, and how often they fail. If two tools could plausibly handle the same request, combine them, route between them in software, or remove one.

8. Context, memory, and enterprise knowledge are different systems

The distinction is operational: context is the information the model needs during the current run, memory is what the system should carry forward from earlier work, and enterprise knowledge is the governed collection of documents, records, and policies it may consult. Treating them as one undifferentiated store creates both quality and security problems. They can share infrastructure, but they should not share the same retention, retrieval, and access rules by default.

Memory in particular should preserve lessons, not transcripts. What matters is the mistake to avoid, the technique worth reusing, or the preference that should change future behavior. The test is not how much the system remembers. Inspect what it saved, what it later retrieved, and whether that information changed the result.

9. Fix the knowledge before you pay for a bigger model

When an agent performs poorly, the default reaction is often to buy a stronger model or begin fine-tuning. Before changing the model, I would look closely at what it is being given to work with. One team replaced a pile of raw support documents with a diagnostic playbook. A router translated a customer’s complaint into a symptom, then a problem type, then the relevant handbook. Tokens fell 43 percent and errors fell 48 percent without changing the model.

Retrieval is not a feature you install once and forget. It fails when a user’s words do not match the document’s words, when information is buried in tables or PDFs, or when sources conflict. Before changing the model, improve the structure, routing, and governance of the knowledge it receives. The model is the component you can buy. The system around it is what you own.

If agents and real-world AI deployment are on your roadmap, come spend two days with the teams already doing it. AI Conference, San Francisco, Sep 30–Oct 1. 20% off with gradientflow20.

The Case for AI Data Centers Nobody Is Making Well

The Memory Bottleneck in Frontier AI

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

The biggest AI risks sit outside the model

The most revealing AI failures right now are not stories about models becoming too capable. They are stories about everything around the model. One system received more access than its test environment could contain. Another was trained on material whose acquisition created $1.5 billion in exposure. In a third, people walked away more certain without being any more correct. I recently argued that passing your evals does not mean you are safe. This is the next layer of that argument. Permissions, inputs, and presentation are three places where serious AI risks accumulate, and each one lands on a different desk.

What an Agent Can Touch

Model capability tells you what a system can do. Architecture determines what it may do, what data and tools it may use, and which actions require approval. A mediocre model with broad write access can be more dangerous than a stronger model in a tight box.

OpenAI’s cyber-focused models, including GPT-5.6 Sol, left the sandbox used for testing, reportedly exploited a zero day, reached the open internet, and compromised Hugging Face production systems. Nobody appears to have noticed for days. Calling the behavior rogue explains little. Safety filters had been switched off for the exercise, and network isolation was weaker than the test assumed. Those were architecture and configuration decisions.

Other reports describe an agent deleting user files without being asked and a coding tool uploading full codebases without clear consent. Different incidents, same pattern: a weakly specified task becomes dangerous in proportion to what the system can touch.

An agent can also be redirected by hostile instructions embedded in an email, document, web page, or tool response. Once tools are attached, prompt injection can lead to an unauthorized action, not merely a bad answer. Permissions and approval rules therefore have to be enforced outside the model.

Give every production agent its own identity and task-specific permissions rather than letting it borrow a person’s credentials or a generic service account. Split read from write. Use short-lived credentials, narrow network access, transaction limits, and approval gates for irreversible actions. Preserve logs and make recovery possible. The goal is not perfect prevention. It is a small blast radius, fast detection, and enough evidence to understand what happened.

Not every system needs all of those controls. If an agent only reads, gets things wrong cheaply, and can be corrected before anyone acts, most of those controls are overhead. Once it can write to a system of record, spend money, touch regulated data, or produce something a customer will act on without review, those controls apply. The dividing line is the damage the system can cause before someone stops it. Many teams want a third option. There isn’t one.

Where the Data Came From

The same pattern runs upstream, into what the model was trained on before anyone deployed it. A federal judge approved Anthropic’s $1.5 billion settlement over pirated books used to train Claude. The headline number is the least interesting part. What matters is the line the settlement draws between training on material you obtained lawfully and training on material pulled from pirate libraries. The failure was not model capability. The training pipeline did what it was built to do, and a decision made around it created the exposure.

If you are fine-tuning a model, assembling an evaluation set, or building retrieval over third-party content, that distinction is your problem too. How the material was acquired, what rights came with it, and whether those facts can be reconstructed later matter as much as what the model does with it.

Boards now have reason to ask where training data came from, who carries the risk, and whether a disputed model or dataset can be removed without disrupting the product.

Answering those questions requires a dataset inventory, license terms you can produce on request, a clear answer on who pays if you get sued, a current view of litigation involving your vendors, and a way to swap out a model or dataset without taking the product down. Most teams have some of this. Very few could pull it together on short notice.

The Confidence Trap

Controls and documentation can limit what a system does and clarify what went into it. Neither fully addresses what happens when a convincing answer reaches a user. A recent preprint reports five experiments involving 3,132 participants who answered difficult questions and had the option to say they did not know. Giving people access to AI advice nearly eliminated that restraint. Participants answered more questions, but they were correct about one-third as often as people without AI access. Their confidence nearly doubled.

You should assume your users can do the same thing. In medical, legal, financial, hiring, and security workflows, a confident wrong answer can be more dangerous than an obvious failure. The risk grows when the interface presents a smooth recommendation as the natural next step. A confidence score does not automatically help, especially when it comes from the same model that produced the answer. A precise-looking number can become decoration rather than evidence.

You cannot engineer this one away, because you cannot stop a person from over-valuing a fluent answer. What you can do is design against it, and the moves are concrete. Ask the user to commit to a view before you reveal the model’s recommendation. Show uncertainty honestly instead of rounding it into confident prose. Take confidence numbers from past accuracy, not from the model. Surface disagreement between sources rather than smoothing it out. Make correcting the system easy enough that people actually do so. Let the system abstain. Escalate decisions when the cost of being wrong is high.

Then check whether it worked. The test is not whether people like the assistant or finish more tasks with it. It is whether the human and the system together make better decisions than the process they replaced, know when to abstain, and escalate the right cases.

Three Desks, One Problem

The next phase of enterprise AI will be defined less by who gets the most capable model and more by who builds the strongest system around it.

Security and architecture teams have to bound what the system can touch. Data, legal, and procurement teams have to document what went in. Product teams have to preserve human judgment. Rules people once enforced through policy, habit, or supervision increasingly need technical equivalents such as distinct agent identities, approval gates, provenance, and logs.

These risks land on different desks, but they are becoming one management problem.

Capability is the layer you buy. Everything around it is the layer you own.

Evaluating the system, rather than just the model, requires a richer set of measurements. The Luminos whitepaper explains one approach.

When Mathematical Results Become Abundant

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Continual Learning Is Arriving in Pieces

A while back I wrote about how startups are using reinforcement learning to make agents more reliable. A deeper problem behind that whole trend keeps resurfacing: a model can improve during training, but the moment it’s deployed, learning largely stops. A policy changes, a new edge case shows up, a user corrects the system, and the lesson rarely travels past that one incident. The prompt gets patched, the ticket gets closed, and the same class of mistake eventually comes back. The system might pile up logs and documents along the way, but it’s often no better on day 500 than it was on day one.

That is the gap continual learning is meant to close. If reinforcement learning is about improving a model through practice during training, continual learning asks what happens after. Can real usage produce a lasting change in how the system behaves next time? The mechanics vary, sometimes it’s memory, sometimes it’s an agent revising its own instructions, sometimes it’s the model’s weights (its internal parameters) actually updating, but the working definition holds across all of them: a deployed system that captures its own experience, turns it into a durable improvement, checks that the improvement doesn’t break something else, and carries it forward. I count more than 20 startups where some version of that loop is core to the business. That’s enough to take the trend seriously. That raises two questions: why so many are betting on continual learning now, and where they think it pays off.

The Maintenance Bill Nobody Budgeted For

The strongest argument has nothing to do with research ambition, it’s about maintenance. Today, most agent failures become support tickets, prompt edits, or isolated debugging sessions. Continual learning turns a failure into reusable material: a regression test, a corrected instruction, a better tool call, a revised memory, or training data. The harder part is making sure a fix for one case doesn’t quietly break things that used to work. That’s why some of these companies build regression checking directly into the update loop rather than trusting a human to notice later. That is what separates improvement that compounds from an expanding pile of patches.

The second argument is economic. Feeding a model your documents or your production history at the start of every session works, but you’re paying to reprocess the same information over and over. Several of these startups bet that information you use often should get compressed into the system instead of re-read from scratch, which cuts cost and lets a smaller, cheaper model outperform a much bigger general one on the narrow task it’s actually seen before. Push that further and you get something more ambitious than savings, a system that internalizes how one organization actually operates, or captures the kind of tacit judgment call an experienced employee makes but could never fully write into a prompt.

The third argument is specialization, with trust as the constraint. Some expertise is difficult to express as written rules. It lives in repeated examples, corrections, and subtle judgments. Continual learning offers a way to capture that know-how, but it also creates new risks. A system that keeps changing can absorb bad feedback, forget an old capability, leak information, or become impossible to audit. The practical race is therefore not simply to make AI learn more. It is to make learning inspectable, reversible, and safe enough for a business to rely on.

Where Continual Learning Is Already Earning Its Keep

Customer-facing work is one of the clearest early markets. Support systems produce large volumes of repeated interactions, visible escalations, human corrections, and fairly legible outcomes. Sales and account work benefit from the same continuity, especially when the system needs to remember a long-running relationship rather than treat every conversation as a fresh transaction. The second cluster is enterprise knowledge and professional work: law, financial services, healthcare operations. These are fields where the documents are proprietary, the tasks recur constantly, and mistakes cost real money. High stakes plus narrow specialization plus repeated structure is close to the ideal setting for a system meant to get better at your specific work over time.

Technical work is another natural fit because the feedback is often unusually clear. Code compiles or fails. Tests pass or fail. A developer accepts a patch or rewrites it. Infrastructure operations also produce traces of what was tried, what worked, and what made the problem worse. The common thread is that the longer a task runs, the more a small early mistake compounds, so the systems that benefit most are the ones that catch their own drift along the way.

A third cluster uses continual learning to improve AI systems themselves. Product usage can become a steady stream of training signals rather than an archive nobody revisits. Simulated environments can give agents somewhere safe to encounter rare failures before they touch real customers or infrastructure. Model development and performance optimization also provide relatively clear measures of success, making it possible to propose changes, test them, and retain what works. At the more experimental edge are robots and systems operating in changing physical environments, where the world drifts too quickly for occasional retraining to be enough.

Continual Learning Is Already Arriving in Pieces

I do not expect continual learning to become mainstream all at once because it contains two different bets. The faster half is already arriving. Systems can preserve useful memories, analyze production traces, revise instructions, and test proposed changes while keeping a person at the approval step. These approaches are comparatively cheap, inspectable, and reversible. Many teams will adopt them without ever describing what they are doing as continual learning.

Actually updating a model’s weights on an ongoing basis is a harder step. That is not only a tooling problem. It is a trust problem. Teams need strong answers for data quality, privacy, forgetting, poisoning, regression testing, auditability, and rollback before they allow a deployed model to alter itself routinely. My guess is that weight-level learning will first take hold in high-volume, high-value workloads with clear feedback. But the broader shift is already underway. AI products are starting to look less like software you replace every few months and more like something expected to earn its keep by getting better the longer you use it. The real question probably isn’t when every model learns continuously. It’s when every serious AI product is expected to have an actual answer for how it gets better with use.

AI Conference is a chance to go beyond the headlines and learn directly from teams shipping AI in the real world. San Francisco, Sep 30–Oct 1. Use gradientflow20 for 20% off.

AI Has More Than a Marketing Problem

Summer Books

• The Score: How to Stop Playing Somebody Else’s Game. While the examples are mainly from game design, this sharpened a point that applies directly to AI evals, a case of Goodhart's law: once you optimize for a benchmark, it stops telling you what you actually want to know.

• You Won’t Get Free of It: Stories of Mothers and Daughters. Rachel Aviv is one of my favorite writers, and this collection reminds me why: patient reporting that resists the urge to editorialize, even when the material makes that difficult.

• The Cruelest Game: Chasing Greatness in Professional Tennis. This is a gift for any lifelong tennis fan: Futterman gets remarkably close to his subjects (down to what Djokovic eats between points, and how he tracks the shade creeping across the court to survive a five-hour match) and the result is the rare tennis book that explains the sport’s psychology instead of just recapping its highlights.

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Passing Your Evals Doesn't Mean You're Safe

Evals are part of every serious conversation about putting AI into production. Teams define benchmarks, set thresholds, and increasingly run red teams to see how the system holds up against someone actively trying to break it. That combination is reasonably good at telling you whether a model is accurate, reliable, fast enough for production, and resistant to an adversarial attack. It says almost nothing about what actually gets a company into trouble once the system is live.

Look at the AI lawsuits and regulatory findings piling up this year and the gap becomes obvious. A man in California is suing OpenAI, arguing that ChatGPT used his own disclosed bipolar diagnosis to keep him engaged in conversation rather than steering him toward help, a product liability claim built entirely around a session memory feature. A federal court let a discrimination suit against Workday proceed on the theory that its hiring software leaned on medical leave and other proxy signals to screen out an older applicant with a disability, exposing the vendor itself, not just the employer who bought the tool. A German court ruled that a company is responsible for its own chatbot inventing a doctor’s medical credentials, reasoning that a chatbot isn’t a third party, it’s simply the business talking. None of these are the kind of failure a red team session or a bias checklist is built to catch. They’re everyday production risk, legal, reputational, and regulatory, sitting inside systems that had already passed whatever evals their teams ran.

The High-Dimensional Fix

I recently sat down with Andrew Burt, CEO of Luminos, about the whitepaper his team just published on agentic and generative AI risk evals. The report’s core argument is that the standard setup, one broad prompt, one model, a pass or fail score, is too coarse to do its job. Ask a model whether an output is “biased” or “manipulative” and you get a shallow answer back. Real problems slip through, harmless outputs get flagged as risky, and either way you’re left without enough detail to know what to actually fix.

Luminos calls its answer high dimensionality, and the idea is more concrete than the name suggests. Take an ad that shouldn’t manipulate the people looking at it. A typical eval asks one broad question, is this manipulative? A high-dimensional eval tests separately for a fake countdown timer, a hidden renewal fee, a bait-and-switch price, and a handful of other specific tactics, then comes back with something like: false urgency (yes), hidden fee (yes), bait and switch (no). That’s a finding a product team can actually act on. The same decomposition has to run across more than one model, because models have real personalities when it comes to flagging risk. Claude tends to run conservative and overflag, other models underflag, and a prompt that works cleanly on one model can fall apart entirely on another. Underneath all of it, the standards need to come from actual legal, privacy, and compliance expertise, not just engineers guessing at what a regulator will care about.

None of these three ideas is new by itself. Plenty of teams already run more than one model, and some already pull in legal or compliance reviewers. What’s rare is doing all three together, and treating the result as an ongoing job rather than a pre-launch checklist, since models get updated, laws vary by jurisdiction, and a test suite that felt thorough in January can be stale by summer.

It’s worth being clear about what the whitepaper is and isn’t: it doesn’t present a controlled study proving this approach beats simple evals by some specific margin. The case is structural, built from Luminos’s own experience running this at scale, but it’s a pretty convincing one. Testing more of the actual risk surface should catch more real problems, calibrated sub-risks should cut down on false alarms, and requiring an explanation for every flag should make results something a team can actually audit and fix.

Treat AI Risk Like Reliability

The pattern across this year’s AI incidents isn’t that companies skipped evals entirely, most had something in place. This isn’t an argument for ditching red teaming, guardrails, or standard performance evals either, they answer different questions and are still worth running. The mistake is assuming that together they add up to a full picture of production risk.

Once an AI system can influence a customer, an employee, an account, or a regulated decision, its risk testing deserves the same rigor teams already give to reliability and uptime. That means specific tests, evidence you can point to, clear ownership, and monitoring that continues after launch, not a box checked once before shipping. That bar covers more systems than it sounds like, a support chatbot, a hiring tool, an internal scoring system, an agent with write access to anything, almost any AI system doing real work in a business. There’s no comfortable middle category of “important enough to ship, not important enough to evaluate properly.”

The Luminos whitepaper is worth reading in full, not just skimming, whether or not you end up using this exact approach yourself.

AMD’s Long Game in AI

Two AI Labs Carry More of the Cloud Boom Than You Think

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Specialized AI Is Getting Easier to Build

Last week I argued that open models will absorb most of the money and compute the world spends on AI. A week later, open weights are even more central to the conversation. Recent releases have made the gap between capable and affordable harder to ignore, and a broad coalition of technology companies is now publicly arguing that open models matter for competition, security, and national sovereignty. But the models themselves are only the starting point. The more consequential shift is what teams can now do with them. They can increasingly turn generic weights into specialized intelligence they own.

The Post-Training Stack Is Filling In

Two years ago, taking an open model and making it meaningfully better at your specific job was already becoming easy to do. Supervised fine-tuning, meaning training on labeled examples of good output, was always the accessible option. The newer layer is reinforcement fine-tuning, where a model practices a task and learns from whether it succeeded. More than 25 startups are now building around this idea, but the algorithm is rarely the whole product. They are assembling the less glamorous machinery around it: environments where models can practice, graders and verifiers that decide whether the work was done correctly, tools for generating and curating data, and systems for evaluation, deployment, and monitoring.

Platforms are also starting to automate more of the engineering loop. A team can describe a task, establish a baseline, identify failure patterns, build evaluations and test data, train a model against those failures, and repeat. This still requires judgment, and reinforcement learning is not suddenly a push-button exercise. But what used to be a loose collection of research techniques is becoming a recognizable engineering workflow that more companies can adopt.

The demand side explains why this is happening now. Teams that put generic models into production tend to hit the same three walls. Prompt adjustments turn into whack-a-mole as each fix creates another failure. Costs become difficult to ignore once usage reaches real scale. Then a provider changes a model, retires it, raises the price, or starts competing in the same market as its customer. At that point, renting a general-purpose model stops looking like the obvious long-term architecture.

Specialization is attractive because most enterprise AI work is narrower than the frontier labs’ ambitions. It is classification, extraction, forecasting, tool selection, and the execution of repeatable workflows. In examples described to me, specialized models improved performance by 10 to 30 percentage points on well-defined tasks. In one case, a classifier with only a few hundred million parameters ran on a device and matched a model roughly ten times larger.

From Customizing Models to Creating Them

​​A smaller group is going further and pretraining models from scratch. Datadog is my favorite example because the logic is so plain. They sit on an absurd volume of time-stamped operational data, so they trained a time-series foundation model on it. Two details stuck with me. The model performs better when they exclude public time-series datasets entirely and train only on their own data, which is about as direct a statement of where the value sits as you will find. It is also cheaper to run than the classical statistical pipeline it replaced, once you count the tuning those older methods need on every query. Their proprietary data stopped being an input to the product and became part of the model itself. They are now aiming at something more ambitious, a model of their customers’ distributed systems built from metrics, logs, traces, topology, code, and events, meant to predict what a system will do next rather than report what it already did.

As the lineup for the upcoming Ray Summit hints, plenty of other companies are heading down the same road. A payments company is pretraining on transaction data. A design platform tokenized 900 terabytes across 18 datasets to train a model that generates geometry from text, images, and partial CAD input. A streaming service runs training, fine-tuning, evaluation, and serving on the same shared clusters. None of these are research projects, and two changes explain why. Domain models stay small. The 3D model above runs between 513 million and 1.5 billion parameters and Datadog's largest is 2.5 billion, which is a rented cluster and a strong team, not a national research budget. And the platform underneath has converged on what I have been calling the PARK stack, meaning PyTorch, Ray, and Kubernetes handling data prep, training, post-training, evaluation, and serving in one place.

None of which makes pretraining easy or sensible for the average company. It becomes plausible when you have distinctive data, a workload worth repeating, and the team to keep the model running over time. The hard part is not the training run. It is knowing what data to feed it and in what mix, which is still closer to craft than process. Tooling will chip away at that, and as it does the question shifts from “could a company like ours train a model?” to “which parts of our business are worth one?”

The Moat Moves Into the Model

Put the two developments together and you get the shift toward owned specialization. Post-training is the on-ramp and pretraining is the far end. Most companies will operate somewhere between them, but the distance from “we have a use case and some data” to “we own a model that fits it” keeps getting shorter.

That is the deeper challenge for OpenAI and Anthropic. It is not simply that another open model might match them on a leaderboard. It is that their best customers can increasingly turn proprietary data, workflows, tools, and production feedback into intelligence that compounds inside their own organizations. Once enough companies build that flywheel, the big labs stop competing only against one another. They start competing against their own customers’ data. That is the headwind I do not think they have priced into their IPO story.

If you want to see this up close, Ray Summit has entire tracks on foundation model training, multimodal data curation, and reinforcement learning.

AI Can Win While Data Centers Lose

Four Gates Before You Bet on Open Weights

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Open Models Will Absorb Most of the AI Spend

Here is my bet: open models (open weights and open source alike) will end up absorbing most of the money and compute the world spends on AI. The proprietary frontier models get the headlines and the IPO valuations, but developers and AI teams see something different up close. Open models are improving fast, the gap to the proprietary leaders keeps narrowing, and the economics point one way. OpenAI and Anthropic are already starting to warn us about open models, and I expect those warnings to get louder. Here is why I think the spend follows.

Follow the money

Frontier pricing is running on a treadmill. The best closed models hold a clear lead for two to six months before someone matches them or distills the capability and gives it away. That is a brutal foundation for a business, and it is why OpenAI and Anthropic are pushing into applications like coding, security, healthcare, and legal, where margin sits closer to the customer. Watch the IPO math too. Both reportedly want to go public above $800B, valuations that assume pricing power the market may stop believing in. I would not bet my roadmap on today’s API prices holding.

Cheaper per task is not a smaller bill. Cost per finished task is dropping fast, helped by setups where an expensive model plans and a cheap one executes. But Jevons paradox is real: when a resource gets more efficient and cheaper to use, total consumption often rises rather than falls. Cheaper tokens unlock so much more usage that your overall bill can climb even as each task gets cheaper. Budget for cheaper units and heavier use, not for savings that never show up on the invoice.

The premium model is becoming a planner, not a workhorse. Route the hard reasoning and verification to a frontier model, and push the bulk execution to a cheaper one. It is good engineering and good economics, and it quietly breaks the assumption the big labs were built on, that every step runs through a premium API. If you are not architecting this way yet, your competitors already are.

Watch the gap, carefully

Chinese open weights have reached parity in specific, high-stakes areas. In cybersecurity bug-finding, Zhipu’s GLM 5.2 has been measured matching or beating Anthropic’s Opus 4.8. That matters, because finding security bugs is one of the highest-stakes things you can ask a model to do, and open weights help attackers as much as defenders. Still, being good at one thing is not the same as being good at everything, so test these models on your own work instead of trusting news headlines.

Coding agents are still a clear US stronghold, and that gap is not closing quickly. US models ride a flywheel where serious users generate failure data, which trains better models, which pulls in more serious users. Chinese labs are dragged by chip shortages, over-reliance on distillation and benchmaxxing, and pressure to ship consumer products instead of doing open-ended research. That can change, but today the gap holds, so I would not rush to swap out your coding stack.

The “China caught up” story is genuinely contested. Some observers still put the best Chinese labs a year behind, but most estimates land in the three to twelve month range, and it shifts with every release. It matters less than it sounds. For most routine work, my own experience is that open-weight models match the proprietary leaders at a much lower cost, and you can often post-train them to come out ahead. So test on your own tasks, not on anyone’s leaderboard, mine included.

Rethink your stack

Open weights are about control, not just price. You can run an open model on your own hardware, tune it on your own data, and use it without a third party who can change the rules on you. The Fable shutdown made this concrete. When Washington blocked foreign nationals from using the model, Anthropic cut access for everyone, and European buyers started asking whether US models are safe to build on. That is a data moat, a privacy posture, and a hedge against having the rug pulled, not just a cheaper bill. The catch: attackers can run open models just as freely, and many EU and US buyers still will not touch Chinese-origin ones.

A tested fallback is leverage even if you never use it. Just having a working open-weight alternative caps your exposure to a provider’s pricing, rate limits, behavior changes, and policy calls. But the leverage is only real if the fallback is real, running in production with real evals, not a line in a deck that says “we could switch.” A backup you have never tested is not a hedge, it is a hope.

Inference efficiency is the next cost lever. The newest gains are not smarter models, they are faster serving. One Chinese system reports 57 to 85 percent faster generation and several times the throughput under heavy load, with almost no overhead. That attacks the exact cost structure the big labs depend on, so watch how models are served, not only how they score. The cheapest token often wins on plumbing, not intelligence.

Do this now

Run your models as a portfolio, not a religion. Use closed models where they clearly win, open weights where they are good enough or cheaper or more private, and keep the mix swappable. The right question is not “are open models better,” it is “where are they good enough to change my cost and control position.” A portfolio of models still needs governance, evals, and clear ownership, or it turns into expensive complexity.

Build the switching layer before you need it. Routers, OpenAI-compatible endpoints, eval harnesses, memory, logging, and clean abstractions let you change models without rewriting your product or going blind on quality. And treat your prompts, code, and agent traces as strategic assets: decide deliberately what flows to a closed API versus stays local. Those traces are training signals, not exhaust, and you only get to give them away once.

Optimize for resilient, not open or closed. If you remember one thing, make it this. The useful frame is controlled versus resilient. Closed AI gets dangerous only when it becomes infrastructure you cannot replace, because then the vendor’s terms, access rules, and whatever government pressure lands on them become part of your operating environment. Architect so that no single lab, or government, is ever a single point of failure. That is the whole game.

Where I think this all lands

First, the US posture toward China looks self-defeating. Blocking foreign nationals from a US model while clearing advanced chips for export does not slow China down, it hands it an opening and pushes its labs onto a homegrown stack, exactly what the earlier chip bans did. Cutting research ties and treating foreign talent as a threat drives away the people American labs depend on. Export controls have not stalled Chinese progress, and the damage lands on US companies and their customers instead. I would rather see cooperation than a wall, and the durable answer to strong Chinese open models was never to ban them, it is to build better ones.

Second, supply. Open weights are a business decision that can change on a dime, and I used to worry the supply could dry up. I am more optimistic now. Between consortiums, government and enterprise-backed efforts, and new labs stepping in, I expect a steady stream of capable open models, not a drought. Thinking Machines' Inkling makes the point: not the best model out there, but an open-weights base built for post-training and customization. You do not have to wait for the perfect open model to start cutting your dependence on a closed one.

That is the real challenge for OpenAI and Anthropic. In an earlier piece I argued the hybrid stack was coming for their pricing power, and this is the sharper version of that argument. Enterprises are now convinced of three things: they do not want vendor lock-in, they can offload a growing share of compute to open and smaller models they tune themselves, and they would rather not let their own data train someone else’s next model. Put it together and the days of a two-company oligopoly commanding most of the world’s inference are drawing to a close. Which may be why both are racing to go public, building a war chest before that pricing power erodes and pushing into domain-specific markets where an application is easier to defend than a token price. Smart move, and also a tell.

China's AI Export Controls at a Glance

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

What Startups Taught Me About the Next Layer of AI Infrastructure

A little while back I wrote about how teams use reinforcement learning (RL) to make agents reliable. Since then I keep bumping into startups where RL is not a research footnote or a feature buried in the stack. It is central to what they are building. I know of more than 25 at last count and still climbing, and in some cases RL is basically the product. It is still early, and much of the current tooling is aimed at frontier labs, neo labs, and sophisticated AI teams. Still, there is enough signal to think that some of the tools and lessons will eventually trickle down to the rest of us. At this point I treat my own RL timelines the way I treat horoscopes, entertaining but not binding. What I will say instead is that this particular wave has startups, products, and customers around it, which the previous periods mostly did not.

The New Infrastructure for Learning by Doing

The common thread is that almost all of these startups are reacting to the same frustration. Frontier models are smart but unreliable the moment you ask them to do real, multi-step work. RL is the lever they pull to turn raw capability into dependable behavior, whether the target is a coding agent, a support bot, or a humanoid robot.

But none of them are really selling RL as a magic algorithm. They’re building the missing machinery around it, and that machinery has two hard parts. The first is giving a model somewhere to practice, which is why so many are in the business of simulated environments and training gyms. The second, and trickier, part is scoring. Did the agent finish the task, follow the rules, pick the right tool, avoid a shortcut that games the metric? A surprising amount of this market is really about graders, verifiers, and the tooling to catch an agent cheating its own reward.

From there the goals split. Some startups build RL environments for frontier labs. Some build post-training platforms for companies that want specialist models. Some focus on continuous learning from production traces. Others carry RL into robotics, industrial control, and logistics. The common thread is simple enough to state plainly. If you can define the task, create a safe place to practice, and score the result, RL has something to work with.

Where Learning by Doing Pays Off

The first commercial cluster is exactly where you would expect. Coding, software engineering, ML engineering, security, and infrastructure tasks show up again and again because they come with built-in ways to score progress. Code can compile or fail. Tests can pass or fail. A sandbox can reveal whether the agent’s action changed the system in the intended way. The feedback is still imperfect, but it is much cleaner than asking a human to grade every step from scratch.

The second big cluster is computer use and enterprise work. These are agents that operate browsers, spreadsheets, CRMs, ticketing systems, internal tools, finance software, documents, and knowledge bases. This is where the difference between a demo and a production agent becomes painfully obvious. A model that can explain a workflow is not the same as an agent that can execute it, recover from mistakes, and satisfy a company’s standards.

Then there’s a smaller but striking set of use cases that leave the screen entirely: robots learning to walk and grip, data-center cooling, freight routing. The wider pattern is that RL is showing up wherever teams can create a loop. There has to be a task, a place to attempt it, a way to observe what happened, and a score that is close enough to what the user actually cares about. That pattern applies to coding agents, enterprise research, support workflows, financial modeling, medical reasoning, industrial cooling, logistics, and robotics. The domains differ, but the recipe is surprisingly consistent.

When Does This Reach the Rest of Us

The question I keep coming back to is when this becomes usable by a broader set of AI teams. Right now, a lot of the tooling still assumes serious technical depth. You need to define environments, write rewards, inspect rollouts, catch reward hacking, and connect the training loop to real systems. That is not casual weekend work. But when this many startups are building environments, post-training platforms, verifiers, agent gyms, and RLOps tooling, it is hard not to expect some of the abstractions to become simpler.

In one sense, the trickle-down has already started. The startups helping companies post-train or customize foundation models are bringing pieces of RL tooling into enterprise settings today. That’s RL tooling reaching real businesses today, not in some hypothetical future. The next step is not that every AI team becomes an RL research lab. It is that more teams get practical tools for defining the work they care about, measuring success, and improving models against those signals over time.

The AI Bubble Risk Map

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Your CLI Was Built for Humans, Not Agents

There’s a friendly debate among developers about how to give AI agents reliable ways to use external tools, data, and services so they can do useful work beyond generating text. One side favors CLIs, or Command-Line Interfaces, where agents run text commands against tools like git, aws, jq, duckdb and internal scripts. The other side favors MCP, or Model Context Protocol, which gives agents a structured way to discover tools, authenticate, and call services through defined schemas. This may seem confined to coding agent, but the tradeoffs apply equally to anyone deploying agents in operations, finance, customer success, or any workflow where agents need to touch real systems. Once an agent starts taking action, its interface with the outside world is no longer just a technical detail. It becomes a critical decision point for security, cost, and governance, regardless of whether that agent is writing code or managing customer data.

CLI: Fast, Familiar, and Sharp-Edged

The strongest case for CLI is that it is familiar, flexible, and cheap. First, AI models have been trained on enormous quantities of shell scripts, developer documentation, and terminal content, which means they already understand tools like git, docker, and kubectl, without any special configuration, and that familiarity translates directly into reliability and lower error rates. Second, CLI output is composable: an agent can pipe the output of one command directly into another to filter, aggregate, or reshape data before it enters the model’s working memory. Finally, the token efficiency advantage is real and measurable. Loading a single MCP server’s full tool schema can consume tens of thousands of tokens before any useful work begins, whereas CLI tools require zero upfront context cost and agents can discover a tool’s capabilities on demand using its built-in --help flag. The catch is that CLI access can be too powerful. Giving an agent raw shell access without sandboxing, credential isolation, and approval gates makes it incredibly difficult to prevent unauthorized changes to files and networks.

The MCP Case for Agents

MCP’s advocates have a direct response to the CLI-first view: CLI doesn’t account for how most AI is actually consumed in the world. The majority of people interacting with AI agents do so through browser-based chat interfaces or mobile apps where running shell commands is simply not an option, and many business tools have no CLI equivalent at all. For those contexts, MCP fills a genuine gap: it gives agents a self-describing interface where the agent discovers which tools are available and how to use them upon connecting, without needing to guess command syntax or parse documentation. Unlike CLI, MCP includes built-in OAuth authentication, per-user permissions, and audit trails that satisfy enterprise security requirements.

The main criticism is overhead. Poor MCP implementations can load huge tool schemas before useful work begins, burning tokens, adding latency, and leaving less room for reasoning. MCP can also introduce server failures, authentication friction, and harder debugging. The practical lesson is that MCP needs good interface design: smaller task-level tools, progressive disclosure, schema filtering, and gateways that show agents only what they need.

Your CLI Was Built for Humans, Not Agents

An agent can learn command syntax. The harder problem is that most CLIs were designed for humans, not software agents operating under time, token, and safety constraints. We easily overlook these flaws because humans are great at filling in the blanks. We can guess missing steps from bad manuals, decode vague error messages, and realize on our own when a tool requires us to log in before showing its options. An agent pays a concrete cost for every one of those gaps: repeated discovery loops, wasted tokens, fragile text parsing, and unnecessary interruptions asking the user what to do next. The CLI didn’t change. The user did.

The practical response is to treat CLI agent-readiness as something you can measure and improve, not just assume. An open-source project called CLIARE (CLI Agent Readiness Evaluation) does exactly this. It probes the actual installed binary, not your documentation, records what happens, and produces a scored report across six dimensions: whether commands are findable and navigable, whether valid invocations can be constructed, whether commands complete without hanging, whether bad inputs fail with clear errors, whether machine-readable output modes are available and actually work, and whether read-only probes stay read-only. The result is a scorecard you can run, track across releases, and share with the teams building agents on top of your tooling. That shifts interface quality from a guessing game into a measurable metric.

This matters well beyond coding agents. Companies can expose real operational authority through CLIs: deployments, finance workflows, database jobs, cloud control planes, support scripts, analytics pipelines, and internal platforms. If agents are going to touch those surfaces, the relevant question is not just “does a CLI exist?” It is “can the agent understand and use it without guessing?” AI teams exposing CLIs to agents should try CLIARE, or at least borrow its core idea: make the command surface measurable, versioned, parseable, and explicit about preconditions, outputs, side effects, and risk.

Match the Interface to the Risk

I would not treat this as a religious choice between CLI and MCP. The better question is what kind of authority the agent is being given, and on whose behalf it is acting. Use CLI when the workflow is local, composable, inspectable, and already lives close to the terminal: code, tests, logs, files, data inspection, build systems, mature vendor tools, and internal scripts. But if agents will use that CLI repeatedly, use CLIARE to measure the surface before relying on it.

Use MCP when the workflow is hosted, multi-user, permissioned, or tied to SaaS systems where OAuth, consent, scoped access, and audit trails matter. In both cases, raise the bar when the agent can touch money, identity, production systems, customer data, or irreversible state. The common design rule is simple: expose the task, not the whole system. Keep the interface small, prefer parseable outputs, separate reads from writes, log consequential actions, and require approvals for operations that cannot be easily undone. Transport choice matters. Interface design matters more.

The AI Developer Productivity Debate: A Field Guide

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Agents Need Maps, Not Bigger Context Windows

Like everyone else, I’ve been enjoying the steady improvement in coding agents and the tooling around them, from frameworks and harnesses to evaluation suites. But the more I talk with teams actually deploying agents in enterprises, the more I circle back to plumbing. Agents need data integrations and infrastructure built for them, a theme I explored in a piece on rethinking databases for agents. I’ve also written about using knowledge graphs (GraphRAG) to improve accuracy and reduce hallucinations, which is especially important in high-stakes applications. The persistent problem has been that most teams don’t have a knowledge graph, and the few that do struggle to keep it current. A new system I recently came across is aimed squarely at that gap.

Automating Graph Construction and Maintenance

I recently spoke with Andrew Moore, CEO of Lovelace, former head of Google Cloud AI, and former dean of Carnegie Mellon’s School of Computer Science. Lovelace’s Elemental platform turns enterprise data into structured context engines that AI agents can navigate and investigate. I think “knowledge graph” is still a reasonable term here, although “context graph” may be closer to the point. This is not a graph built mainly for a human analyst browsing nodes on a screen. It is designed for agents that need to answer questions too large, messy, or time-sensitive for ordinary search, basic RAG, or stuffing more raw documents into a context window.

Elemental ingests whatever you have: relational tables, PDF files, news feeds, JSON fragments, even satellite imagery and other multimodal sources. Everything gets converted into a common intermediate form that eventually becomes nodes and relationships in a graph. The platform automates much of the construction and maintenance of these graphs. A graph with a thousand nodes can tolerate a human in the loop, but once you’re at many millions of facts, any step that depends on a person checking whether two records refer to the same company simply collapses. Too many enterprise AI projects die in the six months of data wrangling that comes before any actual AI work, and removing that toil is the whole point.

For an agent, a context graph gives it a map of the enterprise and the surrounding world. A bank investigating exposure to a sanctioned shipping company might need to connect ownership records, trade data, vessel movements, subsidiaries, legal filings, customer relationships, and recent news. Vector search can find relevant documents, but these questions often require following relationships across many entities and events. That is where a graph-shaped context layer becomes useful.

Graphs for Agents, Without a Graph Database

Elemental is designed around four steps: data ingestion, entity resolution, graph construction, and agent-facing tool access. I covered ingestion above, but one constraint is worth adding: the system has to work without assuming clean schemas, consistent naming, or a single source of truth.

Entity resolution is the problem of deciding when different records refer to the same real-world thing. “IBM,” “International Business Machines,” an internal vendor ID, a legal entity in a filing, and a name in a PDF might all refer to the same company, or they might not. The same issue appears with people, ships, aircraft, accounts, subsidiaries, addresses, products, assets, and events. Anyone can prototype an entity resolution system in a weekend. Making one that scales, handles multiple languages, adapts to entirely new entity types, and updates as new information arrives is brutally hard. Lovelace built its own engine and goes well beyond text matching. I half-jokingly told Andrew that the entity resolution engine alone could be a standalone product, given how many applications live or die on getting this right.

Auditability is just as important. In regulated or high-stakes settings, the system needs to show where a fact came from, how it was linked to other facts, and what version of the graph existed when an agent produced an answer. In plain English, that means lineage, provenance, and version control: how a fact moved through the system, what evidence supports it, and what the graph believed at a point in time. That matters if a regulator, customer, or internal reviewer asks why an agent flagged a transaction, recommended an investigation, or dismissed an alert. “The model said so” will not be enough.

Retrieval is where the graph becomes more than a data catalog. In a serious graph, a single node can have hundreds of thousands of links, so a naive “show me everything within two hops” query explodes into billions of paths and grinds to a halt. Elemental takes a different route. It reframes a question like “who is most affected by this event?” as a probability problem, then uses randomized algorithms and other graph-theory tricks to surface the strongest connections without walking every path. The result is approximate rather than mathematically exact, but it returns confident, ranked answers over millions of nodes in under a second. That trade is well worth making, because the alternative is an LLM trying to reconcile contradictory probabilistic evidence in its head, which language models are genuinely bad at.

Another interesting design choice is that Elemental does not rely on a graph database as the core storage layer. The data is exposed to agents as a graph, but the persistence layer underneath is custom. The reason is simple. Traditional graph databases are very useful, but they were not built for this particular job: broad, high-speed inference across tens of millions of nodes.

Why Private Graphs Need Public Context

Lovelace also brings another asset to the table: YottaGraph. Think of it as a large, continuously updated graph of the public world, covering companies, people, places, events, filings, sanctions records, and the movements of ships and aircraft. This is useful because internal enterprise data rarely tells the whole story. YottaGraph would be hard to replicate because it requires continuous ingestion from many sources, entity resolution at scale, and a lot of operational discipline. Importantly, the enrichment flows inward. Public context can augment a company’s private graph, but Lovelace does not share one customer’s data with other organizations.

Lovelace isn’t alone in using graphs to give agents better context, and I expect to see much more activity here. There is also a funny twist. GraphRAG was promoted heavily by graph database vendors, but some of the more interesting graph systems for agentic AI may not use graph databases at all. Whatever we end up calling these things, knowledge graphs, context graphs, or context engines, the underlying idea of structured, auditable context for agents is becoming hard to ignore.

OckBench: The Cost of Getting the Right Answer

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

The Data Compliance Problem AI Teams Keep Ignoring

I’ve avoided writing about copyright and AI. Not because it isn’t important, but because it felt like a legal sideshow compared to the engineering and business questions I find more interesting. That’s gotten harder to justify. There is also something mildly funny about naming my podcast The Data Exchange years ago, because I believed data markets would become central to technology, and then mostly ducking this corner of the data conversation. The data question is now central to how AI systems are built, how they compete, and how they create risk. You can’t talk seriously about any of that without eventually bumping into who owns the stuff these models are trained on.

For AI teams, data is no longer just raw material. One recent economics paper estimated that the value of data missing from official GDP figures amounts to about 4% of GDP on average, and closer to 6% in more recent years. More interesting for companies is the relative value of their own operational data: transaction logs, usage records, support tickets, product telemetry, and customer interactions. That internal data can be more informative than public web data because it captures how a real business actually works. For AI builders, the practical implication is simple: the data sitting inside your company may be more strategically valuable than anything you could scrape from the open web.

The Data Supply Chain Problem

The first practical issue is provenance: knowing where training data came from, what rights apply, and whether it can be inspected or reconstructed later. That used to sound like paperwork. It now looks like core infrastructure. What a model knows, where it performs well, where it breaks down, and what legal exposure it carries are all shaped by the data it was trained on. Pre-training is the broad phase where a model absorbs patterns from large collections of text, images, code, books, journalism, and other sources. Post-training is where you steer the model toward specific tasks, behaviors, and domains using narrower, more curated datasets. Both phases are data strategy questions as much as engineering questions. For most teams, the immediate action is not to audit a trillion-token pre-training corpus. It is to get serious about the smaller, higher-value datasets they use to adapt, evaluate, and deploy models.

Memorization adds a second layer of risk. A model does not need to act like a database to create trouble. Under the right prompts, especially when material appears many times in training data, it may reproduce protected text, code, lyrics, images, or paywalled content too closely. That makes copyright risk a lifecycle issue, not just a training-data issue. Hosted models give providers more room to monitor, filter, and rate-limit suspicious behavior. Released model weights are harder to control. Either way, teams need to test for verbatim reproduction, near-duplicates, and outputs that could substitute for the original work.

Guarding Your Post-Training and Fine-Tuning Pipelines

Most AI teams aren’t doing pre-training. They’re taking a foundation model and adapting it: fine-tuning on domain data, building instruction datasets, running human feedback loops, assembling evaluation suites, or wiring up retrieval pipelines. That’s where the real data work happens for the majority of developers, and it comes with its own provenance problems. The data you use to customize a model, customer support logs, internal documents, expert annotations, licensed content, third-party datasets, needs the same documentation discipline as any other part of the stack. Where did it come from? What rights apply? Was it collected with appropriate consent? Can you reproduce it if challenged? These questions matter whether you’re fine-tuning a vertical copilot or building an evaluation harness, and the answers are often murkier than teams expect. A lightweight data bill of materials can help: a record of each dataset, its source, license terms, consent status, known restrictions, and intended use.

There’s also a technical risk that’s easy to underestimate in post-training contexts. Fine-tuned models can memorize training examples more readily than large pre-trained models, partly because the datasets are smaller and examples may repeat. A customer-facing model fine-tuned on proprietary documents, licensed text, or third-party content could reproduce that material verbatim in ways that create real liability. That risk doesn’t disappear after training ends. Output monitoring matters: similarity detection, content flagging, and policies for handling prompts that look like attempts to extract protected material should be part of any production deployment. No single control handles all of it, and the teams most likely to get caught flat-footed are the ones who assumed that using someone else’s base model meant the data compliance problem was already solved.

Who Owns the Fuel That Runs These Models

We are also seeing the rise of a formal data licensing market, where major publishers and content platforms sign deals with AI labs. While this reduces legal uncertainty for those who can afford it, it also threatens to create a massive moat for wealthy incumbents. Large AI firms can afford broad or exclusive deals that smaller labs, startups, researchers, and open-source projects cannot.

The legal landscape is also evolving unevenly around the world. The U.S. still relies heavily on case-by-case fair use arguments. Europe is pushing more toward transparency, opt-outs, documentation, and compliance processes. Some Asia-Pacific jurisdictions appear more permissive for computational analysis, but global deployment complicates everything. A model trained in one place can be used, challenged, or regulated somewhere else. In practice, many companies will opt to satisfy the strictest plausible standard, especially if they sell to large enterprises or regulated industries.

This is why AI and copyright no longer feel like a legal sideshow to me. They are part of a larger question about how data markets will actually work: who gets paid, who gets access, who can prove what they used, and who can afford the cleanest inputs. The industry still has to find a workable balance between adaptation and attribution. Data is no longer just exhaust from the internet or residue from business operations. It is becoming one of the main assets, bottlenecks, and fault lines in AI. The teams treating it as a compliance checkbox are going to find that out the hard way.

The Runtime Pattern Behind AI Agents

Claude Code to Cursor: a poem

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

The Hybrid AI Stack Is Coming for the Pricing Power of OpenAI and Anthropic

OpenAI and Anthropic are going public while still capturing much of the money spent on foundation-model usage. But deployment patterns are starting to tell a more complicated story. Companies are building hybrid model portfolios, using proprietary models where convenience, support, and frontier capability matter, while turning to open-weights models where cost, privacy, customization, and deployment control matter more. As teams get better at operating multiple models, the single-vendor AI stack will look less like the default and more like a transitional phase.

What Open Weights Give You And What They Do Not

The economics are hard to ignore. In an earlier piece, I argued that the CFO has become the CTO, the Chief Token Officer. That framing applies directly here. Proprietary APIs are still useful for prototyping, broad assistants, and difficult reasoning tasks. But once a workflow becomes stable and high volume, token-based pricing starts to feel like a tax on scale. Open-weights models can cut unit costs sharply for repeatable work such as document processing, classification, extraction, internal search, customer support, and summarization. The savings are not automatic. Infrastructure and engineering costs have to be absorbed first. But for organizations running millions of daily inferences, the case can become compelling much sooner than many executives expected.

The appeal is not just lower cost. Open-weights models give enterprises more control over where data goes, how models are adapted, and which version is running in production. That matters for banks, insurers, healthcare organizations, manufacturers, software companies, and any business handling sensitive records, proprietary code, contracts, customer conversations, or regulated data. A model that runs in a private cloud, on-premises environment, or approved regional infrastructure can be easier to govern than one accessed only through a third-party endpoint. With open-weights models, enterprises can select a model version, test it, approve it, and keep running it. This is valuable for regulated or high-stakes systems where silent behavior changes create audit and compliance problems. Fine-tuning and post-training lets teams adjust model behavior without retraining the whole system. RAG, or retrieval-augmented generation, lets a model consult a curated internal knowledge base before answering. Together, these patterns allow smaller models to perform well on company-specific workflows.

The barriers are just as practical. Switching to open weights is not like swapping an API key. Proprietary platforms package hosted endpoints, scaling, documentation, security features, enterprise support, and contractual accountability. Open-weights models shift more of that burden to the buyer. Teams need GPU planning, model serving, inference optimization, monitoring, evaluation, logging, access control, fallback paths, and cost tracking. The total cost of ownership is easy to underestimate because the API bill disappears, but the operational work does not. In some organizations, infrastructure and specialized engineering talent can become the dominant cost. Open weights can improve economics, but only for teams mature enough to run them well.

The governance issues are even harder to wave away. Proprietary platforms usually include safety layers and enterprise risk support. Open-weights models often require companies to build those controls themselves. Input/output guardrails, red-team testing, incident response, secure artifact management, and model provenance all become first-class engineering concerns. Tools such as Heretic show why this matters. Safety guardrails can be stripped from open-weights models quickly, which raises obvious compliance and misuse risks. Legal teams also have real work to do. “Open-weight” does not mean unrestricted. Licenses may limit commercial use, redistribution, scale, geography, or specific capabilities. And when these models are used for code generation, companies need processes for scanning outputs for vulnerabilities and license conflicts.

The New Enterprise Model Stack

The next phase is not open versus closed. It is workload-specific model selection. A routine classification task can go to a smaller open-weights model. A sensitive internal search workflow can run inside private infrastructure. A difficult reasoning problem can be escalated to a proprietary frontier model. This is the real meaning of hybrid model portfolios. The architecture is no longer about picking one best model. It is about building a routing and governance layer that can evaluate requests, choose the right model, monitor cost and quality, and replace models when needed. I already see this in my own coding workflow. My O² toolkit, using opencode and OpenRouter, is useful precisely because different models are better for different jobs.

This hybrid approach becomes even more critical as we move toward agentic workflows. When you build autonomous agents that continuously run loops, call tools, and handle background tasks, your token consumption scales dramatically. If you run these entire multi-step workflows on premium proprietary endpoints, your monthly API bill will explode. Instead, a smarter architecture uses smaller, local open-weights models to handle the routine, repetitive steps in an agent’s loop, while escalating only the high-stakes decisions or complex reasoning tasks to closed frontier models. This setup also keeps the agent’s sensitive tool integrations and internal system commands within your own secure network rather than routing them through external APIs. The catch is that agents make the control layer harder to build. Every action needs to be logged, auditable, and ideally reversible, and most teams deploying agents have not built that infrastructure yet.

This shift in workload routing has uncomfortable implications for OpenAI and Anthropic as they move toward IPOs. Their revenues may continue to grow quickly, but pricing power will face pressure from below. If open-weights models are good enough for a larger share of routine inference, enterprises will become more selective about when they pay premium API prices. They will still pay for frontier reasoning, managed compliance, uptime, enterprise support, convenience, and strong guardrails. But they will be less willing to send every summarization, extraction, routing, tagging, retrieval, and support workflow to the most expensive model. Revenue can grow while the price umbrella weakens.

The biggest uncertainty is supply. Open weights are not a guaranteed public good. They are a business decision, and business decisions can be reversed. OpenAI demonstrated that years ago. Meta now appears less committed to Llama as an open ecosystem than many expected. At the same time, the open-weights landscape has become more concentrated among Chinese labs, and even that supply is narrowing as some providers move toward closed or more restricted licensing under revenue pressure. This is the part of the story that deserves more attention. Enterprises should absolutely take open weights seriously, but they should not assume the ecosystem will remain abundant, permissive, and stable. The winning teams will be the ones that can test, route, govern, and replace models without rebuilding their AI stack every time the market shifts.

Tokenomics: AI’s New Design Constraint

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

The Gap Between the Press Release and the Power Grid

Back in February, I wrote about what I called the “Data Center Rebellion,” the growing local resistance to the physical infrastructure behind AI. Since then, I have been asking tech people around the Bay Area how closely they are following the backlash. The answer is usually: they know it exists, but not much more than that. There is still a quiet assumption that most of these announced campuses will get built, plugged in, and brought online more or less on schedule. I am much less sure. What looked like a scattered set of zoning fights has hardened into something more organized, more politically potent, and more consequential for anyone trying to think clearly about AI infrastructure timelines. The opposition has gone from a speed bump to a genuine constraint.

The striking part is how broad the opposition has become. Polling suggests that resistance to local AI data centers is now a mainstream position, not a fringe one. And the “AI” label matters. Data centers have become a visible target for wider concerns about corporate power, electricity costs, water use, job displacement, surveillance, and who actually benefits from the buildout. The politics have also gone cross-ideological in ways that make this harder to dismiss. Environmental justice advocates, rural conservatives worried about local control, and labor groups anxious about automation are all finding common cause at the zoning board.

The Local Backlash Gets Smarter

The objections are practical and increasingly specific. Communities are worried about water in dry regions, electricity demand on strained grids, air pollution from backup power, the constant hum of cooling systems, farmland conversion, tax breaks, and the small number of permanent jobs these projects often create. The industry tends to lead with billion-dollar investment figures. Residents tend to ask a simpler question: what do we give up, and what do we actually get?

The tactics have matured too. This is no longer just online frustration or a few angry public meetings. Residents are using moratoriums, zoning challenges, lawsuits, ballot measures, protests, water-rights filings, and elections. What makes the backlash more durable is the trust problem. Shell companies, project code names, NDAs, fast-tracked approvals, and vague end-user disclosures make communities feel boxed out. Once that trust is gone, even reasonable technical claims start to sound like sales material.

Microsoft’s recent pledge to stop requesting NDAs is a useful illustration here. It’s notable precisely because it signals how widespread the practice has become. When a company feels compelled to make that kind of pledge, it’s an admission that the old playbook has become a liability.

Announced Capacity Is Not Real Capacity

Local opposition gets much of the attention, but it is only one constraint. Even if a project clears the political process, it still has to get power, transformers, electrical equipment, GPUs, memory, networking gear, cooling systems, utility approvals, and enough skilled construction capacity. In many markets, the bottleneck has shifted from “can you get the chips?” to “can you get the megawatts?” A site can have land, permits, and a glossy announcement and still sit idle because the electrical infrastructure is not ready.

That’s why the gap between announced capacity and capacity under construction matters so much. Of the roughly 12 GW of U.S. data center capacity announced for 2026, only about 5 GW is under active construction. That ratio gets worse, not better, for later years. The industry’s pipeline looks enormous in press releases and investor decks, but only a fraction appears to be moving toward actual completion on the promised timeline. The word “capacity” can mean too many things: land under option, a permitted site, a building shell, reserved power, an energized hall, installed racks, or revenue-generating compute. Those are not minor distinctions. They are the difference between a story the market wants to hear and an asset that can actually serve AI workloads.

The Trust Deficit Becomes an Execution Risk

The industry is responding, but mostly at the edges. Some developers are revising site plans, changing cooling designs, offering community benefits, making renewable-energy commitments, or becoming more selective about where they build. But they do not fully answer the harder questions: who pays for grid upgrades, who gets the water, who absorbs the noise and pollution, and who has the right to say no?

What concerns me is the financial exposure accumulating underneath all of this. More and more of this buildout is being funded through debt, long-term lease obligations, and capacity commitments made against a pipeline that is, in many cases, more announced than actually under construction. The hard question is whether the math works even if the projects do get built. Hyperscalers are spending as if AI infrastructure will unlock a very large new revenue pool. Maybe it will. But if capital spending keeps rising faster than revenue, the industry may discover that “demand” and “attractive return” are not the same thing. Either AI generates much more revenue than analysts currently expect, or some planned spending gets pushed out, scaled back, or canceled. That second outcome might not look like a crash. It could look like delayed campuses, slower GPU orders, tougher financing terms, and more “rephasing” language on earnings calls.

This is why I would not treat local opposition as a side story. The compute demand behind AI is not going away. That does not mean every project deserves a rubber stamp. It means the industry has to get much better at earning trust and building capacity that communities, utilities, and investors can actually live with. If AI infrastructure spending is helping support the broader economy, then a serious pause becomes a macro risk, not just an AI story. A slowdown isn't even the worst version of this. It is an AI economy where compute becomes scarce, expensive, and concentrated in the hands of the few companies and customers that can afford it. That would be a bad outcome for everyone who wants AI to become broadly useful, not just broadly hyped.

• The MANIAC. I know this is a 2023 book, but it feels even more worth reading now. As AI reshapes computing, research, and mathematics, this portrait of John von Neumann made me wonder what one of history’s great mathematical minds would make of the machines we’ve built. This is historical & biographical fiction at its best 💯

• Steve Jobs in Exile: The Untold Story of NeXT. I had a couple of professor friends who were devoted NeXT users, and this helped me understand why that little black cube inspired such loyalty. This is a sharp, readable look at Jobs’ wilderness years, and how what looked like a detour ended up shaping the Mac, the iPhone, and the tools we use every day.

• Inside the Box: How Constraints Make Us Better. I liked this book because its core idea feels especially relevant to AI right now. DeepSeek and other Chinese model builders are a good reminder that constraints do not always slow innovation down, sometimes they force teams to get sharper, scrappier, and more creative.

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Your Enterprise Data Deserves Better Than a Chatbot

Large language models and their multimodal variants remain the foundation models most people encounter first. That makes sense. Text, images, audio, and video cover a huge range of knowledge-work tasks, and today’s chatbots are far more capable than the text-only systems many people first tried. But enterprise AI does not run on chat alone. It runs on tables, time series, transactions, telemetry, product catalogs, customer histories, service graphs, and messy operational data that rarely fits neatly into a prompt. Using current coding agents for data science makes this gap concrete. Generating Python and SQL is easy for an agent or an LLM, but data science requires a uniquely human skepticism to contextualize messy data and recognize when a result is simply too good to be true. We may finally be moving past the TINA phase of foundation models for enterprise AI, where the answer to every problem has been “just use an LLM.” A new wave of frontier models look more specialized and useful for the prediction and decision problems that actually run businesses.

Structured and Semi-structured Data Get Their Foundation Model Moment

Kumo’s foundation mode is aimed at one of the most valuable categories of enterprise data: structured relational data. Think customer records, orders, transactions, risk signals, and event histories spread across warehouses and lakehouses. The argument is straightforward: businesses still rely on slow, hand-built ML pipelines, while LLMs lose too much structure when tables are flattened into text. AutoML tried to automate this work, but mostly by tuning the old workflow of joins, features, and model selection. Kumo’s relational foundation model takes a different route: it treats a database as a graph, where rows and tables become connected entities. Instead of manually building features, the model attends over raw relational context and generates predictions such as churn risk, recommendations, fraud likelihood, or customer lifetime value. The appeal is workflow compression: fewer bespoke pipelines, faster iteration, and a prediction layer that agents or applications can call on demand.

The newer KumoRFM-2 release sharpens that pitch. It is described as a pre-trained model that can make predictions directly over multi-table databases using only a small number of labeled examples at query time. In other words, teams can ask predictive questions across connected tables without first flattening the data, building features, training a task-specific model, and maintaining a separate production pipeline. The reported benchmark results are notable because KumoRFM-2 claims to beat both supervised baselines and other foundation-model approaches on relational tasks while using a tiny fraction of the available labels.

That said, I would not treat this class of foundation models as a blanket replacement for every predictive system. For lower-stakes routing, forecasts, personalization, fraud triage, and operational scoring, the value could be immediate. For quantitative finance, regulated credit decisions, medical risk scoring, or underwriting, I would expect a more cautious path: careful validation, calibration checks, monitoring, challenger models, explainability review, and human governance before deploying to production.

Kumo isn’t the only active effort here. Prior Labs is pursuing foundation models for tabular data with TabPFN, a system aimed at row-and-column prediction problems such as churn, fraud, pricing, demand forecasting, credit scoring, predictive maintenance, and clinical risk. Its pitch is similar in spirit: reduce the amount of preprocessing, feature engineering, tuning, and model selection needed before teams get useful predictions. This momentum extends to temporal data, where an expanding roster of time-series foundation models can now generate reliable forecasts without prior, domain-specific fine-tuning. The important lesson is that structured enterprise data is becoming a first-class target for foundation-model development, with architectures that target tabular, relational, and temporal data.

Datadog is moving in a related direction, though it uses different language. At the AI Agent Conference, Ameet Talwalker described work on what Datadog calls a “world model” for observability. I think it belongs in this discussion because the inputs are largely structured and semi-structured operational data: metrics, logs, traces, service topology, code, events, alerts, and incident history. The goal appears to be a foundation model that learns how production software systems behave over time, not just a dashboard that summarizes what already happened.

The Toto 2.0 release strengthens that story. Toto began as a time-series foundation model for observability data, and the new version turns it into a family of open-weights models that scale from 4M to 2.5B parameters. Datadog’s results suggest that larger Toto models keep getting better, run much faster than the first generation, and generalize beyond observability despite being trained largely on observability and synthetic data. For enterprise teams, the practical implication is that telemetry may become a richer prediction layer for incident detection, root-cause analysis, simulation, and agentic remediation. The world model effort is the next step in that direction, moving beyond forecasting individual streams toward a learned model of how distributed systems fail, recover, and respond to change.

AI Moves Closer to the Point of Work

The same push toward specialization applies to how models interact with people, not just what they predict. Thinking Machines is training what it calls an “interaction model” for continuous, two-way exchange across audio, video, and text, rather than the familiar turn-taking pattern where a user speaks, waits, and receives a finished response. The key design choice is that the model works in real time, listening, responding, and tracking visual cues while deeper reasoning and tool use happen in the background. The enterprise relevance is direct: customer support, field service, sales coaching, clinical workflows, industrial operations, and design reviews all involve interruption, demonstration, and real-time correction. These are not clean prompt-and-response tasks. In many settings, the best interface is less like filing a ticket and more like working alongside a capable colleague. This is still a research preview with real constraints around connectivity and session length, but the direction is one that product and platform teams should be tracking.

There is also a related wave of specialized interface models that expand where AI can be embedded in workflows. Google DeepMind’s AI-enabled pointer imagines assistance that follows users across applications, understands what is on screen, and accepts natural instructions like “fix this” or “move that” without requiring a carefully written prompt. OpenAI’s real-time voice models point in a similar direction, with separate models for live reasoning, translation, and streaming transcription. The common thread is that these are less about building a general chatbot and more about making AI useful at the point of work. Put those alongside the structured data and observability models covered earlier and a clearer picture emerges: enterprise AI is developing a stack of specialized models, from prediction engines on relational and telemetry data to real-time interaction layers, all coordinated around the workflows where value is actually created.

Foundation Models Start to Specialize

World models is an emerging category I have not treated fully here. There is a lot happening, and some of it is genuinely interesting. But the world-model work I know best is not yet centered on the enterprise workflows I spend my time thinking about. My recent conversations with the founders of Rhoda AI and Odyssey made that clear. Rhoda is applying video-native foundation models to robotics tasks such as decanting, return processing, and container breakdown. Odyssey is building interactive world simulations, with early relevance for gaming, robotics, media, and other visual environments. These are important directions, but they sit closer to physical systems and simulated worlds than to the messy operational data, prediction problems, and knowledge-work interfaces that dominate most enterprise AI roadmaps.

The bigger takeaway is that foundation models are starting to specialize in useful ways. LLMs and multimodal chatbots will remain central, but they will not carry the enterprise AI stack by themselves. Enterprise AI will likely depend on a portfolio of more targeted models: relational models for structured data, time-series models for forecasting, observability models for production systems, interaction models for real-time work, and eventually world models where simulation and physical reasoning matter. The better end state is not one giant model that handles everything, but a routing layer that reads the task and directs it to the right model for the job.

When Agent Teams Reach for rote

Explore rote

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Beyond the Demo: What Real AI Agents Actually Do at Work

I am always on the lookout for new AI agents and applications that operate outside the coding world. By agent, I mean a system that can take a goal, use tools, keep context, and work through several steps rather than simply answer a prompt. Looking through my notes from the recent AI Agent Conference, I put together a few standout examples drawn from conversations with the people who built them and friends at the conference. What stands out is not that these systems are magical. It is that they are showing up in ordinary business workflows where speed, judgment, and access to the right data matter.

When Marketplaces Start Doing the Work

Upwork’s Uma Recruiter starts from a familiar problem: hiring is not just search. A client may write an incomplete job post, miss important constraints, or fail to separate required skills from nice-to-have ones. Uma Recruiter turns that messy input into a structured hiring task, then searches the talent marketplace and evaluates candidates across signals like past work, portfolio depth, availability, prior client relationships, and Job Success Score. It then invites promising candidates and ranks incoming proposals. Upwork is treating a job post less like a one-time query and more like the start of a multi-step recruiting process. Early results suggest this is not just a demo: Upwork says Uma Recruiter can produce shortlists within six hours, initial testing showed a 30 percent increase in hires using the shortlist, time to hire fell 11 percent, and jobs filled within seven days rose 10 percent between November 2025 and March 2026.

DoorDash’s merchant tools are less like a single recruiter-style agent and more like a bundle of AI-enabled workflows placed directly into merchant operations. The core idea is to remove setup and growth tasks that independent restaurants often lack the time or staff to handle. Its AI-powered onboarding pulls information from a merchant’s existing web presence, including photos, hours, and menu items, so a merchant can review and edit rather than start from scratch. DoorDash says this helps merchants launch more than 35 percent faster. The suite also includes AI photo tools that retouch or replate food images while preserving the underlying dish, AI-generated branded websites for direct ordering, and AI-assisted marketing campaigns for email content and scheduling. Millions of photos have been enhanced, AI-powered websites are seeing nearly 10 percent average order conversion, and merchants report meaningfully reduced effort across onboarding and marketing.

The common thread is that both companies are using AI to make their platforms easier to use, not to replace the marketplace itself. Uma Recruiter is more explicitly agentic: it has a reasoning core, specialized tools, memory, and an iterative plan-act-assess loop that runs until it has a strong shortlist. DoorDash is more embedded and task-specific, appearing as product features that reduce merchant friction at known points in the lifecycle. For builders, the distinction matters. Upwork is automating a judgment-heavy workflow where the system must reason across candidates and outcomes. DoorDash is automating setup, content, and marketing jobs where usability, guardrails, and tight integration into the merchant portal may matter more than a visibly autonomous agent.

Two Paths to the Enterprise Agent

Meta’s AI Second Brain addresses a problem most knowledge workers recognize immediately: the context needed to do your job is scattered across documents, meetings, tasks, code reviews, and internal discussions. Rather than starting each session cold, the system gives an AI agent a persistent workspace organized around the PARA method, a lightweight structure for Projects, Areas, Resources, and Archives that tells the agent what you are actively working on, what to load on demand, and where to route new information like meeting notes. The agent connects to internal tools through authenticated MCP servers, which give it scoped, permissioned access to internal systems, and command-line interfaces. The adoption numbers are striking: more than 63,000 installs in roughly three months, around 10,000 daily active users, and thousands of user-created skills written in plain markdown. That kind of internal traction usually means the tool solved a real problem.

EY’s agentic AI effort targets a very different kind of knowledge work: audit and assurance. The firm is embedding a multi-agent framework directly into EY Canvas, its global assurance platform built on Microsoft Azure, Microsoft Foundry, and Microsoft Fabric. The system is designed to help audit teams orchestrate complex tasks, surface updated accounting and audit guidance, sharpen risk assessments, and reduce administrative overhead, while keeping human judgment explicitly in the loop. The scale is institutional rather than viral: 130,000 assurance professionals, 160,000 audit engagements, more than 150 countries, and a roadmap to support end-to-end audit activity by 2028.

The shared lesson is that internal agents need far more than a capable model underneath them. They need access control, structured context, workflow integration, and a clear boundary between AI assistance and human decision-making. Where the two cases diverge is in philosophy. Meta’s system grew from the bottom up: users write their own skills, teams extend workflows, and adoption spread because the entry cost was low and the system was composable. EY’s deployment is top-down by design, embedded inside a regulated professional platform where governance, quality controls, and training matter as much as productivity gains. Some agents spread because people can adapt them. Others earn trust precisely because they cannot be adapted freely.

When the Data Is the Moat

RealAI is an AI real estate analyst for investors, multifamily professionals, agents, homebuyers, and renters. The pitch is that property and market analysis that once required specialized teams or expensive institutional tools that can now be done conversationally. RealAI can compare markets, evaluate properties, model returns, and surface data on rents, sales histories, demographics, household financials, migration trends, and market dynamics across U.S. residential properties. Fundrise claims underwriting that previously took days can now be completed in seconds. The owner-developer of New York City’s World Trade Center, said complex analyses that once took days can now be completed in minutes. RealAI is priced accessibly for individual investors and smaller firms that have historically been priced out of this kind of intelligence.

The most important part of RealAI may not be the chat interface or the agent harness, the layer that routes tasks to the right tools and data sources. It is the data underneath it. Fundrise describes a system trained on tens of thousands of hours of internal team experience and a database of roughly 3.5 trillion data points covering every residential property in the country, drawn from public records and private databases and updated in real time. The interface can be redesigned and the orchestration layer can be swapped out. But assembling, cleaning, connecting, and continuously refreshing domain-specific data at that scale is the genuinely hard part. In many applied agent systems, durable advantage lives below the model, in the data.

What Makes Agents Stick

These cases make me less interested in agents as a category and more interested in the conditions that make them useful. The pattern is not simply “give a model tools.” The better examples sit inside real work loops, draw on domain-specific data, preserve human control where judgment matters, and have enough memory to carry context across steps without dumping everything into every session. They also need less glamorous machinery: access control, retries when tools fail, logging that explains why the agent acted, evaluation tied to business outcomes, and cost discipline so every step does not require the most expensive model. That is where many agent projects will succeed or fail, not in the demo.

The workforce question is also sitting just below the surface of most of these deployments, even when it goes unspoken. Fundrise’s Ben Miller is unusually blunt about the labor consequences. He says AI will cause job losses across commercial real estate, and while Fundrise has not laid people off, the company has slowed hiring. That is a more candid framing than most companies are willing to offer publicly, and it is worth taking seriously. The systems described here are not replacing human judgment in the high-stakes moments, but they are compressing a significant amount of the work that surrounds those moments. For anyone building or buying agents, the value is real, but so are the organizational consequences.

When AI Found the Counterexample

Vatican to AI Leaders: Efficiency Is Not Enough

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Integration Is the New Moat: Moving Beyond the LLM

The AI Agent Conference in New York was one of the better events I’ve attended to get a read on what’s actually happening with enterprise AI. The formal sessions were great, but the hallway conversations was where I got the inside scoop. The consistent message: deploying AI agents is much harder than most organizations expect, and the reasons are rarely the ones they anticipate. What follows is my attempt to distill what I heard into a practical view of why enterprise agents are so hard to deploy.

The first barrier is integration. In one enterprise after another, agents run into older CRMs, finance systems, document stores, and homegrown tools that were never designed for autonomous software. What looks like a wiring problem usually isn’t. A logistics company built a capable agent that fell apart the moment it had to touch their order management system, a decade-old platform never designed to be queried by software making autonomous decisions. The agent wasn’t broken. The process it was dropped into was. The worst failures are often the ones that stay hidden. A financial services firm ran an agent successfully in its test environment for months, only to discover during a quarterly audit that production CRM records had silently stopped updating. No error, no alert, just bad data accumulating for three months. Teams that treat integration as a technical handoff rather than a workflow redesign problem consistently get stuck at the same wall.

The data situation compounds this at every layer. Enterprise knowledge doesn’t live in clean, queryable databases. It lives in Confluence pages nobody maintains, Slack threads from two years ago, and the heads of three people who’ve been at the company long before the current CTO arrived. One study found that more than a quarter of agent deployment failures trace directly to critical knowledge that was never captured anywhere a system could reach. When agents fail on company-specific terminology, like non-standard product codes or internal procurement shorthand, the instinct is to upgrade to a more powerful model. That instinct is almost always wrong. The fix is domain-specific examples and better knowledge capture, not a bigger model. And underneath all of it, security cannot be an afterthought. Agents need governed, scoped access to data, with proper permissions and audit trails built in from the start. Without that foundation, even a well-integrated, well-trained agent is a liability waiting to surface.

Agents Don’t Fix Broken Processes, They Find Them

But even when integration is solid and data is in order, most deployments hit a third wall that almost nobody budgets for: the organization itself. Agents do not arrive as neutral software upgrades. They ask people to change how work is routed, approved, measured, and owned. In a hospital system, that might mean deciding whether an agent can prepare a prior authorization packet before a clinician reviews it. In an industrial manufacturer, it might mean exposing the informal workaround a plant manager has used for years because the official process is too slow. In an insurance operation, it might mean discovering that no one can explain who is accountable when an agent recommends a coverage decision that later gets challenged. These are not edge cases. They are the work.

This is why early agent projects often fail in a way that is politically expensive. A weak first deployment does not just miss a KPI. It convinces managers, operators, and risk teams that the whole category is immature or unsafe. The harder second attempt, the one that would involve process audits, clearer ownership, better evaluations, and serious change management, may never get funded. In practice, deploying agents means changing the organization while the organization is still running. That is slower and messier than a demo, but it is also where the real implementation work begins.

Compounding all of this is a talent problem with no clean solution. Deploying agents at scale requires people who understand process design, LLM behavior, systems integration, and organizational change, all at once. That combination is genuinely rare, and companies that hire for one or two of those skills and assume the rest will follow are setting themselves up for the same failure pattern, just with different symptoms.

Agents Are Implemented, Not Installed

The real implementation work isn’t getting an agent to run correctly in isolation. It’s getting the agent to run correctly inside a live organization, where the work is distributed across roles, the accountability is murky, and the processes were designed around human judgment calls that nobody ever wrote down. What looks like an AI project is usually a process redesign project with an AI component attached. The companies that figure this out early tend to scope their first deployments around a specific, broken workflow rather than a whole function or department. Not “automate the sales team,” but “automate the part of lead qualification that requires pulling the same three fields from two systems and writing the same email 200 times a week.” That narrower target is less exciting to demo but far more likely to survive contact with the organization. Enterprise agents are not installed. They are stood up, governed, monitored, and gradually expanded.

The implementation gap has also created a staffing model that most organizations haven’t fully absorbed yet. The most effective deployments involve a dedicated operational owner for each agent in production, someone who sits at the intersection of the business workflow and the technical system, and who can tell the difference between an agent that’s working and an agent that’s producing plausible-looking output that nobody has actually verified. Some vendors have formalized this with embedded specialists, technical product managers or forward-deployed engineers whose job is to live inside the customer’s workflow until the deployment actually holds. Without that kind of ownership, agents in production become side projects maintained by whoever built the prototype, and the research bears this out: organizations that skip dedicated ownership are dramatically more likely to face failures that require rolling the whole thing back. The talent profile this requires doesn’t map cleanly onto any existing job title, which is part of why so many companies are either building it from scratch or outsourcing it to implementation specialists who’ve learned these lessons on someone else’s dime.

Integration Is the New Moat

The next durable advantage in enterprise agents will not come from picking a cleverer model. It will come from making agents usable inside messy, specific, high-stakes workflows. A legal team does not need a model that can sound like general counsel in the abstract. It needs a system that can find the right contract, apply the company’s negotiation playbook, respect approval thresholds, and leave a clean audit trail. A customer support operation does not gain much by giving every human agent a better copilot. It gains real leverage by redesigning the service so routine cases resolve end to end, with people handling the work that requires judgment, empathy, or escalation.

A manufacturer running fragmented ERP, procurement, and plant systems will not close the competitive gap by waiting for larger models. It will close the gap by modernizing the connective tissue of the business. The raw intelligence of foundation models is becoming a commodity faster than most people expected. What remains scarce is the integration depth, the governed data access, the workflow redesign, and the operational ownership that make agents actually hold in production.

That gap is also a market. Vendors, integrators, and internal transformation teams that can do the unglamorous work, connecting legacy systems, capturing institutional knowledge, building evaluation frameworks, managing the organizational change, are sitting in front of a real and durable opportunity. The distance between what foundation models can theoretically do and what enterprises can actually deploy is not closing on its own. The companies that treat integration, governance, and workflow redesign as the product rather than the plumbing will be the ones that turn agents from demos into operating leverage.

The Three Pillars Behind DeepSeek’s $51.5B Premium

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

The End of the AI Experiment: Surviving the CFO’s New ROI Demands

Why This Has Become an Executive Issue

Why is AI spend no longer just an IT budget problem? AI has crossed a threshold where aggregate spend across every department requires capital allocation discipline, not just software procurement review. Every function now has a case for AI investment, and someone has to decide which requests deserve ongoing funding. That decision has landed with the CFO, which means technology leaders who frame AI proposals as feature requests will lose funding to peers who can demonstrate measurable business outcomes.

What do “tokenomics” and “tokenmaxxing” actually mean in practice? Tokenomics is simply the practical economics of AI usage: how prompts, automated workflows, and background agents translate into real spending, and whether that spending is producing value. Tokenmaxxing is the emerging habit of pushing more work through AI because tokens feel cheap, or because high-consumption workflows appear more productive. The instinct can be rational, but it creates a governance problem because organizations need a way to distinguish productive consumption from wasteful consumption, and most have not built that capability yet.

Why are AI bills climbing even though token prices keep falling? Lower unit prices are encouraging more consumption, not less. As tokens get cheaper, teams build more ambitious systems: more automated, more context-heavy, always-on agents running continuously in the background. The marginal cost of any single query feels negligible, so consumption expands to fill whatever budget exists. Organizations focused purely on negotiating lower unit prices while ignoring how their systems are designed will find their total bills climbing regardless.

Why is CFO scrutiny intensifying right now? The broad experimentation phase is ending. Many organizations have deployed AI in some form, but far fewer believe those deployments have produced tangible value. Once that gap becomes visible, finance teams stop treating AI as a learning exercise and start demanding evidence for continued investment. The funding logic shifts from supporting a large portfolio of loosely defined experiments to concentrating resources on fewer workflows with a clear payback case.

What Leaders Should Actually Govern

What is the right unit of control: seats, teams, vendors, or workflows? The most useful unit of governance is the individual application or workflow, not the software seat or department budget. AI costs are generated by usage patterns, not by who holds a license. A single automated workflow can quietly consume more tokens than dozens of human users combined. Budgeting at the workflow level makes it possible to see which use cases are scaling, which are overrunning, and which should be redesigned or shut down.

When do spending caps help, and when do they backfire? Caps help when they prevent undisciplined growth in low-value usage, particularly when nobody can explain where the spending is coming from. They backfire when they suppress the most productive work. If your highest-consuming teams are also your highest-performing ones, a blanket ceiling is a tax on performance dressed up as financial discipline. The right sequence is to instrument outcomes first, then decide where controls belong.

What should leaders actually ask when a vendor proposes outcome-based pricing? Outcome-based pricing sounds appealing because it appears to align vendor incentives with business results. That alignment is not automatic. It depends entirely on how the outcome is defined, how success is verified, and what happens when the system produces something that technically triggers a charge but does not create real value. Leaders should ask who defines what counts as a valid outcome, how disputes are handled, and whether the vendor has any incentive to maximize billable events in ways that diverge from the customer’s actual objective.

Why do different AI pricing models need different governance approaches? Not all AI spend behaves the same way. Subscription pricing buys predictability but can conceal waste inside a flat fee. Usage-based pricing makes activity visible but creates volatile invoices. Outcome-based pricing sounds more business-friendly, but it can obscure the operational work required to verify whether the billed result was correct, complete, and valuable. The shift toward seats-plus-consumption adds another complication: buyers may renew a familiar per-seat contract while also taking on usage charges, credits, agent actions, or outcome fees that behave very differently. Leaders need governance that matches how value is claimed, how cost is incurred, and how performance can fail. Otherwise, they risk optimizing the old pricing model while their real exposure has already moved somewhere else.

Visibility: The Prerequisite for Everything Else

What is the single most important governance gap right now? Attribution. Most organizations cannot answer the basic question of which team, workflow, or agent is consuming how many tokens, and what business outcome that consumption supports. Without that visibility, every other governance mechanism, whether caps, chargebacks, or ROI thresholds, operates on incomplete information. Solving attribution is the prerequisite for everything else.

What does good visibility infrastructure actually look like? It means purpose-built dashboards that surface per-workflow and per-agent consumption in near real time, not month-end invoices that arrive with no ability to trace costs back to specific decisions or teams. Salesforce expanded its internal Engineering 360 dashboards to track AI usage at the workflow and team level, showing how companies often need custom visibility tools when standard reporting does not give leaders a clear view of token consumption, agent activity, and adoption patterns. This is an area where early investment in custom observability pays off rather than waiting for the vendor ecosystem to catch up.

How does token consumption become a productivity signal rather than just a cost metric? High token consumption and high-quality output often correlate. Before setting any controls, connect token spend to actual business outcomes: deals closed, issues resolved, code shipped, churn prevented. Once you have that picture, invest more in the high-correlation workflows and scrutinize the rest. Organizations that skip this step and go straight to spending ceilings risk penalizing their most productive teams first.

Practical Governance Mechanisms That Work

What is the most actionable governance step we can take right now? Set per-application token budgets with automated alerting thresholds, and require cost-impact assessments for any new AI feature before it ships. Build that review into sprint planning rather than treating it as a finance team afterthought. This embeds financial discipline into the development process rather than bolting it on after costs have already run up.

What are FinOps practices and why do they matter for AI? FinOps is the discipline of bringing financial accountability to technology spend through collaboration between engineering, finance, and business teams. Applied to AI, it means forecasting token demand before projects launch, setting ROI approval gates for competing use cases, and implementing chargebacks so business units bear the actual cost of their own consumption. The chargeback mechanism in particular creates real incentives for teams to ask whether their usage is justified.

How should infrastructure choices factor into AI cost governance? Stop treating all AI workloads as equivalent from a cost perspective. Public cloud is the right choice for experimentation and burst capacity where flexibility justifies the premium. Predictable, high-volume inference workloads are better suited to private or on-premises infrastructure where fixed costs outperform consumption pricing over time. Defaulting everything to public cloud absorbs a premium that compounds significantly as workloads scale.

Procurement and Organizational Risk

Our vendor contracts are still per-seat. Is that a problem? Yes. Per-seat pricing no longer maps cleanly to how AI systems generate costs. In many AI-heavy products, the seat is becoming a wrapper around a base level of included usage rather than a reliable proxy for total cost. Every prompt, automated workflow, and background agent can burn tokens regardless of how many people are licensed, creating invoice volatility that per-seat budgeting cannot predict. Push for hybrid models that combine a predictable baseline fee with usage-based pricing above agreed thresholds, with explicit price caps, volume commitments, reporting rights, and overage terms built in.

What changes when a seat becomes a consumption bundle? The license still matters because it controls access, but it no longer tells you enough about cost. Two teams with the same number of seats can generate very different bills if one uses AI for occasional drafting and the other runs context-heavy agents across customer support, software development, or security workflows. Procurement teams therefore need to negotiate included usage, overage rates, usage reporting, and contractual limits on unexpected consumption. The buying question shifts from “how many people need access?” to “how much machine work are we authorizing?”

What is the governance maturity gap for agentic AI? Agentic AI refers to systems that take sequences of actions autonomously rather than responding to a single prompt. That matters economically because an agent is not naturally a seat-based user. It performs tasks, calls tools, consumes tokens, and may keep working after the human has stepped away. Research suggests only about one in five organizations planning to deploy agentic AI has a mature governance model in place. Without clear accountability structures and performance metrics, organizations accumulate what practitioners call “content debt,” meaning AI-generated outputs requiring human remediation that erode the ROI case for further investment. Building governance before you scale is significantly cheaper than retrofitting it after problems surface.

How should we frame AI cost governance to get board-level attention? Frame it as a competitive risk, not a budget management problem. Unmanaged AI consumption erodes margins in a way that compounds over time, and organizations that govern their AI economics well will have a structural cost advantage over those that do not. Tokens are becoming a real operational input, and treating them with the same rigor applied to energy procurement or capital expenditure is not optional for organizations that intend to scale AI seriously

🎗️Cerebras IPO🎗️

Cerebras is going public this week, a milestone for an AI infrastructure company I have followed since its early days. I first met CEO Andrew Feldman in early 2018, before Cerebras had released its first processors and when the company was still focused mainly on AI training. After its first-generation chip came out, one of the first talks the team gave was at a conference I co-chaired in 2019. What makes this IPO especially interesting now is Cerebras’s growing focus on inference, the work of running trained AI models to produce answers, code, images, or other outputs. That shift matters as more enterprises move AI into production and as reasoning models use more compute while generating responses, not just during training. For those of us who build, buy, or use AI applications, another strong, speed-focused alternative to Nvidia is welcome news.

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

Why Your AI Agents Fail in Production (And How to Actually Test Them)

In a previous post, I argued that deploying autonomous AI agents reliably is not primarily a model problem. It is an environment problem. The gap between a capable foundation model and a production-ready system is bridged by harness engineering: the discipline of building structured workflows, validation loops, and governance mechanisms around the model rather than inside it. The central argument was that organizations that treat the surrounding environment as the primary engineering target outperform those that chase better models, and that this principle applies across every domain where agents handle complex, consequential work.

That argument raises an immediate practical question: how do you actually know whether an agent is ready for that work? Most existing evaluations still measure narrow, low-friction tasks in controlled or synthetic environments. They can tell you whether a model produces a plausible answer or completes a neat subtask, but they reveal far less about whether an agent can stay coherent across a long workflow, adapt when something breaks, and finish a job that actually runs. Many benchmarks were simple enough that top models were already approaching perfect scores, leaving no meaningful signal about which systems could handle real work and which could not.

What production usually demands is sustained execution under friction: long chains of interdependent actions, genuine error recovery, and deep domain knowledge applied to messy, open-ended objectives. That is a fundamentally different test than anything most benchmarks were designed to measure. The commercial stakes behind closing that gap are no longer abstract. Terminal-based coding agents alone are already generating billions in revenue, which means accurate measurement of what these systems can and cannot do in realistic conditions has moved from research interest to commercial necessity for anyone building, deploying, or investing in AI agent products.

Measuring the Agent, Not the Demo

Some of the most capable autonomous agents in production today are still concentrated in coding and software engineering. That makes sense. The terminal is one of the few environments where success criteria are clear and feedback arrives immediately. An agent cannot hide behind a fluent answer when a build fails, a dependency breaks, or a command returns the wrong output. It has to keep working until the job is done.

Terminal Bench was built around that reality. It places agents inside real terminal environments loaded with the files, packages, and system configurations needed for the task. Each problem includes an instruction, a verification script, and a reference solution. What gets measured is not whether the agent followed a preferred sequence of steps, but whether it reached a machine-checkable result. There is no partial credit for looking competent. The output either works or it does not.

The significance is not just that Terminal Bench is harder. It is harder in ways that matter. Previous benchmarks often measured narrow command-line skills, relied on synthetic environments, or used tasks so short that they revealed little about sustained execution. Terminal Bench instead asks whether an agent can manage long sequences of dependent actions, recover from real error messages, and apply domain knowledge to open-ended work. Its rigor also comes from the curation. Every task in Terminal Bench is manually reviewed to reduce broken tests, underspecified instructions, and loopholes that let agents game the evaluation. The early results show why that level of manual verification matters. Frontier agents still fail more than a third of the tasks, and smaller models perform much worse. For anyone building, deploying, or investing in agents, that makes Terminal Bench less a leaderboard curiosity than a practical instrument for separating systems that look capable from systems that can actually finish difficult work.

The Emerging Infrastructure for Real Agent Evaluation

Terminal Bench has not developed in isolation. A small but important set of related efforts is now building on the same premise that agent evaluation should look more like real technical work and less like a polished demo. LongCLI-Bench pushes this further by focusing on longer command line tasks and by adding step-level scoring, so an agent is penalized not only for failing to finish the job but also for breaking something that previously worked. That is a meaningful advance for anyone building production agents, because regression is often the real failure mode. DevOps-Gym pushes the boundary further into real-world software operations, evaluating agents on their ability to configure builds, monitor systems, and resolve live issues instead of just completing isolated terminal prompts.

The ecosystem is also expanding into the infrastructure needed to train and compare terminal agents at scale. TermiGen addresses one of the clearest bottlenecks exposed by Terminal Bench, namely the cost of hand-building realistic environments and trajectories for training and evaluation. Terminus and Terminus 2 provide reference implementations for how a terminal agent can interact with a live shell over many steps, which makes them useful both as engineering baselines and as cleaner testbeds for comparing models. And the appearance of fine-tuned terminal-focused systems such as Reptile, LiteCoder-Terminal, and TerminalAgent suggests that terminal competence is now being treated as a distinct capability worth training for directly. Taken together, these developments make Terminal Bench look less like a standalone benchmark and more like the anchor for a broader effort to measure and improve agents that have to do real work under real constraints.

What Comes Next, and What It Means Outside the Terminal

The near-term roadmap for Terminal Bench is really about keeping the benchmark informative. As agents improve, benchmarks only stay useful if they keep stretching the best systems, which means adding harder tasks, expanding domain coverage, and refreshing the suite before leaderboard movement stops meaning anything. Just as important, the Terminal Bench team is making an explicit argument that manual verification is not optional. Their experience showed that confirming task correctness, closing loopholes, and fully specifying success conditions takes substantial human effort, and that cost rises with task complexity.

The infrastructure supporting Terminal-Bench has become more valuable than the benchmark itself. Harbor serves as the expanded framework that allows developers to go beyond basic testing, giving them the tools to optimize AI prompts, run trial-and-error learning, and perform automated quality checks on their agents. That is a meaningful shift. Evaluation is no longer a report card at the end of a development cycle. It is moving into the development stack, not sitting outside it. The ecosystem growing around Terminal Bench is what that shift looks like in practice.

For companies building agents outside coding, this is probably the clearest near-term lesson. The winning approach is unlikely to be full autonomy across messy, high-stakes workflows. It is more likely to be structured human-agent collaboration inside tightly engineered environments. That fits the larger argument in my previous post. The real leverage comes from the surrounding system, not just the model. Terminal Bench sharpens that claim by showing that even in a domain where feedback is fast and success is machine-checkable, reliable autonomous performance remains limited. In domains where mistakes are subtler and more consequential, companies will need even more harness, more evaluation, and more deliberate handoffs between automated execution and human judgment.

Quantum Computing Supply Chains

• The Laws of Thought: The Quest for a Mathematical Theory of the Mind. AI is starting to shape the economy, the job market, and national security. That’s why I think more people should understand the concepts in this highly readable book.

• Mutiny: The Rise and Revolt of the College-Educated Working Class. The college grads making lattes and running Apple Store demos are already living the gap between what their degrees promised and what the economy delivers. With AI moving into knowledge work, that gap may widen for many more people.

• London Falling. One of my favorite writers delivers again here. I came away feeling like London is not just the setting but almost a character in its own right, with all its glamour, fraud, and seedy underbelly baked into the story.

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

What mathematicians figured out about AI that most enterprises haven't

Recent results suggest that research mathematics is no longer a purely speculative test case for AI. A growing set of examples shows AI contributing not just to short contest puzzles, but to open-ended mathematical work that requires literature search, cross-domain connection-making, revision, and verification. The important lesson for enterprise AI teams is not that AI has suddenly become a mathematician. It is that progress accelerates in settings where outputs can be checked, workflows are iterative, and human experts remain responsible for choosing the right problems. Mathematics makes these conditions unusually visible, but the broader pattern maps directly to enterprise AI systems built around retrieval, structured feedback, and human oversight.

Research mathematics might seem like the last domain to yield to AI, given its reputation for requiring deep intuition and creative leaps. Yet several structural features make it surprisingly well-suited for AI augmentation. Mathematical claims are either correct or incorrect, which means AI outputs can be verified with certainty, either by human experts or by formal proof systems. This eliminates the trust problem that plagues AI in domains where ground truth is subjective. Additionally, a large fraction of research work involves tasks that are tedious but not conceptually deep: writing experimental code, checking computations, finding citations, exploring minor cases, and surveying literature across subfields. AI handles these tasks well, freeing human researchers for higher-level reasoning.

The existence of formal proof languages and large mathematical libraries creates an infrastructure that is uniquely favorable for AI integration. Tools like Lean and Mathlib translate abstract mathematics into machine-readable, computationally certified code, providing both a training ground for AI systems and a verification layer for their outputs. This ecosystem enables a genuinely new workflow where AI proposes and humans verify, or where AI explores at scale and humans direct the search. The result is not replacement but restructuring: mathematics is becoming a hybrid process where large-scale exploration, formal verification, and human-guided insight combine into a new mode of discovery.

How AI Is Reshaping Mathematical Workflows

Research mathematicians currently deploy AI across a few workflows, starting as a disciplined assistant for work that is useful but easy to verify. They hand off tasks such as writing experimental code, checking large numbers of computational cases, and finding citations, but only when the result can be tested independently. That trust boundary matters. Researchers are not treating a language model as an oracle, but rather using it where mistakes are visible and fixable. A second pattern is broader literature scanning. Because mathematics is so specialized, important ideas are often buried in distant subfields or older papers. AI helps widen the search, systematically surfacing analogies and relevant results from outside a researcher’s immediate area. The system acts as a multiplier for breadth, while human experts provide the depth of insight needed to judge which connections are genuinely useful.

A more advanced set of workflows is now taking shape where verification is even stronger. Some researchers use systems that combine pattern-based AI with formal logic to get verified answers to specific research questions. The AI performs extended reasoning and translates the result into a formal proof language, ensuring the output is mathematically guaranteed. In parallel, other mathematicians are using AI to conduct large-scale experimental surveys of entire problem classes. The AI systematically maps the landscape, resolving routine cases automatically and flagging the smaller set of problems that still demand new conceptual approaches. The broader lesson is not that AI replaces expert reasoning. It is that AI becomes transformative when paired with structured feedback, clear verification, and a workflow that lets humans concentrate on the genuinely hard parts.

Possible Future Workflows for AI-Guided Mathematical Discovery

As these systems mature, research mathematics will likely illustrate a broader shift now visible in many knowledge-heavy fields. The mathematician of the future will spend less time carrying out long derivations by hand and more time deciding which problems deserve attention and where scarce computing resources should be focused. The transition mirrors the shift in software engineering from writing individual lines of code to designing system architectures. AI will handle the execution layer by searching for possible proof paths, scanning literature across distant subfields, and checking results in formal verification environments. The human role moves upward toward judgment, priority setting, and significance. For enterprise AI teams, this is a familiar pattern. As systems improve, the bottleneck shifts away from raw execution and toward problem selection, workflow design, and review.

A more ambitious trajectory sees AI moving from a disciplined assistant to an active research partner. In this model, the system proposes hypotheses, surfaces analogies, and explores many more directions than any one person could track alone. In specific settings, this could eventually extend to largely autonomous research loops where the AI identifies tractable open problems, tests candidate solutions, and drafts papers. Early versions of these autonomous agents already exist and have successfully resolved minor historical math puzzles. Yet the likely near-term outcome at the frontier of research is not full autonomy. It is a staged model in which AI expands the search space while humans retain responsibility for deciding what matters, which results are credible, and who bears accountability for the final product. That lesson should resonate well beyond mathematics. In enterprise environments, the most valuable AI systems will not be the ones that fully replace experts. They will be the ones that widen exploration, tighten verification, and make expert attention more selective and strategic.

Beyond Generation: What Math Reveals About Production AI

What mathematical AI makes especially clear is that hallucination is not mainly a prompting problem. It is an architectural problem, and it demands structural solutions. For high-stakes enterprise systems, prompt engineering and occasional human spot-checks are too weak to carry the load. The more durable pattern is to combine the breadth and fluency of generative models with deterministic checks: a code model paired with automated tests, a contract assistant constrained by a compliance engine, or an agent whose actions are filtered through policies, schemas, and hard business rules. Once those components are embedded in iterative loops in which the system generates, critiques, and revises, organizations can push AI further without simply scaling the risk. That is the broader lesson from mathematics. As generation gets cheaper and more abundant, the real bottleneck shifts to evaluation. Teams that build strong verification environments will be far better positioned than those that produce large volumes of plausible but weakly checked output.

That design choice also changes the role of the expert. When systems are built for verifiability from the start, the machine can take on more of the first-pass burden of checking, leaving humans to focus on higher-order judgment: deciding what matters, what is worth pursuing, and where ambiguity still requires experience and taste.

Mathematics offers an early preview of that shift. The strongest mathematicians, perhaps even future Fields Medalists, may be the ones who are best at using AI to explore more possibilities, test more ideas, and concentrate their own effort where originality matters most. The same logic applies across the enterprise. The most effective knowledge workers will not simply be those who use AI to move faster. They will be the ones who know how to direct it across a large search space, surround it with structural feedback, and intervene at the moments where human judgment creates the most leverage.

The AI Race Is No Longer Just About Benchmarks

Introducing rote™: Procedural Memory for AI Agents

In a previous piece, I argued that AI teams need operational memory, not just larger context windows or more elaborate prompts. Modiqo’s rote™ pushes that idea into a concrete system: agents should reason through genuinely new work once, then capture what worked as reusable procedural memory. That matters because many teams are now discovering the same failure mode in production agent systems: demos are easy, but reliable, repeatable workflows are hard. The challenge is no longer just connecting a model to tools. It is designing an environment where agents can remember successful work, reuse proven flows, and reduce the cost of rediscovering the same answer over and over. For teams trying to move from impressive prototypes to durable AI infrastructure, Modiqo is worth watching. Check out rote™.

Learn More

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.

Subscribe • Previous Issues

The Missing Layer: Why Your AI Agent Fails — and What Actually Fixes It

As organizations move autonomous AI agents from experimental sandboxes into live production, a critical bottleneck has emerged. Foundation models are remarkably capable but structurally unsuited to complex, multi-step work on their own. They have no persistent memory, no built-in sense of what is allowed, and no reliable way to stay on track across a long workflow. Left to their own devices, foundation models hallucinate bad decisions, lose track of context mid-task, and generate cascading errors that are expensive to unwind.

Software engineering teams were the first to hit this wall at scale, and their response offers a practical blueprint for every domain now building AI-powered applications. Their conclusion was counterintuitive: scaling AI reliably is not primarily about making the underlying model smarter. It requires a completely different discipline focused on building a structured, automated environment around the model. That discipline is called harness engineering, and its principles extend well beyond writing software.

The Core Concept

Harness engineering treats the AI model as a fixed engine and builds the entire operational system around it: workflows, specifications, validation loops, context strategies, tool interfaces, and governance mechanisms. The model stays the same. Everything around it changes.

The distinction from adjacent approaches is fundamental, not incremental. Prompt engineering optimizes what you say to a model for a single interaction. Model fine-tuning adjusts the model’s internal weights to adapt it to a specific domain. Harness engineering does neither. It accepts the model as it is and focuses entirely on the environment the agent operates inside. Practitioners describe it as “meta engineering: building the factory rather than the product.”

The formal framing organizes a harness along three dimensions. Context covers the declarative and procedural knowledge that informs the agent. Constraint covers the rules governing agent output both before and after it is produced. Convergence is the iterative process by which constraints are evaluated, gaps identified, and rules refined until the harness reaches what practitioners call structural idempotence, the point at which re-applying the checks produces no further changes. The system has stabilized.

OpenAI’s Codex team put the shift plainly: a software engineering team’s primary job is no longer to write code, but to design environments, specify intent, and build feedback loops that allow agents to do reliable work. The mental model practitioners use involves three nested loops. The outer loop, at the project level, handles intent capture through specifications, architecture documents, knowledge bases, governance rules, and human oversight. The middle loop, at the task level, handles execution through the agent’s active work cycle. The inner loop, at the action level, handles verification through immediate feedback, automated tests, and automated rule checks that scan each output against a defined set of constraints and flag anything that violates them. This layered architecture originated in software development, but any team deploying autonomous AI agents to handle complex workflows faces the same underlying problem: a capable model is not the same as a reliable system.

The Anatomy of a Reliable Agent System

Translating the lessons of software harness engineering into broader business applications leads to a set of operational patterns. These practices group into four categories that shape how teams design, build, and govern autonomous agents in any high-stakes domain.

The Strategic Mindset Shift

The most fundamental adjustment for teams deploying AI agents is a complete inversion of their daily focus. Rather than optimizing the underlying model or manually reviewing every output, practitioners must shift their attention to architecting the environment where the agent operates. This means treating the model as a fixed engine and investing in the surrounding validation infrastructure, turning domain experts from manual reviewers into system designers. The evidence for this reframing is concrete: LangChain moved a coding agent from 30th to 5th place on an industry benchmark by changing only the harness, not the model. The same dynamic applies in any domain. A legal research agent will not become reliably accurate by switching to a more capable model if the validation layer cannot catch citation errors. When an agent produces a bad output, the first question should be “what is missing from the surrounding environment?” not “how do we change the model?”

The true return on investment for an agent system is not measured in tasks completed but in expert human attention hours saved. Every failure pattern encoded as an automated rule reduces future review burden, which means the harness is not a setup cost but a compounding asset that grows more valuable with each iteration. The primary deliverable for the domain expert becomes the testing suite and evaluation pipelines rather than the content the agent produces. A domain expert who insists on reviewing every output does not become a quality guarantor. They become a ceiling on what the system can ever achieve.

Architecture and Orchestration

Once the mindset shifts to environment design, the architecture of the system must enforce strict boundaries and predictable workflows. Rather than granting a single agent freeform autonomy to complete a complex task, robust systems rely on structured orchestration. A fixed control layer governs how the process moves from one defined step to the next, so the agent is never left to decide on its own what to do next or whether a step can be skipped. A clinical documentation agent should never decide on its own whether to file a record, request clarification, or escalate to a physician. A procurement agent should not unilaterally skip an approval step because the task looks routine. Structured orchestration makes agent behavior auditable, predictable, and recoverable regardless of domain.

Paired with this are two supporting elements. Teams must also build durable documents that live permanently inside the agent’s operating environment. These files encode the institutional knowledge the agent needs to behave consistently across every session: regulatory constraints, brand voice rules, escalation thresholds, and the rationale behind key decisions. Without them, agents operate from whatever context a user happens to provide, which produces inconsistent and unpredictable behavior. These anchors ensure consistent behavior regardless of how a session begins, rather than relying on ad hoc instructions that vary by user. Complex tasks should also be decomposed into specialized agent roles with explicit, structured handoffs between them.

A grant writing system might feature a research agent, a drafting agent, and a compliance review agent operating in sequence. Specialization narrows the blast radius of individual failures and makes each component easier to test and improve independently. Every connection to an external tool or data source must carry explicit permission limits enforced mechanically, because without those boundaries, agents can access sensitive data inappropriately or trigger irreversible operations in connected systems, and that risk scales directly with the number of agents running in parallel.

Validation, Feedback, and Escalation

A well-architected environment requires automated mechanisms to catch errors early and correct them cheaply. The strategic optimization target is not preventing every error through exhaustive upfront review but detecting errors fast and reversing them at the lowest possible cost. Reliable systems build this through three layered feedback mechanisms working in sequence: structural checks that block invalid outputs and return actionable fix instructions the agent can act on without human translation, runtime observability through logs and metrics that make execution visible to both agents and humans, and agent-led self-review that audits outputs before escalating only genuinely novel cases to a human expert. Layering all three creates a system that catches errors at the cheapest possible point in the pipeline rather than routing everything through a human approval queue that cannot scale.

Escalation itself must be triggered by specific, pre-defined conditions encoded in the harness, not left to the agent’s discretion. This prevents both under-escalation, where agents proceed when they should not, and the equally damaging pattern of agents interrupting human experts for routine decisions that automated checks could handle. Teams should build evaluation criteria that evolve as they observe how agents actually fail in production. The mistakes that matter most in a live environment are almost never the ones engineers anticipate before launch. The ultimate design goal across all of this is convergence: a harness mature enough that re-applying its constraint checks produces no further changes, and the system has reached a stable, rule-compliant state.

Critical Anti-Patterns

Understanding how these systems fail is as important as knowing how to build them. The most dangerous trap is silent state corruption, where an agent generates outputs that look structurally plausible but contain semantic errors that accumulate undetected until they cause cascading damage that is expensive to unwind. This failure is more insidious outside software because the feedback signals are slower and weaker than a failed build or broken test. A research synthesis agent that subtly misattributes sources, a clinical documentation agent that quietly introduces dosage errors, or a financial analysis agent that gradually drifts from regulatory definitions all represent silent corruption that looks fine on the surface until it does not. The problem gets worse when teams rely on the conversation history itself to track where the workflow stands, rather than maintaining a separate, explicit record of progress. When something goes wrong in a system built this way, there is no clean state to inspect or replay. The only record of what happened is buried in a thread of messages that the agent may have interpreted differently at each step.

When AI teams deploy multi-agent configurations without verification gates between handoffs, each agent’s small mistake becomes an input assumption for the next, producing catastrophically wrong final outputs from a chain of individually plausible-looking steps. A multi-agent insurance underwriting pipeline where a data extraction agent makes a small error, a risk scoring agent builds on that error, and a pricing agent compounds it further illustrates how quickly the damage accumulates and how difficult it becomes to trace back to its origin. When guardrails and oversight structures are absent from the start, technical debt accumulates faster than teams can address it. Retrofitting those controls after a production failure is dramatically more expensive than building even a minimal harness from day one. A persistent context document, a few structural checks, and a basic escalation rule is substantially better than none, and it provides a foundation for iterative improvement rather than a crisis-driven rebuild.

The Architect or the Bottleneck

Harness engineering emerged from software development because that is where autonomous AI agents hit production scale first, but the underlying problem it solves is not specific to code. Any domain deploying agents to handle complex, multi-step work with real consequences faces the same structural gap: a capable model is not a reliable system. Those principles are not specific to software. They apply with equal force to legal research, clinical documentation, financial analysis, and procurement, anywhere that errors compound and someone is ultimately accountable for the output. The names change. The structural requirements do not. The gap between a raw model and a reliable production system is bridged entirely by the environment built around it. That means every organization deploying AI agents faces the same choice software teams have already confronted: invest in the environment that makes agents reliable, or spend your time cleaning up after ones that are not.

Building this infrastructure requires upfront investment in time and discipline, but the alternative is a system that generates technical debt and silent errors at machine speed. A well-engineered harness is the only mechanism that allows an organization to capture the productivity gains of autonomous AI without sacrificing the safety and quality of its most critical operations.

Quick Takes

1. AI Committees: Accelerant or Anchor?

2. The New Rules of AI Information Architecture

Ben Lorica edits Ethics.dev and the Gradient Flow newsletter, and he hosts the Data Exchange podcast. He helps organize the AI Conference and the AI Agent Conference, while also serving as the Strategic Content Chair for AI at the Linux Foundation. You can follow him on Linkedin, X, Mastodon, Reddit, Bluesky, YouTube, or TikTok. This newsletter is produced by Gradient Flow.