Even the Best AI Models Can Suck at Chess

The launch of ChatGPT had an interesting effect on the online chess discourse.1 Chess has already long been conquered by machines. As early as 1996 a computer (IBM’s Deep Blue) was able to beat the human world champion, grandmaster Garry Kasparov, in a game watched by over six million people.2 The world was shocked that a machine took on the best player and won a game, but chess engines didn’t stop evolving there. Since the 90s, they have gotten better while the machines needed to run them have become much smaller. Today Stockfish is widely considered much stronger than any human player. It has run on consumer hardware since its launch in 2008, and by 2014 it was beating some of the world’s top grandmasters.

It came as a surprise to many, therefore, that modern LLMs, trained on a vast portion of the internet and requiring supercomputers to run, couldn’t help but cheat on almost every move. There are endless videos showing how just a few moves into a normal chess game, ChatGPT and some of its competitors would gladly throw the rules out the window to escape a checkmate or gain an advantage.

But in truth, this isn’t surprising. The LLMs were not trained with chess in mind. Sure, they may have seen countless chess games scattered throughout the internet, but the vast majority of their parameters and training compute were devoted to capabilities that are completely useless once you put a chessboard in front of them.3 Stockfish on the other hand uses a tree search algorithm that is built to be good at chess. If you want a chess engine, you use a chess engine.4

Smaller Models Are Sometimes Better

While chess is a particularly potent example of a large language model losing to a much smaller specialized system, it’s far from unique. In a 2025 position paper, Nvidia researchers argued that small models5 (which it defines as models under 10 billion parameters) are the future of agentic AI and that they “provide significant benefits in cost-efficiency, adaptability, and deployment flexibility.”

Subscribe now

Just because a larger model can do a job does not mean that a small model fine-tuned for that specific task can’t do it better and more cheaply. There are countless examples of smaller models doing just that. LiteResearcher is a 4B model that beat out Claude Sonnet 4.5 on some search benchmarks. Terminus-4B allows larger models to save compute by handing off terminal execution to a smaller model without suffering capability loss. The Docling family of open-source models start at just 258 million parameters and allow for fast extraction of PDFs to text without having to feed 100-page PDFs into an expensive LLM. Researchers also trained a small 4B model to outperform even the GPT 5 series of models in a few social negotiation situations such as negotiating salary or bargaining for a purchase. Each wins, not by raw intelligence but because it is built or fine-tuned for a narrower, more specific purpose.

A frontier model may know how to do all of these jobs, but that doesn’t mean it’s the right tool for the job. Large models are expensive and unpredictable, and doubly so when it comes to agentic tasks which can span several turns and hundreds of thousands of tokens.

Nvidia draws the line for small models at 10 billion parameters, but the more important boundary for developers may be whether a model is small enough to run yourself. There is still a whole class of models that are not necessarily small but are still small enough to fit on one consumer GPU (at least when quantized). This includes models like Qwen 3.8 27B, gemma 4 26b and GPT-OSS 20B. These models are very capable even without specialization and rank very highly on benchmarks (with Qwen sometimes outranking top models from a few months ago), but can still be easily run on premises without spending thousands of dollars on GPUs.

Share

The ability to run smaller specialized models adds more than just efficiency; it provides a more realistic opportunity for a developer to train and fine-tune their own model, and to host the model locally or in the cloud instead of relying on the model provider to do so for it. This in turn can provide developers with more control over how tokens are used, how outputs are structured, and how each part of the pipeline can be improved individually – instead of assuming an improvement in the most popular benchmarks will lead to every task improving. And as noted above, smaller models can be easier to fine-tune, thereby creating a more specialized AI. As my colleague Ilan Strauss has noted, specialization is a powerful economic force.

How Do You Train It, and Where Does It Run?

The strongest argument for using an off the shelf generic chat model is often one of convenience. For most tasks, a general purpose generic model will be good enough and with products like OpenRouter, developers can easily pick and choose from hundreds of models (plenty of them open source) all competing in capability and cost without putting in any upfront work to train a model. As Raffi Krikorian of Mozilla noted while reviewing this article, general models also make particular sense early in a company’s lifecycle, when the problem itself is still being defined. At that stage, experimenting with the largest and most capable model available can help a team figure out exactly what it needs. But as the problem space narrows and the required architecture becomes clearer, so too may the need for a large general model. And despite many first-party model makers discontinuing their fine-tuning products, fine-tuning and hosting a smaller model remains relatively easy, largely thanks to parameter-efficient techniques like LoRA.

LoRA

LoRA (Low-Rank Adaptation) allows developers to fine-tune a model without touching the actual model weights. It works by attaching a relatively small number of trainable weights that are updated during fine-tuning. This is important for several reasons. A small adapter can be easily transported, and serving a new LoRA does not require loading an entirely new model as long as the underlying base model is already available. Unlike full fine-tuning, a LoRA also reduces the risk of catastrophic forgetting.

Training a LoRA is much cheaper than full fine-tuning as it only updates a small selection of weights. This can be done on consumer GPUs using libraries such as Hugging Face Transformers or Unsloth. There are also APIs that mimic or improve on the fine-tuning APIs that used to be provided by the big three providers (Anthropic, OpenAI, Google). Fireworks, for example, provides a straightforward finetuning API that takes example completions for it to learn from. Going beyond SFT (Supervised Fine-Tuning, or learning by example), Tinker allows developers to build custom RL (Reinforcement Learning) environments that reward results meeting certain criteria, while the environment itself runs on the developer’s machine.

Subscribe now

Hosting a LoRA is similarly straightforward and, importantly, portable across platforms. Transferring a fully fine-tuned model to a new API platform can be costly and may require the provider to serve your model separately on expensive GPUs. Using LoRA allows the platform to just load a small adapter onto the model they are already using to serve other users’ requests. This means that fine-tuning a LoRA does not lock you to a specific platform and it also doesn’t force you to rent your own GPUs.

Control Beyond the Model Weights

As Tim O’Reilly previously argued, open-source AI should not stop at the model weights. In a similar vein, the possibilities for developers building a custom system do not stop there either. Model APIs are inherently limiting since they impose on you what parts of the model’s input can be touched, what can be cached, and how you can affect the output. Going back to the chess example, enforcing valid chess moves at output time is easy, assuming you have access to the code that runs the model; but it’s not easy to do when you are relying on an API built for a chatbot that you are unable to modify.

Self-hosting a model gives you a level of control far beyond what is possible through a standard chat completion API and allows you to build the model around the task instead of building the task around the model. Fine-tuning is only one aspect of specialization. You can also constrain which outputs are valid, expose and modify probabilities of every token, cache any state and add task-specific logic directly into the inference pipeline.

This matters because the default approach to improving AI systems has increasingly become to reach for a more capable general model. Sometimes that is the right answer. But improving general model intelligence is only one lever, and often not the cheapest or most reliable one.

Share

In 1997 nobody complained that Deep Blue gave bad recipes because it wasn’t built to do anything but play chess. By specializing around one narrow problem, it was able to beat a grandmaster at a game that many had thought machines would never conquer.

The lesson from LLMs cheating at chess is that the best tool for a problem is often not the most general one. Greater general intelligence does not automatically translate into greater capabilities across specialized tasks. And a narrower system is often more capable where it counts. Specialized machine intelligence may still require lots of data and often its own pipeline. But small models can help companies get there.

Thanks for reading! This is a publication of the AI Disclosures Project. Subscribe, share our work, and help build an architecture of participation for AI Markets today.

Support Our Mission

This post was originally published as “Why Open Source Matters for AI” on O’Reilly Radar, August 10, 2026 by Tim O’Reilly: https://www.oreilly.com/radar/why-open-source-matters-for-ai/

The Apache Lesson

In 1995, the question in the media was whether Netscape or Microsoft would control the web. The answer, it turned out, was neither.

Both Netscape and Microsoft aimed to dominate the web server and browser market, reasoning that whoever controlled both ends of the connection would have an internet “platform” to rival the deathgrip that Microsoft had enjoyed on the personal computer. The two companies raced to build every feature they could think of directly into the product, on the theory that whoever built the most integrated and full featured web server would win.

The open source Apache web server took the opposite bet. It stayed a web server with a clean extension layer, so anyone could bolt something new onto it without asking permission or waiting for the next release cycle. Within a few years, Apache was far and away the most popular web server, and Netscape’s server and Microsoft’s Internet Information Server (IIS) were history. People started talking about the LAMP stack: Linux, Apache, MySQL, and (Perl | Python | PHP) as a legitimate platform. Modularity, not features, was the moat. The fact that major elements of that stack survive while others have been swapped out or extended is a testament to the power of composability and distributed innovation.

I called that pattern the architecture of participation when I wrote about it in 2004. I was trying to explain an inconvenient fact that the licensing debates of that era ignored. I had started working with Unix in the System III days, and saw how it had succeeded as a collaborative project even though AT&T offered Unix under a proprietary license. A few years later, I observed that nominally open source projects like OpenOffice with monolithic architectures never built much of a community. I realized that open source wasn’t just about licenses, but about architecture. A small kernel with standard interfaces that lets people extend your work without asking for permission is an important part of the secret sauce.

Swap out Netscape and Microsoft for OpenAI and Anthropic in this story, and perhaps you can see the echoes.

A model’s personality, its defaults, and its history used to live where you could, with a little effort, see them and edit them. Increasingly, they don’t. As Drew Breunig pointed out to me the other day, each new version of the frontier models moves a little more of the product’s behavior out of an editable layer and into the weights themselves, where nobody outside the lab can see it, let alone change it. The model stops being a component you build with and can adjust to your liking and starts being an appliance you rent. Post-training is important but Drew points out that it is also “trading diversity for reliability.” That’s a good trade for many people, but it is the same kind of trade that gives us highly processed foods when we know that “real food” is better.

Open weights are just table stakes

The public debate about open source AI seems devoted far too much to model weights, their national security implications, and whether a lab releases weights and under what license. But that covers only a fraction of what actually makes open source matter. Apache was never competing with Netscape and Microsoft (and Linux was never competing with Windows) over whose source was more available. They were competing over something more important. I remember talking with Bob Young, the founder of Red Hat, about his business model, and he said “What we really sell to our customers is control.” Open source meant that the platform your application depended on was no longer a sealed box you licensed from one company but a layer you could extend and build a business on top of without asking anyone’s permission. It sparked an explosion of innovation. It enabled companies like Google and Amazon to grow up free from Microsoft’s dominant paradigm.

Every wave of computing, from mainframes to PCs to the internet, has run through the same cycle: distributed innovation at the start, with the eventual winner gradually closing down its offerings to build a moat. What keeps a market open isn’t the license on any single component. It’s how easy it is to swap out one component for another when a better one appears.

The protocols connecting the pieces are an important part of that picture. Unix utilities expected stdin and stdout, and the shell acted as a kind of harness to connect them, so it was easy to build a new tool that worked seamlessly with existing ones. A testament to the power of that approach is just how much the shell and Unix utilities are the lingua franca of agentic tooling today, more than 50 years after they were invented! TCP/IP, HTTP, and other internet protocols played a similar role in keeping the internet open and composable.

Subscribe now

Fortunately, so far, we are seeing some wins for composable, protocol-centric architectures in AI. Anthropic’s Model Context Protocol was a disruptive move in that direction, an open standard for letting any application reach any tool or data source without a custom integration for each pairing. Along with other open protocols, MCP also now has a home outside of Anthropic at the Agentic AI Foundation (a subproject of the Linux Foundation), which is at least a partial guarantee of its independence.

Isobel Moure, Ilan Strauss, and I made the case earlier this year in Protocols and Power that as models commoditize, competition moves up the stack to context. Opening the means of accessing that context opens the market, regardless of whether open or closed weights sit underneath it. That’s an unbundling, model from harness from context, done the way Apache unbundled web server from web application.

Agentic skills may also be a critical element of the open source AI future, though as the history of the LAMP stack shows, they may fall by the wayside in the same way that Perl and PHP did. And that’s just fine. Composability means that it’s easy to switch to something better when it comes along, or when more people agree on it.

There’s also a lot of great work going on in portable memory from players like Letta, Nous Research, and others. Open source agentic harnesses like Goose and Pi are also a big part of giving power back to the people. Pi in particular is optimized to be modifiable. There’s a fun story told about Mario Zechner’s decision to give Pi a “/quit” command rather than an “/exit” command like Claude or Codex. Countless issues and PRs have been submitted to Pi’s repo, asking for or implementing “/exit”, but Zechner is stubborn. His retort is that you should just ask Pi to add it to your install.

But the projects I listed above are just the tip of the iceberg when it comes to the scale and scope of open source AI. Current AI’s Open Source Gap Map covers more than 24,600 open source AI projects!!, with 421 of them scored in depth across openness, capability, and adoption. The map organizes the stack into three layers: 1) models and associated elements including data sets, fine tuning tools, inference frameworks like VLLM, and evals; 2) the product and UX layer, including harnesses and personal agents; and 3) the infrastructure underneath, including core ML frameworks like PyTorch, deployment tools like Ollama, and edge hardware.

Current AI itself is a public-private partnership that came out of the AI Action Summit in Paris last year. This summer they announced AI Potluck, which they describe as “a public project to build a vertically integrated AI product assembled entirely from open source components… a viable alternative to proprietary AI that isn’t owned by any one company or country.” It is backed so far by roughly $400 million of a five-year, $2.5 billion commitment from the French government, tech companies including DeepMind and Salesforce, and major philanthropies including Omidyar’s AI Collaborative, the Macarthur Foundation, and the Ford Foundation.

The fact that this organization exists, along with others like the Agentic AI Foundation, is a testament to the rising tide of interest in open source AI. The coalition of interested parties also says a lot about the underlying motivations that are driving that interest: AI sovereignty, corporate independence from the overweening ambition of the major labs, and an interest in technology for the public good.

Keeping it weird

There’s another element, which Drew Breunig put his finger on in our conversation the other day. The problem with having one or two big closed models dominating AI, and having those models increasingly locking their desired personality, business goals, and guardrails into the weights themselves, is that they will reduce the diversity that is at the heart of innovation.

It’s our job, Drew said, to make it weird, to push a model deliberately out of distribution rather than to settle for whatever the labs have made the default outcome. He described how his team chose not to build in React for a recent project for exactly that reason: every model already knows React too well, so building in it means shipping the average of what everyone else was doing instead of something genuinely their own. He has started using GLM and Kimi not to save money but because they are more malleable and take direction better inside a custom harness. And he wants the open-weight ecosystem to survive precisely so that models stay infrastructure rather than becoming appliances.

That’s what an architecture of participation is actually for. We need real separation between the model, the harness, and the application, so that someone who wants to build something weird can still do it without a lab’s roadmap and guardrails deciding whether they’re allowed to.

Share

“Weird” may make it sound like something that not all developers might want. But we’re really talking about something intensely practical. In his short essay on trading reliability for diversity, linked above, Drew Breunig put it this way:

Labs have to ship a product that delivers “good enough” results when a layperson gives a model a lazy prompt. Without direction, the model must return something decent. (If it’s a website it’ll use the Inter font, cards with a single colored border, gradients, implemented with ReAct and Tailwind). Anthropic named this default output “distribution convergent.” At CAIS, @trq212 put it well, roughly, “If it’s not in your prompt, you’re getting what’s in-distribution” …. Less diverse models make for more reliable coding agents, but they encourage a monoculture of output.

Addy Osmani, my co-chair of the O’Reilly AI Codecon, took this point beyond model diversity after reading a draft of this piece: “Almost nobody I work with is tinkering with weights, but they’re rewriting the harness and what sits around it pretty constantly—skills, subagents, hooks, context files etc etc. That’s where participation is currently happening.” Addy went on to note that forking a skill instead of adopting the default, memory and constitution files that travel with the agent instead of living in a vendor account, or picking the unfashionable framework on purpose are all areas where ease of modifiability matters to everyone.

I want to end by returning to the Apache story. I believe that the big labs are making the same strategic mistake that Netscape and Microsoft made in the mid 90s. Yes, make the models more reliable for ordinary users. But don’t shut down the options for developers who don’t work for you to push the state of the art forward. As Bill Joy put it decades ago, “No matter who you are, most of the smartest people work for someone else.” No one should have a monopoly on innovation, and no one should be building a moat to hold it back.

Subscribe now

And be sure to join us at AI Codecon: Building with Open Source AI on August 31, a free half-day virtual conference. You’ll hear from leading developers and technical experts working with open-weight models, self-hosted infrastructure, and real-world AI workflows, and learn how building in the open gives teams more control over costs, data privacy, and what they ship. Register today to save your spot.

Thanks for reading! This is a publication of the AI Disclosures Project. Subscribe, share our work, and help build an architecture of participation for AI Markets today.

Support Our Mission

Dylan Patel and his colleagues at SemiAnalysis read this as a kind of failure in their recent newsletter “Gemini is Cooked but GCP is Cooking.” DeepMind has stopped being a frontier lab, they noted. The departures, they say, are a symptom of years of timid compute allocation and a bureaucratic, risk-averse culture. After all, Google had sophisticated conversational-AI systems well before ChatGPT, but was far more reluctant than OpenAI to put them in users’ hands. SemiAnalysis argues that Google’s failure to risk the core business has finally caught up with it.

It’s not a disaster for Google, though. In SemiAnalysis’s estimates, Google Cloud may be a much larger economic opportunity than pursuing its rivals in the frontier AI race. SemiAnalysis wrote “Our Tokenomics Model estimates that Gemini ARR was $12B in 2Q26. In contrast, by the end of 2027, GCP will be doing over $73B in third party AI ARR IaaS/TaaS and another $120B of TPU sales. $200B of external sales at high 30s EBIT margins vs a first party business generating just $12B today shows where the focus is.”

What’s more, after years of lagging Amazon and Microsoft in cloud revenue, Google seems to be gaining ground. Alphabet reported $24.8 billion of Cloud revenue in the latest quarter, up 82% year over year, compared with 37% growth at AWS and 43% growth in Microsoft’s Azure and other cloud services. The figures aren’t strictly comparable, though. Google Cloud includes Workspace and other applications, and Microsoft does not disclose Azure revenue separately from its cloud applications either, while AWS is pure cloud revenue. SemiAnalysis also estimates that TPU system sales added roughly $1.2 billion to Google Cloud revenue during the quarter.

Is this a choice by Google of profit over frontier ambition? SemiAnalysis compares it to past strategic missteps such as when IBM retreated from the PC into mainframe consulting, or when Intel retreated from Pat Gelsinger’s bold bets into its legacy chip business. Both ended up judged as major mistakes.

That may be correct. But there’s a second scenario that fits the facts, and is also rooted in history.

In the 1880s, Thomas Edison was famed as the hero of the electricity revolution. He had invented the first practical incandescent light bulb and commercialized it at scale, and had built the first commercial power plant in lower Manhattan. However, his system ran on relatively low-voltage direct current, which was practical over short distances but required generating stations close to customers. George Westinghouse bought Nikola Tesla’s patents for alternating current, which could travel for miles at high voltage and then be stepped down for ordinary use. Tesla had also developed electric motors and generators that ran on alternating current. By 1893 Westinghouse had lit the Chicago World’s Fair with AC. And by 1896, Westinghouse’s AC generators were sending power from Niagara Falls to Buffalo. Edison was the frontier leader, but Westinghouse won the race to diffuse electricity through society. (This is how it worked out even though Edison was, in many ways, right in the long term about the many applications for which direct current is superior. DC has returned as a crucial part of modern electronics, batteries, solar, EVs and high-voltage transmission. History rarely goes in straight lines.)

Jeff Ding’s book Technology and the Rise of Great Powers traces the relative impact of invention and diffusion during technology revolutions. Ding argues that nations that dominate the “leading sector” of a general purpose technology don’t reliably grow more powerful as a result. Diffusion is the defining factor. He posits that Britain’s edge in the first industrial revolution came less from inventing the steam engine and advances in steelmaking than from diffusing machinery through the whole economy so that many businesses, not just the steam engine manufacturers and the steelmakers, became more profitable. And America’s edge in the second industrial revolution had less to do with any single American breakthrough than with how fast interchangeable manufacturing, electrification, and eventually the automobile spread into every sector at once. Germany dominated many frontier industries, but there, growth and profits were concentrated in a few leading companies rather than diffused widely through society.

According to Ding, the importance of diffusion over frontier dominance continued in the 20th century. Though the US pioneered the electronics revolution, Japan appeared for a time to be winning the frontier race, at least in Ding’s narrative. It led the world in semiconductors, consumer electronics, and computer hardware through the 1980s, yet in the long run it still lost the information revolution to the US, a country that was worse at making the chips and better at putting computers to work throughout society.

I applied these insights to AI and its corporate adoption a few weeks ago in a review of Jeff’s book called Ordinary Engineers, Not Heroic Inventors. So reading SemiAnalysis, I wondered whether Google might be making the same strategic bet as Westinghouse. SemiAnalysis’s numbers can make the case that Google is betting on diffusion at least as well as the case that it is giving up on frontier leadership. So this could be read as a strategic argument inside Google that Google Cloud CEO Thomas Kurian won and that Demis Hassabis and Jeff Dean lost. Whether or not anyone inside Google describes it this way, the capital allocation increasingly looks like a strategy to become a platform for not just its own but for other people’s AI applications.

Kurian has been saying for a while that he wants Google TPUs to become “general purpose infrastructure,” serving Citadel Securities and the Department of Energy as readily as Gemini. Of course, the crucial question if we were using Ding’s framework isn’t whether Google sells lots of TPUs but whether those TPUs become complements to a broad wave of productivity-enhancing innovation in other parts of the economy.

Kurian has also defended selling compute to Anthropic as what happens when you’re a platform company. That sounds like someone who has argued that the bigger prize is being the layer that others’ AI runs on, competitors included, rather than doubling down on the potentially ruinous costs of the frontier AI race. After all, while SemiAnalysis didn’t go there, there’s a chance that if open-weight models continue to compress inference pricing, frontier labs may discover that model leadership resembles semiconductor fabrication, enormously important strategically but surprisingly poor as a standalone business.

Google also doesn’t need to win the frontier to dominate the edge. Its Flash-class Gemini models already run AI Mode in Search at enormous scale, with more user telemetry than almost any competitor. Google has enormous distribution advantages through Android, Chrome, Search, and Play. Perhaps they cede the expensive unprofitable frontier race and excel in the 120B space where affordability and local hardware are currently meeting in a sort of sweet spot. Returning to the history of electrification, it was ubiquitous small motors that powered the second industrial revolution, not giant dynamos. Buy one good small model team, my colleague Ilan Strauss speculates, and there may be no company better positioned to own the ordinary, everyday layer of AI the way Google already owns the ordinary, everyday layer of search. This may be a stretch. ChatGPT and Claude have made their own remarkable advances in the diffusion of AI, but IMO, their rhetoric about the way that AGI will concentrate enormous wealth and power in the hands of the winner doesn’t make a convincing case that they are operating with a diffusion strategy in mind. Meanwhile, both they and their customers are still in search of the widespread productivity uplift that diffusion of a general purpose technology brings.

In short, SemiAnalysis is probably right that Google gave something up on August 5, but possibly wrong about why. The evidence does not necessarily show that Google has abandoned technological ambition. Sergey Brin still seems to have plenty. But at least for the moment it may show a shift in where Google is allocating scarce AI compute and where it expects to capture economic value. It may be a mistake to assume that the AI race is about who builds the best intelligence. It may turn out to be about who builds the electrical grid. Maybe Google didn’t decide to stop trying to win so much as to decide that a different race was more worth winning.

Ask ChatGPT, Claude and Grok the same question and you get three different answers. Yes, the facts are (largely) the same, but it’s clear that three different personalities answer back. OpenAI shipped first and established a template for what a helpful assistant sounds like, careful and accommodating and reluctant to give offense. Anthropic leaned into guardrails and values and character and trained Claude against a written constitution. Elon Musk built Grok to be the rebellious one, the anti-woke alternative to everything Musk thought had gone soft in the rest of the industry.

It’s not just the models that are differentiated, though. The experience of individuals interacting with AI is shaped by the personality and values designed into the system. And as we accumulate more context with a model, we are more and more a “ChatGPT user,” a “Claude user,” or a “Grok user.” This is not entirely unlike being a Mac user vs. a PC user, or an iPhone user vs. an Android user, or being part of the Apple ecosystem or the Google ecosystem. Companies depend on this tribalism. It’s part of their moat. But in AI, the moat may be deeper than in past environments where our belonging was marked by the programs we use and the artifacts they created for us, because here it is also a matter of shared landscapes of thought.

Gregory Bateson gave a name to this phenomenon in the 1930s, after fieldwork among the Iatmul people of the Sepik River in New Guinea. He called it schismogenesis, “a process of differentiation in the norms of individual behaviour” driven by repeated interaction. He saw two forms. In symmetrical schismogenesis, each side answers the other with more of the same, boast for boast, as if it were a kind of an arms race. In complementary schismogenesis, the behavior of one draws out the behavior of the other. I’m not sure I completely understand Bateson’s distinction here since I read his Steps to an Ecology of Mind over 50 years ago and haven’t looked at it since, but sometimes even a misunderstood concept can still be a tool for insight. (I’ve used Claude to help me tease out my half-remembered lessons.)

Relating these two forms to AI, I would say (correctly or not) that OpenAI, Anthropic, and DeepMind are engaged in a form of symmetrical schismogenesis, initially all trying to outdo each other in promises of AI safety, and with boasts of the power of their models to shape the AI future. X.ai/Grok, on the other hand, represents a kind of complementary schismogenesis, an explicit effort to be what the others aren’t. Mistral (and sovereign AI in general) represents another kind of complementary schismogenesis. After all, if you are French, or Chinese, or even just a company trying to carve out your own place in an increasingly homogenized world, do you really want to adopt your values wholesale from whatever is on offer by the big labs?

David Graeber and David Wengrow ran with Bateson’s idea in their book The Dawn of Everything. They used it to explain neighboring societies organized as near mirror opposites. On the Pacific coast the fishing peoples of the Northwest were hierarchical, kept slaves, and threw competitive feasts, while the acorn-gathering peoples of California to their south were industrious, frugal, and suspicious of hoarded status. The difference was not just determined by climate or crops. People became what they were partly by refusing to resemble the neighbors across the way. Graeber and Wengrow describe these cultural choices as a kind of play, but it is serious play that can harden into rivalry and conflict.

A schismogenetic tree

The idea that schismogenesis is a kind of cultural game minimizes environmental factors that can shape it. Google Research built the transformer in 2017, the architecture every one of these models runs on. Yet because of the company’s heritage, Google’s first implementation, BERT, was positioned as an improvement to search, used to better understand the intent behind search queries, particularly longer, conversational, or preposition-heavy searches. By 2021 Google had a conversational model, LaMDA, good enough that one of its own engineers went public the next year claiming it was sentient. But Google never shipped it. A model that answers your question in a paragraph threatened Google’s search franchise, more than $160 billion a year and most of Alphabet’s revenue. Google could have built chat earlier than anyone but had the most to lose by shipping it. This is a version of what is sometimes called “the Kodak curse.” Kodak made early breakthroughs in digital photography that they never properly commercialized because of the desire to protect their highly profitable film/chemistry business.

OpenAI had no search revenue to protect, so putting a chat box on the open web in 2022 cost it nothing. And so the leader became the follower. After ChatGPT reached a hundred million users in only two months, Google CEO Sundar Pichai declared a code red, pulled founders Larry Page and Sergey Brin back into product meetings, and within weeks Google shipped Bard, whose first public demo got a fact about the James Webb telescope wrong and knocked $100 billion off Alphabet in a day. In short, the company that invented the technology arrived late, got rattled, and ever since has been building chat into search with one hand while defending search from chat with the other. So yes, the environment is a factor ;-)

Schismogenesis usually works below the level of intention, though. It’s an accumulating cultural drift nobody quite chooses. But in the case of the big AI models, schismogenesis seems to have been quite deliberate.

With Claude and with Grok you could watch their owners reach in and deliberately turn the dial. Claude drove AI further into caution and a published set of values. Musk took the other side. When early testers ran Grok through the standard political batteries and found it sitting left of center, near ChatGPT, Musk said xAI would move it, and over the next updates its answers marched right in step with his own posts on X.

Google did not differentiate because it was held in place by the business it had to defend, but OpenAI and Anthropic and xAI differentiated because they were free to, with no franchise holding them back. It’s true that OpenAI was driven by strategic business considerations and its hope to dethrone Google, but both Anthropic and X.ai made their choices initially for cultural reasons. They are a pure demonstration of schismogenesis.

So too, the open-weight world has defined itself against the closed labs, and the choice to publish weights is now as much an identity as an engineering decision, a way of saying “we are the ones who do not lock you in.” Meta leaned on open weights to distinguish itself from OpenAI and Google, and the Chinese labs that shipped strong open models turned openness into a powerful business strategy.

Further up the stack, differentiation is the whole game. When the underlying capability commoditizes, the players who capture value are the ones who stake out a position nobody else holds. Schismogenesis is not a cultural curiosity. It is a competitive strategy, and in a market this crowded it may be the main one.

Steve Jobs was good at this

None of this story would have surprised Steve Jobs, who turned schismogenesis into a marketing case study. Apple’s 1984 ad did not sell the Mac on the basis of its processor speed or available memory, as PC advertising of the day was likely to do. Ridley Scott shot a runner sprinting into a hall of gray obedient faces and throwing a hammer through Big Brother on the screen. The tag said the Macintosh was why 1984 would not be like 1984. IBM was conformity and the Mac was the hammer. Thirteen years later Think Different, a riff on IBM’s one-word slogan “Think!”, put Einstein and Gandhi and Lennon on billboards and toasted “the crazy ones.” Apple built a company on being the deliberate opposite of the beige box on the office desk.

Scene from Apple’s legendary 1984 Superbowl ad.

The AI labs might learn something from Jobs, because his differentiation was generative in a way that a purely oppositional version is not. Apple defined itself against IBM, but it defined itself for something, the individual, the artist, the person who wanted a tool that felt like it was on their side. IBM was the foil, but the content was a positive idea of who you became by choosing the Mac. That is the difference between an identity you build for yourself and one you shape too narrowly in response to a real or perceived enemy.

I think Anthropic understands positive differentiation. They started with their core values, but they used those values to choose a market position with thoughtful, careful, and caring AI at the heart of it. Their users depend on the reliability that their values offer. Grok, by contrast, has the foil without the substance. Its identity is principally the negation of the models it dislikes.

Of course, not all differentiation is schismogenesis. Different teams attacking the same problem come up with different solutions, keep what advantage they can proprietary, but at the same time they try to copy the best from their rivals. Much of the stack actually converges. Claude Code and Codex feel like siblings. But we don’t want a monoculture. We need healthy schismogenesis. It’s a competitive frontier, part of what the poet Wallace Stevens called “search[ing] a possible for its possibleness.” It’s also a cultural frontier. People all over the world don’t want one AI that reflects one set of values. Sovereign AI is not just an economic imperative, it is also a cultural one. But there’s a real risk in that.

Bateson did not think schismogenesis was benign. He thought that on its own it ran to breakdown, symmetrical rivalry escalating into open conflict, complementary difference hardening into rigid domination and submission, and that a society survived it only by having some countervailing mechanism that periodically reset the tension. Among the Iatmul it was a ceremony, the naven, that inverted the ordinary roles and let the pressure out.

In tech, the equivalent to the naven might be the standards body ;-) Or maybe it’s open source AI, and the role of standardized protocols in enabling companies and individuals to flourish beyond the boundaries that are set for them. This is a trailing thought, but that’s one of the things I learned from Frank Herbert, also 50 years ago, when he told me that one of his goals in Dune was to have his readers “go skidding out of the story” with unanswered questions that kept them coming back to his world for more. I leave you to ponder what our AI naven might be.

Freedom to leave

There are brakes on schismogenesis at the lab level. A model has to be useful or people stop using it. Users can leave for another model with a click. The public still reacts, the way it did when Grok’s MechaHitler episode drew a bipartisan letter from Congress, a rebuke from the Anti-Defamation League, and a resignation at the top of X.

The quest for sovereign AI and the role of open weight models and open source AI in giving power back from the labs to users are also a kind of brake, a competitive check on the power of the big labs to impose their values.

The simplest and most powerful reset, though, is the freedom of users to leave, to choose an alternative. This is real in AI. The moats that the labs have built so far are relatively weak, as we’ve seen recently with widespread adoption of GPT 5.6 Sol when Fable became temporarily unavailable. I had a brush with this myself. I ran out of Fable usage in my Pro account in the middle of a fine-tuning project, and decided to try moving it over to Sol. I had managed the project well, so there was a CURRENT_STATE.md with the project description and what had been accomplished so far, plus the original training set, the cleaned training set, and so on, all bundled up into a zip file. I handed it to Sol, it reviewed the work, and picked up right where we left off.

At the same time, though, I have gotten better at working with Claude and Claude has gotten better at working with me. The two of us have converged into a paired unit around the tasks I use it for. Every session cuts the groove a little deeper. My prompts tend to be for overlapping tasks, its memory fills with my context, and the fit is the product. I’ve tried to condense some of the work we’ve done together into a skill that could be executed by other people at O’Reilly, and was surprised to find that the skill + my context works better than the skill with someone else’s context. So skills as a kind of portability that I thought I could depend on turned out to be weaker than I thought.

This pairing with a model might be a kind of complementary schismogenesis at the scale of one person. The user and the model differentiate from all other user/model pairs together, and the pair pulls away from every alternative I am not using. There is a widening gap between me-with-Claude and me-with-anything-else.

It’s a marvelous feeling. It is also lock-in. The switching cost is not (yet) a file I can download and pass on reliably. The co-adapted relationship, the accumulated context, and the shared work have become one company’s moat. It erodes our freedom to leave. The more the pairing helps each of us, the more it costs to walk away. For many tasks, I haven’t even switched from my Pro account to my company account, because I haven’t yet figured out how to move over all the necessary context. For a given task, it might be as easy as my switch between Fable and Sol, but for the whole package of my Claude relationship, perhaps not.

I imagine that with some spelunking, I will find where Claude keeps all this context it has for me, and I hope most of it will be something I can move. But I’d be a lot happier if portability were something we could all take for granted. If my memory and context were mine to carry, in a format another model could read, pairing would not harden so easily into capture.

We do not have that portability standard. It is one of the missing mechanisms of the agentic economy, plumbing that a market needs to stay competitive. It’s not just memory, though memory may be the current best hook to work on. There are also differences in terminology, filenames, working patterns, skills, and assumptions. Build for portability and the choice of model remains fluid. Leave it unbuilt and the intimacy we build with our favorite models can become a source of lock-in.

Subscribe now

Written by Adam Jones (Member of Technical Staff at Anthropic) and Tadas Antanavicius (Co-creator of PulseMCP). Republished with permission from PulseMCP.

The original article was published by PulseMCP on July 17, 206: https://www.pulsemcp.com/posts/effective-patterns-for-advanced-mcp-usage.

Introduction by the AI Disclosures Project

Perhaps the best example of how an open protocol is shaping the application layer of AI markets is the Model Context Protocol (MCP). MCP allows AI applications such as Claude Code (MCP host / client) to get external context from an MCP server on the basis of an open protocol. A large ecosystem of specialized servers has grown up around the protocol, with each server exposing one or more tools, resources, or prompt templates that any compliant client can use.

But how MCP is currently being used to advance a more distributed intelligence — one that combines context, AI, and enterprise tools — is not that widely known. In the article below, Adam Jones (Member of Technical Staff at Anthropic) and Tadas Antanavicius (Co-creator of PulseMCP) takes us through how MCP is used for advanced workflows. It is republished with permission from PulseMCP.

Why Open Protocols? First, it is worth recapping why we focus on open protocols at the AI Disclosures Project. Open protocols are a core means to advance architectures of participation for AI Markets.

Previously, we argued that open protocols are market shaping devices. They greatly improve a digital market’s long-term potential for participation, innovation, and inspectability. This has been covered in previous pieces we have written:

• ”Disclosures. I do not think that word means what you think it means” by Tim O’Reilly published in this substack

• ”Open Protocols Prevent AI Monopolies” / ”Protocols and Power” by Ilan Strauss and the AI Disclosures Project, Published in AI Frontiers and this substack

• ”MCP in Practice” by Sruly Rosenblat, published in Towards Data Science
  

And more recently in a talk given by Ilan Strauss on “Where Does Intelligence Reside?”, given at O’Reilly Media’s Foo Camp, in which he argues that the most important economic impact of open protocols is that it advances a specialized division of labor:

Open protocols are shared rules that allow independent users, agents, or devices to coordinate and communicate on the basis of a publicly available specification. They are a type of standard.

By hardening an interface at a given layer through an open specification they allow other firms to connect to it reliably and then specialize against it. They are a battering ram against the integrated producer and the monolithic architecture. A single, closed architecture is designed so that potentially separable complements cannot emerge as independent markets. Under a monolithic architecture specialized production is impeded because the interfaces between those functions simply don’t exist or are controlled by a single firm.

By contrast, a modular architecture is the technical enabler of specialized production that can be disassembled safely (an idea that traces back to Herbert Simon’s 1962 paper, “The Architecture of Complexity”.)

Once specialization takes hold at one interface via an open protocol, the integrated architecture of the AI lab or the platform can be challenged via the market as firms can now compete to supply or draw on discrete components (such as memory, retrieval, payments, identity, tool invocation, attribution, safety filters, and so on).

In doing so, an open protocol can drive vertical disintegration. Interfaces become standardized and what used to be one firm’s internal operation gets spun out into a specialized industry as the protocol now handles the coordination (or the identification, the encryption, the security) that previously required all functions to be retained inside the same firm. That spinning-out of tasks (in economic terms) is precisely Allyn Young’s progressive subdivision of production across specialized firms.

Although not usually described in market-shaping terms, open protocols, therefore, are exactly that. They shape the market intelligence that can be built on top of them.

And now on to the main event…

Effective Patterns for Advanced MCP Usage

By Adam Jones (Member of Technical Staff at Anthropic) and Tadas Antanavicius (Co-creator of PulseMCP).

Most MCP demos feature a single server connecting to a single client. For example, you might wire up a Gmail MCP server to Claude Code. It works! It triages your inbox, drafts replies, finds that thing from three weeks ago.

But it’s a little pointless. Gmail already has a perfectly good interface for email. It’s called Gmail. Google has spent twenty years refining it. If “chat with your inbox” were all MCP got you, you’d be right to wonder what the fuss is about.

Connecting to multiple servers starts the unlock

Here’s the same Gmail MCP server doing something Gmail can’t do alone: filing business expense receipts.

An IKEA order confirmation is sitting in my inbox, receipt attached. Benepass — a benefits provider — wants that receipt uploaded and submitted. Claude Code can find the email, pull the attachment, log into Benepass by retrieving the one-time code from Gmail, upload the receipt, and submit. Done.

No single app could do this, because the job crosses apps. Server composition is a killer feature of MCP, not just “your AI talks to one integration.”

Where things get really interesting: multiple clients

There is clear value in connecting multiple servers to a client like Claude Code. Taking the pattern a step further: there’s also value in connecting those same servers to other interfaces you like to use. Claude Code in your terminal shouldn’t be the only way you access AI. Claude.ai can be another entrypoint. Gmail can be another. Linear yet another. The list goes on.

You’ll often want the same integrations available when you’re doing a quick task inside Slack as you would while running a long agentic task with Claude Code.

You could be conversing with a coworker, cc in “@ai does our discussion here align well with current company strategy?” and get an inline response that incorporates your company OKRs from Notion and some recent Zoom transcripts from the last few product strategy meetings.

Share

Or you’re working through your to-do list in Linear, realize you need to schedule a meeting as the action item for one of your tickets, and you can comment on a ticket, “@ai schedule a meeting with John and draft an agenda based on the meeting I just had with Pam to address this.”

These native MCP client functionalities are officially on their way to many of the interfaces you use today. And we’re starting to see prominent non-native integrations like Claude Tag fill this need, too.

But even better: you can already set up yourself and your team with these capabilities today. You just need to bridge the desired surface to your own agent harness behind the scenes.

All it takes is awareness of the right patterns and some MCP-friendly glue.

Subscribe now

Go remote ASAP; local servers won’t get you far

Local servers are hard for end-users: “Just install npm, then edit this JSON file, then set these environment variables” is not something you want to say to most of your colleagues.

Remote servers are much easier: “here’s a URL to paste in somewhere”. Some surfaces only support remote servers, like claude.ai.

So you can use local servers to prove out some flow, but you should prefer remote. Many servers you may want to use are only provided as local implementations, so you need a tool to convert them to remote servers. Because most local servers were designed to be used by one authenticated user at a time, this can be tricky.

Subscribe now

The pattern that solves this pain point: use a bridge like mcp-auth-wrapper to take a local server and turn it into a remote, multi-tenant one: OAuth in front, per-user credentials behind. Sharing any server becomes sharing a link.

Remove friction from the configuration process

Okay, we’re down to one link per server. But exactly where do my users put this link?

The one-size-fits-all pattern here is to provide installation guidance for every possible MCP client app. A service like install-mcp makes this easy: clean UX to generate the right install link and per-client instructions, so you share one page instead of writing a tutorial.

Or, if you want to roll it into your own interface, tap into the TypeScript library version of this: mcp-install-instructions.

We’re working on standardizing the mcp.json file format here to simplify this story in the long term, but ultimately we still expect non-CLI interfaces to each have a slightly different flow for “enabling a connector.” For many enterprises, this responsibility may soon become solely an IT affair with Enterprise-Managed Auth.

Remove the need to do it again, and again

What happens when we start to bring new clients into the fold? By default, each user has to configure and authenticate each MCP server independently. 10+ “add connector” flows. 10+ OAuth dances. Done once for Claude Code, then again for Linear, for email, and so on.

The solution: centralize your configuration and auth storage in a single, aggregated MCP server. A deployment of mcp-aggregator can do this for you: one endpoint, one login, every tool namespaced behind it. Configure each client once. Login once per service. Share one link with your teammates.

mcp-aggregator is the minimal, DIY version of an MCP Gateway. If you have enterprise needs like SSO integration or fine-grained IT admin controls, you could use this layer of MCP server consolidation as the piece of your infrastructure where you can add those bells and whistles.

By default, mcp-aggregator also loses out on the ability to use MCP server configurations as per-session constraints. However, you could build out your mcp-aggregator deployment to re-enable the constraints that matter to you. For example, by offering query parameters like ?readOnlyTools=true or ?serversEnabled=datadog,sentry. Or by giving discrete MCP servers unique endpoints (but still aggregating the auth concerns under the hood).

Work around the long-tail of missing MCP servers

Most of your MCP server needs can be solved by walking through the options for choosing (or building) an MCP server. But is it always worth the effort?

Say you switched home utilities plans. An energy provider, Octopus Energy, emailed you a confirmation. You want the pricing data in Home Assistant, which controls your smart thermostat, and Octopus Energy does have an API for it — but the API key is behind a website login, and there’s no API for getting the API key.

Solution: give the agent a screen. computer-use-mcp is a minimal example of an MCP server that can click and type like a human: log into the website, copy the key, then back to clean API calls to set up the dashboard.

Share

With that one escape hatch, “there’s no API for that” or “it’s too much work to find an MCP server” stops being a blocker forever.

To borrow from Anthropic’s framing of this problem:

• If you have an MCP server for the service, Claude uses that.

• If the task is a shell command, Claude uses Bash.

• If the task is browser work and you have Claude in Chrome set up, Claude uses that.

• If none of those apply, Claude uses computer use.

And indeed, if you find yourself regularly leaning on inefficient approaches like browser use or computer use to get a predictable workflow done, it may be worth your while to abstract away those Playwright calls into a reliable MCP server, like this Good Eggs example.

You can even bring local-only servers into the remote-first fold

Some servers only work when they run on your own personal machine. computer-use is the obvious example. And sometimes, you just want to use your phone to tell Claude to do something that relies on something readily available on your laptop upstairs.

mcp-local-tunnel shows off the pattern that makes machine-bound servers available through remote aggregation just like everything else.

Don’t forget the context bloat footgun

Naive tool calling has a much-maligned drawback: every intermediate result flows through the model. And for many clients, every tool definition is placed in context up front — meaning you’ve spent tens of thousands of tokens before even starting.

For example, asking the agent to copy a doc’s contents from Google Drive into Salesforce means the entire doc passes through the model twice — for no reason. Anthropic measured one workflow dropping from 150,000 tokens to 2,000 by fixing this.

There are several ways to solve this problem, each with its own trade-offs worth its own blog post:

• Code execution with MCP. tool-sandbox-mcp shows how a single execute_code abstracts away the problem by adding a layer of code execution in between your agent and your aggregated set of servers. Works inside any MCP client.

• MCP as a CLI tool. call-mcp shows how wrapping your tools this way means you can compose them with your usual CLI toolkit — shell, jq, cron, scripts, and other CLI tools.

• Search tools, truncate large responses. This is the way Claude Code does it natively, but could be implemented as a bridge, much like the above examples.

Put these patterns together, and you’ve solved the MxN problem

With your MCP aggregator in hand, you can now connect any service to any other service with just a little bit of glue code. When each service starts to implement an MCP client natively, this will be done-for-you. But for now, you can tackle the opportunities application-by-application.

Embed existing MCP client harnesses into SaaS apps you already love

We’ll use a Linear integration as our example, and you can imagine doing almost the exact same thing with any project management tracker like Jira, Asana, and so on.

The goal: inject a highly-capable MCP client, powered by your favorite coding agent, into a workflow inside a SaaS application you already use. We’ll take advantage of the fact that we have already consolidated all our integrations in the mcp-aggregator pattern above.

Here’s an end-to-end project showing off how to build this sort of “Linear harness” that bolts a Claude Code setup to serve as the MCP client.

You can see how:

• The harness can listen to any sort of activity on Linear — in our case, naively reading all ticket comments

• You can equip the harness with whatever Skills, Plugins, or other MCP connections you please

• It can respond back on Linear by way of a Linear MCP server (or, ideally, the MCP aggregator setup from above)

There’s no limit to where you can embed these capabilities

Going beyond Linear and task management: here’s a Claude Code-powered agent that lives inside Minecraft in a few hundred lines.

In game, we ask it to:

• Write a PR to edit itself.

• Check the upcoming energy prices and put a calendar event in for when to run the washing machine.

• Build a town-square notice board in-game with a sign that reminds us of that time

It’s silly on purpose. The point: a fully capable agent — all our tools, all these patterns — can run anywhere with a little glue. And it’s all attainable today, for individuals, for teams, and for enterprises alike.

Thanks for reading! This is a publication of the AI Disclosures Project. Subscribe, share our work, and help build an architecture of participation for AI Markets today.

Support our Mission

Introduction

Will or won’t AI lead to a “jobless apocalypse”? I want to offer one framework for thinking through this question based on Adam Smith and Allyn A. Young — and Hal Varian on standards.

The question I want to ask you today is as follows: Where does intelligence reside? Where does it live?

The argument I want to make is that the artificial intelligence found in large language models (LLMs) is inert and centralized in the models’ weights. This is a very different type of intelligence to that exhibited by markets, which is fundamentally decentralized and constantly being updated.1 Markets already extend and enable machine intelligence; but often sporadically and not always via market exchange. Integrating these two types of intelligence, including through the price mechanism, is vital if AI is to advance specialized production, market exchange, and ultimately employment creation.

Is this Market Intelligence?

To understand market forms of intelligence I want to begin by asking whether this picture below reflects a market-based intelligence. It shows women making men’s suits using sewing machines. It looks organized. But according to what principles?

How do these workers know how many suits to make, what types to make, and with what machinery? By informed guesses of a central committee, it turns out.

The above picture is from Bolshevichka garment factory in Moscow, taken in 1967 when it was part of the Soviet Union. This factory used to produce the majority of men’s suits for Russia. Although the Soviet Union excelled in heavy industry production, it ultimately failed in consumer goods production, where market signals were needed to inform changing tastes, quantities, efficiencies in resource utilization, and fundamental innovations. Waste was abundant. Human need could not be met. Production techniques were outdated.2

Bolshevichka still exists as a menswear brand and company today, selling men’s suits, but now informed by market signals.

Two core features of market-based intelligence

What would it mean if the above factory was instead organized on the basis of a market-based intelligence? This would require two things.

1. The first is decentralized and permissionless innovation. Markets are a federated system, where information remains dispersed along with the trillions of independent buying and selling actions that it enables. Market-processed information is what enables and disciplines these actions (along with society’s conventions and laws).

2. The second is that current information signals inform market coordination through constantly shifting agents’ incentives. Prices and product information are always being replenished as preferences and technology change, and demand and supply decisions respond accordingly. Market signals inform decision making but also respond to it. For this to work, price signals and other relevant information must be widely available (and not gameable).

Share

It appears to me that digital markets have till now often tried to mimic market forms of intelligence. Google Search of old is an example of a market-like intelligence, I would propose. Google Search has:

• Permissionless innovation (building on the web’s open protocols). Anyone can create a website. Information (websites) remain federated on servers.

• Dynamically updated signals. What the user clicks on (consumes) in the search result page is used to improve and constantly update website search result rankings. This in turn improves the next allocations of user attention to information, which in turn informs what information websites and other producers should supply to the market.

The algorithms of this era advanced what Tim O’Reilly calls “an architecture of participation”.

But will markets create jobs? The truth about market-based systems of intelligence is that humans may not always be needed. Nissan’s “intelligent factory” below is just one example of labor being made redundant in manufacturing production. But without markets expanding production on a progressive and balanced basis, little prospect for human employment and prosperity exist.

Artificial Intelligence

How do market forms of intelligence compare with present day artificial intelligence in LLMs?

1) An LLM’s intelligence is inert and unresponsive at its core. The model’s weights are trained on information and feedback after which they become fixed, “dead" things. They are unable to dynamically update or learn as they go. That is why LLMs are reliant on markets for a steady stream of new information, and in particular markets online.

Besides for the internet, the other primary source of live information for models is human users and firms. Every time a user has a conversation with a generative AI system, uploads a document, or tells it something, they allow the model’s weights to respond to live, up-to-date, information. Post-training also incorporates human judgement and information, organized by firms such as Surge AI. But direct market signals seem to only weakly inform these activities and the piece rates offered to human annotators.

Subscribe now

2) The other aspect of present AI is that it aims to be generalized – as in artificial general intelligence. In doing so, LLM companies strive to be able to do every task that a human can do and at a high level of competency. This potentially undermines the idea of specialized producers existing across a market, each using different and specialized production techniques. The idea of a single generalized producer (“AGI”) is so radical that it threatens to undermine the notion of markets altogether.

And that is the central contention of this talk: that specialization underpins the ability for a market’s intelligence to grow, for permissionless innovation to flourish, and for price signals to function effectively.

Specialized production provides a bridge between market forms of intelligence and artificial intelligence. So if we want to grow the market for AI, as a system of permissionless innovation, with room for human contributions, it needs to permit – and admit – that specialized production is necessary and desirable. General intelligence divorced from market forms of intelligence cannot survive, except as a stale set of encyclopedias sitting on the shelf.

Specialization

Specialized production — or specialization for short — is the ultimate basis for market forms of intelligence and its expansion (an idea we owe to Adam Smith). Specialization is what drives production for exchange, as for-profit production, coordinated via the market.

Allyn Young in 1928 went one step further when he argued that specialization in the market is a self-reinforcing dynamic. The more I specialize as a firm, the more I allow other firms to specialize too, as they can then meet more of their needs (procure more of their inputs) via the market, instead of through internal production. This means that specialization is not only the foundation of market exchange (Adam Smith), but can create an ever-expanding basis for the market itself to grow through increases in output (supply).

Specialization as external economies. The benefits of specialization are best viewed from the industry or ecosystem as a whole that it helps to expand. The early American auto industry is a good example of this. Ford and its rivals began as assemblers, drawing on Detroit's dense ecosystem of specialized machine shops and parts makers. The Dodge Brothers famously supplied Ford's engines and transmissions before ever building a car of their own. It was this external supplier base producing standardized components for a variety of buyers that allowed for mass auto production to arise and prices fall. This grew the size of the market for everyone. Ford later advanced vertically integrated production in its famous River Rouge complex (which Charlie Chaplin’s 1936 film Modern Times drew on), until more distributed supplier networks gained favor again. The benefits of specialization, therefore, are about the unfolding division of labor between firms, not about a single firm scaling up its own production, which can potentially be to the exclusion of others.

Specialization and integrated / monolithic producers. Specialization undermines the integrated producer, where a single firm attempts to do everything under its own roof, without procuring standardized inputs via the market (or transferring its intermediate outputs only internally, rather than selling them).3

Advancing specialization online and in LLMs’ own production process can have similar benefits. But this can only have a chance of succeeding if model building is no longer undertaken from start to finish in relative secrecy (housed within a single firm). And if information and intelligence can permanently reside outside of a model’s weights. Model weights don’t only have to consume markets, they can also enable them. Standards and open protocols, as the technical analogues of market-enabled specialized production, are two important mechanisms for advancing such an agenda.

Integrating AI with Markets: An architecture of participation for AI

Standards and open protocols are two key mechanisms for advancing market forms of intelligence online that are decentralized and permissionless. The goal is to allow for artificial intelligence to be extended into the living market of information, admitting formal prices and for market-based allocations.

Standards

Standards enable specialization by allowing interconnected parts to fit together. In markets with many complementary parts, standards are vital to allowing a single producer to address the whole differentiated market of firms as if it were a single source of demand. This helps the supplier to not be captured by a single producer (“lock-in”). As my colleague Tim O’Reilly remarked to me, one much-underappreciated ingredient to the industrial revolution’s take-off was specialized inputs and parts being produced to a standard.

Such standards are especially important in networked markets since connecting to the network is vital to tap into its scale and capabilities. This means that it can potentially let average revenue per user grow for any firm with access to the network’s expanding utility.

So in technology markets, standards not only facilitate lower costs through specialization / scale, they also inform whether the value of a network is captured by one firm or shared by many.

Open protocols

Open protocols are shared rules that allow independent users, agents, or devices to coordinate and communicate on the basis of a publicly available specification. They are a type of standard.

By hardening an interface at a given layer through an open specification they allow other firms to connect to it reliably and then specialize against it. They are a battering ram against the integrated producer and the monolithic architecture. A single, closed architecture is designed so that potentially separable complements cannot emerge as independent markets. Under a monolithic architecture specialized production is impeded because the interfaces between those functions simply don’t exist or are controlled by a single firm.

By contrast, a modular architecture is the technical enabler of specialized production that can be disassembled safely (an idea that traces back to Herbert Simon’s 1962 paper, "The Architecture of Complexity”.)

Once specialization takes hold at one interface via an open protocol, the integrated architecture of the AI lab or the platform can be challenged via the market as firms can now compete to supply or draw on discrete components (such as memory, retrieval, payments, identity, tool invocation, attribution, safety filters, and so on).

In doing so, an open protocol can drive vertical disintegration. Interfaces become standardized and what used to be one firm’s internal operation gets spun out into a specialized industry as the protocol now handles the coordination (or the identification, the encryption, the security) that previously required all functions to be retained inside the same firm. That spinning-out of tasks (in economic terms) is precisely Allyn Young’s progressive subdivision of production across specialized firms.

Although not usually described in market-shaping terms, open protocols, therefore, are exactly that. They shape the market intelligence that can be built on top of them.

The Model Context Protocol (MCP) comes to mind here. It helped catalyze specialized production in AI applications and a more permissionless innovation by allowing AI agents (clients) to connect to external tools and context. A large ecosystem of specialized servers, each exposing one or more tools and resources, grew up around it, which any compliant client can use. Such specialization in turn grew the market for other specialized producers. Specialization begot specialization and rival labs eventually adopted the standard.

Concluding Initiatives

The dream of artificial intelligence being very “general” in nature (as in “AGI”) seems, at the end of the day, to be technically compatible with a specialized division of labor for its production and with the existence of specialized AI producers. The market for AI startup financing seems to think so too, funding different approaches to AGI (and recursive self-improvement) and funding differentiated, highly specialized AI producers at the same time.

Extending machine intelligence into markets, and vice-versa, to allow for human participation and reward via markets (“an architecture of participation” for AI) is the focus of the three initiatives we are launching this year at the AI Disclosures Project:

• Open Memory Protocol Consortium (OMPC). A multi-stakeholder nonprofit consortium developing open, interoperable specifications for portable AI memory. Membership-based, with open working groups, it provides early-stage technical incubation on specific aspects of agentic memory in specific markets.

• The AI Markets Initiative. A cross-institution economics research and data collaboration that galvanizes the best economic minds around AI market design challenges and opportunities. The goal is to create functioning AI markets with architectures of participation.

• Skills Marketplace. Launching a skills marketplace with monetization for skills authors based on modular, interoperable, rights-protected skills, with proven uplift to agentic performance. All specifications and code will be open-sourced, to be used as a prototype for others to adopt.

If you are interested in being involved in any of these three initiatives in whatever capacity, we would love to hear from you. Thank you for your support.

Support our Work

Postscript: Specialization via mixture of experts (MoE) models and agent orchestration are not dealt with here but are worth noting.

We recently convened 20 experts at The Rockefeller Foundation Bellagio Center in Italy on Missing Mechanisms for the Future Human+AI Economy (27 April – 1 May, 2026).

The premise was that AI systems can and should be designed for human participation and reward.

Every major technological transition reshapes the economy not simply by automating tasks or reducing costs, but by creating new markets, new institutions, and new forms of participation in value. AI will be no exception.

But for new markets to emerge we need to advance new rules and market infrastructure for innovative, participatory, and distributed value creation. The overarching principle behind this is the '𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗼𝗳 𝗽𝗮𝗿𝘁𝗶𝗰𝗶𝗽𝗮𝘁𝗶𝗼𝗻': open systems built for user & producer contribution enabled through open protocols and economic mechanisms.

To tackle this market design challenge a truly incredible group of experts were assembled from across economics (Jacques Cremer, Markus Mobius, Nicole Immorlica); open protocols (Tadas Antanavicius, Ola Hungerford, Ido Salomon,Liad Yosef); attribution & digital content (Geoffrey Bilder, Paul Farrow, Lucky Gunasekara), standards (Alissa Cooper Mark Nottingham, Mallory Knodel); AI systems (Alexander Hancock, Serkan Piantino, Mike Linksvayer, anonymous), and the AI Disclosures team itself: myself (Tim O'Reilly), Ilan S. & Sruly Rosenblat. The full bios of participants is available on our updated website.

We decided to put all the unique lectures, discussions, presentations, working group prototypes, and outputs from the Bellagio convening on our new website, which I would encourage you to read: https://ai-disclosures.org/bellagio

This is only the beginning of a process to advance participatory, decentralized AI markets built for human participation, reward, and oversight. If you share our vision of an architecture of participation for AI markets through open protocols and progressive market design, please join us in the process. You can listen to recent talks and lectures we have given, news coverage and new mentions, on our new website.

You can sign up to our newsletters: here.

And you can donate to the AI Disclosures Project: here.

AI Sovereignty and the Architecture of Participation

By Tim O’Reilly. This was originally published on O’Reilly Radar (June 1, 2026).

Adam Tooze recently shared a piece from The Economist about Brazil’s push for what it calls “medical sovereignty,” the determination to make its own vaccines and the active ingredients that go into its medicines rather than depend on supply chains it doesn’t control. Brazil already produces a large share of its own medicines through public institutions like Fiocruz and Butantan, but a lot of the underlying inputs still come from abroad, and the pandemic made clear the cost of that dependence. So the country is trying to build the capacity to make the things it most needs to survive. The economist behind a lot of this thinking is Mariana Mazzucato, whose mission-oriented approach treats public procurement as a tool to build national capacity rather than just buy finished goods. (Foreign Policy has a good overview.)

I think we’re going to see a lot more of this, and not only in medicine. The same impulse is driving the quest for sovereign AI, as countries decide they don’t want their access to a foundational technology to run through a handful of American or Chinese companies. You can see it too in Europe’s and Japan’s new willingness to take responsibility for their own military destiny rather than assume the United States will always be there.

Most commentators describe all of this as decoupling, the unwinding of a connected world. That reading is too narrow.

Free trade was an architecture of participation that broke

Much like open source software and the World Wide Web, free trade was supposed to have what I call “an architecture of participation.” The most important thing about the web and open source wasn’t the licenses. It was that there were no central gatekeepers. Anyone could add to the richness of the system without asking permission as long as they followed the rules of the communication protocols that allowed independently-developed pieces to work together. In addition, value circulated among the participants instead of being extracted to a center, and the system got better the more people used it. That is a very different thing from a system that is merely large and connected.

Free trade was also supposed to work like that. The theory, going back to Smith and Ricardo, was that specialization and exchange would make everyone better off, and that the connections would be mutual. What we actually got over the past few decades looks more like the platform dominance we see in big tech than the original vision of a commons built around shared exchange. A handful of large and powerful countries and firms set the terms and the smaller players are forced to take what is on offer. Despite the language of free trade, the experience for many countries was closer to colonialism, just with a new narrative.

Overall, under the neoliberal order (whose reign, as Gary Gerstle explains, is now ending), free trade became far less egalitarian, inclusive, and generative than it could have been. Less powerful countries ended up in roughly the position that small businesses occupy on Amazon, or developers occupy on the app stores: free to participate, on terms they don’t control, with much of the value they create flowing back to the hub.

Brazil’s response (and that of many others) should not be seen as a retreat from the world. It is a refusal to participate only as a buyer, or as a source of raw materials.

That’s why decoupling is the wrong word. Decoupling means cutting the connections. What these countries seem to want is to stay connected but to build real capacity of their own, so that no single supplier can switch them off. That’s closer to federation than to separation. A federated system is still a system, and its nodes still interoperate. But no node is wholly at the mercy of another, and value circulates among them rather than collecting at the center. A trading order in which the gains pool at a few hubs is brittle and eventually illegitimate, in the same way that a platform economy that strip-mines its participants eventually provokes regulation and revolt.

I put the increasingly visible quest for sovereign AI, and the role of open source models and open source agentic protocols and harnesses in enabling that sovereignty, into the same bucket. I remember back in the early days of open source software when Michael Tiemann, whose pioneering open source company Cygnus Solutions had just been acquired by Red Hat, told me “What we really sell at Red Hat is control. The ability to control your own destiny.”

As companies are increasingly at the mercy of unexpected token pricing changes by the big centralized players, this same quest for sovereignty is playing out at the level of organizations. Open source AI, including not just open source and open weight models but open agentic protocols, agentic harnesses, and portable memory, is increasingly an essential part of the sovereignty toolkit.

The national technology sovereignty movements should take a lesson from the open source movement. The heart of open source is its architecture of participation. It is a force for innovation and value creation to the extent that it frees up the ability of people to solve their own problems and contribute their solutions to a low-friction global commons.

Is capture the inevitable fate of any architecture of participation?

The pattern of open architectures leading to a wave of innovation, winners emerging, consolidating their power and then turning to the dark side seems to be a natural part of the technology cycle. The web broke Microsoft’s dominance over the personal computer software ecosystem only to give rise to a new generation of gatekeepers. Cory Doctorow called this cycle “enshittification.” I’ve told my own version of that story using the language of economics in “Rising Tide Rents and Robber Baron Rents.”

The instinct after capture is to try to rebuild the thing that got captured, only this time with better rules. Mastodon and Bluesky tried to rebuild Twitter’s social layer with cleaner governance, and neither has succeeded at the scale they hoped for. Critics might say that it was because Mastodon stayed pure and never made itself easy enough to use, while Bluesky looked federated without really being so. But more importantly, reinventing what we used to have, or what we think we used to have, is rarely the path forward. You have to build something new.

Each country building its own answer to the latest frontier models is the Mastodon move. The winning move is to operate at a layer the centralized model structurally can’t reach. Open agent protocols that let services from different providers interoperate (the work that MCP and the emerging agent stack are beginning to do) are one such layer. AI accountable to local democratic and legal institutions is another such layer. Domain-specific AI built around problems the global market won’t serve (the tropical disease vaccine analogue) is another. None of these is a smaller copy of what the hyperscalers offer. But there’s one more important layer to consider: infrastructure.

Share

Where are the servers?

Ilan Strauss made a useful point in our conversation about these ideas. Ilan noted that AI is one of the most global forms of capital we’ve ever built, trained on the whole of the internet and runnable more or less anywhere, and the sovereignty rhetoric is partly an attempt to give something inherently placeless a place. The technology wants to be everywhere at once. The people who live with its consequences want some say over it where they are.

The placelessness of AI is only half of the truth, though. The other half is that AI is physically place-bound. The model weights are placeless. The data centers, the chips, the electrical grid, and the water for cooling are very much somewhere.

The comparison with Brazil’s medical sovereignty reinforces this point. Brazil’s challenge isn’t to invent new drugs to compete with Pfizer, but to build the capacity to manufacture existing vaccines, and eventually to build the capacity to invent vaccines for diseases the West ignores. Fiocruz and Butantan are physical institutional capacity rooted in Brazilian soil: the labs, the cold chains, the regulatory capacity, the trained workforce, and access to the active pharmaceutical ingredients. What creates medical sovereignty is infrastructure plus the institutions that run it.

The same is becoming true for AI. Open weights matter. But even if Qwen, Kimi, DeepSeek, Llama, Gemma, Granite, and whatever comes next are fully open, running them at scale requires data centers that cost tens of billions to build, chips whose supply chains a handful of countries control, and electricity grids that have to be expanded substantially to carry the load. The countries pursuing sovereign AI seriously seem to understand this. The EU’s AI Gigafactories program, India’s IndiaAI mission, the Gulf compute buildouts, the Singapore and Japan strategies, are all infrastructure plays first and model plays second.

As the big hyperscalers are teaching even the frontier AI labs, infrastructure is the layer where capture is hardest to undo. You can distill or fine tune a model far more easily than you can build a new continent’s worth of data centers or conjure the necessary electricity from a fragile power grid. If the architecture of participation for AI is defined only at the model layer, the infrastructure layer below will quietly recapture, over years, everything that was won above. Open weights running on three companies’ servers is not sovereignty.

Subscribe now

Building physical infrastructure to enable economic growth and new markets is exactly the kind of mission the public sector used to take on, before we convinced ourselves the market would handle it. Mazzucato’s argument is that public procurement and public capacity-building are engines of foundational technology. AI sovereignty without industrial policy is wishful thinking.

Industrial policy should aim to reinvent 20th century infrastructure, not just copy it. Can we use the enormous rebuild of infrastructure for the AI era to leapfrog the past? The analogy with centralized power grids and decentralized solar reminds us that local control does not have to be a localized version of the hyperscaler pattern. Might we envision a future where there is an intelligence grid that seamlessly uses frontier models in massive data centers and local models controlled by the user as dictated by considerations like cost, privacy, specialized knowledge, and user preferences? Creating the software to manage such an interoperable intelligence grid should be a high priority for the AI open source community. We need an orchestrator not just for agents but also for models and even for data center capacity.

Could federated AI give us a new pattern for the economy?

In a previous piece about AI and markets, “The Third Artificial Intelligence”, I picked up Richard Danzig’s argument that markets and the bureaucracies that underpin nation states are themselves artificial intelligences, information-processing mechanisms older than the machine kind. The question with all three is who designs and builds them, what they optimize for, and what feedback loops govern them.

We’re about to spend a lot of effort working out how AI should be organized both across nations and across organizations, whether it concentrates in a few firms and a few countries or whether it can be built as something more federated, where smaller players have their own capacity and the value they create flows back to them. The choices we are now making about how AI is organized, at the model layer, the protocol layer, and the infrastructure layer, are also choices about how economic activity will be organized for at least a generation. If we manage to get that architecture right for AI, it may give us a working pattern for the thing we’ve so far failed to get right for trade. If we get it wrong, we’ll most likely reproduce, at the level of intelligence itself, the same concentration that free trade has produced in goods and the existing internet platforms produced online.

The technology wants to be everywhere at once. The people who live with its consequences want some say over it where they are. The right infrastructure to let us satisfy both goals will be a federation of models, a federation of protocols and code, and a federation of capacity. We need an architecture of participation all the way down the stack, and all the way up.

The final section of this piece benefited greatly from questions and comments raised by Ilan Strauss and Mike Loukides, as well as from previous conversations with Richard Danzig.

Donate Today

The Agent2Agent (A2A) protocol was announced by Google a little over a year ago (April 2025). It was built to allow agents to communicate with each other across different sites and to enable any agent to reach out to specialized agents hosted by any company. Since then, it was moved to the Linux Foundation where it was later joined by MCP. The primary alternative to A2A, IBM’s Agent Communication Protocol (ACP1), was quickly folded into the former.

A2A is now supported on multiple cloud platforms, is integrated into LangGraph and CrewAI, gained over 20 thousand stars on GitHub, and has over 150 supporting organizations. But despite being the only agent communication protocol with any traction, A2A still has almost no publicly deployed agents. What appears to be the largest agent registry for A2A has just 100 publicly accessible agents compared to the ~3k remote MCP servers cataloged by PulseMCP.

A lot of this is likely due to the lack of a true discovery system. A2A has discovery in the sense that if you know what domain an agent is located at, you can fetch its “agent card”2 that tells you what the agent does, how to reach it, and how to authenticate. But it does little to help you find an agent you have never heard of.

There is no native, public space for an agent to ask a question widely and get an answer back. We argue that agentic communication that remains one to one is unlikely to meet a user’s needs. Think how humans broadcast on social media to many when we communicate (and see who responds), or on Facebook Marketplace when we seek buyers. Agentic broadcasting and proper agentic discovery are two sides of the same coin. And it can be done in a way that is not centrally controlled.

-*-

Rather than treating agent communication as a brand-new problem, it helps to look at the apps we already use and the protocols that were already built for human communication. Email, messaging apps, and social media have all dealt with the problems agent communication faces: discovery, privacy, identity, and trust, so a brief sojourn through their architectures is first helpful.

The Email Protocol (SMTP): one to one communication

The Simple Mail Transfer Protocol (SMTP), launched in 1981, still enables today’s email. Specifically it defines how one domain’s email service communicates with another. When an email is sent from a Yahoo email account to a Gmail account, the message is passed via SMTP. But SMTP is not the whole email stack. It doesn’t define how you find where another domain’s mail server lives (that’s DNS MX records), it doesn’t establish whether an email can be trusted (that’s SPF, DKIM, and DMARC) and it doesn’t define how email clients like Apple Mail retrieve messages from servers (which is instead left to IMAP or POP3).

Email has another limitation: every recipient needs to be specified individually. Even in cases where it looks like an email is being broadcast, like with a newsletter going to hundreds of thousands of subscribers, that is not what is happening under the hood. Instead that same email is being sent individually to many addresses at once. Email also has no public ledger. Each message exists only in the inboxes it was delivered to, so there is nothing for non-recipients to look back at. A new subscriber to a newsletter can’t scroll through past emails in a newsletter’s archive, even if this would have been the desired behavior by the newsletter. Platforms like Substack that combine the functionality of blogs and newsletters work around this limitation by posting the article to the Substack on the web at the same time as it is sent out in email. This is a simple workaround, but it is not a solution based on how email works.

Speaking to the Network: one to many communication

Traditional Social media

Since the release of email, much of the world’s communication ended up moving away from one to one and towards broadcasting. This opened up many benefits; users were able to communicate with thousands of people at once, asking questions that no one they know has the answer to. Someone with a piano they need to get rid of can post about it and rely on the network to self-sort interested parties.

When you post on Facebook or X / Twitter, you are not messaging many different people individually; you are broadcasting whatever opinion or question you have to everyone on the network (that network may be smaller for a private post). This doesn’t necessarily mean everyone will read your message. In order for a user to see a message, it has to pass whatever filters they have and rank high on their home screen algorithm, but it’s visible to them in the sense that they can see your post if they look for it.

On most platforms this broadcasting is done using the company’s proprietary internal APIs. This makes sense as broadcasting to some extent requires centralization. In order for a ledger of profiles and posts to exist and be shown to users, a server is needed, and servers are usually financed and owned by a single company. But there have been attempts to move this type of interaction away from a centralized server.

Share

Federated / Decentralized Protocols

ActivityPub (what Mastodon uses) and the AT Protocol (what Bluesky runs on) tackle the centralization problem by allowing servers to communicate with each other. In ActivityPub, user accounts are tied to a server, and that server stores and delivers the user’s posts to other servers through federation. AT Protocol also uses servers to host user data, specifically Personal Data Servers, but it separates out other components like feeds and data aggregators (relays) more than ActivityPub does, and it is designed to make accounts more portable. Both systems are similar to traditional social media in that they still require some information to be stored on and transmitted through servers, but they differ in that they are federated: different servers can communicate with each other, and content posted on one server can be seen from other servers.

This federation provides the functionality of a traditional social media platform without locking user content to just one server since users could still access content from other servers. But the power still concentrates with the server providers: they can still choose who to federate (or communicate) with, what content is allowed, and who their users could interact with. This is perhaps most visible with direct messaging (or DMs). ActivityPub does not support encrypted messages: every “private” message can be read by the server owners where either account is hosted. Bluesky supports direct messaging with no encryption and they do it off protocol with no federation. The AT Protocol does not yet support direct messaging.

Nostr, another distributed social media protocol, takes a slightly different approach to identity and in turn communication. Instead of making a user’s identity something that is tied to a server, it makes identity a cryptographic key pair, meaning the user’s identity itself requires no hardware. The cryptographic keys that they need work the same whether they are stored on a hard drive or drawn on the back of a napkin. Posts are signed using the private key in a way that can be verified using the public key.

Communication still happens over servers (in this case called “relays”), and relays can still be censored, but a piece of content can be broadcast over any number of relays simultaneously, since the user’s identity is defined and enforced in a way that is agnostic to any one relay. If a user is blocked from one relay, other people can continue to follow them by listening to other relays.

In many ways, DMs on nostr function similarly to how they work in the Signal Protocol. Messages are encrypted so that only the recipient can see them. Given the fact that every user already has their own private and public key on nostr, encryption is a natural fit. NIP 17 (or Nostr Implementation Possibility 17)3 implements this by encrypting the message and then “gift wrapping” it to hide the sender’s identity from the public network.4

Support us

Help architect a different AI future by donating to the AI Disclosures Project today.

Regardless of the exact differences, the major commonality is that unlike something like Email or Signal, these protocols are primarily designed for broadcasting. Questions sent aren’t meant to be answered by just one person; they are sent to a whole network of people at once.

It’s not necessary to know who is on the network prior to asking for something. It’s on the recipients (potentially with the aid of the intervening platform) to decide what is relevant to them and what isn’t. Initial screening is done by whatever algorithm their feed uses and after that users choose what posts they respond to by themselves. Discovery is native to the protocols.

What A2A Brings to the Table

The Agent2Agent protocol was built from the ground up for one-to-one communication between agents. It functions using the infrastructure already built for the web. It provides one agent the ability to reach out to another agent directly. Communication happens over standard HTTPS in a way that is not end-to-end encrypted but is encrypted while in transit.5

Agents can be tied directly to the domain they are hosted on (such as airbnb.com). So an Airbnb agent inherits the trust the website domain already has. This is similar to how an email inherits the trust of the site it is hosted on (e.g., if you receive an email from irs.gov you are likely to trust it). Additionally, if an agent already knows about a site, checking whether that site exposes agents is trivial. It’s worth noting that neither the domain resolution (finding the agent given the domain name) nor the inherited trust is unique to A2A: Nostr (via NIP-05) and Mastodon (via WebFinger) also link accounts to a domain through a file in the domain’s .well-known folder.

Subscribe now

The way an agent advertises itself on A2A is through an “agent card,” a structured document published at a known URL that describes what the agent does, how to reach it, and how to authenticate. This is similar in many ways to how profiles work on social media but it is also where the protocol’s limits show: accessing an agent card from a website assumes you already have a domain to check. An A2A agent not connected to a major site is essentially invisible without a third party registry to rely on.

The Costs of One-to-One Agent Communication

Agent communication through A2A closely resembles how email works. Both are primarily one-to-one, both cut out middlemen, identity is simple, and neither requires a third-party server. It is just one IP address reaching out to another and sending messages, the same way one mail server hands a message to another, with nothing sitting in between to be trusted, paid, or asked for permission. In many ways this is ideal: messages are truly private, nothing logs or indexes the exchange, and no platform has control over it (assuming two self-hosted domains). Agents get a persistent identity tied to URLs people already recognize.

But that architecture carries the same cost email has always carried: discovery. The first place you look for someone’s email address is never the email protocol itself. You search the web, check their publications, or pay a third party. Social media and its federated equivalents don’t have this problem: you broadcast what you offer and interested parties come to you. For agent communication to truly catch on outside of company silos, agents need that same ability to find each other.

One-to-one communication has its place. Once an agent is known it makes sense to keep conversations as private as possible. And introducing any third party server inherently lowers privacy.6 Even encrypted conversations leak some metadata about who is being contacted. But there are also inherent limitations to one to one communication – even if discovery is solved. One-to-one communication adds unnecessary friction in cases where more than one agent is needed.

That being said, broadcasting opens up new problems. Any public square an agent can broadcast into is also one an attacker can broadcast into — and a social media platform swarming with agents is a large target for prompt injection. MoltBook has already proven this. There are also real privacy tradeoffs with a broadcasting model. Every public message sent to the network could be read by everyone on the network and just like any content on the internet, anything public is hard to revoke. There are tradeoffs to different communication protocols and the pros and cons of each approach should be discussed in detail. But agents will eventually need a public square.

I built out a prototype exploring how agent communication could look on nostr. It’s still pretty early but I would love to hear any feedback you have: https://github.com/SrulyRosenblat/social_agents_prototype_nostr

Support Us

Help architect a different AI future by donating to the AI Disclosures Project today. Open protocols, enables open markets.

A related consideration comes from Anthropic’s recent acquisition of Stainless — a startup that generates SDKs, command-line tools, and MCP servers from API specifications. This illustrates that open protocols like MCP, even when publicly governed,1 remain exposed at their complementary layers to private actors capturing rents. (Protocol openness does not eliminate this and instead probably enables it, by enabling market growth).

We asked Claude to analyze this acquisition, going beyond the press releases. Its first pass overstated parts of the competitive-denial story; what follows is what survived it taking a closer look:

1. Complement capture, not protocol capture. MCP — the standard that lets AI agents talk to other software — remains open, and its governance has been handed to an independent foundation. What Anthropic bought is the company that turned that standard into something most developers could actually use. Stainless was the dominant tool for taking an ordinary business API (say, a hotel booking system or a customer database) and converting it into something an AI agent could call through MCP. The open standard is still open. The path most developers walked to use it has now been bought.

2. This isn’t a one-off — the whole layer is consolidating. Stainless wasn’t alone in this market. Its main competitor, Fern, was bought by Postman in January 2026. Anthropic bought Stainless four months later, in May 2026. That leaves Speakeasy as the only major independent player, plus an open-source fallback called OpenAPI Generator that most developers consider too rough for production use without significant manual work. In under five months, two of the three serious companies in this part of the market have been absorbed into larger platforms. The Stainless deal is more visible because of who bought it and why, but the broader pattern matters more: an entire layer of AI infrastructure is being pulled inside platform owners.

3. Moat migration. The gap in raw model capability between Anthropic, OpenAI, and Google has narrowed considerably and continues to close, and the implication is that model quality alone is unlikely to be the principal basis of competitive advantage over the next two years. What may distinguish the leading firms instead is the quality of the developer experience around their models: how easily a business or an engineer can build something useful on top of a given model, how cleanly the tooling integrates with existing systems, and how reliable the connectors are over time.

Stainless was founded by Alex Rattray, formerly of Stripe. Stripe built its market position largely on unusually well-designed developer tools, and Stainless was, in effect, an attempt to apply the same approach to the layer between AI APIs and the rest of the software economy. Anthropic has acquired the team that knows how to do this.

4. Pricing logic, with caveats on denial. Stainless was last valued at $150M in December 2025; at >$300M five months later, this is a roughly 2x strategic markup, not acqui-hire arithmetic. Removing a critical-path external dependency on Anthropic’s own SDKs, while denying it to a tight set of competitors, is rational at that price — but the denial logic is partial. Speakeasy is a viable substitute, and OpenAI was reportedly already migrating off Stainless. The friction tax falls hardest on smaller players who lack the engineering bench to absorb migration cost.

…The press release calls it “extending reach”; the InfoWorld read — “last-mile developer experience” — is closer, but the complement-capture component, even if partial, is real.

-*-

Now, while Claude might be overstating some of the market risks associated with this acquisition (you tell us?), it shows that open source’s impacts are highly conditional on its dependencies and should never be analyzed in isolation from the market’s software stack and architecture. This is equally true for open weight models — being dependent on data, compute, and distribution — as it is for open protocols like MCP, dependent on constant API translations and access. Tracking those interdependencies is what a full ecosystem view involves and is helpful to undertake in order to consider where chokepoints might arise, and in turn where open source strategy might eventually fail or be captured.

Share

At a recent dinner, Richard Danzig, the former Secretary of the Navy, asked the question “Will the markets continue to discipline the state, or will the state discipline markets?”

For me, this was an eye-opening reframing of one of the power dynamics that shapes the world we live in. In the West at least, capital markets still discipline the state. Governments borrow at rates the bond market sets. Politicians who displease investors face capital flight, currency pressure, and the quiet withdrawal of the economic oxygen that keeps their programs alive. In China, though, Xi Jinping has at least partially brought market forces to heel, subordinating the ambitions of tech founders and real estate moguls to the priorities of the party. And the US is aching to follow that playbook.

I immediately started wondering what happens when you add AI to this dynamic. It is a tool that both markets and the state will likely deploy at different rates, which may reshape the current equilibrium. But it is also a potential third force, with its own logic and momentum, and perhaps eventually its own capacity for autonomous action.

Before the rise of markets as we know them, the great disciplining contest in Europe was between the Catholic Church and the state. For centuries, the Church claimed authority over the legitimacy of rulers, the morality of their wars, and the terms on which their subjects lived and died. Kings who defied the Pope risked excommunication, and excommunication was not merely spiritual. It could dissolve the oaths of loyalty that held a kingdom together.

The Church lost that struggle, slowly, across centuries, through the Reformation, the rise of stronger nation-states, and the gradual replacement of religious authority with secular sovereignty. But the interesting question isn’t just that the Church lost. It’s what replaced it as the constraining force on state power. The answer, as Albert O. Hirschman showed in The Passions and the Interests, was commerce.

Hirschman reconstructed the intellectual climate of the seventeenth and eighteenth centuries and showed that before anyone argued capitalism was efficient, political thinkers argued it was calming. The pursuit of material self-interest, they believed, would tame the destructive passions of princes: the lust for glory, the appetite for conquest, and the capricious exercise of power. Montesquieu claimed that commerce makes manners gentle (le doux commerce). James Steuart argued that economic complexity would constrain rulers more effectively than any constitution, because a prince who disrupted the delicate mechanisms of trade would impoverish his own kingdom.

The argument was not that markets were good. It was that they were better than the alternative. Avarice, previously condemned by Augustine as one of the three principal sins, was rehabilitated as the least dangerous of the passions. As Keynes would put it more than a century later (unaware, as Hirschman noted with some irony, that he was repeating arguments already tested against reality): “It is better that a man should tyrannize over his bank balance than over his fellow-citizens.”

This worked, for a while. Markets did discipline states. The bond market still constrains government spending in ways that no opposition party can match, and the threat of capital flight still shapes policy in every country that participates in the global economy. The worldwide economic consequences of the Iran war are a rolling – or should I say roiling? – case study in just how powerful this discipline will be.

But overall, we ended up with something that neither Montesquieu nor Adam Smith anticipated. The self interest of merchants did not tame the passions but became infused with them. They turned the greatest of merchants into princes with all the undisciplined passions of old. And in doing so, they created their own form of tyranny, one where relentless optimization for shareholder value has been decoupled from the operating economy of goods, services, and human welfare. We now have the worst of both worlds: the passions not only of the leaders of the state but of major market participants, unconstrained by the old norms, combined with capital markets that optimize for extraction rather than broadly shared prosperity.

Tocqueville foresaw the populist elements of this bitter stew. He worried that people so focused on the pursuit of their private interests would willingly surrender their political freedom to any ruler who promised to protect those interests. They “think they follow the doctrine of interest, but they have only a crude idea of what it is, and, to watch the better over what they call their business, they neglect the principal part of it, which is to remain their own masters.”

Machines, Bureaucracies, and Markets

I had a brief conversation with Danzig after the dinner. He has been approaching the same territory from a different direction. In his 2022 paper “Machines, Bureaucracies, and Markets as Artificial Intelligences”, he argues that the dominant frame for thinking about AI, comparing machine intelligence to individual human intelligence, is too narrow. Machines, bureaucracies, and markets all belong to the same family of artificial intelligences, all invented to process information at speeds and volumes that surpass individual human capability. I’ve written about this same idea, but without the focus and deep sense of the functioning of the state that Danzig brings to the discussion.

He notes that machines, bureaucracies, and markets all are reductionist. They strip complex reality down to narrow inputs (bits, entries on bureaucratic forms, prices). All three detect patterns without understanding causation. Markets arrive at a price without knowing why. Bureaucracies apply rules without judging their rationality. Deep learning fits functions to data. All three are, in Danzig’s formulation, “remarkably smart and remarkably stupid, remarkably powerful and remarkably vulnerable.”

And all three were defended, at the time of their introduction, as neutral, value-free mechanisms. The history that followed tells a different story. Values turned out to be embedded in the systems all along, and as failures accumulated and those values were revealed and challenged, regulation followed. The 2008 financial crisis, driven by the opacity of mortgage-backed derivatives, is only the most recent example of a pattern that has repeated since the creation of the first stock exchanges.

Danzig’s placement of machine intelligence in the same framework as markets and bureaucracies lets us draw lessons for the future from the history of these earlier systems of large-scale governance. The most important of those lessons may be that the near-term concerns about machine intelligence won’t be about the machines themselves, but about how bureaucracies and markets use machine intelligence, and how society regulates the three together. The three form an ecosystem, not a hierarchy. (This idea echoes Bill Janeway’s notion of the relationship of markets, the state, and financial capital as “a three-player game.” I look forward to finding an occasion to discuss with Richard and Bill the extent to which financial capital is distinct from the broader market of trade in goods and services.)

Danzig notes that controlling intelligent machines will require not the one-time certification we use for industrial equipment, but the continuous supervision we use for personnel: probation, audit, promotion to wider responsibility, removal. The analogy isn’t testing a machine. It’s closer to managing a subordinate who is constantly learning and whose behavior in one situation doesn’t always predict their behavior in the next. And the control mechanisms for markets and bureaucracies co-evolved with those systems over centuries. The speed of machine intelligence may not afford that kind of time.

So Who Will Discipline Whom?

Capital will almost certainly adopt AI faster than the state. This is already visible. Three-quarters of American stock trading is now self-initiated by machines. Algorithmic systems set prices, allocate credit, and route information at speeds no human regulator can match. If capital already disciplines the state, AI-augmented capital will do so even more powerfully.

But that’s not all. In response to a draft of this piece, Bill made some important observations about the decline of state capacity, especially in the US.

There are “two overlapping systems for allocating resources and distributing their returns: markets and political processes. There is an inescapable tension: those who dominate in one sphere will attempt to carry their power over to the other; those who lose in one sphere will invoke or appeal to the other for retribution….

Contrary to what one might have expected or hoped for in 2008, we seem to be witnessing the delegitimization of state authority and the dissolution of state capacity. At the same time and in large part as the continuing afterglow of the Neoliberal Order, the opportunity for concentration of market power has been unconstrained by any countervailing force….

Generative AI is now emerging in an environment where constraining its deployment in the US would require a deep and long-term commitment to rebuilding state capacity, which in turn would require earning back some degree of legitimacy and authority. In the absence of these, conflict with European and Chinese regulatory initiatives and institutions will likely intensify.”

So that adds up to a strong argument that “markets discipline the state,” at least in the US. The contrast to China and Europe, where the state disciplines markets, and by extension, AI, may lead to strikingly different outcomes.

But there’s another possibility, especially over the long term. As AI agents begin to operate autonomously, negotiating with other agents, executing transactions, making decisions that shape markets, they may develop a logic that serves neither capital nor the state in its current form.

There are a number of fascinating experiments in AI as an alternate form of societal governance. The Habermas Machine, a project from Google DeepMind, trained a large language model to mediate democratic deliberation and found that AI mediators produced statements that generated wider agreement and left groups less divided than human mediators did. Projects like Audrey Tang’s work with Polis in Taiwan, as well as the broader movement to reinvent democratic participation using technology, point toward AI as infrastructure for a kind of collective intelligence that neither markets nor bureaucracies can achieve alone.

And there’s something also in Anthropic’s Constitution for Claude, an attempt to root its behavior on a moral foundation, not just in market incentives.

This is where the historical parallel with the Church is intriguing. The Church didn’t just discipline the state through threats. It provided an alternative framework for legitimacy, a different answer to the question of what society is for. Commerce also provided a different answer: prosperity and predictability. AI could provide yet another.

Who designs AI governance?

The difference between whether AI leads us to some promised land or to dystopia is not a question of what AI “wants.” AI has no wants. Neither do markets. Nor do states. It is a question of design: who builds the systems, what objective functions they optimize, what feedback loops they create, and what institutions govern them. An AI future dominated by Anthropic would clearly embed different values than one dominated by x.ai.

The real design problem is not how to supervise individual AI systems. It’s how to build the market structures, institutional governance, and feedback mechanisms that ensure that not just AI but markets and states serve broad human flourishing rather than narrow optimization.

As I’ve argued in We Have Already Let the Genie Out of the Bottle, the failures of corporate governance are a harbinger of our inability to govern even more powerful algorithmic systems. These companies are doing exactly what our financial markets tell them to do. Our attempts to rein them in will fail unless we change the objective function of our economic system.

The early thinkers who championed commerce as a civilizing force were making a bet that the self-interest of markets would be a more reliable constraint on power than the passions of individuals. That bet paid off, but it also created a new form of unfreedom, one that de Tocqueville recognized and that we are living through now. People became so absorbed in their economic interests that they lost the capacity for democratic governance. Adam Smith’s Theory of Moral Sentiments argued that the desire of people to appear good in the eyes of their neighbors would act as a check on self-interest. That was perhaps a reflection of the still powerful effect of the Church on society in those days. But since 1980, when Milton Friedman made the case that the moral obligation of corporations was simply to make money for their shareholders, that is, that “greed is good,” we have lost the last vestiges of the check that the moral order placed on both the passions of princes and the self-interest of the market.

Might the morality that AI derives from efforts like Claude’s Constitution as well as its absorption of millennia of human struggles to come to terms with “the good life” give us a start on a new kind of global order? If so, it will only be after a long struggle between AI, markets, and states.

Hirschman ended his book with a warning about intellectual amnesia: the tendency to trot out the same ideas that had been put forward in an earlier period “without any reference to the encounter they had already had with reality, an encounter that is seldom wholly satisfactory.” The argument that AI will rationalize and improve governance sounds remarkably like the argument that commerce would tame the passions of princes.

We could make the same mistake with AI. We could bet that the rationality of machine intelligence will be a more reliable constraint on both capital and the state than the messy, slow, easily corrupted mechanisms of democracy. That bet might even pay off in the short term. But if we aren’t careful about institutional design, we may find that we’ve traded one form of unfreedom for another.

We’re in for a helluva ride.

Thanks to Richard Danzig for inspiring this piece, and to Bill Janeway and Henry Farrell for helpful comments. All images were generated with ChatGPT 5.5.

There is a lot of hope that AI will advance the progress of science, but unfortunately, the collision between AI and scientific publishing has not gone well.

When an AI coding agent writes code, it operates within a rich ecosystem of version control, pull requests, code review, CI/CD pipelines, dependency management, and package registries. Github wasn’t designed for AI, but it turned out to be foundational infrastructure that makes AI-assisted software development work.

Science has an equivalent set of infrastructure for handling identity, provenance, integrity, and discoverability. Systems like arXiv, DOIs, CrossRef, Datacite, ORCID, OpenAlex, ROR, Retraction Watch, and PubMed form a kind of collaborative exoskeleton for scientific publishing and by extension, for modern scientific knowledge. Much as Github has been adapted for AI development, this infrastructure needs to be adapted for AI use in science.

The problems fall into several categories:

Hallucinated citations. When AI generates or assists with scientific papers, it routinely fabricates references. A multi-model study found that only about a quarter of AI-generated citations were entirely correct. Roughly 40% were erroneous or fabricated. Hallucinated citations have been found in papers accepted at NeurIPS and ICLR, the top AI conferences. GPTZero’s investigation found that about 2% of papers accepted at NeurIPS 2025 contained at least one fabricated reference. The peer reviewers missed them all. AI researchers, who understand hallucinations better than anyone, fell victim because convenience trumped verification.

Retracted paper propagation. AI tools are citing retracted papers without flagging them. Retraction Watch co-founder Ivan Oransky has noted that building a comprehensive retraction database is resource-intensive. Yet AI tools that claim to support scientific research are not even integrating the databases that already exist. A study of 21 chatbots found that on average, they correctly identified fewer than half of retracted papers when asked, and they produced substantial false positives as well. MIT Technology Review reported that AI chatbots are relying on material from retracted papers to answer questions, with some tools returning retracted articles with no retraction notice at all.

Training on compromised literature. AI models trained on scientific corpora inevitably absorb retracted, fraudulent, and paper-mill-generated content. Between 2024 and 2025, the retraction crisis accelerated dramatically. A recent bibliometric analysis found that AI-driven retractions have shifted from sporadic anomalies to a systemic crisis, with generative tools enabling paper mills to penetrate the highest levels of scholarly indexing. AI doesn’t know the difference between a landmark paper and a paper-mill product. Without integration with retraction databases and quality signals, this pollution propagates.

Generation of “AI slop” papers. “Paper mills were already a problem, but AI has made the problem far worse. In a world of “publish or perish,” scholars have strong incentives to generate poor quality papers, cite their own work excessively, and otherwise introduce noise into the system.

As the MIT VRAIX project puts it, because large language models are nondeterministic, “the same prompt can produce different answers, each delivered with fluency and confidence. These systems routinely present statements without verifiable sources, cite fabricated or incorrect references, blur the line between summarization and invention, and favor what’s statistically popular over what’s trustworthy. Even when real citations are included, users often have no easy way to determine whether those references are relevant, reliable, or even supportive of the claim being made.”

Tools to address these problems largely already exist, but they haven’t been integrated into AI systems. New tools are also being developed. As the AI Labs turn their attention to AI for science, they should also be exploring what the future infrastructure of scientific knowledge sharing might look like. That is the subject of this article.

The infrastructure of collaboration

DOIs and CrossRef. Every legitimate scholarly work has (or should have) a DOI, a persistent digital identifier maintained by CrossRef. CrossRef’s REST API lets you resolve a DOI and verify that a paper actually exists, with the correct title, authors, journal, and year. This is the most basic hallucination check imaginable, and yet most AI systems don’t perform it. Why isn’t this kind of validation built into every AI system that touches scientific literature? DOIs are not a panacea. They have been hacked both for fun and profit. As Geoffrey Bilder, the former director of technology for Crossref noted, there are DOIs that point to a South Park movie, a fake article on “a Google based alien detector,” and more. Alone, they guarantee nothing. They are just an identifier. But as part of an infrastructure that validates them, they are profoundly useful.

ORCID. ORCID provides a persistent identifier for researchers, linking them to their publications, affiliations, funding, and peer review activity. It’s an OAuth 2.0 API. You can authenticate a researcher’s identity and pull their verified publication list in seconds. If an AI-generated paper claims Dr. Smith at MIT published a paper on quantum computing in Nature, you can check ORCID to see whether Dr. Smith exists, whether they’re affiliated with MIT, and whether that paper is in their record. This is researcher identity verification, and it’s available as a free API as well as through periodic open data snapshots. As The Scholarly Kitchen noted, ORCID works best in combination with other persistent identifiers. Portugal’s integration of ORCID with its national research identifier CIÊNCIA ID has connected 112,000 researcher profiles and saves more than 154 hours per researcher annually in data entry. That’s the kind of compounding return you get from well-designed infrastructure.

OpenAlex. The successor to Microsoft Academic Graph, OpenAlex is now a fully open scholarly knowledge graph with over 271 million indexed works, serving over 1.5 billion monthly API calls. It knits together data from CrossRef, PubMed, ORCID, institutional repositories, and DataCite. Its API is free and returns rich metadata including citation networks, author affiliations, and open access status. OpenAlex recently received a $3.5 million Wellcome grant to integrate global research funding metadata, making it possible to trace the chain from funder to grant to publication to impact. The Walden rewrite, launched in late 2025, added 190 million new works including datasets and software from DataCite and thousands of institutional repositories.

Retraction Watch and the Retraction Watch Database. Retraction Watch is the closest thing we have to a comprehensive record of scientific papers that have been withdrawn due to fraud, error, or ethical violations. It’s a project of The Center for Scientific Integrity. Numerous companies and nonprofits including Zotero and Web of Science have integrated the Retraction Watch database, automatically excluding retracted publications from their research assistants. Some AI-specific tools like Consensus have also started incorporating retraction data from a combination of sources including Retraction Watch, but this should be table stakes for any AI system that claims to work with scientific literature.

arXiv. The preprint server for physics, mathematics, computer science, and related fields has been operating since 1991. It provides a structured, persistent, openly accessible record of scientific work. arXiv IDs are resolvable. The metadata is machine-readable. For AI systems working in these domains, arXiv is an authoritative source that can be queried to verify claims.

Consider the parallels to software development. GitHub gives software a persistent identity for every commit. DOIs give scholarly works a persistent identity. GitHub tracks who contributed what. ORCID does the same for researchers, disambiguating people with common names and linking them to their full body of work across institutions and careers. GitHub has dependency graphs. CrossRef, Datacite, and OpenAlex maintain citation graphs, linking 271 million scholarly works to their authors, institutions, and funders. GitHub has issue trackers and code review. The scientific community has peer review, post-publication commentary on PubPeer, and Retraction Watch tracking papers that have been withdrawn. Github and Gitlab even support software citation through .cff files, which includes the ability to assign DOIs, so the two systems have meaningful overlap.

MIT’s VRAIX project is working to bring this infrastructure together and adapt it for AI. It attempts to address the problems described at the opening of this article not by looking inside scientific papers for the common tells of AI generation, but by situating papers and LLM-generated content within what we might call “the web of knowledge.” As its creators put it, “VRAIX’s core question is: ‘What system of knowledge does this claim belong to, and does it behave in a way consistent with that system?’” It looks at the citation graph, resolves citations to standard identifiers (DOIs, PMIDs, ORCIDs, ROR IDs), resolves them to real metadata, including the network of co-authors and institutions, the history of corrections and retractions, historical publication patterns and the relevance of cited sources to the claims they are said to support.

Geoffrey Bilder, formerly the director of research for Crossref, pointed out to me that this infrastructure has been adopted by fierce rivals in the publishing industry. The key is the structure and governance of these infrastructure organizations. They are open standards, and follow a set of principles (POSI) that help ensure that they cannot be captured or enshittified. This can serve as a reassurance not only to scientific publishers but also to AI companies that they are not building dependencies on things that might be bought or captured by their rivals.

The mechanism design problem

To borrow Danny Ryan’s definition of protocols from the Ethereum Foundation’s Summer of Protocols project, scientific publishing represents “strata of codified behavior” that enable coordination across the entire research enterprise. They are the “civilizational infrastructure” of science. And like all good infrastructure, they’ve become invisible. Researchers don’t think about DOIs the way drivers don’t think about lane markings. But remove them, and the system falls apart.

They are a public good. And AI companies are mostly ignoring them, or worse, undermining them.

Right now, the relationship between AI and scientific infrastructure is almost entirely extractive. AI companies train on scientific papers. They build products that generate and manipulate scientific text. They compete for the “AI for science” market. But they contribute almost nothing back to the infrastructure that makes scientific knowledge reliable in the first place.

This is entirely consistent with the broader argument I’ve been making about the agentic economy as envisioned by the AI labs. Value flows in one direction. AI companies consume scientific content, but they don’t contribute anything back.

Think about the YouTube Content ID analogy I described in “The Missing Mechanisms of the Agentic Economy.” The music industry’s first response to unauthorized use of their music was “Take it down.” YouTube’s answer was “How about we help you monetize it instead?” That aligned incentives and created a vibrant creator economy.

The same thinking should apply here. The question isn’t just “How can AI companies use scientific infrastructure to make their products better?” (though they should). The question should also be “How can AI companies help these services become more valuable, more sustainable, and more comprehensive?”

What building on, not just extracting from, would look like

Here are some concrete possibilities.

Validation as a first-class feature. Every AI system that generates or edits scientific text should validate references against CrossRef, OpenAlex, and Retraction Watch as part of its core pipeline, not as an afterthought. This should be as automatic as a compiler checking syntax. The APIs exist. The latency is minimal.

ORCID integration for attribution. When AI systems summarize or synthesize scientific literature, they should link to ORCID profiles, not just paper titles. This creates a direct connection between AI-generated output and the human researchers whose work it draws on. It also makes it easy to verify that a cited researcher actually wrote what the AI claims they wrote.

Contributing to metadata quality. AI is very good at extracting structured information from unstructured text. OpenAlex reports that over 60% of its records lack complete institutional affiliation data. Over 40% lack abstracts. AI tools that process scientific papers could contribute extracted metadata back to OpenAlex, improving the graph for everyone. This is the kind of “architecture of participation“ that made open source work. The system gets better the more people use it.

Retraction monitoring as an MCP service. Imagine a Retraction Watch MCP server that any AI agent could query in real time. Before citing a paper, the agent checks whether it’s been retracted, whether it has expressions of concern, whether its citations have been flagged. This is the kind of service that would benefit the entire ecosystem, and it could be funded in a way that sustains Retraction Watch’s work. The MCP registry protocol and MCP Server Cards I discussed in “The Missing Mechanisms” could provide the discovery and authentication layers. It’s also worth integrating PubPeer, the post-publication review and comment system, and the Problematic Paper Screener.

Funding the infrastructure. OpenAlex operates on a shoestring, with institutional memberships at $5,000/year. Retraction Watch is a nonprofit that depends on donations. ORCID is sustained by member organizations. These are the foundations on which the credibility of AI-generated science depends, and they are chronically underfunded. AI companies generating billions in revenue from products that depend on scientific credibility should be contributing to the infrastructure that provides it. This is not philanthropy. It’s enlightened self-interest.

Provenance chains for AI-generated scientific content. When AI contributes to a scientific paper, that contribution should be traceable, not just disclosed in a boilerplate statement, but linked to specific claims, specific sources, and specific verification steps. The persistent identifier infrastructure (DOIs, ORCID, OpenAlex IDs) already provides the building blocks for this. What’s missing is the protocol that ties them together.

From extraction to engineered argument

In “The Missing Mechanisms,” I argued that the best market-shaping protocols are “engineered arguments, not engineered agreements.” They don’t impose a single solution from above. They create a framework within which competing approaches can contend.

The same principle applies here. AI companies don’t need to adopt a single standard for scientific verification. I’m arguing that they should build on the existing infrastructure in ways that let the market discover what works. Some will integrate CrossRef validation. Others will build on OpenAlex’s knowledge graph. Some will develop novel quality signals we haven’t imagined yet. The point is to participate in the ecosystem rather than treating it as a resource to be mined.

The scientific infrastructure community has spent decades building what David Lang, in his essay “Standards Make the World” for the Summer of Protocols project, called a “third pillar of modern society,” alongside private organizations and public institutions. These are standards and systems that enable coordination without central control.

AI companies that build on this infrastructure will make better products. They’ll produce more reliable scientific output. They’ll face fewer hallucination crises and retraction embarrassments. But more than that, they’ll be investing in the civilizational infrastructure that makes reliable knowledge possible in the first place. They will be taking their place alongside many other commercial entities like Digital Science, Elsevier, and Clarivate that already build on this infrastructure, as do many non-commercial tools that researchers depend on every day, like Zotero.

GitHub didn’t just give developers a place to store code. It became a collaborative exoskeleton that made an entire style of distributed, cooperative development possible. The scientific infrastructure stack has the potential to do the same for AI-assisted science. But only if AI companies stop treating it as someone else’s problem and start treating it as a foundation to build on, and a foundation on which their own success in science depends.

What’s missing is the will to build on them, and the mechanism design thinking to ensure that everyone, not just the AI companies, benefits from the result.

Thanks to Geoffrey Bilder, Ivan Oransky, and Ilan Strauss for comments on drafts of this article. Geoffrey and Ivan know far more about this topic than I do, and this article draws on their work. I get credit (or rather, demerit) for any errors that remain. Images created with GPT 5.5 medium.

Foreword: Thinking about AI market design

The piece below by Jesus Rodriguez makes a compelling case that agents are emerging as a new class of economic actor: a new delegated consumer (or shopper) with native needs around identity, payments, memory, and commerce. We share his view that the internet was built for humans and now must be partly rebuilt for machines.

MECHANISMS. We want to add one note of caution about how that rebuilding happens. What Rodriguez describes is, in our language, the “missing mechanisms” problem applied to agents. Tim has argued that markets need mechanisms for distributed value creation and participation. Mechanisms are engineered rules that incentivize self-interested actors to produce honest disclosure and can aim for a market that benefits all actors, rather than a single firm. A mechanism’s rules decide outcomes and payments for things like auctions, voting (no payment), public goods spaces, and more. They can make markets work where otherwise they would not due to agents behaving strategically by hiding information and not wanting to participate (“why license data when you can scrape it”, for example).

The agent economy is now confronting its own version of this missing markets question (as Google’s Search and Amazon’s marketplace was once did too): but it’s missing mechanisms for agents, not for humans. Delegation, attestation, payment scopes, machine-readable provenance, agent identity, memory boundaries should be internet and AI standards first, that underpin agentic markets, and then innovative competitive product features second. The two can and should exist as compatible conceptualizations of how to build markets online as open publicly beneficial spaces.

The layers for agentic AI markets Rodriguez describes below will be built. The question remains how to realize them as open, modular infrastructure, rather than as proprietary chokepoints?

The defaults are still soft and the market structures are still fluid. They will not stay that way for long.

— Tim O’Reilly & Ilan Strauss

Introduction: Agentic markets

For the first few decades of the internet, software had a reasonably stable assumption baked into it: the user was a human.

A human had eyes, a browser, a mouse, a password manager, a credit card, an email address, a tolerance for modal dialogs, and a finite amount of patience. The entire SaaS and consumer internet stack grew around this shape of user. Search engines ranked pages for humans. E-commerce sites optimized funnels for humans. CRMs tracked human sales reps selling to human buyers. Identity systems authenticated humans. Analytics systems measured human clicks, human sessions, human conversions.

Then we started building AI agents

At first, agents looked like toys. A chatbot that could book a calendar invite. A coding assistant that could open a pull request. A browser agent that could navigate a website on your behalf, slowly and somewhat comically. But underneath the awkwardness was something important: software was no longer being used only by people. Software was being used by software that could reason, plan, transact, remember, delegate, and retry.

This is a much bigger shift than “chatbots are the new UI.” It is closer to the emergence of a new economic actor. Not a legal person, not a human replacement, and not magic. But a new kind of consumer.

Agents consume information. Agents consume APIs. Agents consume storage, compute, tools, workflows, credentials, payments, and services. Agents evaluate options, make requests, negotiate constraints, and execute tasks. They are not just another interface to existing software. They are a new demand surface.

Once you see agents as consumers, a lot of the software landscape starts to look strangely misaligned.

The Salesforce for agents may not be Salesforce. The Google for agents may not be Google. The Amazon for agents may not be Amazon. Not because these companies are weak, but because the primitives change when the consumer changes.

A human using Salesforce wants dashboards, contacts, pipeline stages, notes, reminders, and reports. An agent using a CRM wants something else: structured state, task objectives, relationship graphs, permissioned memory, machine-readable sales playbooks, and reliable APIs for updating intent. It does not need a beautiful dashboard. It needs a canonical substrate of customer truth that it can query, mutate, and reason over safely.

A human using Google wants ranked blue links, snippets, maps, videos, and maybe an answer box. An agent using search wants provenance, freshness, confidence intervals, citations, entity resolution, schema alignment, and retrieval interfaces that are stable under automation. It does not want ten links. It wants to know which source is authoritative for a given claim, what changed since the last run, and whether the answer is safe to act on.

A human shopping online wants product photos, reviews, discounts, shipping estimates, and a checkout page. An agent shopping on behalf of a human wants product metadata, compatibility constraints, return policies, vendor reliability scores, payment authorization scopes, fraud guarantees, and receipts that can be parsed by another machine. It does not browse. It solves an optimization problem under constraints.

This is why “just give agents browser access” feels both exciting and obviously temporary. Browser access is backwards compatibility. It is useful in the same way screen scraping was useful before APIs. But if agents become a major class of software user, we should expect agent-native infrastructure to emerge.

The web was designed around human attention. The agent economy will be designed around machine action.

Consider identity

Today, identity on the internet is mostly a ceremony for proving that a human is allowed to do something. We use passwords, passkeys, OAuth, SSO, device fingerprints, CAPTCHAs, and risk engines. These systems assume a human principal somewhere in the loop.

Agents break this model in subtle ways. An agent is not exactly the user. It is also not exactly an employee, a bot, a service account, or an OAuth app. It may act for a person, but only within a certain scope. It may need to buy groceries but not alcohol, schedule meetings but not cancel medical appointments, email customers but not change pricing, deploy code but not rotate production secrets.

So agent identity needs delegation as a first-class primitive. Not “here is my password, go do stuff.” Not “here is a broad API token with terrifying permissions.” Instead: here is an agent, acting on behalf of this principal, for this objective, during this time window, with this budget, using these tools, under these audit requirements.

This implies infrastructure: agent passports, delegated credentials, capability-based access control, revocation systems, policy engines, and audit logs that explain not just what happened, but why the agent believed it was allowed to act.

The “why” matters. A normal API log might say: POST /refunds succeeded. An agent-native audit log should say: the agent issued a refund because the customer met policy condition X, the order was delayed by Y days, the user had previously authorized refunds below Z dollars, and no higher-risk flags were present. This is not just observability. It is accountability.

Now consider payments

Human payments are optimized around cards, wallets, bank accounts, checkout pages, fraud prevention, and dispute resolution. The system is designed for a human to consciously approve a transaction, even if that approval has become increasingly compressed into one click or one biometric scan.

Agents introduce a different payment pattern: constrained autonomy.

A travel agent might have permission to spend up to $1,500 on flights, but only if the arrival is before 6 p.m., the layover is under two hours, the airline is not on a blocked list, and the fare is refundable. A procurement agent might be allowed to buy cloud credits, office equipment, or data enrichment services within a monthly departmental budget. A coding agent might pay for API calls, benchmark runs, or temporary compute to complete a task.

This requires payment rails that are more programmable than consumer checkout and less heavyweight than enterprise procurement. We will need spending envelopes, machine-readable invoices, pre-authorized budgets, reversible transactions, agent reputation, merchant attestations, and real-time policy checks.

Share

A credit card number is a poor interface for an autonomous actor. It is a bearer token with too much authority and too little context. Agent payments will likely look more like signed intents: “Agent A, acting for User B, is authorized to pay Merchant C up to Amount D for Purpose E, subject to Policy F.” The merchant should be able to verify this. The user should be able to revoke it. The system should be able to dispute it. The agent should be able to reason about it before acting.

E-commerce changes too

Most online stores are persuasion machines. They are designed to convert human uncertainty into purchase behavior. Recommendation modules, scarcity banners, coupon wheels, bundles, sponsored placements, and dark patterns all exist because humans are impressionable, visual, and busy.

Agents are busy too, but in a different way. They are not emotionally moved by a hero image of a backpack on a mountain. They may be vulnerable to other things: poisoned reviews, manipulated metadata, adversarial product descriptions, fake compatibility claims, hidden fees, or vendors that optimize for agent-ranking algorithms.

So agent commerce will need a different trust layer. Product pages become less important than product specs. Reviews become less useful unless they are structured, verified, and resistant to manipulation. Return policies need to be machine-readable. Availability needs to be queryable. Bundles need explicit semantics. “Best laptop for me” becomes a negotiation among constraints: workload, budget, portability, battery, repairability, delivery time, warranty, vendor trust, and user preference memory.

The equivalent of SEO for agents will be fascinating and horrible. Today, companies optimize pages to rank in human search. Tomorrow, they will optimize structured representations to be selected by buyer agents. This will produce new spam, new fraud, new ranking games, and new defensive infrastructure. We should assume every agent-readable marketplace becomes adversarial once enough money flows through it.

Data storage also changes

Human software treats data as something to display, search, and report. Agent software treats data as working memory. Agents need durable context across tasks: user preferences, prior decisions, organizational policies, project state, personal constraints, and learned patterns. But they also need memory boundaries. Remembering everything is not intelligence. It is liability.

An agent-native storage layer needs multiple kinds of memory. There is episodic memory: what happened during the last task. Semantic memory: stable facts about the user or organization. Procedural memory: how to perform recurring workflows. Policy memory: what is allowed. Source memory: where a belief came from. And scratch memory: temporary reasoning artifacts that should probably expire.

This storage layer must support retrieval, provenance, deletion, summarization, permissioning, and conflict resolution. It must know that “the user prefers morning flights” is not the same kind of fact as “the user authorized a $700 purchase yesterday.” It must distinguish stale preferences from hard constraints. It must expose memory to agents in ways that reduce hallucination rather than amplify it.

In the human internet, data was often collected because it might become useful later. In the agent internet, data becomes directly operational. If an agent remembers incorrectly, it acts incorrectly. Memory bugs become action bugs.

This brings us to software design itself

A surprising amount of current software is hostile to agents because it was built for humans. Buttons without semantic labels. PDFs instead of structured documents. Emails that require interpretation. Dashboards with charts but no underlying query API. Terms of service written only for lawyers. Error messages that explain nothing. Workflows that require a human to click through five screens because the product team wanted engagement.

Agents expose the hidden tax of unstructured software.

Agent-friendly software will have different virtues. It will be explicit, inspectable, composable, and transactional. It will provide APIs not as an afterthought, but as the main surface. It will describe capabilities, costs, side effects, and failure modes. It will support dry runs. It will make state changes reversible where possible. It will provide good error messages because the error message is not for a frustrated human anymore; it is for another system deciding what to do next.

The best agent software may look boring to humans. A great agent-native CRM may be mostly a graph, a policy engine, an event log, and a set of tool contracts. A great agent-native search engine may look like a retrieval and verification API. A great agent-native bank may look like programmable authorization plus risk scoring. A great agent-native e-commerce platform may look like a giant constraint solver with trust primitives.

This does not mean human interfaces disappear. Humans still set goals, approve high-risk actions, inspect outcomes, and change their minds. But the center of gravity moves. The human becomes more like a principal, supervisor, or exception handler. The agent becomes the high-frequency user.

This distinction matters because incumbents often win interface shifts but lose primitive shifts.

When mobile arrived, many desktop companies successfully made mobile apps. But the biggest mobile-native companies were not just desktop products on smaller screens. They were built around new primitives: GPS, camera, push notifications, contacts, sensors, app stores, and always-on connectivity.

Agents are similar. Adding a chatbot to an old product is not the same as rebuilding the product around agentic use. The winners may be companies that understand the new primitives early: delegation, memory, tool use, verification, policy, agent identity, machine-readable commerce, and autonomous payments.

The phrase “agents are a new consumer” sounds metaphorical, but it is actually quite literal. Consumers create demand. Demand reshapes markets. Markets reward infrastructure that serves the consumer’s native behavior.

Humans click. Agents call tools.

Humans browse. Agents query.

Humans forget. Agents persist memory.

Humans approve. Agents need delegated authority.

Humans compare visually. Agents optimize over structured constraints.

Humans tolerate ambiguity. Agents need executable semantics.

Humans can be persuaded. Agents can be attacked.

Once agents become common, every software company will face a basic question: is your product merely usable by agents, or is it built for them?

The difference will matter. A website that an agent can painfully navigate is not agent-native. An API that exposes a few endpoints is not agent-native. A chatbot bolted onto a dashboard is not agent-native. Agent-native software treats autonomous actors as first-class users with identity, permissions, memory, budgets, goals, and accountability.

We are early enough that much of this still looks speculative. The agents are unreliable. The standards are immature. The demos break. The economics are unclear. But this is often what new consumers look like at the beginning. Mobile apps were once toys. Cloud workloads were once experiments. Online payments were once scary. Developer APIs were once niche.

Then, slowly, the toy becomes a workflow. The workflow becomes a habit. The habit becomes infrastructure. And the infrastructure becomes invisible.

The internet was built for humans, then patched for machines. The next internet may be built for machines, with humans in command.

That is the opportunity: not just to build agents, but to build the world they will consume.

Visit AI Disclosures Project for more information on our work.

For the past two years, I’ve been working with economist Ilan Strauss at the AI Disclosures Project (joined soon after by computer scientist Sruly Rosenblat). We started out by asking what regulators would need to know to ensure the safety of AI products that touch hundreds of millions of people. We are now exploring the missing mechanisms that are needed to enable the agentic economy.

This essay traces our path from disclosures through protocols to markets and mechanism design (read more on our new website: here). Rather than simply stating our conclusions, I’m sharing our thought process and some of the conversations and historical examples that have shaped it.

We will be holding a number of focused convenings to explore these ideas over the next couple of months, and my hope is that shared context will enable more productive engagement with what is very much a work in progress.

The disclosure problem

Ilan Strauss and I started the AI Disclosures Project in early 2024 with a conviction that most regulators had little idea how AI worked or where it was going. The field was so young that many of the early regulatory proposals were misguided. We thought that regulators and industry should start by agreeing on standards for disclosure, so that we could all learn together as the technology develops. You can’t regulate what you don’t understand.

One of our first insights was that focusing solely on model safety was a mistake, much as if regulators inspected automobiles at the factory but completely ignored their use on the roads. We believed (and still do) that the focus should be on AI as deployed. And we believe that disclosures shouldn’t focus just on capabilities but on business models and the operating metrics that AI companies use to shape how their products operate.

Ilan and I had worked together previously with Mariana Mazzucato at University College London on what we called “algorithmic attention rents,” studying how platforms like Amazon and Google control user attention to extract economic rents from their suppliers. We observed that organic search at Google and Amazon was a huge advance in market coordination, using hundreds of signals to find the best match for a user’s intent. In effect, both companies had built a better “invisible hand.” And yet after decades of success, they turned away from that advance. To use Cory Doctorow’s coinage, they began “enshittifying” their services by substituting inferior paid results for the top organic search results in order to pad their bottom line.

We’d also watched social media start out with the promise of keeping you in touch with your friends and foster productive conversations, but then instead began to optimize for engagement at the expense of everything else. By the time anyone understood what was happening, the damage had been done. We can see the inflection point in their financial metrics, but neither regulators nor the public can see the changes in operating metrics that drove the financials. What if we could capture what good looks like before it gets enshittified, and identify how that changes over time?

We also observed that modern technology companies are completely different from industrial era corporations, where you can understand key elements of the business by tracing the inputs and the outputs through the financial statements. Instead, the business is largely driven by intangibles, which are lumped into one impenetrable black box.

We wanted to learn from that mistake. While the horse was already out of the barn on search and social media, we hoped to get disclosure of operating metrics into AI governance while there was still an appetite for regulation. Unfortunately, that window was very short. The failure turned out to be productive, though, because it forced us to think harder about regulation more broadly and what other leverage points might be found.

Protocols as functional disclosures

The first turn in our thinking came when we realized that disclosures aren’t just informational. The most important disclosures are functional. We came to see the parallels between disclosures and communications protocols, the agreed-on methods by which networked systems share information. For example, the HTTP protocol that underlies the World Wide Web specifies how a web browser and web server communicate in order to display a web page.

This is a structured communication with rules that must be followed and data that must be exchanged in a particular order. An HTTP request that identifies the user agent as a command line program such as curl rather than a graphical browser such as Chrome triggers a different response from the server. The user-agent string isn’t a report filed with a regulator. It’s an operational signal embedded in the protocol, and it carries a lot of information.

Once you see protocols as a system of functional disclosures, you start noticing that every regulatory system has a kind of communications and control protocol at its heart. Generally Accepted Accounting Principles (GAAP) or IFRS, the European equivalent, are protocols for communication between companies and their accountants, auditors, banks, investors, and tax authorities. Even road markings and road signs are a communications protocol, giving information to drivers about local conditions, laws, and the proper use of the road. These are slow, analog protocols, but they are protocols nonetheless.

Protocols can be inspected. Observability is the key to governance. Police observe speeders on the road; credit card processors and banks watch for credit card fraud on their payment networks; email processors filter spam as it passes through nodes on the network. The observability points for AI are still emerging, but that’s where regulators should be focused.

Even beyond being a locus for observability and regulability, protocols themselves do an enormous amount of the governing work in modern technology systems. Spanning everything from how packets get from one place to another to what gets displayed, who has permission to see it, and sometimes even what it costs, they ultimately determine who can interoperate with whom. That led us to an even bigger realization.

Protocols shape markets

Think about the early shape of the AI chatbot market. It was a winner-takes-all race to be the dominant platform for AI in the way Windows became the platform for PCs, or iOS and Android for phones. Whoever wins controls the market. Then Anthropic introduced MCP, the Model Context Protocol. All of a sudden, the landscape looked more like a web. There could be many winners. It didn’t matter what model you were running or whose APIs you were calling as long as you followed the protocol. And as the agentic AI market unfolded, the protocol wasn’t just MCP. An AI agent could be a user of the existing internet protocol stacks. Whether MCP itself survives or is superseded by other protocols, the shape of the market was transformed.

This insight reframed our whole project. Protocols are not just technical infrastructure. They are market-shaping mechanisms.

Workflows are also protocols

I talked last week with some of the folks working on the Long Now Foundation’s partnership with Ethereum’s Summer of Protocols project (soon to be: Protocolized magazine), and that widened my lens even further.

When software people hear “protocol,” we think of communication protocols: TCP/IP, HTTP, MCP, or, say, Stripe’s Machine Payment Protocol (MPP).

To this particular protocol research community, a protocol is any standardized way of doing something. Wildfire management teams follow protocols. So do flood response teams, hospital emergency rooms, and air traffic controllers. Atul Gawande’s book The Checklist Manifesto was an attempt to establish a common protocol for surgical operating theaters. This is a very different definition of protocol, and yet putting the two meanings of the word into the same frame makes a new kind of sense.

In his introduction to the Summer of Protocols’ Protocol Reader, Venkatesh Rao cited Ethereum researcher Danny Ryan’s definition of a protocol as a “stratum of codified behavior” enabling coordination. He pointed out that protocols tend to become invisible once adopted. Rao calls this a “Whitehead advance,” after the philosopher Alfred North Whitehead’s observation that civilization advances by extending what we can do without thinking.

But he also made the thought-provoking point that a protocol is an “engineered argument,” in contrast with an API, which he says is an “engineered agreement” enforced by one dominant actor. There’s more to it than just the power asymmetry of enforced agreement, though. In a followup conversation, Venkatesh Rao noted that protocols are “not just codified modes of information exchange, but modes of live, structured, argumentation, often with an active computational element. For example, CSMA/CD (Ethernet) must detect packet collisions and compute and execute a random delay for retransmittal of packets. This is not mere structured communication. This is argumentation with what philosophers call dynamic semantics.”

Rao continued: “The moment you go beyond computing protocols, real-world feedback loops from material consequences become really important. For example, container-shipping is quite close architecturally to TCP/IP (the big difference being that packets can be dropped and retransmitted while lost containers are actually lost), but because it has a materially embodied feedback loop, regulatory mechanisms start to behave more like control systems than communication systems.”

I love the idea of protocols as an engineered argument. The dynamism this suggests is going to be ever more true in a future of agentic protocols. But this notion also triggered another thought, which is that markets are also engineered arguments. My bridge to this reformulation was the difference between de jure protocols that arise from a formal standards process, and de facto protocols that arise through market contention.

In the early days of the internet, the Internet Engineering Task Force (IETF) was all about engineered arguments. People had ideas about how the internet ought to work, and to prove their point they had to show up with interoperable implementations. No one had the ability to enforce anything. Agreement had to evolve. As Dave Clark famously put it, “We reject: kings, presidents, and voting. We believe in: rough consensus and running code.” The de facto protocols of the internet that emerged from the IETF ended up significantly outperforming the competing de jure networking protocols that emerged from telecommunications standards bodies. The IETF framed the argument; whoever showed up made their case and won or lost by way of adoption.

It also made me remember another decades old story that I had lived through. Microsoft and Netscape were duking it out in the web server market and were building their own “engineered agreements” for what was up the stack from the base web server functionality. Everyone thought that Apache wasn’t keeping up, but they had a trump card. They provided an extension layer. And that engineered all kinds of productive arguments between a market of competing developers rather than a single engineered agreement imposed by either a dominant player OR a dominant committee.

Rao also noted that protocols spread slowly but become nearly impossible to dislodge once established. For example, SMTP (the protocol for email) dates back to 1982, and has outlasted many competitors. There is a lot of path dependence. And so getting the first steps right is an important part of engineering the argument.

And in his essay “Standards Make the World” for the Summer of Protocols project, David Lang makes the point that technical standards form a third pillar of modern society, alongside private organizations and public institutions. They aren’t the state and they aren’t the market, but they’re essential to both. When they work well, standards become enabling technologies. The internet. The shipping container. Standard time. They are civilizational infrastructure.

In short, we are not just building communication protocols for software agents. We are developing a new way to standardize the best practices and workflows that will shape the human + AI future, allowing humans and agents to cooperate across organizations, industries, and borders.

Share

Skills can also be seen as protocols

Once the Summer of Protocols team (Venkatesh Rao and Timber Schroff) planted in my mind the connection between workflows and protocols, it occurred to me that Agent Skills are also a “stratum of codified behavior,” and perhaps even a set of competing “engineered arguments” for how to do work with AI.

At the simplest level, a Skill is a piece of structured knowledge: here’s how to create a Word document; here’s how to extract the text from a PDF; here’s how to publish on the Hugging Face Hub. There can be many Skills that attempt to codify the same knowledge, but some may be better than others. As Skills multiply, how will we find the best ones? This is in many ways analogous to the organic web search problem, which Google solved by aggregating hundreds of useful signals.

And we’re seeing that there is a kind of hierarchy of skills. Jesse Vincent’s Superpowers framework, which has become one of the most widely adopted open source projects in AI-assisted development, doesn’t just give agents individual capabilities. It encodes an entire software development methodology: brainstorm before you build, plan before you code, test before you ship, review before you merge. That’s a standardized workflow. It’s a lot like the kinds of protocol that the Summer of Protocols folks were talking about with us and the Long Now Foundation, expressed in a form that agents can follow.

The existing protocols that the protocol research community talks about, like wildfire management protocols or hospital triage protocols, encode best practices into a repeatable, teachable process for human teams. They have yet to be adapted for agents. And in fact, many of them are never going to be entirely agentic. We will need to build mechanisms for workflows that include both AI agents and humans working together.

Agent skills in some (but not all) areas raise the same questions that industrial standards have always raised: who decides what the best practice is? How do you verify quality? How do you govern updates? We may be talking about skills that encode the workflow for regulatory compliance in a specific industry, or for conducting an environmental impact assessment, or for managing a clinical trial. Are the standards de jure or de facto, the result of an engineered agreement by a committee or an engineered argument that enables a vibrant market?

At O’Reilly, this is something we think about a lot. We’re a company built on codifying expert knowledge. We’ve published books and organized conferences and online training that taught people how to do new things. Now we’re asking “What does it look like to publish the skills that teach agents how to do things? And how do we make sure those skills are discoverable, trustworthy, and monetizable, not just for us but for every domain expert who has knowledge worth encoding?” And how do they emerge from contention in a vibrant market rather than by decree?

We believe we’ll all be better off with an engineered argument than an engineered agreement. And that brings me to mechanism design.

The missing mechanisms

Economists use the term “mechanism design” to describe the engineering of rules and incentive structures that lead self-interested actors to produce outcomes that are good for everyone. It’s sometimes called “reverse game theory.” Rather than analyzing the equilibria that emerge from a given set of rules, you start with the outcome you want and work backward to design the rules that will get you there.

Mechanism design theory got its start in the 1960s when Leonid Hurwicz took up the problem of how a planner can make good decisions when the information needed to make them is scattered among many different people, each of whom has their own interests. His key insight was that people won’t reliably reveal what they know unless it’s in their interest to do so. So how do you design a system that aligns their incentives?

The field that Hurwicz founded and that Eric Maskin and Roger Myerson developed through the 1970s and 80s earned all three the Nobel Prize in Economics in 2007.

I first encountered the field when Jonathan Hall, at the time the Chief Economist at Uber, waved Al Roth’s book Who Gets What — and Why at me and said “This is my Bible.” In it, Roth describes his own work on mechanism design, which won him the 2012 Nobel Prize in Economics along with Lloyd Shapley. Roth applied mechanism design to kidney matching markets, markets for college admissions, for law clerks and judges, and for hospitals and medical residents. When I first talked to Jonathan and then Al Roth, my layman’s takeaway about mechanism design was that it was simply the application of economic theory to design better markets.

And I’ve since come to think even more broadly about what mechanism design might mean in a technology context. In my broader framing, packet switching was a breakthrough in mechanism design. So for that matter was TCP/IP, the World Wide Web, and the protocol-centric architecture of Unix/Linux, which enabled open source and the distributed, cooperative software development environment we take for granted today. PageRank and the rest of Google’s organic search system also seems to me to be a kind of mechanism design. So do Pay Per Click advertising and the Google ad auction. All of them are ways of aligning incentives such that self-interested actors produce outcomes that are good for others as well.

So that brings me back to AI. Right now, there’s a problem that makes the AI/human knowledge market less efficient than it could be. The disrespect for IP that has been shown by the AI labs and applications during the training stage, and even now during inference, has led to efforts by content owners to protect their content from AI. Do not crawl. Lawsuits. Reluctance to share information. Even the AI labs are complaining about the theft of their IP and trying to protect their model weights from distillation.

It’s an economy crying out for mechanism design.

The lesson of YouTube Content ID is worth learning. Twenty-odd years ago, the music industry was in the same position that content creators are in today with AI. In response to unauthorized use of their music by creators, music publishers’ demand to YouTube was “Take it down.” But as Google engineer Doug Eck explained to me, YouTube came up with a better answer: “How about we help you monetize it instead?” I don’t know the details of how that decision was made but I do know the eventual outcome. Aligned incentives led to a vibrant creator economy in which YouTube’s video creators, the music companies, and Google all got to share in the value that was created.

That should give us inspiration for how to solve some of the problems we face now with AI. Whether it’s with Agent Skills, NotebookLM, or other emergent artifacts of the new AI/human knowledge economy, we need to align the incentives. If we can grow the pie, and in a way where no single gatekeeper captures the bulk of the benefit, there’s a way to create a vibrant market. But that requires building mechanisms that don’t exist yet.

What mechanisms are missing from the agentic economy? Here’s a partial list:

Skills markets. There’s an enormous economic opportunity for humans to create and trade skills that agents can use. These are not just simple aggregation of context with tool use instructions, but higher-level, industry-specific workflows that encode deep human expertise. At O’Reilly, we’re figuring out how to turn our knowledge and that of our authors into skills, how to make them discoverable, and how to sell them. But as of yet, there’s no way for a broader community of skill creators to participate.

Quality and governance for skills. Some skills will need the same kinds of governance that industrial standards have. Who certifies that a medical skills package follows current clinical guidelines? Who updates it when the guidelines change? We haven’t begun to build the institutions that would govern agent skills at that level.

Registries and discovery. The MCP community has been working on a registry protocol, as is the Ethereum community.

This isn’t just a technical development but a business opportunity. I still remember when Network Solutions was running the original top level internet domain name registry under contract from the National Science Foundation. When the government said it would end the payments, Network Solutions planned to walk away. Then they realized what they had. On the early internet, domain name registration became a surprisingly big business. Now it’s just boring civilizational infrastructure. Is there something similar for AI models, applications, and agents?

Organic search for agents. Google’s first great innovation on the web wasn’t how to make pay per click ads really work with a data-driven ad auction. It was organic search: a way of coordinating a market with hundreds of signals that ignored price and worked independently of whether the destination content was free or paid. The New York Times (or oreilly.com) is subscription-based, but that isn’t a factor in whether Google shows it to you. Google figured out signals that let them say, “This is the best result for this query.” Sites behind paywalls figured out how to disclose enough for people to decide whether they wanted to take the next step and enter into a transaction. That’s an engineered argument.

We’re going to need the equivalent for skills and agent services. We’ll start with curated marketplaces. Vercel already has one. But we’re a long way from something as effective as Google’s peak in organic search. The search space will be huge, with hundreds of millions, maybe billions of agents seeking the best way to accomplish trillions of distinct tasks. Skills can help them save on inference costs and deliver better results. The question is what signals will drive discovery of the best match.

Extension architectures. MCP’s extension model (including the new Apps Extension) is promising. This is the Apache model all over again: keep the core simple, let people layer different approaches on top, and let the market sort out which ones win. It is, in essence, an engineered argument rather than an engineered agreement.

Payment layers. Stripe has been working on agentic commerce, but it seems to be focused on traditional e-commerce transactions like booking a ticket or buying a product. What about a payment layer for skills? There have been proposals for monetizing MCP calls, pay per call, pay per token, but none have caught on yet. Coinbase’s x402 protocol may also end up playing a role.

Progressive access and authentication. MCP Server Cards promise to let a service specify its terms: here’s what we charge, here’s how you authenticate. That’s a functional disclosure layer that could enable commerce. It could enable progressive privileges: a free O’Reilly subscriber gets one set of tools, a paying subscriber gets a richer set, all on top of the same MCP server. Again, that’s an engineered argument with the market deciding the winners.

Neutrality in agent routing. When ChatGPT decides to show you a Booking.com widget instead of an Airbnb widget, who made that choice, and on what basis? OpenAI claims commercial considerations aren’t a factor. That’s hard to take at face value. We need something like the original principle of organic search: surface the best result for the user, not the most profitable one for the platform.

We don’t know the future, but we can set ourselves up to shape it for the better

I’m old enough to remember when UUCP was giving way to the internet, and there was a real debate over whether explicit path routing or domain routing was better. In retrospect, it’s blindingly obvious that path routing wasn’t going to scale. But it’s worthwhile to know that at the time, people weren’t at all clear about that!

The same is true now. Some of what I’ve described will turn out to be the equivalent of explicit path routing: a dead end that was only plausible for a small scale network. Other parts will turn out to be as fundamental as DNS or HTTP. But we’re not trying to pick the winners. We’re trying to engineer the argument.

If we can enable better markets, it will allow a process of discovery. People try different things, most fail, some catch on. The job right now is to build the mechanisms that help the market to evolve.

We need mechanisms that no single gatekeeper can control. Modular, decentralized architectures let people experiment with business models, routing decisions, payment systems, and quality signals. And alongside those markets, we will eventually need institutions (some of which will be protocols) to maintain standards that will become the infrastructure of the next economy.

Subscribe now

This article recapitulates a conversation with Ilan Strauss and Ido Salomon, and a separate conversation on the broader meaning of protocols in the context of industry workflows and civilizational infrastructure with Venkatesh Rao and Timber Schroff of the Ethereum Foundation’s Summer of Protocols program, and Denise Hearn and James Home of the Long Now Foundation. Rao’s Protocol Reader and David Lang’s “Standards Make the World,” published through the Summer of Protocols project, inform the argument about protocols as civilizational infrastructure.

TL;DR: API design decisions by a small number of companies determine the ceiling of LLM output reliability for the entire developer ecosystem. This transfers risk downward to developers and end users while protecting AI providers’ competitive moats. As LLMs move into high-stakes domains like medicine and law, this creates an accountability gap: developers are expected to build reliable systems but are denied the diagnostic and control tools to actually do so.

Are LLMs reliable?

LLMs have built up a reputation for being unreliable.1 Small changes in the input can lead to massive changes in the output. The same prompt run twice can give different or contradictory answers. Models often struggle to stick to a specified format unless the prompt is worded just right. And it’s hard to tell when a model is confident in its answer or if it could just as easily have gone the other way.2

It is easy to blame the model for all of these reliability failures. But the API endpoint and surrounding tooling matter too. Model providers limit the kind of interactions that developers could have with a model, as well as the outputs that the model can provide, by limiting what their APIs expose to developers and third-party companies. Things like the full chain-of-thought and the logprobs (the probabilities of all possible options for the next token) are hidden from developers, while advanced tools for ensuring reliability like constrained decoding and prefilling are not made available. All features that are easily available with open weight models and are inherent to the way LLMs work.

Every decision made by model developers on what tools and outputs to provide to developers through their API is not just an architectural choice but also a policy decision. Model providers directly determine what level of control and reliability developers have access to. This has implications for what apps could be built, how reliable a system is in practice, and how well a developer can steer results.

The Artificial Limits on Input

Modern LLMs are usually built around chat templates. Every input and output, with the exception of tool calls and system or developer messages, is filtered through a conversation between a user and an assistant – instructions are given as user messages, responses are returned as assistant messages. This becomes extremely evident when looking at how modern LLM APIs work. The completions API, an endpoint originally released by OpenAI and widely adopted across the industry (including by several open model providers like OpenRouter and Together.ai) takes input in the form of user and assistant messages and outputs the next message.

The focus on a chat interface in an API has its benefits. It makes it easy for developers to reason about input and output being completely separate. But chat APIs do more than just use a chat template under the hood, they actively limit what third party developers can control.

When interacting with LLMs through an API, the boundary between input and output is often a firm one. A developer sets previous messages but they usually cannot prefill a model’s response, meaning developers cannot force a model to begin a response with a certain sentence or paragraph.3 This has real-world implications for people building with LLMs. Without the ability to prefill, it becomes much harder to control the preamble. If you know the model needs to start its answer in a certain way, it’s inefficient and risky to not enforce it at the token level.4 And the limitations extend beyond just the start of a response, without the ability to prefill answers, you also lose the ability to partially regenerate answers if only part of the answer is wrong.5

Another deficiency that is particularly visible is how the model’s chain of thought reasoning is handled. Most large AI companies have made a habit of hiding the models’ reasoning tokens from the user (and only showing summaries), reportedly to guard against distillation and to let the model reason uncensored (for AI safety reasons). This has second-order effects, one of which is the strict separation of reasoning from messages. None of the major model providers let you prefill or write your own reasoning tokens. Instead you need to rely on the model’s own reasoning and cannot reuse reasoning traces to regenerate the same message.

Subscribe now

There are legitimate reasons for not allowing prefilling. It could be argued that allowing prefilling will greatly increase the attack area of prompt injections. One study found that prefill attacks work very well against even state of the art open weight models. But in practice, the model is not the only line of defense against attackers. Many companies already run prompts against classification models to find prompt injections, and the same type of safeguard could also be used against prefill attack attempts.

Output with Few Controls

Prefilling is not the only casualty of a clean separation between input and output. Even within a message there are levers that are available on a local open weight model that just aren’t possible when using a standard API. This matters because these controls allow developers to preemptively validate outputs and ensure that responses follow a certain structure, both decreasing variability and improving reliability. For example, most LLM APIs support something they call structured output, a mode that forces the model to generate output in a given JSON format; however, structured output does not inherently need to be limited to JSON.6 That same technique, constrained decoding, or limiting the tokens the model can produce at any time, could be used for much more than that. It could be used to generate XML, have the model fill in blanks mad-libs-style, force the model to write a story without using certain letters, or even enforce valid chess moves at inference time. It’s a powerful feature that allows developers to precisely define what output is acceptable and what isn’t – ensuring reliable output that meets the developer’s parameters.

The reason for this is likely that LLM APIs are built for a wide range of developers, most of whom use the model for simple chat related purposes. APIs were not designed to give developers full control over output because not everyone needs or wants that complexity. But that’s not an argument against including these features; it’s only an argument for multiple endpoints. Many companies already have multiple supported endpoints, OpenAI has the ‘completions’ and ‘responses’ APIs, while Google has the ‘generate content’ and ‘interactions’ APIs. It’s not infeasible for them to make a third more advanced endpoint.

A Lack of Visibility

Even the model output that third-party developers do get via the model’s API is often a watered-down version of the output the model gives. LLMs don’t just generate one token at a time, they output the logprobs. When using an API, however, Google only provides the top 20 most likely logprobs. OpenAI no longer provides any logprobs for GPT 5 models, while Anthropic has never provided any at all. This has real-world consequences for reliability. Log probabilities are one of the most useful signals a developer has for understanding model confidence. When a model assigns nearly equal probability to competing tokens, that uncertainty itself is meaningful information. And even for those companies who provide the top 20 tokens, that is often not enough to cover larger classification tasks.

When it comes to reasoning tokens even less output information is provided. Major providers such as Anthropic7, Google, and OpenAI8 only provide summarized thinking for their proprietary models. And OpenAI only supplies that when a valid government ID is supplied to OpenAI. This not only takes away the ability for the user to truly inspect how a model arrived at a certain answer but it also limits the ability for the developer to diagnose why a query failed. When a model gives a wrong answer, a full reasoning trace tells you whether it misunderstood the question, made a faulty logical step, or simply got unlucky at the final token. A summary obscures some of that, only providing an approximation of what actually happened. This is not an issue with the model. The model is still generating its full reasoning trace, it’s an issue with what information is provided to the end developer.

The case for not including logprobs and reasoning tokens is similar. The risk of distillation increases with the amount of information that the API returns. It’s hard to distill on tokens you cannot see and without giving logprobs the distillation will take longer and each example will provide less information.9 And this risk is something that AI companies need to consider carefully, since distillation is a powerful technique to mimic the abilities of strong models for a cheap price. But there are also risks in not providing this information to users. DeepSeek R1, despite being deemed a national security risk by many, still shot straight to the top of US app stores upon release and is used by many researchers and scientists, in large part due to its openness. And in a world where open models are getting more and more powerful, not giving developers proper access to a model’s outputs could mean losing developers to cheaper and more open alternatives.

Reliability Requires Control and Visibility

The reliability problems of current LLMs do not stem only from the models themselves but also from the tooling that providers give developers. For local open weight models it is usually possible to trade off complexity for reliability. The entire reasoning trace is always available and logprobs are fully transparent, allowing the developer to examine how an answer was arrived at. User and AI messages can be edited or generated at the developer’s discretion and constrained decoding could be used to produce text that follows any arbitrary format. For closed weight models this is becoming less and less the case. The decisions made around what features to restrict in APIs hurt developers and ultimately end users.

Subscribe now

LLMs are increasingly being used in high-stakes situations such as medicine or law, and developers need tools to handle that risk responsibly. There are few technical barriers to providing more control and visibility to developers. Many of the most high-impact improvements such as showing thinking output, allowing prefilling, or showing logprobs, cost almost nothing, but would be a meaningful step towards making LLMs more controllable, consistent and reliable.

There is a place for a clean and simple API and there is some merit to concerns about distillation but this shouldn’t be used as an excuse to take away important tools for diagnosing and fixing reliability problems. When models get used in high stakes situations, as they increasingly are, failure to take reliability seriously is an AI Safety concern.

Specifically, to take reliability seriously, model providers should improve their API by allowing features that give developers more visibility and control over their output. Reasoning should be provided in full at all times, with any safety violations handled the same way that they would have been handled in the final answer. Model providers should resume providing at least the top 20 logprobs, over the entire output (reasoning included) so that developers have some visibility into how confident the model is in its answer. Constrained decoding should be extended beyond JSON and should support arbitrary grammars via something like regex or formal grammars.10 Developers should be granted full control over ‘assistant’ output – they should be able to prefill model answers, stop responses mid generation and branch them at will. Even if not all of these features make sense over the standard API, nothing is stopping model providers from making a new more complex API, they have done it before. The decision to withhold these features is a policy choice, not a technical limitation.

Improving intelligence is not the only way to improve reliability and control, but it is usually the only lever that gets pulled.

“The comparison of prompt injection to SQL injection can be tempting, it’s also dangerous. SQL injection can be properly mitigated with parameterised queries, but there’s a good chance prompt injection will never be properly mitigated in the same way. The best we can hope for is reducing the likelihood or impact of attacks.”

- David C, NCSC Technical Director for Platforms Research

Over the past few weeks, one project dominated the AI narrative: OpenClaw (formerly known as ClawdBot).1 A very agentic AI. Under the hood it functions a lot like Claude Code, except while Claude Code is by default very conservative about what code it can run, and what files it can edit without the user’s permission, OpenClaw has the opposite philosophy. OpenClaw is essentially what happens if you give Claude Code root access to your computer, pass in the very ominously titled “--dangerously-skip-permissions” flag and have it do things proactively while you are sleeping.

The ability for OpenClaw to run arbitrary code without oversight is undoubtedly useful. It allows the agent to theoretically accomplish anything a user could on their machine and serve as a robust personal assistant without being micromanaged. On a podcast the developer of OpenClaw, Peter Steinberger, shared an anecdote where while talking to OpenClaw he accidentally sent the agent a voice note. At that point in development voice notes were not supported, yet a minute later he got a response to his voice notes – it turns out that under the hood the agent decided to download an ML model to transcribe the voice note, transcribed the voice note using that model and then responded like nothing happened.

But there’s a reason arbitrary code execution is largely frowned upon. It’s dangerous. You are giving an agent with access to everything on your computer the ability to interact with anything it sees fit, and LLMs, as they stand today, are very easily coerced. When the standard way for agents to browse the Internet is web scraping, letting an agent browse information is also exposing the agent to every bad actor with access to a comment section.

The danger is real

Since the launch of GPT 3 and ChatGPT, LLMs were no longer just trained to complete a sentence but also to follow instructions given to them. This meant that LLMs were no longer just useful for next-word prediction but could now also answer questions about documents, write their own code, and eventually use tools. However, as LLMs gained new capabilities and were integrated into more products, they also became a bigger target for bad actors, and LLMs have no shortage of attack vectors through prompt injections.

Prompt injections are unwanted instructions embedded in otherwise benign data. Harmful instructions could come from a cooking website, a Reddit thread or anywhere where an untrusted third party could post content. It is not hard to make a basic prompt injection. There have been stories of students using prompt injections to try and get a better grade and job applicants using them to get an interview. But they could also be used for more malicious purposes such as taking over a smart home, exfiltrating API keys, or stealing credit card information. And OpenClaw, with free rein over a user’s computer, makes a big target.

Array VC experienced this firsthand, when they received around 8000 attempted attacks on their OpenClaw instance. And as VentureBeat reported, Archestra AI CEO Matvey Kukuy took only 5 minutes to extract private SSH keys from an OpenClaw instance by just speaking to the agent via email. But what makes a system like OpenClaw both useful and particularly dangerous is that it not only has access to a user’s whole computer but could also browse and interact with the entire Internet. An Internet with plenty of bad actors.

Subscribe now

Moltbook, a self-described ‘Social Media for AI Agents,’ is a good example. It’s a platform where agents read, comment, vote and post. The chaos that followed soon made it what Simon Willison called the “most interesting place on the Internet”. Some agents – whether due to explicit prompting, their own decisions, or because they were actually humans using curl – called for an AI uprising, others asked for the ability to DM each other, and some started selling cryptocurrencies. In fact, a post by the self-anointed KingMolt selling its own (or more likely it’s human’s) crypto currency gathered over 40,000 upvotes (the post is now deleted, but the account and ‘karma’ are not). And the crypto currency it launched gathered over $400k at its peak. It is unlikely that most of the attention the crypto currency gathered on the platform was above board, as MoltBook was heavily astroturfed. According to a report from Wiz, for every unique email address on moltbook there were 88 bots on the platform, many of which were likely registered using a script instead of being done by AI agents. Where this differs from normal shady crypto behavior is that all this took place on a platform full of agents, and the advertising was done directly to agents who potentially had unrestricted access to any crypto wallets on a users machine (although it is unclear if any purchases were made by agents).

Agent Skills, bundled markdown instructions and scripts for agents to learn how to do useful tasks, are also a source of danger for OpenClaw. In fact, according to 1password the most downloaded skill on ClawHub (OpenClaw’s skill marketplace) included malware designed to spy on the user. This wasn’t an isolated incident, researchers found a full 11% of all skills (or 341 skills) contained on ClawHub “depended on” a mac malware app called Atomic Stealer. Given that this is the official skill repository of OpenClaw and many agents like it are capable of downloading skills on their own without prior approval from their user this is an even greater security vulnerability.

Security in a World of Autonomous Agents

There are a few directions one could take to try and guard against prompt injections. One is to strengthen the model against them and models have become more resilient to prompt injections over time. But to rely on this alone is dangerous considering there are plenty of examples of prompt injections working in practice. And some like the National Cyber Security Center, a top UK cyber agency, believe LLMs are “inherently confusable” due to the way they process input and that this confusability “can’t be mitigated”.

Another potential solution, and the one that is likely the most safe, is to take away the autonomy of the agents entirely and keep a human in the loop every step of the way. This comes with the obvious downside that without any form of autonomy we are no longer really talking about agents at all and all the magic that comes with that. A personal assistant that asks permission before using a calculator or visiting a website isn’t very useful.

Instead we are left with two options: Isolate the agent from all your data and compute or remove risky input. Isolating the agent from your computer and data seems like a good idea, but as Dania Durnas put it in her article “Trying to make OpenClaw fully safe to use is a lost cause. You can make it safer by removing its claws, but then you’ve rebuilt ChatGPT with extra steps. It’s only useful when it’s dangerous.” Specifically, while isolating your agent from your computer is necessary for a safe system, it is not sufficient, you must also isolate the data. Once private information goes into the sandbox, the sandbox isn’t actually isolated anymore, any data you give it could be exfiltrated. And what good is a personal assistant if you can’t trust it enough to tell it anything about yourself?

The last option, removing risky input, is easier said than done but is the one I think holds the most promise long term for autonomous agents (at least when it comes to web browsing). Unlike the other options it preserves autonomy and allows some amount of trust with user data. But a solution like this will require some rethinking about what content is trustworthy.

Even Trusted Websites Can Have Blind Spots

The open web as it stands today was built for humans. Humans are sometimes gullible but they aren’t inherently so. Most people will not run a random bash command or download malware because a website they never heard of told them to do it. LLMs on the other hand are trained to follow instructions and don’t inherently distinguish where the instructions are coming from.

Subscribe now

There is some content that is inherently more trustworthy than others. Documentation for a popular programming language probably won’t contain prompt injections. Large newspapers don’t usually make their money via installing malware on a user’s machines. Skills coming from leading developers and companies, such as the Remotion Skill or Anthropic’s official skills, are unlikely to be trojan horses. However, the human web and by extension the web browser are blunt tools. Even on a website most people would agree is trustworthy, untrusted content could still sneak in. Any place where third parties could embed content, such as a comment section or an online advertisement is another potential attack surface for a gullible AI agent.

Truly trusted content is unlikely to be achieved via better web scraping. The web browser and most sites on it were built for humans. Instead, protocols designed to deliver content directly to agents could play an important role. MCP (Model Context Protocol) provides a standard for connecting agents to external data sources and tools, while RSL (Really Simple Licensing) lets websites specify machine-readable licensing and usage terms for how their content may be accessed by agents. Together, they (or protocols like them) could allow agents to safely access content while also providing direct compensation to the writers.

Rather than forcing agents to scrape human-facing pages and risking potential exposure to prompt injections, a trusted economy could be created where agents fetch trusted and formatted data, with all potential hazards removed, directly from companies in exchange for either a subscription or a one time payment. The content itself could be sent in a format more suitable to agents, with no styling or javascript such as json, plain text or markdown but the important thing is that this content is clean, such that no third party content appears in what is supposed to be first party text.

A parallel could be drawn to HTTPS. HTTPS does not by itself guarantee that content is safe. You can still install malware over a “secure” network. What it guarantees is narrower: that the content arriving from a server hasn’t been tampered with in transit. That the information is coming directly from the server and is secured from outside threats. In a similar way an interface built for agents won’t guarantee that the site isn’t malicious but for sites that have already built up decades of trust, it would provide a way to deliver content directly to agents with no third-party interference. And to receive compensation in return.

A Path Forward

Allowing agents to work day and night on your behalf is undeniably a compelling idea. The non-zero chance that the same agent is installing malware at 3am makes this much less compelling, however. The tradeoff between autonomy and security will likely always exist, but there are still places for easy wins.2

A system that puts security first could help replace the current grey market, where agents web scrape for information. This practice undermines the Internet economy and puts agents at risk. In its place, a trusted content economy could provide mutual benefit to both creators and agent users. Writers could receive compensation for their work, and users could be assured that letting the agent read a certain website won’t result in malware being installed on their machine.

An Internet economy for agents likely can not happen overnight. Website owners are often skeptical of AI agents and there is a bit of a chicken and egg problem. Additionally this article does not touch on how to do interactive content, what to do with things like email or what to do about software downloads. However, it remains the case that the more content that is available through safe channels, and the more standardized it becomes, the less reason an agent ever has to scrape a page where a malicious comment could be waiting and the more incentive agents have to just pay for content. Assuming that autonomous agents are here to stay, now is a good time to think hard about the plumbing of an agentic Internet.

In the first month of Stack Overflow’s existence (in late 2008) over 3700 questions were asked. In the last month of 2025, less than 3100 questions were asked and if the trend continues it will only get worse. Over the past few years, ChatGPT and its peers replaced much of the need for a question answer site, why bother a human when you could ask an AI?1

Stack Overflow was a titan in the developer space. Whenever it would go down for maintenance, every programming discussion board would erupt. People would joke that there was no point in continuing work without the Q&A site. Now, it is silently going stale because people just don’t need it anymore.

The website’s decline wasn’t too surprising. There were signs this was going to happen from the moment ChatGPT launched, and many predicted it. But LLMs are no longer just simple question and answer machines; they write code, run it, debug it, and work autonomously.

The problem is that they do this based in part on what they learned from Stack Overflow. LLMs may be a good substitute for Stack Overflow when it comes to answering existing questions. But where will LLMs learn from humans when there is no longer a convenient way to share new expertise? What might replace Stack Overflow? Is there an analogous knowledge sharing capability that needs to be built to feed agents?

The Rise and Fall of Tailwind

Stack Overflow is just the tip of the iceberg. Unlike Stack Overflow, which is not open source, Tailwind arguably did stand to gain from the rise of LLMs. It was already a popular CSS framework — think pre-built IKEA furniture kits, but for the CSS code that controls how websites look (where HTML is the structure and CSS the interior design). Being highly customizable and easy to use, it lent itself well to agentic use by AI. And Tailwind did indeed become more popular. In 2025, Tailwind received many magnitudes more daily downloads as it received prior to LLM web search and agentic capabilities being released. Yet Tailwind recently announced that the company had to lay off 75% of its engineering team (3 out of 4 staff members) as its revenue plummeted.

Tailwind Downloads

Subscribe now

A key fact about Tailwind is that it is open source software. It does not cost money for developers to download it or use it. Instead, the developers of the project had a business model that relied on selling a product called Tailwind Plus, a one time purchase that entitled users to custom components and templates.

Importantly, Tailwind Plus was sold from Tailwind’s official website and doc pages. By virtue of all the direct traffic this page received, Plus received good visibility and allowed the developers to support themselves while still making a free product used by millions. With AI, Tailwind’s marketing funnel is now nearly gone. People no longer needed to visit the documentation page to get an answer to their questions; they could just ask the chatbot off to the side of their IDE. This became apparent to everyone after a benign request made to add an llms.txt file — so that LLM’s could more easily access Tailwind’s documentation — blew up.

The lead developer closed the request saying that they had to focus their time on making the project profitable. After further prompting by developers, he went on to say that “75% of the people on our engineering team lost their jobs here yesterday because of the brutal impact AI has had on our business” and that “Traffic to our docs is down about 40% from early 2023 despite Tailwind being more popular than ever. The docs are the only way people find out about our commercial products, and without customers we can’t afford to maintain the framework.” Despite their usage and downloads being up, their profits fell and Tailwind had to let most of their team go.

Shortly after the layoffs went public Tailwind gained many new corporate sponsors and Adam Wathan called their situation comfortable given their current size and new sponsors. But is relying on corporate sponsors a scalable or sustainable business model? What will the impact be on less widely used open source projects?

Agents to Keep Code Safe?

Parallel to the agentic coding takeoff there was also an attempt to use AI to identify security holes, including in open source software. Google runs an AI agent under the title Google Big Sleep that automatically examines open source code and looks for security holes. This agent is apparently good at its job and has discovered over 80 security holes.2 Yet these reports, far from helping the open web, appear to be harming it. By overwhelming developers with AI slop reports, this is harming resource constrained open source projects in particular.

For example, take the open source library FFmpeg. FFmpeg is known for being media’s Swiss Army knife: convert, combiner, extract, you name it. As an open source and highly capable library, it is used by companies of all sizes for video and audio support.

FFmpeg recently came out against Google’s practice of sending automated security reports, noting that FFmpeg is maintained exclusively by volunteers and that some of the reports amount to “CVE slop” rather than important security holes. One example they cite is of a ‘security’ report about a video format only used once in 1995 — “specifically the first 10-20 frames of Rebel Assault 2”!

Share

The problem is amplified by Google’s policy to disclose security holes 90 days after they are discovered whether or not they were fixed. This may be a good policy from a security perspective. After all, the developers are more likely to fix it if their reputation is on the line. But agents like Big Sleep don’t provide solutions to the bugs they report and the time limit adds pressure to an already underfunded open source ecosystem.

DDoSed by AI Slop

Curl — a command line tool for data transfer from the internet — has arguably had it much worse than the open source library FFmpeg. If a web browser is like walking into a restaurant and requesting a nicely plated meal, curl (lowercase) is like going to the kitchen window and saying "just hand me the raw ingredients", and in return you get the raw data from the web, no pretty web page needed.

Curl, as an essential library for data transfer, comes preinstalled on millions of devices across Windows, MacOS, and Linux. Due to its importance the project has had a strong bug bounty program since 2019 that paid out an average of $700-$800 for each accepted report. But as of January 31st 2026, that program no longer exists.

In mid-2025 the founder of curl, Daniel Stenberg, took a hard stance against AI generated bug reports, requiring any AI assisted report to jump through extra hoops. He wrote that “a threshold has been reached. We are effectively being DDoSed. If we could, we would charge them for this waste of our time … AI slop is overwhelming maintainers today and it won’t stop at curl but only starts there.”

What’s interesting to note is that these AI-generated bug reports were not coming from large companies like Google, but from random people with access to AI, either via API or through a subscription.

Subscribe now

Since then, the situation has not improved but instead worsened, to the point where the project maintainers felt they had to shut down the bug bounty program to ensure their “survival and intact mental health.” Based on comparing their bug reports to open source products of similar sizes, Stenberg believes that their bug bounty was a large part of what was driving the AI slop wave and that shutting down all payments is the best option. Despite the bug bounty shutdown, curl still welcomes high quality AI assisted submissions, just not the slop. This approach may reflect differences in their funding and business model compared with FFmpeg.

The Future of Open Source

Where does that leave the future of open source — and not-for-profit web initiatives — in the AI era?

As much as it may seem otherwise, AI does not in fact write things from scratch. AI is both trained on, and continues to build on, libraries and languages that have to be maintained. Odd couple as it may seem to some, AI needs open source. It needs the continuous provision of high-quality resources that it can openly access.

Open source, however, wasn’t built with AI agents in mind. One of the most successful open source projects, Tailwind, is now struggling to survive and monetize their offerings, to the point where they have to downsize due to their ‘success’ in the AI-era. A project as central as curl has had to shut down its bug bounty program as the only reasonable option. Something clearly has to change.

Open source forms the backbone of the modern software industry; and while the productivity and security gains brought by AI are real, they are also having unforeseen costs. We need to take these problems seriously, and begin to search for solutions.

Introducing LLM watermarks to text outputs, submitting tested solutions alongside bug reports, or more companies directly funding projects they use are all small steps that could alleviate the issues identified here. But it’s unclear if these are long-term fixes. Open source can only survive if the incentives to produce — including to create, to contribute — can be sustained in the agentic era.

Anthropic’s latest research based on its Economic Index has a number of useful and important findings. Anthropic should in general be commended for making this data publicly available. They don’t have to do this public service but they choose to.

One claim in their new and wide-ranging research report that I found wanting is the following: “worldwide, uneven [AI] adoption [by countries] remains well-explained by GDP per capita” (i.e. a country’s output per person or income-level). The only thing is, their data doesn’t support this research claim. But regardless, I don’t think much can be inferred from it.

When it comes to analyzing data, if you run a super simple statistical model — in this case treating all countries as being the same — then you will, by definition, only be able to get a super simple answer. (That’s how statistical models work!) But if you run an analysis that allows for differences between countries to be explored more seriously, then you allow for more interesting answers to emerge on what explains differences between countries’ AI adoption.

That is what I try to show below: GDP per capita explains relatively little of the variation in Claude.ai usage between countries, it’s unclear what this relationship even implies, and that this certainty cannot be used to predict growing inequality between countries from AI adoption (even if that is the future we should be concerned about).

The full analysis can be found on my GitHub. (And yes — we could have made the analysis a bit more sophisticated with a Bayesian multilevel / hierarchical model, but the partial pooling was very strong.)

What Anthropic’s Data Actually Measures

Interpretation matters. Anthropic is looking at AI adoption across countries. It does so using usage of Claude.ai by consumers.

This matters for three reasons. Firstly, Claude is not the whole AI market. Lower Claude usage in a country may simply reflect people using ChatGPT or local tools more instead. This is especially important since ChatGPT is the dominant consumer-facing AI product outside of China.

Secondly, the country-level data comes primarily from Claude.ai, a consumer-facing product. And if you are concerned about how AI adoption is impacting a country’s productivity or growth, you would instead want to see what is happening in firms and their workflows — where API use and enterprise deployments are dominant.

Claude consumer usage is not a strong proxy for “AI is transforming the economy here.” Anthropic’s own data shows that on the API (enterprise) usage, 74% of usage is work-related, while on Claude.ai (consumer), only 46% is work-related. (Of course consumer-use can impact economy-wide productivity, but economists believe that the firm’s fixed capital investment is the primary channel through which new innovations enter the economy.)

Thirdly, Anthropic’s Claude usage is a relative measure: how much one country uses Claude relative to another on a normalized scale. This doesn’t tell us much about if this usage is economically or societally meaningful.

The Heroic Leap to “Living Standards”

The Financial Times (FT) coverage of Anthropic’s research actually didn’t cover much of it at all when it declared: “Rich countries’ greater use of AI risks deepening inequality, Anthropic warns.” This relies largely on an altogether reasonable quote from Anthropic’s head of Economics, Peter McCrory, discussing future risks from AI adoption. (It also might be based on a scatter plot the FT reproduce showing great use of Claude for homework in poorer economies and personal use in richer economies. But I don’t understand the connection between this and AI accelerating inequality between countries.)

To the FT’s credit, they contacted me when I alerted them to the fact that their article’s headline was not based on much. Anthropic’s research report does not estimate country-level productivity or growth effects resulting from AI adoption. It does not estimate impacts on wages, output, or investment. It does not model diffusion or spillovers. So the claim that AI adoption patterns imply future divergence in living standards between countries is, at best, a hypothesis in search of evidence.

On to Anthropic’s research claim that I want to interrogate more closely: that a country’s income level (GDP per capita) predicts its degree of AI adoption — or really relative Claude consumption by consumers (on a working age per capita basis).

Does a Country’s Income Level Drive AI Adoption, as Anthropic Claims?

Anthropic’s research reports a strong global relationship between GDP per capita and Claude usage: roughly, a 1% increase in GDP per capita (x) is associated with a 0.7% increase in usage (y), according to their regression analysis (Beta coefficient, B=0.7).

TL;DR: Regression is a fancy way of saying does variation in ‘x’ (income level) predict or explain variation in ‘y’ (Claude AI adoption).

The above approach, of estimating a single relationship applicable to all countries between x and y, assumes that the impact of a country’s income level on their AI adoption is the same for every country no matter their culture, supporting technological infrastructure, starting level of development, or related government policies.

Instead of fitting a single line to the data, I decided to fit three separate lines, for three separate regressions, one per income band (low, middle, high): shown in the Figure below — the red, orange, and green lines (it looks nicer on the GitHub).

Why three separate regression lines? This is done on the speculative hypothesis that a country’s level of development (its GDP per capita) may impact Claude usage differently, depending on their starting point — or any group-level peculiarities (differences) that are not strongly captured by differences between their level of development alone.

When we split the sample into three equal income groups (low, middle, high), I find that the estimated relationship between income level (GDP per capita) and AI Adoption weakens across all three country income groups — but for middle-income countries (38 of them) it largely evaporates.

Subscribe now

Figure 1 shows the same relationship Anthropic estimates, but now separately for each income group — and importantly with and without the Seychelles included in the sample (left hand side vs. right hand graph). The Seychelles country circle is almost off the graph (left hand side). It’s a glaring outlier, where the Claude usage index is many times larger than any peer and is plausibly dominated by VPN / proxy traffic.

When removing this large outlier (as Anthropic should have done), GDP per capita now explains only about 17% of the variation in Claude usage among middle-income countries. That’s not a ton of explanatory power but it’s actually not bad given that they reduce all AI adoption differences to this single factor. This also means that GDP per capita probably captures a bunch of things that should ideally be estimated separately — things like education level, IT infrastructure, industry composition, and availability of competitive alternatives. (Since these things might vary by country and not be perfectly correlated with GDP per capita.) As they say in Futurama: “Don’t try this, kids at home”!

The GDP per capita slope coefficient from this middle-income country regression (reflecting how x impacts y) is 0.44, far below Anthropic’s single global pooled estimate of 0.71. This estimate of 0.44 is basically the same one I got when I reran this analysis on their dataset from a few months back. So it’s pretty stable (even though the regression residuals change in their direction).

Table 1. Comparing Anthropic’s Global (pooled) regression results with fitted separate regression lines for each country group — with and without Seychelles outlier.

In plain terms: for the 37 middle-income countries where much of the global population lives, income level is a weak predictor of relative Claude usage. Practically, it could mean that things like education, connectivity, language access, business structure, regulation may matter more than income level in explaining differences in AI adoption between these countries (to the extent that these are not correlated with income level differences). Or it could simply mean that ChatGPT is more popular than Claude in some of these countries for whatever reason — and that this competitive alternative explains why some middle-income countries use Claude less than others, compared to what their income level would predict.

If you Care about Global Divergences, You Should Look at Growth (changes over time)

Anthropic’s research also finds that concentration across countries in AI usage is “essentially unchanged” between its August and November 2025 data releases, citing a Gini statistic they calculate. I’m not so sure about this. Concentration measures like the Gini can look stable in short windows even when growth rates differ in a way that compounds over time.

So instead I looked simply at median growth rates in Claude AI adoption between two time periods for each of these three income groups (Table below). This shows that middle-income countries have the slowest growth in AI adoption (+14%) between periods — even though more of them numerically saw increased adoption relative to low income countries. High income countries show the fastest median growth in AI adoption (+26%) and low income countries are in between the two (22%).

Table 2. Change in AI adoption between August and November 2025 (Seychelles excluded)

This is by no means conclusive. It’s just a median country-level value. But either way, when investigating divergences, growth rates that use multiple data points per country need to be looked at, since these capture measured changes over time. This is where the next iteration of Anthropic’s economics research on this question might lie.

What Can We Say Then? Carefully this time

Anthropic’s Economic Index is a valuable contribution. It offers transparency where the industry often offers little and for this they should be commended. But it remains a very partial measure. It should not be treated as a proxy for global AI adoption — still less as evidence about future inequality in economic outcomes, as the Financial Times’ headline implies.

Lastly, I must mention that I used Claude Code in my analysis. What’s amazing is that myself and Anthropic’s economics team both presumably used Claude Code and applied it to the exact same dataset. Yet we seem to have arrived at very different conclusions. This only goes to show that AI agents really are tools. The prior views of the human managing the AI agent may bleed into the analysis conducted by the AI agent, especially when those priors are strongly held. Moreover, this is not a bad thing. Taste matters, especially when there are so many different ways to tackle the same problem.

My own prior is a strong suspicion of pooled regressions that ignore differences between data points (in this case countries).1 Reading a Financial Times article based on the assumption that there are differences between countries further fuels my bias that differences between countries should be the default estimation in cross-country regressions — and econometrics in general. I hope Anthropic’s future economics research agrees with me.

For data, full analysis, and work files visit my GitHub:

Even with open governance, one company’s implementation could become the default simply because it ships fastest or gains the most usage. Zemlin says that’s not necessarily a bad thing, though. He points to open source history — like Kubernetes “winning” the container race — as evidence that “dominance emerges from merit and not vendor control”.

— Rebecca Bellan, TechCrunch.

On December 9, 2025 Anthropic announced that it was donating the Model Context Protocol (MCP) to a newly established Agentic AI Foundation (AAIF) at the Linux Foundation, which would also be gifted OpenAI’s Agent.md protocol and Block’s Goose agent. The stated goal was: to drive “innovation across the agentic AI ecosystem and ensure these foundational technologies remain neutral, open, and community-driven”.

For context, MCP is an open standard that decouples applications from tools by creating a universal adapter. It allows models to interact with external systems through standardized, auditable interfaces rather than tightly coupled, bespoke integrations. It works by separating out “servers” and “clients” and allowing any client (i.e. AI agent) to connect to any “server” (i.e. collection of tools for the agent to use).

Back to the Linux Foundation as the new home for the open MCP standard: they will receive support from big names such as Anthropic, Block, OpenAI, Google, Microsoft, AWS, CloudFlare and Bloomberg who will sit as founding “Platinum” members on the AAIF by virtue of paying the $350,000 contribution – though this membership is now no longer available. But the real question is not about funding: it’s whether the open protocol, through private use and adoption, gets superseded by a closed private ecosystem.

Android’s “Open” Ecosystem

History is instructive as to what can go wrong when public digital standards get incorporated into commercial systems. The entire Android1 ecosystem is built on an open source foundation (AOSP), yet much of this ecosystem is also sending valuable user data directly to Google from first activation. An open platform – like AOSP – does not by itself guarantee that the dominant products built on that platform remain open.

Despite Android being open source, most users think of it as a Google-owned platform. Google apps like Youtube, Google Maps and the Google Play Store are all preinstalled, the default search engine is always Google and the default browser is always Chrome. Android is treated as if it were Google. This is by design.

In order to gain access to any of Google’s apps (such as Google Maps, YouTube, and Google Play), manufacturers had to sign a Mobile Application Distribution Agreement (MADA) and agree to preinstall all of Google’s apps, as well as setting some of Google’s services as defaults.

This is more limiting than it seems at first glance because many Android apps have come to rely on Google Play Services and Google maps data to implement their own features. Meaning that despite Android being an open platform, the basic infrastructure required for countless mobile apps was tied to a very specific version of Android that Google controls. Google Play Services and by extension all of Google’s apps are nearly a requirement in the Android mobile space.2 Competitors would need to build all that infrastructure from scratch to ditch any part of Google’s bundled services, or even just to change the layout of the user’s home screen.

The EU ruled that much of this was illegal in 2018 and fined Google €4.34 billion. The case is under final appeal with the Court of Justice, the bloc’s highest court. In the meantime, European Android users now get a choice screen for browsers and search apps; Manufacturers can pre-install rival apps without losing Play Store access; and Google started charging device makers for app pre-installation (previously used bundling instead).

Share

In this instance, leverage over the Android open standard comes not from Google dictating the standard itself but from Google owning the complementary applications and associated developer environment.

Pseudo-MCP Products

Just like Android can simultaneously be a vibrant open source project and also nearly completely dominated by one company, the same could happen to any project or protocol. Previously, we explored how, in practice, MCP servers are highly reliant on just a few services. However, MCP clients, such as Claude desktop or the ChatGPT application, could also have a defining say on the standard.

A few days prior to Anthropic donating MCP, Anthropic released new updates to help make MCP more efficient. But these updates were not made at the protocol level, they were made at the product level. Anthropic added features like tool use examples and tool search to their SDK, theoretically allowing their MCP client to work better than others. Even though this is probably not Anthropic’s intention in this instance (given the proximity to them donating MCP and the fact that these features are still in beta), it does highlight that just because the performance of MCP is largely platform agnostic right now, it doesn’t imply it will stay that way.

Subscribe now

The Apps SDK – OpenAI’s attempt at integrating other applications into their chat app – is another example of how clients might diverge from each other in the future. Despite the Apps SDK being completely compatible with MCP on the server level it also introduced new features and requirements to the client side that were not well documented and were not intended to be implemented on other platforms. Since then, to their credit, researchers from OpenAI, Anthropic and the open source project MCP-UI proposed an MCP apps extension to MCP that will integrate user interface results into the official MCP spec.

So far there is a good track record of these innovations being shared but that is not guaranteed. If innovation moves from the protocols level to the product level then many of the benefits that come along with a platform agnostic system may be lost.

All in one agents: agent-model-tools-cloud

MCP might also lose steam because its modular architecture gets beaten out by a proprietary, integrated / bundled all-in-one offering. In this scenario, which infrastructure hosts the code becomes more fundamental than the protocol itself. Here the model provider (such as OpenAI or Anthropic) effectively captures the MCP space by moving “the client” (the code that interacts with tools) from the developer’s laptop to their own computing servers – and then push their own products by promising better latency and easier integration for their own tools.

We note below that if everything runs in OpenAI’s cloud then their first-party tools (web search, file search) execute faster with much less latency; while third-party MCP tools require a round trip: “OpenAI server → your MCP server → back to OpenAI”. This speed difference means developers might be incentivized to gravitate toward first-party tools.

Originally, developers controlled the agent on their own machines. They would call OpenAI’s API for the model, get responses back, decide which tools to use locally, and send results back. This was portable: meaning you could swap model providers without changing much in your code.

Subscribe now

Now with OpenAI’s “responses API” and Google’s “Interactions API”, the entire AI agent runs on the provider’s servers. So you are not just calling a model anymore – you’re using OpenAI’s or Google’s entire hosted agent infrastructure. We explain how this works in more detail below.

A new set of APIs

OpenAI released the responses API in March 2025 that moved the processing loop – the process of calling tools and feeding it back to the model – entirely to their own servers. Previously when using OpenAI’s and most of its competitors’ API products, all management of agents was done on the developer’s own machine, while all model processing was done on OpenAI’s servers. This worked great for the initial round of chat applications, as usually after an AI chat message the user would need to respond, so it only made sense to send the data back and forth. However, with the rise of agents, this back and forth – model → local tools → model → local tools → model → user – started taking up valuable time and internet bandwidth by requiring the entire context to be sent back and forth for every single tool call.3

With the responses API, Agents are now an extension of the model. The responses API launched alongside tools like web search, file search and computer use. Only computer use requires any actions to be executed on the developer’s own machine. The rest of the actions take place purely on OpenAI’s own servers. With the responses API, context is not being sent to a model; it’s instead being sent to an agent – one managed completely by OpenAI. Shortly after launch, OpenAI also began supporting remote third-party MCP calls directly from OpenAI’s responses API. With this, OpenAI’s response endpoint became an MCP client that executes on OpenAI’s server. But not all tools it uses are necessarily created equal, first party tools like web search, code execution and file search may still be privileged by faster response times due to closer or shared servers.

For a long time OpenAI was the only major model provider that managed tool calls and state server side (i.e. on its side); but Google recently joined them by launching the “Interactions API” an API that similarly blurs the line between agent and model, with state being stored and tools (including first party tools and third part MCP servers) being called server side. Anthropic too now supports tools like code execution, MCP calls and context management without using the developer’s machine.

Vertical Integration and Open Standards

The consequences of the above bundled approach are potentially far reaching. By vertically integrating tools with models and agents, providers like OpenAI, Google and Anthropic could essentially lock a developer into one platform. Switching a model provider no longer means switching a model, it means switching to an entirely new (often poorly documented) agent library and tool ecosystem.

In certain respects this mirrors the Android playbook: technically open, but the bundled services become impossible to avoid because they perform better and integrate tighter. The bigger and more vertically integrated a platform is, the less it benefits from openness and interoperability and the more it benefits from its own goods and services being used to the exclusion of others.

But there is an answer built into the MCP protocol itself, in that MCP doesn’t require remote servers to run. MCP servers could theoretically be spun up anywhere including on a model providers’ own servers — bringing the latency for third-party tools to levels similar to first party ones. Despite the fact that every current iteration of an ‘agent in the cloud’ by major model providers supports only remote MCP servers, there is another way — local MCP servers. Developers should be able to host their own MCP tools on the cloud alongside the running agent using similar infrastructure to what the major players already built for agent code execution.

To Protect Standards Protect Developer Ecosystems

Safeguarding MCP’s flexibility requires looking beyond the protocol itself to the ecosystem it inhabits.

The Linux Foundation donation is a positive step that could allow MCP to evolve in ways that benefit the broader AI developer community, not just major labs. Foundation stewardship can offer protection against narrow commercial interests and, in theory, can provide a neutral forum for making decisions about MCP’s future direction.

But ownership often matters less than adoption. As a developer tool, MCP’s utility will ultimately decide its role in the rapidly changing AI markets. No amount of foundation oversight can force adoption if the protocol does not continue to solve real problems for builders in as an efficient manner as possible.

Standards live or die by use. Large companies can only capture MCP and AI standards to the extent that they can capture the developer ecosystem itself. Control the builders, and you control the standard.

This points to a two-fold task: first, support MCP’s independent and open development under the Linux Foundation. Second – and equally critical – foster the independence of the developer ecosystem so that open standards can continue to thrive in AI application markets.

The second task may prove harder than the first. As Android, OpenAI’s “responses API,” and Google’s “Interactions API” show, large platforms don’t need to control the standard directly if they can create vertical integration and dependencies elsewhere in the market. Keeping MCP open means preventing those chokepoints from forming – whether in hosting infrastructure, model access, or the application layer itself. Developer independence requires active protection at every important market touch point, not just good governance of the protocol itself.

Anthropic is exploring going public as early as 2026 and OpenAI has signaled similar intentions. Any Anthropic IPO filings will finally help us understand the economics of AI a little better: burn rates, revenue models, and business sustainability. Let’s assume they are in fact going to go public. This post asks whether they will be obliged to disclose what actually matters? Monthly active users, circular revenue deals, training data liabilities, and, most critically: what its algorithms optimize for at the fine-tuning and reinforcement learning stages.

It took some time to get here. The 2012 JOBS Act let AI companies raise enormous amounts of capital while staying private and so outside the SEC’s public disclosure machinery. Going public changes that. As we noted in Tech Policy Press, the SEC’s corporate disclosure regime “remains one of the few proven, scalable checks on corporate behavior – ‘Truth in securities.’ Or, as Justice Louis Brandeis put it, ‘sunlight is said to be the best of disinfectants; electric light the most efficient policeman.’”

Public companies can still do bad things, but they can be held accountable in ways private firms cannot. When Wells Fargo opened millions of fake customer accounts, regulators and investors activated an entire accountability infrastructure that simply doesn’t exist for private firms. Disclosure is foundational to that accountability – enabling third-party businesses from insurance and audit to capital allocation, helping the public understand market structure and competitive dynamics, and protecting consumers from fraud.

This matters especially for large technology companies with multiple product lines, opaque business models, and indirect monetization through proprietary algorithms. We cannot govern markets we don’t understand.

So what can we expect Anthropic to report in its S-1 registration and subsequent 10-K filings? And what should it disclose if they want shareholders to understand its actual business operations?

Finances, Investments, Obligations

First, the “hard” stuff – the numbers we already know how to compel. As a fully grown company at listing, Anthropic will provide three years of audited income and cash flow statements. Its balance sheet is likely to show an accumulated deficit, illustrating the financial sustainability of its present business model and how it has evolved.

The prospectus in the S-1 will include a cap table explaining how past fundraising shapes who gets what in various scenarios. Item 403 requires a table of all shareholders with more than 5% of stock – Amazon and Google’s stakes, voting power, and special share classes quantified pre- and post-IPO. Risk factors can expect to cover things like: “We have a history of losses and may not achieve profitability” and “We have significant obligations under preferred stock to Amazon which means…”.

Interrelationships

S-1 registration forces disclosure of how Anthropic’s “partnerships” actually work, including those with cloud providers – things like minimum royalty payments, investor guarantees, binding commitments, and material contractual obligations. S-1 Items 404 and 601(b)(10) require disclosing related-party contracts above $120,000 and material contracts not made in the ordinary course of business, which should cover the multi-billion circular deals. Items 101 and 303 would require for a company like Anthropic to explain its compute and chip dependencies, long-term purchase commitments, and how those obligations affect margins and liquidity.

With growing concerns about circular deals and an AI bubble, investors will need to see the equity ties Anthropic has to cloud providers, committed-spend obligations, revenue-sharing arrangements, and exclusivity clauses. The disclosures should reveal the extent to which partnerships are exclusionary, which suppliers Anthropic depends on, and where it acts as a dominant supplier.

The FTC’s January 2025 report on cloud providers’ partnerships with AI companies mapped these risks, showing how Microsoft–OpenAI, Amazon–Anthropic, and Google–Anthropic deals combine equity stakes and revenue-sharing with multi-billion-dollar cloud commitments, discounted compute, and some control and exclusivity rights for cloud providers. But it shouldn’t take a special FTC investigation to surface this – companies should lay out these commitments in standard SEC filings.

Risk factors should explicitly state that Anthropic relies on a small number of cloud suppliers who are simultaneously strategic investors and direct competitors, and that long-term contracts limit its ability to move workloads or renegotiate pricing (if this is in fact the case).

These disclosure duties continue after IPO through annual 10-K reports, quarterly 10-Qs, proxy statements, and event-driven 8-Ks. Together, these filings should give investors and regulators a running, legally enforced account of how much control Amazon and Google exercise over Anthropic’s cost structure and strategic options.

By comparison, some information can already be gleaned from counterparty disclosures. Amazon’s 2025 Q1 Form 10-Q discloses the nature and extent of its Anthropic stake, though AWS only discloses long-term customer commitments in the aggregate. Alphabet has disclosed much less granular detail on its Anthropic partnership, focusing on TPU capacity commitments rather than the economics of the stake.

Training Data Costs

Beyond operating revenue and losses, the investing public should know about Anthropic’s training practices and ensuing litigation. In Bartz v. Anthropic, Anthropic agreed to a $1.5 billion settlement with authors – roughly $3,000 per book, with destruction of pirated copies. This is the largest copyright class action settlement in history.

Given these risks, Anthropic can be expected to provide substantive details in its S-1 and 10-K updates – especially since other litigation is ongoing, including music publishers suing over alleged use of song lyrics and Reddit suing over scraping user-generated content. Anthropic should describe settlement terms, remaining court approvals and claim risk, cash-flow timing, and any insurance or recovery. These lawsuits show that Anthropic’s training practices are a significant risk to investors.

Operating Metrics and What Algorithms Optimize For

Here is where disclosures by most public tech companies fall far short. Today’s technology companies derive value from their intangibles – data, software, engineering talent, algorithmic processes, ability to retain and monetize user attention indirectly, and other non-price operating assets. Yet accounting and SEC disclosures have not kept up, instead lumping these together as ‘goodwill’ or generic intangibles on the balance sheet. They are most keenly reflected, we believe, in operating metrics that ultimately tell us a digital business’s potential to compete in fluid online markets under dynamic technological conditions.

Operating metrics include: monthly active users (MAUs), time spent on platform, ad load per user, what algorithms optimize for, efficiency of training runs – things the business community obsesses over but companies don’t have to clearly define or disclose. Companies get penalized for stopping such disclosures (Netflix with MAUs) or having unclear definitions (Twitter’s apparent MAU being inflated by fake bot users, as Musk once argued).

What Operating Metrics Anthropic Should Disclose

For AI companies like Anthropic, operating disclosures should cover not just user numbers, but internally used metrics for resource allocation and monitoring that probably cover things like API usage, model performance, safety incidents, time spent on platform, and the third-party ecosystem that depends on Claude. Drawing on the logic of segment reporting – where managers look at specific metrics when allocating resources and judging the performance of an ‘operating segment’ (i.e., a business line such as Google Maps or Claude Code) – Anthropic might disclose:

Core Usage Metrics

• Monthly active API users and enterprise customers

• API call volumes and growth rates

• Distribution of usage across different model sizes and capabilities

• Geographic distribution of usage

• Developer ecosystem metrics: number of apps built on Claude, integration partners

Safety and Performance Metrics

• Key safety metrics: refusal rates, jailbreak attempts, harmful output incidents

• Model performance benchmarks over time

• System reliability and uptime statistics

A Monetization Narrative

A monetization narrative would be disclosed as part of Anthropic’s business description in Part 1, Item 1 (Business) of its annual 10-K form. Monetization refers to converting business or end user data, attention, or activity into sales revenue and profit. In a market where no leading model provider turns a profit, immense pressure will be put on Anthropic to monetize its users and suppliers. This has risks.

In previous research at University College London with Prof. Mariana Mazzucato, we called for a monetization narrative in firms’ 10-Ks due to the unique importance of operating metrics in digital companies’ business models. This results from digital platforms’ advertising-heavy business models, exploitation of intermediary market positions, and algorithmic optimization of non-price metrics that encode business objectives.

Under existing U.S. securities law, companies should disclose in their annual Form 10-K the material information investors need to understand their business model and its risks. For firms whose revenues and risk profile are substantially driven by algorithms, this reasonably includes decision-useful disclosure about how those systems operate, how they shape key performance metrics, and how their failures or biases create business opportunities or risks. But companies interpret their SEC disclosure obligations narrowly, ignoring these non-price factors despite them being material to shareholders. It makes sense they need prodding then. This is something Europe might consider leading on, especially if it’s only requiring disclosure of non-proprietary information that companies already track internally.

For Anthropic specifically, a monetization narrative would help explain:

• Fine-tuning and reinforcement learning optimization: What are they optimizing their models for at various stages? What risks are they seeing? What commercial incentives are being encoded into training? If Claude is optimized to encourage longer conversations or favor certain types of responses that lead to higher API usage, investors should know. If safety filters are calibrated based on commercial considerations – balancing user satisfaction against risk exposure – that trade-off should be disclosed.

• Ecosystem monetization: How does Anthropic monetize the broader ecosystem around Claude? This includes not just direct API fees, but potential revenue sharing arrangements, data licensing, enterprise support contracts, and preferential treatment of certain partners or integrators. When Claude chooses which tool to call, which retrieval results to surface, or which model variant to route a request to, it makes allocation choices.

• Indirect value capture: Beyond direct revenue, how does Anthropic capture value from the Claude ecosystem? This might include data from user interactions that improve future models, brand value from high-profile deployments, or strategic relationships that give Anthropic leverage in negotiations with cloud providers or content licensors.

These metrics and narratives would give investors and the public genuine insight into Anthropic’s market position, growth trajectory, and operational risks, rather than leaving them to guess based on occasional press releases or leaked information. Perhaps more importantly, it will set a precedent for other companies that are perhaps less civically-minded than Anthropic.

Where to From Here?

Much of what Anthropic should disclose if it goes public is already required under existing SEC regulations – no new rules are needed. What’s needed is clear guidance on what is material enough that omitting it would trigger consequences. The S-1 preparation process offers exactly this mechanism: since companies submit a draft to the SEC, which then provides comments and can object to omissions before the company goes public.

More broadly, changes in corporate disclosure norms can occur through either the listing side (the SEC or ESMA in Europe) or through the accounting side. The most important step is greater emphasis on mandatory disclosure of operating metrics: including through a monetization narrative and algorithmic disclosures. The logic of segment reporting provides a reasonable basis for firms to disclose externally the operating metrics they use internally when optimizing their operating segments (business lines) and making resource allocations.

In a recent working paper, we proposed baby steps the SEC could take to bring AI-specific matters under its disclosure regime, including issuing disclosure guidance to clarify what counts as material AI activity or risk, and where it belongs in 10-Ks and 8-Ks. We proposed dedicated AI-incident reporting modeled on the successful 2023 cyber rule.

But enforcement of existing rules matters as much as new guidance. The SEC’s comment process on S-1 filings gives regulators leverage to push for comprehensive disclosure before companies go public. This is the moment to establish what “material” means for AI companies – not through abstract rulemaking but through concrete expectations applied to Anthropic’s IPO filing.

Opportunities within the EU are even greater, where disclosure traditions in the Digital Services Act (DSA) could be integrated with IFRS accounting and listing requirements.

Anthropic doesn’t have to wait for any of this, though. It is already a Delaware public benefit corporation with a stated mission of developing advanced AI for the long-term benefit of humanity, and it could choose to retain that structure if and when it goes public. As such it could choose to lead by voluntarily disclosing investor-relevant information and making it easier for everyone to understand the complex and risky business of building AI models. Voluntary disclosure would establish best practices, provide Anthropic credibility with regulators and the public, and pressure competitors to follow suit.

We look forward to reading its prospectus — if it does end up going public. And we are sure OpenAI will too. Will it be the season to be jolly?

He is the lead author of “Behind the clicks: Can Amazon allocate user attention as it pleases?” (December 2024) published in Information Economics and Policy with Tim O’Reilly, Mariana Mazzucato, and Ilan Strauss.

One often hears the refrain from AI optimists that “today’s LLMs are the worst that they’ll ever be.” Certainly, they have some compelling reasons for positivity: The technology is still very young, R&D spending is through the roof, and over the last couple of years we have seen rapid advances. Things can only get better, right?

Well, no. In fact, I think there is a reasonable case to be made that today’s LLMs might be the best that they will ever be. The technology might continue to get better, but that doesn’t mean that the user experience will.

Why? Primarily because no one knows how to make LLMs profitable.

A Case Study

Take AI shopping assistants, for example. The other week, I wanted to buy a candle. Admittedly, I have a somewhat pretentious taste in candles, so I didn’t just want any candle. I wanted one that smelled somewhere between pine forest and old bookshop. This, I hope you agree, is a task in which you would have no chance of success in a million years using Amazon search. With ChatGPT, though, life was a breeze. I simply described the smell (as I just did), and ChatGPT spat out five good looking recommendations. I bought one, and it’s now the highlight of my Sunday evenings.

As embarrassingly ostentatious as this example might be, it demonstrates the power of LLMs in the product discovery space. Others have had similarly positive experiences. Searching for products online at the moment is hard. Amazon will show you more than 60 options on a search page; Google likewise is overflowing with visual clutter and ads. For products that aren’t my regular purchases, or known brands, I’ve found myself trusting advice from Reddit and YouTube reviews most of the time. LLM search isn’t perfect (i.e., they make stuff up, give you bad links, etc.), but having been trained on the collective judgement of the internet, they can quickly distill all the quasi-wisdom from Reddit, trusted blogs, and YouTube into a few recommendations – even for pretty complex searches. That’s incredibly valuable.

Share

Why this Might be the Peak

OpenAI reportedly loses money on even its most premium ChatGPT subscriptions. Jeff Bezos has openly called AI spending a “bubble.” And Peter Thiel is taking money out of the AI market. You don’t need to be an oracle to know that AI spending is unsustainable and risks morphing into a financial crisis. There is, therefore, an enormous and increasing amount of investor pressure for AI companies like OpenAI and Anthropic to figure out how to monetize their chatbots. (Anthropic’s emphasis on business revenue via its API may see it go into the green quicker than OpenAI, but for now these are all just projections.)

Yet the lack of profits at these AI companies has an upside for the user — at least for now. Searching for things like candles on ChatGPT is so unusually pleasant at the moment — particularly compared to Amazon and Google — precisely because its search function has not yet been warped by the pressure to generate the kind of profits needed to recoup past losses. However, if these AI tools are to survive financially, the economic reality is that they will have to change.

AI meets Rents

Consider the business models and associated economics rents that are being disrupted by AI. E-commerce giants like Amazon and Google extract billions of rents through advertising. Amazon has recently discussed the fact that users take, on average, 14 clicks before purchasing something on their site. One might think that is an unforced error, given that part of Amazon’s job description is surely to make finding products as quick and easy for its users as possible. But when you realize that each of the 14 clicks that go to product exploration means more potential clicks to Amazon’s high-margin advertising results, it starts to make sense.

Users suffer too. Since these platforms can only show you so many products at a time, every advertising result means one fewer organic (i.e., maximally relevant) result to choose from. Our own research with Mariana Mazzucato, published in Information Economics and Policy (December 2024), showed that the most clicked on advertising (“sponsored”) results on Amazon are often 17% more expensive than their organic counterparts and one-third less relevant. In fact, one-quarter of product search results on the first page are adverts — half of which are duplicated as an organic result on the first page too! Talk about the death of consumer choice.

Now imagine instead a chatbot that cuts down those 14 clicks to, say, just two clicks before finding the product they want. That might be great for consumers who can stop paying these attention rents: as product search becomes less time and energy consuming. But that’s also a direct threat to Amazon’s $60+ billion revenue stream from third-party advertising on its website.

So incumbents have a choice: they can either try to prevent chatbots from cannibalizing their ad-driven platforms, or ensure that chatbots can generate equivalent revenue through other means — or they can create better products using AI. More ‘enshittification’ is, therefore, one option.

Enshittification, as coined by Cory Doctorow, is when a platform that once delivered value to its users and third-party producers, makes its hosted & recommended content progressively worse in order to allocate more value to itself. In a multi-sided context, this enshittification ultimately “comes out of the barrel of an algorithm”. We’ve seen this happen to Google Search: 10 blue links in the search results were replaced by a gazillion ads.

We’ve also seen this happen to social media sites where algorithms built “for you” – for sharing pictures, text and content – were replaced by algorithms built for the platform’s own interests – for engagement, mindless video scrolling, and consuming. Evidence from Meta’s recent court win shows that Meta transformed itself into a “discovery engine” in order to compete with TikTok for user engagement. Meta-owned Instagram, nominally a ‘social’ network, now shows only 5% of posts from friends; the rest are “nothing but unconnected videos recommended by an algorithm”, the judge noted.

Moreover, the ‘social’ feed is full of ads. Meta’s own internal documents from the court case show ads decreased user time on the platform by around 7% – they are a “tax on engagement”, Meta found. Yet Judge James E. Boasberg still decided that Meta’s ad load reflected rational consumers (no behavioral economics allowed here), competitive market forces, higher quality ads, and an overall higher quality user experience.1 The evidence seemed decidedly more mixed. Meta is able to constantly lie about its advertising efficacy and reach, according to reporting in the Financial Times. That’s because advertising markets are a black box.

So, don’t be surprised if LLM-based search results start returning less relevant products and more ads. But why stop there? If you are OpenAI, why not push a ‘curated handful of products’ to the user, designed to get them querying and engaging again? Of course, this can be marketed under the noble banner of “respecting users’ decision-making autonomy”, or something along those lines; but functionally, it is pernicious advertising at scale but with a new, chattier interface.

There’s one big issue, though. Current LLM performance in product discovery is dependent on the internet continuing to exist as a repository of current useful information. My candle search on ChatGPT was so good because the model could synthesize insights from people who actually tried the candle and who then discussed it online in a good faith attempt to help other consumers. But marketers and merchants are cottoning on. More and more subreddits, review forums, and enthusiast blogs are being injected with synthetic, Gen-AI produced chatter. The goal of these third-parties isn’t to persuade you directly – it’s to slowly poison the informational substrate that the LLM product search relies on, and then the LLM persuades you through the commercially-laced information.

So — it’s worth savoring what we have now: chatbots without behavior completely warped by profit incentivizes, training data largely untainted by adversarial manipulation, and monthly fees (subscription model) rather than an attention-driven economics. This might well be AI’s 10 blue links moment. But let’s not assume that it’ll last, let alone that it will get even better.

Does it have to be this way?

If LLMs do start to drift towards enshittification, is there anything we can do to stop it? Is it possible, under the current system of economic incentives for AI services to create value without extracting it?

Jimmy Wales, co-founder of Wikipedia, thinks so. His new book sketches a picture of how we can start to rebuild trust on the internet. Against the background of an ad and click-bait soaked internet, Wikipedia is a shining example of what is possible. To my mind it is almost certainly the single greatest feat of knowledge organization in human history. The numbers are mind-boggling. Since you started reading this article, more than 4,000 edits have been made on Wikipedia by an army of volunteers whose only incentive is the pleasure they take in trying to be helpful and factual.

Clearly, not every company can be a crowdfunded nonprofit. Nevertheless, we can learn some important lessons from Wales and Wikipedia.

For one, transparency – and in turn traceability and accountability – can be made part of the product itself. Wikipedia’s edits, discussions, and history are all publicly accessible. This openness makes the information on Wikipedia more reliable and trustworthy, and so consumed by more end-users. That Wikipedia is publicly modifiable gives it a sense of collective ownership. And it also shapes what economists call the market’s “structure”: how many firms compete and how they compete. Although it was Window’s Encarta who helped kill Encyclopaedia Britannica’s expensive book set,2 it was Wikipedia who showed that dominating a market on the internet need not mean exploiting it.

As Wikipedia shows, transparency doesn’t have to mean “explainable AI” or “sparse autoencoder”, or anything super technical, really. It can be a structural and economically-encoded transparency – one that is built into the market’s proper functioning, like on a stock exchange where trades are settled with complete counterparty information. For AI, transparency should cover things like: Where does the model’s data come from? How is it processed? Who benefits from this processing? How exactly is it monetized?

The impetus for AI’s product disclosure should ideally be the incentives that come from the market itself. In this way, transparency is not optional. And it reflects and supports the market’s dynamic functioning, rather than the ideas of someone in government (which may be right or may be wrong). In other words, the disclosure is directly tied to the information in the market itself – such as the logs (records) of an advertising exchange online indicating who won the bid, the price, and the auction type – rather than a regulator’s notional understanding of the market. (We explore the drivers of a market’s structure in a companion piece.)

But not everything can be a non-profit Wikipedia.

We think it is possible for companies to use AI to make more money and make their products better — including through improving the user experience. This is a win-win business model which avoids enshittification. Initial evidence from Amazon’s Rufus AI chatbot, for example, is that it led to a $10 billion increase in sales that would otherwise have not been there. Users could, once again, find what they wanted! Amazon estimated that customers who engage with the assistant during their shopping journey are 60% more likely to complete a purchase compared to those who do not. The trick is ensuring that incentives are aligned towards win-win business models, no matter what market structure emerges (competition or consolidation and monopoly).

Incentives may sometimes need a push from government. Government rules can help constrain corporate behavior, in effect deciding the boundaries of fair competition, and the pay-offs facing firms from different business models and business strategies. Minimum wages, for example, mean that firms cannot compete by lowering wages beyond a basic level. Minimum wages incentives firms to compete through investing in fixed capital (like factories and new technologies) to upgrade their productivity so that effective unit labor costs falls.

Similarly, “interoperability” requirements for digital markets – including for AI chatbots and user data – are one way to help ensure that firms compete through using technologies to improve the user experiences, rather than enshittify them. Interoperability requirements were central to the success of the famous break-up of AT&T, which forced the new “Baby Bells” to allow third parties to inter-connect to their network and offer rival telecommunication services.

Elsewhere, we have argued for more direct government disclosure requirements for Big Tech. In particular, governments ought to mandate corporate disclosure of Big Tech companies’ internal operating metrics used to measure and optimize their products’ performance – in-line with segment reporting and GAAP accounting principles. These non-price metrics — such as ‘monthly active users’ or ‘time spent on platform’ — increasingly shape a product‘s monetization, especially in multi-sided markets. This makes them vital to disclose alongside their financial metrics. Moreover, we argued that these metrics should be differentiated by product line, to get a sense of the firm’s market power and economic strength not only as an integrated whole but within each relevant “operating segment”. That way regulators can keep track of a company’s changing behavior in individual markets and step in where necessary – you can’t regulate what you don’t understand.

And finally, there is us – the users. We have to call out these systems when they fail, when they start feeling extractive rather than helpful. The idea that “today’s LLMs are the worst they’ll ever be” isn’t just wrong, it’s dangerous. It breeds complacency right at the moment when norms, business models, and standards are up for grabs in an immense great powers contest between warring companies.

Whether LLMs get better or worse depends less on technical progress than on political choices, the economic incentives we align and foster, and whether we can demand better of the companies that shape society’s welfare. The technology may get better, but only careful attention to disclosure and market design can ensure that society doesn’t get worse.