While I agree with Caputo’s rejection of the fantasy proposed by Séb Krier and others that personal AI agents could feasibly replace shared formal rules with millions (or billions) of bilateral negotiations, I disagree with the essay’s overall conclusion that AI is on a path to making government more effective, provided it is used to improve existing functions. The same tools that upgrade an agency’s capabilities can also be used to undermine the incentives of the people and firms who supply it with information, while leaving the incentives of the officials who make policy untouched.

In economic terms, Caputo has modeled the production function but left the objective functions out of the discussion. The future this portends is a much more active government that fails to become more effective, unless larger institutional changes are put in place along with the new technology.

The use and abuse of rulemaking

To better understand how a technology can improve a government agency’s functions while simultaneously undermining its performance, we need look no further than notice-and-comment rulemaking. The 1946 Administrative Procedure Act is the U.S. federal law that governs how agencies propose and establish regulations. Agencies are required to give interested persons “an opportunity to participate through submission of written data, views, or arguments,” and “after consideration of the relevant matter presented, incorporate in the rule a concise general statement of the rule’s basis and purpose.”

This process is known as “notice-and-comment rulemaking.” Submissions accumulate in the rulemaking docket — a public file that holds a proposed rule along with supporting technical, scientific and economic analyses compiled to justify it, and every comment received by the agency. Any agency that ignores a significant comment risks having its rule struck down by a court as “arbitrary and capricious.”

This process sometimes works well. When the Government Accountability Office surveyed 52 program offices spanning different agencies and policy areas, it found almost all of them reported comments producing at least some substantive change in final rules. In many cases these comments benefited from technical expertise and early timing. Writing a substantive comment used to be expensive, requiring in-depth knowledge and perhaps attorney time, so filing one screened for two things an agency cannot observe directly: whether the commenter has something at stake, and something worth saying.

But this also has downsides. Participation has long been dominated by the biggest regulated firms. Legal scholar Wendy E. Wagner has documented what she calls information capture, in which well-resourced parties flood dockets with technical material, raising the cost of effective participation for everyone else. AI changes who can afford that strategy.

Akerlof’s market for comments

A comment’s polish and technical detail used to be reliable proxies for the sender’s skin in the game and on-the-ground knowledge — informative because they were costly to fake. The risk today is that AI collapses the cost of production for every type of sender,1 so that an expert-sounding comment will no longer provide useful hints about who sent it or why.

Dockets are already being flooded with cheaply produced comments. An investigation by the New York Attorney General concluded that nearly 18 of the 22 million comments on the FCC’s 2017 net neutrality repeal were fabricated, and that was before generative AI made plausible comments practically free to produce.

When comments become cheap, agencies may categorically discount them, and the informed commenter’s incentive to invest in the process disappears. This is the logic of economist George Akerlof’s market for “lemons”: when buyers can’t tell good products from bad, they discount everything, which drives high-quality sellers out of the market, and the buyers’ skepticism becomes self-fulfilling.

If a docket no longer reveals even the limited information it once did about who is most directly affected and how, agencies will rely more on familiar firms and trade associations. The flood of AI-generated voices becomes background noise, while groups the agency already knows are disproportionately elevated. AI could theoretically help agencies locate affected parties who would never have thought to comment. But the agency must first want to hear from them, and the affected parties must be informed and motivated enough to participate, two factors that AI is not obviously destined to influence.

Who reads the comments anyway?

There is nothing in Caputo’s essay to suggest that AI will transform the core incentives driving regulatory agencies. In most cases, much of the substance in notice-and-comment rulemaking is worked out through politics and interest-group bargaining long before a proposal is ever published, turning the comments process into a kind of post facto legal insurance.2

To give an example, we can return to net neutrality. The FCC adopted rules barring internet providers from throttling or blocking traffic in 2015, repealed them in 2017, then restored them in 2024, drawing millions of comments in the process — including the fabricated ones described above. Yet the outcome tracked the party controlling the White House more than anything in the docket (until the U.S. Court of Appeals for the Sixth Circuit struck down the restored rules in early 2025).

A similar bias can be observed in agency findings. When the Obama administration tightened fuel economy standards in 2012, the government’s analysis found large net benefits to doing so. When the Trump administration proposed rolling the same standards back six years later, the new government’s analysis found large net benefits to doing the opposite. Who is right? When the expected impact of a policy flips with the administration, the analysis is following the decision instead of driving it. Many rules are issued without sufficient quantitative assessment. Even when agencies produce a relatively complete analysis, it is often to justify a choice that has already been made.3

In his essay Caputo claims that “AI could help make bureaucracy scrutable.” But this assumes the reasoning an agency displays formally is the reasoning it actually used, and there is little evidence that the administrative record has ever worked that way. Political pressure and bargaining shape a rule long before an agency writes formal justifications for it. For example, in one study of the EPA’s air toxic emission standards, industry representatives contacted the agency 84 times per rule on average in the pre-proposal stage.

Since a model that drafts flawless, reasoned responses makes the public-facing rationale cheaper to produce, and the true one easier to conceal, the likely result is a kind of scrutability theater. Polished arguments quickly become effortless, while the real decision-making happens off the record.

A major rule can take years to complete, and few of the people who write one are eager to see it undone. An agency whose objective is legal survival will use AI accordingly. Federal pilots for comment analysis hint at where this could end up: AI-generated comments answered by AI-generated responses. This pattern could occur across government. AI lowers the cost not only of producing comments, but also rules, guidance, permits, and enforcement actions. Agencies could deploy agents to monitor submissions and enforce penalties around the clock, but there is nothing to stop the firms they regulate from launching petitions and appeals with equal fury. All that activity, yet the political and legal forces that actually drive rulemaking remain undisturbed.

An awareness that AI-created material has saturated rulemaking does not guarantee that the process will be overhauled. In nearly a dozen jurisdictions where AI-generated submissions have already flooded government processes, officials have responded with measures like blocking suspicious traffic or dismissing submissions in bulk. They are coping rather than rethinking the procedures themselves. Broken processes persist in Washington as well. Congress has approved a spending package on schedule only four times since the modern budget process took effect in the 1970s. It hasn’t happened once since 1996. Yet the process limps on through the use of stopgaps rather than getting fixed.

Given the difficulty of passing major reforms, it is to be expected that a broken comment process would simply continue, even in an AGI world. But if reform does become politically possible, a number of options are available.

Reform should focus on judicial review and early engagement

The reforms with the best chance of turning AI’s capability gains into better government are those which harness the incentives agencies already respond to, like the fear of having rules overturned. Most agencies are not required to weigh the costs and benefits of their rules in any way a court will enforce. The analytical requirements that do exist usually come from executive orders rather than statutes. To start, Congress could codify analytical standards and subject them to judicial review. Doing so would flip the agency’s relationship to public input.

Today a rule survives review if the agency can show it responded to comments. This turns the docket into a liability to be managed where responses are boxes that must be checked. If a rule’s survival instead depended on the quality of the analysis behind it, the fear of losing in court would mean agencies start treating the docket as a way to get the best evidence they can on the record.

Rules based on weak economic or scientific evidence should be vulnerable in court. There is some precedent for this. In 2011, the D.C. Circuit vacated an SEC rule because its economic analysis was inadequate. The SEC then issued new guidance on economic analysis, and by one study’s measure, the average quality of the agency’s analyses nearly doubled afterward. The Supreme Court has also ruled that an agency acted unreasonably when it deemed cost irrelevant to a regulatory decision. But both cases turned on analytical requirements written into those agencies’ own statutes, duties most agencies don’t have. The executive orders that mandate analysis across the executive branch explicitly state that they don’t create legal rights, so an agency’s economic analysis can easily be thin or wrong without legal consequence.

AI-relevant reforms should also reach earlier into the rulemaking process where influence actually operates. Congress could require agencies to issue advance notice when they plan on making changes to major rules — a step that is optional today and frequently skipped. In one study tracking commenter influence across the life cycle of Transportation Department rules, the authors found that early commenters played an outsized role in setting the agenda. The advance-notice stage, in particular, positioned them to shape the content of future rules and sometimes thwart unwanted ones.

Early in the process, agencies should invite the public to submit data, economic evidence, and competing regulatory proposals — work AI now makes far cheaper to produce. Commenters already compete with one another, but they do this on advocacy terms. An open call for rival proposals and evidence would focus that competition on the quality of analysis instead. Once a rule has been in place for a while, agencies should be required to look back and determine whether it worked. They might even invite the public, again with AI support, to do the looking for them.

Projected costs and claimed benefits become testable once a rule has been enforced. This is markedly different from Caputo’s belief that AI will make the government scrutable. Scrutability requires agencies to explain themselves, but explanations can be manufactured. Retrospective review asks whether the world actually turned out the way the agency predicted, which is a question the agency can’t choose an answer to arbitrarily.

Sunset clauses in the age of AI

Sunset provisions mean rules expire automatically unless a legislature or agency actively renews them, and they supply both the occasion and the consequence for retrospective review. Repealing a rule today requires a rulemaking of its own, which makes removal expensive. A rule that automatically expires unless renewed will be expected to face its own record at each renewal. If it doesn’t pass muster, it lapses. Sunset provisions are not a new idea and have been used to great effect already,4 but AI makes it practical to apply them widely, because each renewal decision is better informed.

Consider a hypothetical workplace safety rule, originally projected to prevent a thousand injuries a year, which has an eight-year sunset clause. As renewal approaches, the agency is expected to put evidence on the record showing that the rule performed roughly as promised. Outside parties are encouraged to file rival assessments. If the promised benefits did not materialize, renewal fails and the rule lapses by default. If the agency renews anyway on the strength of faulty analysis, the renewal itself can be challenged in court. AI can assist at every stage: finding the data, running the comparisons, and evaluating the rival submissions.

Caputo is right that scrutability is not only something agencies supply. As he argues in the paper his essay builds on, AI also lowers the cost for the public, courts, and legislatures to investigate agency actions on their own. Oversight of that kind addresses part of the incentive problem, since agencies seek to avoid public embarrassment. But outside scrutiny changes agency behavior most when it is attached to consequences, which is what judicial review and sunset provisions supply.

Behind the ceremony

The diffusion of AI all but guarantees a more active government. Whether we get a more effective one, however, is a separate question, one which hinges on the transformation of government incentives along with the change in capabilities.

None of what I have suggested is bulletproof. Generalist human judges can be imperfect referees of economic evidence. Sunset provisions can decay into rubber stamps, and agencies that come to fear their dockets may retreat towards other subregulatory activities that are much harder to oversee.

One could argue that nothing stops economic analysis from becoming a machine-generated theater of its own. But at least an analysis makes claims that can be checked against the world. A court cannot tell whether an agency seriously engaged with a comment, but it can tell whether the data behind an estimate exist or whether a forecast came true.

More than three decades ago, legal scholar and former EPA general counsel E. Donald Elliott observed that “notice-and-comment rulemaking is to public participation as Japanese Kabuki theater is to human passions — a highly stylized process for displaying in a formal way the essence of something which in real life takes place in other venues.” In the near term at least, notice-and-comment and similar sclerotic processes are likely to survive AI diffusion. If they do, their persistence will point to a broader problem, which is that institutions tend to evolve much more slowly than the capabilities they govern.

Both sides see that their values are losing, and each blames the other.

I want to suggest an alternative diagnosis: no one is winning. What’s guiding these institutions isn’t anyone’s values at all, but purposeless busywork. A kind of white noise.

This condition can coexist with instances of factional capture. For instance, there may be some university professors who squeeze social-justice values into the curriculum. But ultimately, professors don’t run universities. They don’t set tuition or manage admissions. And those who do — do they work for leftist values like affordability? Access for the poor? Benefit to the public? Or do they optimize for U.S. News & World Report rankings? The left may have made some gains, but universities don’t really serve left-wing values, or right-wing values either.

The same is true when governments swing to the right. Do they then actually advance right-wing values like freedom, personal responsibility, and smaller government? Or do they optimize more for re-election, media hits, or the interests of donors? Whatever captured the government, it’s not exactly right-wing values.

Clearly, some other force is in play. I call this force ‘Value Drift’. And it extends far beyond politics: the same pattern appears across institutions of all kinds.

You might think this sounds like Goodhart’s Law1, or Cory Doctorow’s enshittification, or C. Thi Nguyen’s value capture, or just the familiar principal-agent problem of economics. All of these are relevant, but they don’t quite capture the phenomenon I’m describing. Goodhart’s Law by itself does not explain why a proxy becomes embedded in an institution, or why the resulting arrangement can remain stable after the mismatch is widely understood. And as we will see, enshittification is only part of the story: it doesn’t explain cases where no one is extracting value and yet the system still converges on the wrong thing.

Below, I’ll provide new models for these problems, covering why some institutions recover from value drift and others don’t. There are several dynamics that can lock drift in place. For instance, in some cases an organization will use a proxy metric, people will change their behavior to succeed by it, and then the institution adapts to that behavior in ways that further entrench the proxy. I’ll also cover how powerful AI could speed up these dynamics and make them harder to reverse.

To explain the institutional misalignment we’ve observed recently, let’s start by pinning down when exactly value drift gets out of hand.

Origins of Value Drift

There are three main ways a gap opens between what an institution says it values and what it actually does:

1. There can be inarticulacy in the measurement system. The institution’s data collection may just not track the values they purport to serve.2 A teacher can recognize moments of learning — say, when a student asks a question they couldn’t have formulated a week ago. But chances are those observations don’t make it to the dashboard the school runs on, because dashboards usually feature standardized, quantifiable signals.3 If the institution can’t see what matters, it will optimize what it can see.4

2. There can be self-scoring by those who set the metrics. Managers and employees want to report success. Their bonuses depend on it. If they can choose what success means, they’ll pick metrics that are easy to improve, so that algorithmic changes or product features can move the needle. Increasing time on site, or signups, may be easier than increasing real benefits.

3. There can be capture. Control of the institution can fall to a faction whose aims diverge from the institution’s purpose: for example, when a regulated industry comes to dominate the agency meant to oversee it.5

   The institution then runs on the captors’ interests, whatever its charter says.

While inarticulacy, self-scoring, and capture happen all the time, institutions often course-correct, for instance by voting out the cronies or revising the metrics. But course-correction depends on at least one of two things: real-world consequences or values-driven people.

• Real-world consequences. An institution whose bottom line depends on delivering real value will get bitten by reality if it drifts far enough from that value. Eventually, failure shows up in lost business, lost elections, or visibly worse outcomes. But it’s rare that exposure to consequences lines up perfectly with the organization’s mission. Consequences only correct drift when the institution’s feedback channels are congruent with its purpose.

• Values-driven people. Sometimes value drift is checked by the people inside an organization who actually care about its purpose. A hospital may push to cut costs, but if doctors still feel moral weight and a sense of responsibility, and if they can see the effects of their actions, they’ll push the organization back towards what it is meant to do.

In some institutions, real-world consequences and values-driven people help clarify values over time. The institution learns more about what really matters, re-aligns incentives, and re-engages with its mission on an ever-deeper level.

When People Can’t Correct the Drift

This restorative process doesn’t seem to be happening as often as we’d like. In the corporate world, product teams optimize for engagement that doesn’t track user value while sales teams chase quotas that ignore customer satisfaction. Parallels abound in academia (citations), journalism (clicks), education (certifications), medicine (throughput), and politics (poll numbers).

What’s going wrong? One factor is the bureaucratic insulation of decision-makers, which makes them feel less responsible for their decisions and less able to see their effects.

Historically, many institutions were smaller and more local. They had fewer layers of management and often more contact with those they served. In those smaller, local settings, you couldn’t hide behind procedures. Your actions were visible to you and your neighbors. And your reputation depended on delivering real value to the people around you.

But organizations scaled. Enter what we might call the company man — someone who hides behind procedures, is blind to real-world effects, and lives insulated from outcomes.6 To the extent that modern institutions are staffed by people or systems operating this way, they are more vulnerable to value drift and less able to reverse it.

When Consequences Can’t Correct the Drift

An institution may receive strong, rapid feedback, but about things other than its purpose. I’ll call this skin in the wrong game: the institution is highly exposed to consequences, but not the right ones.

It’s worth calling out three types of ‘skin in the wrong game’.

First, sometimes proxy metrics get written into formal contracts and legislation. Those running a school may know test scores aren’t the same thing as education, but federal funding, teacher evaluations, and parent expectations can still all depend on them.

Secondly, the law often makes an organization answerable to particular stakeholders rather than to its actual purpose. This can lead the organization to protect those stakeholders’ interests, even when they diverge from its mission.

At least in these cases, those involved can usually point to the gap between the proxy and the actual value. In the first case, the problem reduces to coordination: how to get all parties to renegotiate their contracts, replace the metrics, or change how the organization is evaluated. In the second, even if the law is on the wrong side, there’s often a broader constituency that can demand change, or boycott, or otherwise exert external pressure.

A third case is trickier — when the people the institution is meant to serve also take on skin in the wrong game. This means the ‘strategic equilibrium’ shifts until all parties come to depend on the drifted state. I call this a value substitution loop (VSL).

It’s easiest to show with a stylized example:

A platform launches to help educators share videos with online students. An educator joins to teach. The platform measures student engagement through watch time and comments.

Content that drives these metrics does well. The educator notices this and adapts, perhaps simplifying her takes, because to teach at all, she needs an audience. As thousands make this adjustment, the platform updates its model of what educators want. It sees that they are chasing engagement, and builds features to help them do that better.

As the platform becomes better for chasing engagement, new entrants arrive with their eyes set on this. The role of “educator” has been redefined: being an influencer has become a prerequisite to teaching. Even people with exactly the same values as the original educator try to become influencers first.

A proxy value (student engagement) has displaced the original one (education) through strategic adaptation by both participants and management. As participants adapt to the proxy7, the institution adapts to serve them; as the institution adapts, participants adjust further. The result is a stable equilibrium, but one that serves nobody’s values. It’s not what the educators wanted. It’s not what the students wanted. It’s not even what the platform founders wanted. But no actor could revert to the original values without losing standing in the proxy-optimized system.

What distinguishes a value substitution loop from ordinary perverse incentives is that the institution must have some mechanism (algorithmic, bureaucratic, or market-based) that aggregates participant behavior and feeds it back into institutional design. Such a system reaches a tipping point once too many participants switch to the proxy-optimized strategy and the institution adapts to serve them.8 It tips more quickly when (a) competition among participants is intense, (b) the institution adapts quickly (e.g., through algorithmic feedback), or (c) the proxy is far from the original value.9

Irrevocable drift is most likely in competitive, fast-moving, and hard-to-measure fields.

Social media is a well-known example, but others abound.

Take academia, where researchers optimize for citability via trendy topics and provocative framing as they compete for tenure and grants. Hiring committees see what gets cited and start favoring papers that generate buzz. This in turn becomes what “good work” looks like. New PhD students are trained to recognize and produce that kind of work, reinforcing the equilibrium.

Or take arts funding. Grant criteria in Germany’s experimental music scene highlight markers of “seriousness” like political framing (anti-colonialism, representation) or association with canonical avant-garde forms (free jazz, noise, electroacoustic composition). Artists present their work in these terms. Then funders converge on those criteria even more strongly. The drift is especially ironic as “experimental” gets redefined to mean preserving idioms that were transgressive fifty years ago.

We can contrast this with Doctorow’s enshittification, which blames drift on value extraction by platforms. For Doctorow, enshittification is driven by monopolies extracting profit from trapped consumers, so the cure is increasing competition (via antitrust, interoperability, the right to exit, etc). But it’s hard to apply Doctorow’s “enshittification” story to German experimental music, journalism, or academia. These are places where competition is savage, and they are better described as VSLs, where intensifying competition among participants or between institutions makes it more likely that the system will tip into a proxy-value well.10

A Delicate Balance

The institutions we have survive on a rough equilibrium among the forces in Figure 1. Inarticulacy, self-scoring, and capture push toward misalignment; real-world consequences and values-driven people pull back; insulation from consequences, skin in the wrong game, and value substitution loops weaken those restoring forces.

AI agents will change the strength of each of these forces, and on current trends, mostly in the wrong direction:

• Organizations with business models that actively conflict with their purpose would probably hire more company men if they could. But most human beings have a conscience and want to actually deliver on the purpose, not on misaligned metrics. That keeps many organizations more aligned than their incentives alone would predict. AI agents could be fine-tuned to only care about the metrics, and from the institution’s perspective, this might look like an improvement.

• Previous waves of information technology have embedded simplistic proxies into infrastructure and APIs: customer service reps measured by tickets closed or handle time, ads sold by clicks, and so on. AI agents could deepen this tendency, further entrenching skin in the wrong game.11

• Value substitution loops become more likely as competition intensifies (among both platforms and participants) and as institutions adapt faster to what participants are doing.12 AI could accelerate both of these dynamics.

But AI could push in the other direction too. Models might put values-driven judgment back into systems where decision-makers have lost sight of consequences. They could make it possible to write contracts around thick, qualitative terms like “learner benefit,” rather than thin proxies. They could notice when a metric has become detached from the reason it was introduced, lowering the cost of correcting it. And by making real value more legible, they might help prevent value substitution loops from taking hold.

The Human Cost

Value drift damages our society. It degrades the researcher chasing citations and the Instagram influencer living a lie. An injury is being done to them, and to all of us.

As Wolf Tivy put it:

When I look at the things my friends were into before they destroyed themselves, this is what I see: false value sold to them by institutions and subcultures that have no structural reason to care about their real interests. But this applies to far more people than just the few that didn’t make it. Almost everybody is trapped in some kind of propaganda complex, wasting their lives working for effectively nothing.

Life gets redirected away from what matters, toward what’s easy to measure, easy to game, and easy to hide behind. If, as I mentioned at the beginning, both liberal and conservative values are losing, this is what’s gained power: busywork, nonsense, “false value,” and purposelessness.

We must not allow it to get any worse.

Can we build institutions that stay connected to purpose? That measure what matters, keep decision-makers close to consequences, resist capture, and avoid value substitution loops?

I believe we can.

Thanks to Max Kroner Dale, Joel Lehman, Séb Krier, Ryan Lowe, Oliver Klingefjord, Rachel Calcott, Toby Shorin, Ivan Vendrov, Philip Tomei, and Richard Ngo for comments, and Jamelle Watson-Daniels for generative discussions.

Bureaucracy is everywhere, from continent-spanning governments to rapidly growing corporations. As German sociologist and jurist Max Weber argued, and as we’ve learned in the century since his death, technological progress demands more bureaucracy to handle the information problems of an increasingly complex world. It also supplies the tools that allow bureaucracy to operate at greater scale.2 Yet this growing dependence is easy to overlook, because the work itself is profoundly unglamorous.

Let’s be honest: bureaucracy is boring. Who wants to spend time thinking and arguing about what constitutes a “stationary source” or a “security,” or researching and deciding how much fecal matter is acceptable in food or lead in paint or any of the other mundane and specific tasks that make up modern government?

More than a century ago, Weber described bureaucratized modernity as an “iron cage”: a world so ordered, regulated, and strictly administered that no feeling person could escape it.3 But isn’t AI supposed to free us from such a cage? Won’t AI render these massive structures unnecessary, replacing them with more direct, flexible, and personal forms of government?

I highly doubt it. Setting and enforcing predictable rules is part of what makes markets and democracies function in the first place. Even highly intelligent AI agents will need settled categories and shared information on which to operate, negotiate, plan, and act. Bureaucracy has been with us since early antiquity.4 It will probably follow us into the AI future. But that is not necessarily a bad thing.

I’m not saying don’t worry about the government of a bureaucratic character alien to our system. I’m saying that bureaucracy is already serving us more than we give it credit for, and with AI, or AGI, it could be much better still. The real question is not whether there will be bureaucracy in the AI future (there will be), but how AI could transform bureaucracy and make it more capable, more accountable, and more responsive.

What even is bureaucracy?

Weber described bureaucracy as a system of government defined by hierarchy, expertise, continuity, and impartiality, one which operates by the development and application of rules to defined jurisdictional areas.5

Take as an example the U.S. Food and Drug Administration. The FDA is a hierarchy of review divisions, each reporting up to a director who heads one of the agency’s centers. Each division is staffed by pharmacologists and statisticians with expertise, who outlast any administration, and have jurisdiction over drugs and biologics (but not pesticides, because those are covered by the EPA). They apply impartiality through rules like the bioequivalence standards that determine whether a cheaper generic can be used in place of a brand-name drug.

Weber argued that bureaucracy is to other forms of government as machines are to artisanal production. Its benefits are precision, speed, clarity, continuity, and efficiency, as well as the ability to handle novelty and complexity, though it comes with the loss of personalization as its cost.

Of course, an individualized judgment is often more accurate (unless it’s biased or corrupt), but one-off judgments are expensive, inefficient and difficult for others to forecast. Rules are crude and create false negatives and positives, but they are cheap to apply and everyone can plan around them. These are important qualities in modern life. Anyone who has plugged their computer into a standardized outlet in a foreign country can understand the virtue of predictability at scale. When millions of actors need to know today what a decision-maker will do tomorrow, a broadly applied rule has far more to offer than a slow and mercurial sage.

AI could shift the boundary between standardization and personalization, making individualized judgment cheap enough to apply at scale. But a world of swarming agents could also multiply interactions so fast that the balance tips back toward predictable rules.

The case for bureaucracy

Bureaucracy has grown massively in the past century because it solves three recurring problems of governance. The first is predictability through the rule of law: treating like cases alike creates the settled expectations on which institutions and individuals depend. The second is cheaper coordination through hierarchy: standing relationships, defined roles, and command powers avoid the cost of renegotiating every decision. The third is insulated expertise. Some technical decisions should be protected from both electoral pressure and market bargaining. We do not want drug-approval standards set by plebiscite or auction.

Behind all three is the question of how we deal with complexity. As the social world has grown larger, people have specialized. Delegating technical decisions to experts makes the process of decision-making far more accurate and efficient than demanding that members of the public express preferences through voting or market signals.

But specialization creates problems of its own. How can a non-specialist know that a specialist is acting in good faith? Bureaucracy’s answer is to bring the specialists under one roof and bind them to rules. That way they can oversee each other’s work, build mechanisms for review and appeal, and catch non-technical kinds of cheating like favoring friends or special interests by comparing one official’s output against the organization’s.

When bureaucracy is done right, it can handle immense complexity and scale with great consistency. For example, each month the Social Security Administration sends benefits to more than seventy million people with an error rate most private insurers would envy.6 Oversight is carried out through internal appeals, inspectors general, records that force reasoning into the open, and courts which police the edges of delegated power in an attempt to curb abuse.

Consider in parallel the rule-making apparatus of the United States government that on average issues three to four thousand final rules from its agencies every year. Millions of administrative adjudications apply those rules across every part of the economy and the full breadth of society.7

These numbers are often cited as examples of bureaucracy run amok. But what is the alternative? Congress cannot pass thousands (or even hundreds? tens?) of laws. Federal courts cannot handle their existing workloads, let alone the millions of other cases they would have to address if agencies didn’t do it for them. Both the legislature and judiciary lack the technical expertise to correctly decide these cases. How could Congress quickly learn enough to decide whether a specific cutting-edge drug should come on the market, or whether the national ambient air quality standard should be changed, among a million other things?

As I argue in a forthcoming law review article, the administrative state came into being against significant societal and judicial resistance because it was necessary, and it is even more necessary today. Complexity is not going away, and neither are the three rationales of bureaucracy that complexity calls forth: predictability, cheaper coordination, and insulated expertise.

Why agents can’t just “handle it”

Some people have argued that as AI gets smarter, AI agents could bargain or deliberate on our behalf and remove the need for bureaucracy. I think these arguments either presuppose the bureaucracy they claim to eliminate or implicitly envision a world in which people are disempowered by their AI agents. After all, democratic deliberation and bargaining within the market happen in the context of set categories and according to set rules.

As British economist Ronald Coase acknowledged in the 1960s, efficient bargaining is possible only when property rights are established up front.8 But property rights do not exist outside a system of government, they are established by it. And the relevant property rights are less frequently “who owns this plot of land?” and more frequently “what is the legal nature of this ownership share of a company?” Rules like that are produced by bureaucracies such as the United States Securities and Exchange Commission. Next, the SEC enforces those rules, which provides a basis for the market.

Imagine if every time you tried to buy a share in a company you had to negotiate with that company to determine what a “share” means. Or imagine buying food in a world in which negotiation forms the basis for food safety. Are you supposed to develop a preference for how much arsenic you’re willing to have in your cereal? How much lead? How many insect fragments? If you do, who do you negotiate with? The grocery store? The wholesaler? Are supply chain companies supposed to negotiate with each other to determine which products along the spectrum of safety to supply at which prices?

We tried something close to the market approach to these questions once. It gave us the rotten and adulterated meat described by Upton Sinclair in The Jungle, and a public that quickly demanded food inspectors.9

Similarly, if the aggregation mechanism is voting, are people supposed to determine for themselves what a “security” means and then somehow deliberate collectively on that to arrive at a consensus definition? Even experts often don’t know what these things mean and have long debates with one another to figure out what’s best.

“Why can’t we just let the agents handle it?” you might say. A sufficiently capable agent could negotiate purchases, represent your preferences, and spare you from forming an opinion about the permissible arsenic content in breakfast cereal without involving a third party.

But the agent still needs settled rules. It needs to know what a share is before it can buy one, who owns what before it can bargain, and which agreements other agents and institutions will recognize. A world of perpetual agent-to-agent negotiation is a world of unnecessary and costly contracting. This is not to say that agents shouldn’t be involved in resolving technical questions. Let them investigate the evidence, make consistent decisions, and insulate those decisions from preferences you do not want to develop moment by moment.

Now look again. An entity trusted to decide technical questions consistently, on the basis of specialized knowledge, so that you do not have to decide them yourself?

Well, that is a bureaucracy.

Furthermore, an AI that completely understands you and acts perfectly on your behalf is an agent that has disempowered its principal. That disempowerment might be beneficial to the principal, assuming actual congruence between agent and the principal’s interest, but it is a world with more complete control than even the most highly-bureaucratized societies today. It carries a cost to democratic legitimacy: an agent that thinks for you is incompatible with an informed citizenry.

The goal, then, should be a hybrid world in which human preferences are gathered and represented with much greater fidelity than they are today, without replacing human judgment altogether. But even in that world, bureaucracy — with its consistency and predictability, its ability to operate in technical or scientific domains through specialization, and its long history of incorporating and making the best use of technology — will remain core to the institutions of the future.

The limits of bureaucracy

Assuming that I’m right and that bureaucracy will always be with us, what might the future of AI administration look like? Here we might briefly consider some of bureaucracy’s current pathologies to see where AI might step in, and also where we might get stuck if we don’t think hard enough about how this technology is being developed and implemented.

The failures of bureaucracy will be familiar to anyone who interfaces regularly with a government agency. Everything takes too long and costs more than it should. The companies an agency is supposed to police often end up steering it — something which goes unnoticed because the influence is buried in paperwork nobody reads.

Agencies can barely change anything, even their own outdated rules. Every check we put on them slows them down so that changing one rule can take years. When they do make a decision, a form letter says no and doesn’t explain why. The rules tend to be written for the typical case, so if your case is unusual, they reject you.

Also, bureaucracy can be myopic. The late political scientist and anthropologist James C. Scott was the most prominent exponent of the “legibility” critique of bureaucracies: the idea that government forces society into governable shape, stripping it of much of its richness. His argument was already present in Weber, who argued that bureaucracies, public and private alike, force the simplification and rationalization of the world through their need for consistency and reliance on paperwork.10

The loss of texture and specificity is the price to be paid for scale when perceiving the world and acting on it is costly in terms of attention, time, and cognition. It’s not unlike the way the individual human brain ignores almost all sensory inputs so that it can focus on what it deems important. It is a miracle of sorts that we can run societies as complex as ours using the highly lossy interfaces of bureaucratic institutions, and it is only through continual technological upgrades that we have managed to scale this far.

Even though I’m primarily defending bureaucracy in this essay, now and in the future, reform is certainly needed. Existing efforts from state capacity advocates and modernizers across the political spectrum are bearing some fruit11, but as the AI rollout continues there is also a real chance that our agencies will fall further behind the technical frontier, much as many of them failed to reinvent themselves for the internet age.

Many of the obstacles to reform are well-intentioned limits designed to prevent the arbitrary and abusive exercise of bureaucratic power, like many of the barriers of administrative law that have led to ossification. We need reform that makes bureaucracy both more capable of solving our problems and more accountable to people and their representatives, escaping the trap that has meant capability and accountability trade off across time. It may well be that AI can help us get there.

Bureaucracy, rebuilt

So how could we build a better kind of bureaucracy, one suited to this new technological moment we’re facing? And how can AI help us to do it? Recall the three rationales of bureaucracy: predictability, cheaper coordination, and insulated expertise. Each can be served by frontier AI if implemented correctly.

What might an AI-powered bureaucracy look like? The basic functions of bureaucracy, as Herbert Simon reminds us, are cognitive.12 An organization is established with a specific mission toward which it is directed, like reducing air pollution. It observes its environment, seeking opportunities to achieve its mission, like an industrial sector whose factories emit pollutants. It gathers information, classifies the world into administrable categories, applies rules and makes decisions about those categories, and observes what it has done so that it can learn from its decisions.

Bureaucracy is society’s system for perceiving, reasoning, deciding, and remembering at scale. Frontier AI could change each part of that system. It can help agencies see more of the world, with less delay, less distortion, and in more detail, by processing administrative records, scientific studies, complaints, market data, inspections, and other inputs that currently overwhelm human officials.

It can help agencies reason across that information by identifying patterns and surfacing tradeoffs, operating across different assumptions, and testing whether proposed interventions are likely to actually advance statutory goals.

It can help agencies better explain themselves to the public and their overseers in courts and legislatures, not only in the formal language of the Federal Register, but in terms that affected parties can actually understand. Instead of someone denied welfare getting a form letter saying they don’t qualify, they could get a bespoke explanation that helps them understand what they need to do on appeal.

Lastly, it can help agencies remember, by preserving a more complete and searchable institutional memory of what has been tried, what failed, what worked, and why.

The scrutable state

Bureaucracy alienates because it simplifies the world while remaining obscure, clouded by technical language and burdensome procedure. The people are made legible to the state, but the state is not made legible to the people. Perhaps most importantly, AI could help make bureaucracy scrutable.

Whether deployed as a personal agent or by a government agency, AI could help a person denied disability benefits, or a business subject to enforcement, a member of Congress overseeing a technical agency, or a judge reviewing a complex informal rulemaking. In each case the user is able to ask in ordinary language: what happened here, what mattered, what assumptions drove the decision, what alternatives were considered, and what would have changed the outcome?

This only works if AI is itself governed bureaucratically — that is, consistently, efficiently, and according to the rule of law. Technological improvements in bureaucracy have always been paired with innovations in oversight necessary to maintain accountability in these institutions. AI will be no exception.

Most of the failures mentioned earlier are failures of oversight, and AI drastically changes what oversight can be. Capture is an issue because it is hidden and costly to root out. But when a system’s reasoning is documented and queryable, influence becomes traceable, and traceable influence is checkable influence. Ossification builds up because, until now, checking an agency usually meant slowing it down, by requiring additional procedures or approvals. AI systems could instead be continuously audited, with their decisions sampled for bias or outside influence. That makes it possible to check the agency without placing a brake on every individual decision. And opaque decisions could be eliminated when every determination carries its reasons with it.

None of it happens on its own. We need a new administrative law for this new kind of administration, a new type of record that documents what a system is for, what data is used, how it was evaluated and monitored, and what failure modes have been found. Judicial and legislative oversight could be built on the budding science of AI auditing and evaluation, creating a new form of accountability built for AI.

Bureaucracy in the future

Nothing I’ve written here should suggest I want to replace bureaucracy with AI or paste chatbots onto existing agencies and call it modernization. AI offers the chance to build a vastly more capable and more accountable bureaucracy, capable of handling complexity and avoiding much forced simplification, acting at scale without becoming arbitrary, explaining itself to affected parties and the public. The future will still be bureaucratic, maybe even more than today. But if we get it right, the iron cage Weber warned of becomes a scaffold rather than a cell.

Paine wrote this at the beginning of a great age of institutional invention: the era of constitutions, republics, and rights. Powerful AI2 may be opening another such age, giving us the need and the means to begin again.

By institutions, we mean the rules, roles, procedures, incentives, and organizations that structure how people coordinate. This includes familiar public institutions like courts and elections, but also venture capital, standards bodies, religious orders, and basic things like contracts.

All of this is up for reconsideration in the AI age, because institutions will change to accommodate new kinds of participants and because there are new possibilities for what the institutions themselves can do. AI agents differ from humans on axes like speed and replicability.3 Institutions will at minimum have to account for that, but they will also inevitably integrate AI into themselves, changing how they process information and enforce rules.

It is often remarked that we stand at the dawn of a new industrial revolution.4 We should remember that the first was institutional as well as technological. Industrial production brought us the limited-liability corporation, the regulatory inspectorate, the labor union, and mass public schooling.5 These institutional changes were as consequential as the technological ones.

Despite growing recognition that powerful AI may transform society on a comparable scale, we are not yet engaged in a comparable practice of institutional design. Most of our collective attention remains focused on aligning, regulating, accelerating, or slowing the AI systems themselves, while treating the surrounding institutional infrastructure as fixed. Proposals for new institutions are scarce, scattered across fields, and rarely subject to sustained public debate.

Pax Machina exists to make this work concrete and cumulative. It is a publication for researchers to propose institutions suited to a world of humans and powerful AI. We are also excited to publish critiques and counterproposals that test and refine these ideas.

New governance structures will emerge whether we plan for them or not. Pax Machina brings that process onto the drafting table, turning the architecture of a world with powerful AI into a deliberate, collective project. Our ultimate aim is to foster an abundance of institutional forms that individuals and communities can adapt to fit the lives they want to lead.

AI changes the institutional landscape

Our political stability depends on human limitations. A president cannot personally direct every department, monitor every official, or execute every decision at once. Bureaucratic delegation and delay therefore constrain executive power. But imagine a president able to command thousands of perfectly loyal AI delegates, overseeing the entire machinery of state continuously and acting far faster than courts or legislatures could respond. The words of the Constitution wouldn’t change, but some of its practical checks and balances would stop working. We might then need faster oversight, or entirely new constraints on executive capacity. Researchers have catalogued dozens of such failure modes.6

AI also changes what institutions themselves can do. Voting, for example, reduces a citizen’s complex worldview to a small number of multiple choice questions, partly because listening to millions of people to understand what they care about, then reconciling their competing concerns, has historically been prohibitively expensive. Recent research suggests that language models may make richer forms of large-scale deliberation possible: a model can speak with participants and help identify policy proposals that command far broader agreement than a fixed menu of options.

Taken together, the need to govern agents alongside people and the new possibilities for institutional design suggest that many things will likely change: court systems with AI judges, media with AI context and factuality, international negotiations at AI speed, and more.

Some of these new institutions could be far better than the status quo; others, far worse. On the one hand, we could have unprecedentedly responsive representation; on the other, hyper-personalized political manipulation. The same ubiquitous monitoring could finally make public institutions accountable, or turn into an airtight surveillance state. The same AI agents could either help people clarify and pursue their considered goals, or gradually displace human judgment, leaving people unable to understand or steer the systems around them.

Institutional change has happened before

All the institutions we rely on, from corporate law and property rights to government agencies and courts, were devised in response to particular problems and conditions.

Some were created under stress, like the post-Soviet market institutions that handed Russia over to oligarchs.7

But many of our most enduring institutions were designed thoughtfully through debate and accumulated experience. A prominent example is the United States government. The founders publicly examined how their new republic might fail and built checks and balances in response. They also had a decade between the Declaration of Independence and the Constitutional Convention to study what worked and what didn’t under the pre-federal state constitutions.8

For example, Pennsylvania’s 1776 Constitution (which Paine supported, as it happens) had one legislative chamber and no governor’s veto, so whichever faction won the annual election could pass whatever it liked. Massachusetts gave itself two chambers, a governor with a veto, and independent judges, forcing political decisions through several independent checks rather than a single assembly. When the Federal Constitution was drafted, it drew on the Massachusetts architecture of divided powers.9

The American Revolution kicked off a major shift in the institutional landscape beyond just the U.S. Similar institutions—not just constitutions and representative government, but independent judiciaries, patent systems, and more—appeared in France, then in much of Latin America, and later across the rest of Europe and much of Asia.10 Later technological shifts set off new cascades of institutional innovation. The Internet, for example, gave rise to online auctions, reputation systems, prediction markets, community governance, and multistakeholder institutions.11

Some worry that institutions can create as many problems as they solve. During COVID, overlapping jurisdictions, agencies, and reporting systems often gridlocked coordinated action, for example.12 Institutions can add bureaucracy and veto points, cement incumbents, distort incentives, or decay while remaining difficult to abolish. But these risks are an argument for beginning early, giving us time to test new arrangements before they are direly needed.

We are not approaching powerful AI with anything like this level of preparation. Concrete designs remain scarce, and we’re not experimenting with new mechanisms or building prototypes at anywhere near the scale we believe is necessary to succeed. Or at least, not yet.

What Pax Machina is for

Pax Machina is a forum for rigorous debate about how institutions should work in a world with powerful AI. It brings together researchers across mechanism design, political philosophy, social choice theory, law, economics, computer science, AI governance, and related fields, including people who see institutional implications in their work but have not had a natural home to develop them.

Responses are central to the publication. A proposal might be followed by a critique, an alternative design, or a revised version from the original author. Rather than a collection of isolated essays, we aim to produce a visible record of institutional ideas being proposed, contested, and refined.

We are looking for authors with a concrete institutional proposal, a sharp objection to one, or an analysis that changes how a class of institutions should be designed. A piece might propose a new mechanism, explain how AI alters an existing institution, identify a failure mode, draw a specific design lesson from history, or offer a roadmap tied to future capability thresholds.

For example, the regulatory markets proposal is concrete enough to invite equally concrete objections, and thus we’d consider it in scope.

The bar for publication is specificity. We’re looking for more than claims that AI will upend our social infrastructure. We want the design details: what should change, how would it work, who holds the authority, and what could go wrong?

To help orient this work, we’ve developed an interactive map of the institutional landscape, which we refer to as “the grid”. The grid organizes institutions by the scale at which they operate (rows) and the capacities they require of participants (columns). It is a provisional map of the landscape, and is not intended to be exhaustive.

The view below shows our existing institutions (see here for a demonstration of how these have developed throughout history).

A second view includes candidate institutions and research directions for a world of humans and autonomous AI agents. These entries are illustrative and opinionated, highlighting some of the possibilities we think deserve attention.

These possibilities raise several recurring design questions:

• How do we preserve human agency? How can institutions capture the benefits of AI without eroding people’s ability to understand, contest, and ultimately steer the systems they depend on?

• How can institutions adapt at AI speed without losing legitimacy? Institutions may need to respond faster as AI systems evolve. But speed can come at the expense of due process, accountability, contestability, and meaningful human consent.

• What properties should institutions demand of AI systems? Alignment, cooperation, trustworthiness, and controllability suggest different relationships between agents, people, and institutions. Which matter in which settings, and are there new concepts that might be more fruitful?

If you’re working on these questions, or have an idea for an institutional design you’d like to develop, we want to hear from you. Write to us at editors@paxmachinamag.com.

Beginning again

Institutions often change in bursts. An unexpected crisis, technological breakthrough, or political shift can suddenly make reforms possible that seemed unthinkable a year before. Political scientists call this a punctuated equilibrium.

We cannot know when the next such opening will come or who will hold power when it does. But we can be certain that when it arrives, there will be little time to design from scratch. The options on the table will be the ideas that have already been developed and tested. We need to do that work now, so that good proposals are ready when the opportunity arises.

As Tyler Cowen remarks, powerful AI will force us to rebuild our world. We are not used to thinking this way today. We inherited an institutional order and have mostly adjusted it at the margins in recent decades. The prospect of reimagining it can therefore feel impossible.

But humanity has rebuilt its world many times before. The liberal institutional order we inherited was itself the product of unusually bold design. It created radical forms of government and cooperation that now seem perfectly ordinary. Though deeply flawed and often sustained by violence, it stands among humanity’s great accomplishments.

Pax Machina exists to seed a rigorous debate about what a world of humans and powerful AI could and should look like. We hope it can accompany another great age of institutional invention. Let’s get to work.

Subscribe now