I’ve led a locally logical life, each step had some coherence. Were they truly predictable? For better and worse, my steps and missteps were locally explainable, mostly suboptimal, and generative of who I am today. I wonder, could it have been an optimal path? Certainly, but the obvious counterfactuals are badly constructed. My 22-year-old self would never accept the advice I’d give him today. I can explain why younger me was incapable of taking advice that now obviously seems prudent, but can no longer fully reconstruct why he would be unreceptive, or stubborn, or naive. He would be equally baffled by where I am today.

I could tell a story about how I got here. But it would be a fabrication, a rationalization constructed to make my current self make sense. Some broad strokes are probably true. I met my future wife and her friends, and it taught me that there were worlds and careers beyond the military and the government. My father died and I spent years alternately struggling with it and avoiding it. But other elements are probably retrofitted. I read the right book at the right time, met the right professor, learned the right skill, and gained experience. Interview answers. All just so.

The story glosses over the truth of the matter. I have died a thousand times before today. I am no longer 10-year-old me, sure. Obvious. But I’m also not 22-year-old me. Or 30-year-old me. They died. Not at a specific moment, just somewhere along the way. When I embarrassed myself that one time. When I read that book that changed my mind. When I talked to that one person who impressed me. When I chose to work out, and when I didn’t. When I ate poorly, drank too much, when I nourished and cleansed myself. Each moment takes something away and adds something else. Eventually, a previous version of me becomes unrecoverable.

I can’t make perfect sense of these selves, though I know they are continuous and contiguous. Never identical point to point, but always coherent. Feelings like regret often get confused about this. My previous self might have made a different local decision, but he couldn’t have been a fundamentally different person. I can dislike what he did without pretending that he had access to the things I know, value, or desire today.

That’s continual learning. It is simultaneous destruction and construction. Learning nuance and detail kills absolutes. Learning to think differently kills a previous misunderstanding. But not all experience is growth. Comfort kills discipline. Avoidance destroys the capacity to face things.

Learning is not just facts overcoming misconceptions. We learn tastes, ambitions, aversions, social intuitions, we collect norms and habits, we generate and refine categories and fill them with things that map onto and away from our interests. Experiential learning changes what is salient before it changes explicit beliefs. Competence destroys insecurities. Embarrassments destroy self-conceptions. Romantic relationships destroy entire theories about the future and what success is.

Among all this destruction and construction something remains. But what remains is not necessarily an essence, a fundamental self. The most interesting continuities are dispositions, it’s not always clear why they persist. Some survive because they are useful for problem solving, or serve as a crutch, or provide shelter from daily torments. Some survive because they are cultivated, others are anchors we fail to cut loose. At any time along this continuous path, you’d recite core beliefs if asked. How many of them were core, in the end? If you enumerated core beliefs over time, how many had foundations of mud collapsing at the first real challenge?

We often tie changes in ourselves to some event. Sometimes that is accurate. But events are also narratively convenient markers placed over a process that was already moving. Events can resemble discontinuities, but we’re often more primed for a shift than we let on. The event serves as the narrative reason, a way to explain why we no longer recognize our younger self. It is one way we convert a continuous process into a narratable sequence of causes.

When I judge my former self, I should distinguish two counterfactuals. Could I have made a better choice as the person I was? Or could I have been the kind of person who would naturally make the choice I now prefer? The first has a real, and potentially useful answer. The second relies on the decisions of a fictional person, never existing.

So what do we owe to our future self, the one in the long future who has shed beliefs we see as core, who has more refined taste, or new aversions, or new social norms? If my beliefs might rest on mud, and my habits might vanish, and the very person writing this essay will be foreign to me, how can I make a decision today that will be useful in twenty years?

Mors Janua Vitae, death is the gateway to life. The death of my previous selves is the condition of ongoing development.

But it’s also a condition that drives devolution, and there’s the rub. Devolution is the same process as learning, but with the gradient pointed toward diminished capacity for self-correction. The self emerging from a devolving process does not feel the loss, the process killed the self that would understand what went wrong. The mechanism doesn’t care which way it runs.

Devolution is locally logical, too. Each step down has its reasons. That’s what makes it hard to arrest. I can judge my past selves because in some dimensions I acquired new capabilities they lacked. Nothing guarantees my future self will stand the same in relation to me. He may know more than I do, or he may lose certain capacities rendering him less capable of judgment. That makes the present a point of leverage. While I can still recognize my own crutches, evasions, and bad habits, I can act on them. My future self may not have either the desire or the capacity to do so.

Our current judgment of our past selves generates a new local responsibility. We know our current self is temporary. So our task is not to preserve who we are, it is to cultivate the next transformation without knowing what it will be, and what will survive it. Much like our dispositions that survive transformation, cultivation is non-specific. I do not owe my future self a specific set of outcomes, I owe him optionality and capacity.

I can cultivate things that enlarge the possibilities for my future self without dictating what they ought to become. He will judge me the way I judge my younger self, if he can. He will dislike some of what I did, even if he knows I couldn’t access the things he knows, values, or desires. So I can at least be honest about what actually transfers. My beliefs may not. Nor will my plans, optimized for a person who will not exist. I can transfer skills, dispositions, and certain material conditions like health and financial resources. Of those, the dispositions need the most sorting. Some of mine are useful for problem solving, and those I can cultivate. Some are crutches, potential precursors to devolution, and I should be aware of which. Some are anchors, and I must choose whether I cut them loose now or leave that work to the future.

I ought to build dispositions and conditions that remain useful for many of my possible successors, rather than optimizing narrowly on the preferences of the person I currently expect to become. It’s not about becoming the person I want to become today. It’s about leaving a future me with enough capacity and freedom to become someone I can’t yet imagine.

My latest essay, “Mesa-Optimization is Destroying Education,” is out now in Palladium Magazine.

I borrowed a highly specific concept from machine learning and mapped it onto human sociology, so inevitably there are some rough edges. But it’s a better way to think about certain forms of institutional failure. I chose it deliberately, because other commonly used models fail to describe what is actually happening inside our schools and other institutions.

So I’ll go beyond the essay to explain why that framework is useful for understanding the collapse of institutional metrics, and what it means for the future of the classroom.

Subscribe now

Why Other Models Fail

When diagnosing institutional failure, the instinct is to reach for commonly-used concepts like the principal-agent problem, or Goodhart’s Law. Neither accurately captures the modern education crisis.

The principal-agent problem assumes a principal who holds the true objective while a self-interested agent diverges from it. But the degradation of educational rigor doesn’t come from this kind of divergence, because education has no such principal. Parents, states, and administrators each optimize their own local targets, and no one’s incentives point in precisely the direction of actual competence and scholastic excellence. Teachers and administrators aren’t actively plotting to develop worse human capital; they are making rational, localized decisions to avoid certain frictions.

Goodhart’s Law (and by further extension Campbell’s Law) states that when a measure becomes a target, it ceases to be a good measure. This describes a metric’s value changing under pressure — a real but limited failure mode. For example, the UK’s NHS set a target for patients to be treated within 4 hours in waiting rooms, so hospitals forced ambulances to wait outside with patients to keep the clock from starting. That’s a pure Goodhart failure. The metric itself drove poor service, and removed ambulances from the road. Education, on the other hand, isn’t based on one top-down metric that gets optimized; it is millions of micro-measurements, distributed across tens of thousands of districts, heavily influenced by parental pressure, funding mechanisms, and social expectations. And unlike the aimless metric drift of Goodhart, the system fights back! And yet every intervention gets absorbed. Ed-tech, performance-based funding, testing reforms, and other standardizations all get swallowed and repurposed back toward the proxies. Goodhart says metrics falter when they become the goal, but that alone doesn’t drive what’s happening in education. The education system is an optimizer actively fighting metric corrections.

The Local View of Correlates

This is why mesa-optimization becomes a better lens than our other options.

In machine learning, a mesa-optimizer is a subsystem that emerges to pursue its own internal objective—an objective that is locally correlated with the base objective, but ultimately diverges from it.

This captures the emergent dynamics of the education system. The system optimizes for local correlates of learning: GPA, graduation rates, college admission rates, and job placements. Because these metrics once correlated with actual education, the system blindly optimizes for them. There is no explicit choice to hollow out the curriculum. It is simply a subsystem running its own objective function based on survival constraints and local incentives, steadily drifting away from the base objective of actual competence.

Mesa-optimization as a concept allows us to diagnose the decay without the unproductive morally- and politically-loaded blame assignments, and drives us to identify the broader problem. The system is just doing what complex systems do when innumerable feedback loops break.

The Schoolhouse Model

In the essay, I argue that the only way to arrest this decay is to implement gaming-resistant AI assessment systems. But that doesn’t entirely remove the human element.

Currently, we force teachers into a deeply compromised position. They must be both the uncompromising judge and the supportive ally. Those two mandates are often at odds. If a teacher acts as the strict assessor, they might alienate the student and face parental wrath. If they act as the mentor, they are incentivized to soften the edges of the assessment—which triggers the exact mesa-optimization that leads to broad grade inflation.

A “schoolhouse model” functionally unbundles the educator.

We use AI as the tool to assess, and we use humans to generate moral formation. Use the tool to sharpen; use the human to broaden horizons, and to speak with authority about the world as it is.

In handing the assessment layer to an AI, the machine becomes the physics engine of the classroom. It is an unyielding, consistent standard that cannot be bargained with. It allows students to go as far as they are able without the worry that the rest of the class can’t keep up. But it also doesn’t leave a kid that struggles in the dust, it holds them back until they get it.

Because the adult is no longer the one handing out the ‘F’, they can sit on the same side of the table as the student. The teacher’s role shifts from potential, context-dependent adversary to pure mentor. They are no longer evaluating the student and they are free to help the student achieve. They are teaching them about the world, not the curriculum. Removing grading allows adults to step up and do the vital, un-automatable work of mentorship, discipline, and presenting the meaning of the world to the next generation.

Over the last year I’ve been interested in what’s going on with fertility rates. Certain wonks think it’s one of the most vexing and consequential problems of our time. But in my view, nobody has provided a satisfactory explanation why fertility rates are falling. The most popular theories always tend toward explanations that happen to align with pre-existing ideologies. For example, market fundamentalists say it’s housing issues, certain religious groups and conservative organizations say it’s a loss of traditional values, and others think it’s the pill. All of those are fine explanations, and probably have some impact at the margin, but are ultimately unsatisfying. Each of those factors vary widely across nations and cultures.

Subscribe now

And the fact is the fertility rate is falling globally. No nation is exempt. Poor countries, rich countries, deeply religious countries, democracies, autocracies, monarchies. None are bucking the curve. The decline used to track national wealth pretty well, but lately the decline is accelerating the most in the poorest countries.

The breadth of the effect indicates to me that local explanations don’t work. Any good story explaining fertility decline must have features that align with the facts that the effect is global, it crosses cultures, norms, and ideologies, and it tracks with wealth, though not as much recently.

I started interrogating the issue on my own, starting with a simple idea: how can I measure time expenditure and its relationship with fertility. My initial dive into IPUMS data found a large and obvious effect, which somewhat aligned Gary Becker’s fertility model. One of his theories is that the opportunity cost of time explains fertility choices. And indeed, in the US I found ongoing evidence for his 60-year-old theory — that unemployed women have more kids than employed women when controlling for age and other factors. However, I also find that the effect is much larger at low incomes than higher incomes. This stresses Becker’s explanation, because one would think the employment effect would be larger at middle- and upper-incomes where there are more ‘strivers’. It’s worth noting that there’s a reverse causality, women who have kids are more likely to exit the workforce, but I think Becker is onto something. The problem is he’s only pricing market time, I think there are more claims on time that depress fertility, and better explain low-income fertility disparities and long-run decline.

Let’s think about the system that generates births. In modern societies, what’s considered a ‘normal’ birth comes at the end of a long pipeline. There are encounters with potential mates, courtships, couplings, and conceptions. Those require two things: first it requires time in the environment where such encounters happen, and second it requires a willingness to terminate the search phase once a potential mate is found. Becker’s story says that education and careers pull people out of that pipeline, reducing the rate of ‘collisions’. That’s true, but even in wealthy countries most people don’t take the long education path. For time spent out of the environment to matter, it has to affect more than the well-educated.

A better explanation is that increased complexity drives time spent outside of the mating system, not wealth alone. Complex societies, whether wealthy or not, don’t just expand optionality, they induce additional time-expenditures. Every idle hour now has a high-value claimant.

I’ve called this phenomenon neverboredom, though it’s not entirely about boredom per se. It combines a lack of idleness and the overwhelming ubiquity of options. In some ways, idleness was a hidden subsidy to family formation. Now idleness has been competed away. Many people don’t actively decide against having children. In the US, the data show that women are having fewer kids than they want to. But in a world where there’s a claim on every minute of time, micro-delays compound, and many people end up in their mid-30s without feeling like they’ve ever made an explicit decision about family formation.

Historically, urbanization has tracked with declining fertility, starting well before the modern era. The standard explanations for urbanization’s effect are country-specific grab bags. But if you read it through the frame of neverboredom, attention pressure explains more than things like living square footage. There’s more available mating choice, and the amount of choice makes it harder for people to terminate search. There’s more entertainment, more non-romantic social relationships, all contributing to time outside of the mating environment, which all adds up to urbanization being the first phase of neverboredom. Cities were claiming time long before electrification. Now we have smartphones which deliver a city’s worth of distractions anywhere at any time. But smartphones aren’t the whole story, if it wasn’t smartphones it would be something else.

To test the neverboredom theory, I looked at entertainment technology bundles, and when they came online in different countries. In the chart, below, you can see the story in the US. The first wave is television, which is then followed by the internet and then mobile subscriptions. Each one corresponds with a decline in fertility rate on about a 10-year lag. It suggests that adolescents growing up with the technology tend to have fewer kids. For completeness, I should note that the television effect is confounded by the rapid decline at the end of the baby boom, but the internet and mobile diffusion don’t have the same kinds of convenient confounders. I’ve done more detailed econometric analysis that suggests these entertainment bundles, particularly video-capable mobile devices, may be a leading indicator of decline globally, some of that evidence is in the data appendix at the end of this piece.1

Why entertainment technology bundles? Neverboredom covers a number of potential time expenditures, but I wanted to focus on time expenditures that have more ‘empty calories’. Education has clear benefits, careers have financial benefits, travel and leisure have psychological benefits. Entertainment technology may have some benefits, but we generally understand that those benefits are limited and have diminishing returns.

It’s unsurprising that I’m not alone in noticing the effects of entertainment technology. Eliana La Ferrara found interesting fertility impacts in Brazil driven by television entertainment, particularly soap operas. Robert Jensen and Emily Oster found similar impacts in India driven by cable television access. Their work often focuses on the transmission of westernized or urbanized norms through those channels, but I think that story is incomplete. The effects of entertainment technology cross firewalls, content control regimes in authoritarian countries, and strict, traditional norm-enforcing forms of entertainment.

Not every neverboredom time claim is about fun. Donald Moynihan, Pamela Herd, and Hope Harvey point to the time expenses coming from administrative burdens. Each of these micro-claims extract from real leisure time, reducing further the available time to meet and couple. Leisure time can grow on paper, seemingly expanding the time we’d have to nurture romantic relationships, but at least some of those open minutes are claimed by administrative burdens on top of omnipresent entertainment.

So What?

If this is all true, then it’s clear that fertility responds like a price. It’s universal in reach and responds to costs. Even subgroups and nations with strong, and strict pro-family norms aren’t exempt. In the western world, Israel has one of the highest birth rates and strongest norms around family formation among wealthy nations, and yet theirs is still in overall decline. Same with communities in the US, like the Amish, or the LDS community, all with strong family formation norms, but still having fewer kids than they did before. Policy responses are a drop in the bucket. While they have some impact on fertility, the cost-effectiveness is modest. Which is to be expected when working against a powerful global downward pressure.

If we must act, policies like expanded childcare or child tax credits should be lower on the list of things we should try. Instead, it would be better to invest in reproductive technology. In the US, if we take women at their word, they are saying they want to have more kids than they are having. Reproductive technology lowers the cost of delayed childbearing, and potentially enables women to have as many kids as they want. Policy buys marginal births at known, poor returns. Fertility technology extends and de-stacks the reproductive window, and raises late-age success rates. On the long horizon, ectogenesis is an R&D option with asymmetric upside, because it targets the actual mechanism: accumulated delay.

To be clear, technology only solves one of our attention-collision model mechanisms: the biological window. The social problem, declining partnership, remains and requires different solutions.

But another option is that we simply accept declining fertility as the reality of a modern, complex world. My conjecture is that a rebound has already begun. The 2010’s “you don’t need kids” register is gone. Gallup’s ideal family size is higher than it’s been since the 70s. It isn’t farfetched that children will become status symbols for later cohorts. And underneath this conjecture is a slower, surer mechanism. Subcultures that currently have elevated fertility rates compound their population share generation over generation. I don’t expect that the vibe rebound or subculture population share will end up recouping the population losses already being priced in, but it’s a clear glide path to a new normal in the long run.

So we can spend on technology that attacks the mechanism, or accept the compression and let the selection and preference cycle work. We shouldn’t fight the margin with subsidies. It’s useless to pay retail price against a wholesale force. We should either fight with tech, or let it be.

Additional Data Appendix

TFR vs Entertainment Bundles for Selected Countries

I’m testing a number of different indices for the entertainment bundle, but this is what’s called a ‘min-max’ index, which stacks a cumulative entertainment bundle effect. You can see the downward fertility trend for four different countries from different regions as entertainment bundles compound.

Age-Specific TFR Contribution for Entertainment Bundles

This shows is that entertainment bundles coincide with demographic transition, meaning women tend to have more kids at older ages as entertainment diffuses, but it does not offset the large losses at younger ages.

Long Run Fertility at 10- and 12-Year Time Horizons in High- and Low-income countries

This table shows the effect of entertainment bundles in countries for which we have long run data. I’ve excluded the boomer period, but when they are included the effect is stronger. It shows the entertainment bundle pressure is higher for countries that already have low TFR. This indicates a multiplicative effect, in countries with higher TFR, other factors contribute to decline, but once those modernizations are baked in other extraneous time expenses have a larger effect.

Why did I exclude the baby boom? Because it’s anomalous. This ERR line in this chart shows that even when adjusting for the fact that fewer children survived into adulthood in the past, the baby boom was a huge blip.

Two interesting AI moments happened recently. First, OpenAI’s model disproved a long open conjecture about the upper bound of the unit distance problem. It’s not the first time AI developed a solution for an open, unsolved math problem, but it’s the most interesting problem an LLM made progress on. Previous LLM-based math results had mostly discovered solutions to problems that didn’t get much attention and that had fairly trivial solutions. So this is different — it’s a bigger deal.

Second, a Commonwealth Short Story Prize regional winner was likely completely AI-generated. It’s no secret that artistic and literary communities have generally been opposed to AI. In this case, their opposition blinded them to the imposter in their midst. Most avid AI users would easily recognize the very AI-flavored tics in the text. But whatever else, AI managed to get by a selective human filter.

So: AI is a brilliant mathematician that can write award-winning prose. One might think a broadly capable intelligence would be having more impact on the labor market right now. A model that produced the unit distance result can trivially produce reasonable financial projections and Excel models. A model that wrote a well-received short story could surely build a passable slide deck. So why hasn’t it taken over the kinds of jobs made up of these tasks? The standard answers that “it takes time” or “it isn’t good enough” are both inadequate. Something else is going on, and it’s not about AI, it’s about how we conceive of, and measure, what jobs are.

Our conception of jobs, of what tasks within them are valuable, and of what counts as success, is poorly specified. Many people in and around AI mismeasure jobs by crudely accounting for some enumerated tasks and a wage value that might be saved. It’s an application of Marx’s notably incorrect labor theory of value, conflating time spent on tasks with the value of the job. These poor definitions drive us to mislabel jobs as highly AI exposed because they involve a higher percentage of computer-based tasks.

I’m going to explore four non-exhaustive, contributing reasons why we poorly predict job disruption, at least in the short- and medium-term. First, the commonly drawn equivalence of tasks to jobs is inadequate, we need to think differently about what a job is. Second, we need to evaluate the quality floor and risk tolerance of job functions. Third, we should consider the time horizon for failure identification for job functions. And finally, we should consider the problem of defining what “success” looks like for certain tasks, particularly those somewhat analogous to writing a story for the Commonwealth Short Story Prize.

Task Value: Training Time Is an Important Consideration

To start, it’s better to think of task value as the training time required to get a human with sufficient prerequisite knowledge to do the task correctly, especially for highly paid jobs. Too often we think about tasks as components of a job that AI will replace, and we count up how much time those tasks take. That’s precisely how Anthropic’s Economic Index thinks of jobs, for example. But time-share is a measure of volume, not value. The critical information is the combination of task training time, supply of sufficiently skilled labor, and salary value. If a task is cheap to learn, and there are millions of people equipped to learn it, then the task probably does not explain the wage premium of an expensive job.

When we hire, baseline competence is but one screen. Once we’ve found competence we assess other aspects of fit, often unwritten and amorphous ‘cultural fit’. For now, let’s assume that we’re going to continue sending people to school and college and internships for the foreseeable future. Then the right question for AI exposure isn’t how much of the workweek a task fills. It’s how long it would take to train a human with baseline competence to do the task correctly.

Consider financial modeling. A junior financial analyst spends a decent share of their week modeling. Time-share frameworks would read that as a heavily exposed job. But if it only takes a few hours to train someone with the requisite math and Excel skills to make a baseline financial model, and if the whole job were financial modeling, then finance jobs wouldn’t command the salaries they do. What are companies paying for, if not for the tasks? I’m not sure it matters too much. Perhaps it’s the future value of that employee after they’ve learned from senior people, perhaps it’s a set of relationships they can bring or build. But the wide gap between task trainability and job compensation shows the task poorly represents the job.

This can invert the standard exposure picture. Jobs that look most exposed by time-share often have the lowest-value tasks at risk. Jobs that look less exposed often have higher-value tasks that AI can also do. The tasks worth disrupting aren’t the ones that fill the most hours. They’re the ones that require years of accumulated judgment and experience. The tasks that are ripe for disruption have long training times.

Quality Floor: Firms Don’t Often Pay for Excess Quality

Companies don’t typically pay for surplus quality in support functions. Some people assume companies will sacrifice quality for the savings they get from AI. That misses how many job functions are actually valued. Companies often buy the minimum quality they can tolerate. They aren’t paying for extra-good accounting, extra-good compliance, or extra-good operational controls just in case. They pay for work that clears a floor. If AI falls below that floor, or changes the characteristics of the floor, then cost savings convert directly to risk.

Conside accounting quality more specifically. Companies pay for sufficient accounting to keep them compliant and their business inflows and outflows readable. If AI does the work and misses the mark more often than their human controls system does, the consequences aren’t merely lower-quality books. They are fines, regulatory action, and, in the worst cases, jail time for the CFO. The quality floor is a hard floor.

Error rates aren’t the only measure that matters. AI can hit the same error rate as a human, or a lower one, but make different kinds of errors. Companies have built their detection infrastructure around common human failures. Auditors are trained to spot typical human mistakes, and controls are designed for the kinds of things experts have learned that a junior accountant might get wrong. Swap in AI and the taxonomy of errors changes, and different errors might slip through systems that weren’t built to catch them. Whether AI changes error rates or error forms, the risk problem is the same.

So it’s not that AI is ‘bad at accounting’. Even excellent AI can’t trade margin for cost in a function priced at the floor, because there’s no margin to trade. The tasks that are ripe for AI disruption have low quality floors, lower than the company’s current pricing.

Error-Detection: Horizons, Failure Size, and Who’s to Blame

Finally, we need to account for errors and the error-detection horizon. AI benchmarks often report task error rates, pass rates, or average performance, but that’s not enough. More important are when the error is detected, how big the cost is when the error hits, and who is hit with the cost. A coding error often shows up immediately. The cost is small and the developer absorbs it. A bad accounting judgment may take months to show up, and the cost lands on the company. A bad strategic branding decision may go unnoticed for years, the damage might never be accurately attributed, and the cost is the company’s market position. When error detection is slow and the cost is large, the question of who is on the hook becomes more salient. And it’s a question companies have already answered, in ways that don’t include AI.

Strategy consulting looks ripe for AI disruption. The work is full of tasks like data analytics, slide decks, and market research, all stuff that AI does well right now. Companies could absorb more of the work in-house with AI assistance. But that misses what they’re actually buying. It isn’t the slide deck or the flashy analysis. It’s outsourced judgment, and perhaps more importantly, outsourced accountability. A consultancy helps to absorb the blame for long-horizon decisions that companies can’t confidently make themselves. The slide deck is simply an artifact. The consultant diffuses responsibility and stakes.

This is why strategy consulting is a multi-billion dollar industry in the first place. Companies aren’t outsourcing solely because they lack the skills for certain analyses. They’re outsourcing because they can’t afford to own the decision when the time horizon is long enough that the decision will outlast the executive who made it. The fee buys a second set of eyes and someone to point at. The tasks that are ripe for AI disruption have short error horizons and incur lower error costs.

Putting it Together: Why Math and Programming Are Vulnerable

Coding, and for similar reasons math, are the rare case where all three previous dimensions favor AI substitution. The tasks are core to the job, they require long training time, and are exceptionally hard to learn. The quality floor is low for most production code like simple apps, internal tools, or basic scripts. Bugs are mostly recoverable. In math, there is essentially no cost to untrained math hobbyists trying their hand at problems with AI assistance. In both fields, error detection can be fast. Bad code doesn’t compile, or doesn’t pass unit tests. Math proofs are eventually checked by experts and, when necessary, formalized in Lean.

Software is especially vulnerable because the enforced quality floor is often lower than engineers like to admit. If the app behaves correctly, the UI works, and the output is usable, ugly internal code is often tolerated. AI may lower that floor further by making software more disposable. If code is cheap to generate, maintainability matters less at the margin.

There are many who’ve pointed out that AI is built in a way uniquely structured to solve math and programming. But training time, quality floor, and error horizons better explain why AI automations of very similar tasks, like financial modeling or accounting, are not as impactful. Most knowledge work tasks fail one of the three tests, and those failures better explain diffusion lag than the idea that companies can’t figure it out.

This is also what makes coding and math misleading as a leading indicator. Their features are unusual.

Returning to the Short Story: Self-Defeating Success?

Our regional short story winner illustrates a fourth, and more important, problem for AI. Most of the job tasks I’ve discussed so far have fairly well-defined success criteria. Accounting is successful if it’s accurate and properly categorized, and that rarely changes. Coding is generally successful if the code compiles and serves the right output.

What made “The Serpent in the Grove” successful? Well, presumably it was written well enough for the judges. But, now that it’s been accused of being AI-generated, do you think another similar story will win a similar prize? The next story written in the same style won’t win, because the judges and the industry have already updated their beliefs. Any short story with those characteristics will be labeled as AI-generated, whether it was or not. The success of the story can’t be replicated. And even if the story wasn’t AI-generated, something too similar in style and substance would be read as derivative.

Literary prizes are a category where success criteria change rapidly in response to exploits. AI can match a stable target, but it can’t currently keep up with a thermostatically moving target. OpenAI’s unit distance result is real progress because the rules of math don’t change. The rules of literary judgment do, and nobody writes them down. So do the rules of branding, strategy, sales tactics, and many other job functions. That is to say, perhaps we overestimate just how logical human work is.

What Would This Predict?

If my framework is useful, it will predict something that the current conventional wisdom doesn’t. I’ll offer a couple examples based on what I’ve already discussed in the piece.

First, I predict that accounting, and specifically audit functions, aren’t substantially impacted over the medium term. Most “AI is coming for these jobs” lists put accounting near the top, but I think the risk profile of accounting makes it difficult to automate with LLM-based AI. Some trends may hold, like more outsourcing of low-level functions, but instead of pushing them overseas they’ll get pushed to AI. The ongoing challenge, then, will be to train the next set of senior accountants when they rarely engage with the lower-level tasks. The valued skills will shift substantially, focused more on process design than on technical accounting capabilities.

Second, management and strategy consulting will grow in overall value as their services come in more demand as customers, themselves, automate and need accountability for those automations. Companies will still want to retain a “second set of eyes” on these harder problems to manage and diffuse risks. Consultancies will automate some artifact generation, and technical tasks, but the bulk of the work will still be about relationships. Selection processes for junior roles will change, as they focus more on client readiness and less on technical capabilities, since there’s less need to tradeoff on technical skills.

Watch these fields over the next 18-24 months. If I’m wrong, then I’ll have made the latest in a long line of bad predictions.

James C. Scott writes in Seeing Like a State that governance requires simplification and compression to understand the facts of its world. It generates reductive artifacts that enable the state to grasp what it is governing. They are important proxies for local and tacit knowledge. If a state measures grain production and grain stores, they don’t need to understand how to work the land. The measures are a compressed but manageable proxy for productivity, land value, worker skill, and more.

“Data-driven” governance is not new. Rome ran censuses, tracked taxable property, and knew who was eligible for conscription. Ultimately, every civilization is data-driven. What changes is the algorithm that processes the data. Sometimes it’s the local chief’s gut instinct. Sometimes it’s a massive bureaucracy synthesizing reports and modern data streams into executive action.

And in every society, the leaders processing that data are ultimately beholden to sentiment. Sentiment is not necessarily opinion polls, it’s the actual mood of the citizenry. It’s obvious in a democracy, but Hume tells us it’s true of autocracy as well. Viktor Orbán just lost an election in Hungary despite sixteen years of tilting the playing field in his favor. Scott Alexander recently made the point clearly: modern autocrats calibrate fraud, coercion, and institutional meddling to what the public and key elites will bear. Sentiment is the ceiling every ruler operates under. It’s also incredibly difficult to measure directly, which is why governments build elaborate information channels to approximate it. They track resources, behaviors, and a suite of outcomes as proxies for the mood that ultimately determines their legitimacy.

But despite every government’s great efforts to process information that converts into effective action, every great society has eventually declined. There are other causes, but one driver, consistently, is that every declining society loses some connection with and control over its citizenry. Formalized information channels fail. Governments falter when information is corrupted. This is easiest to see at the level of metrics, our consistent, repeatable measurements of what’s happening in the world. Every metric has something like a half-life. From the moment a metric is adopted, its relationship with the underlying condition it seeks to quantify erodes.

Formalization of a metric generates a new world condition. It alters incentives, changing the behavior of the people within the process it is measuring. It narrows the focus of governments and other organizations, to the detriment of other information that could be considered. And once a metric starts decaying, it is impossible to right the ship without redefining the metric or adopting a new one entirely. Such adjustments happen, but generally institutions are slow to make these changes, often in order to keep longitudinal comparisons in force.

Others have made similar observations. Goodhart’s law tells us that once a measure becomes a target it ceases to be a good measure. Campbell’s law says that when quantitative measures are used for high-stakes decision-making, it corrupts the social process it’s meant to measure. Taken together, we see that every metric has an attack surface.

Metrics are subject to four categories of attack: underspecified definitional changes, incentive distortions, administrative manipulation, and broader changing world conditions.

Consider the crime rate. It’s the state’s chosen method for measuring the amount of disorder in a community — that’s the core intent, the function of the metric. But “disorder” is the tacit thing; “crime” is already a narrow formalization of it. In defining the crime rate, we make choices about what counts as a crime. When the metric is established we already bake in certain rules that diverge from the core intent of the metric. For example, speeding tickets and other “infractions” aren’t counted in crime rates, and yet every speeding driver marginally increases the danger in a community, they generate disorder.

Over time we make more adjustments to our definition of crimes. So-called victimless crimes, like drug possession, public intoxication, or loitering, are obviously excluded from violent crime rates, and are reported inconsistently elsewhere. Over the last three decades, there has been a broad movement to decriminalize these acts. How could you presume to compare an arrest rate in 2026 to an arrest rate in 1990, in a world where many drugs are mostly legal and harm reduction is the method du jour for dealing with addiction? Public intoxication, open-air drug activity, and visible street disorder persist whether or not they are classified as criminal activities. They contribute to disorder in the community, and our crime rate now has a weaker relationship with the disorder it’s meant to describe.

There are administrative decisions, too. Even if our definitions were stable, it’s well-established that the process of turning incidents into records is discretionary at every step. A DA can decline to prosecute, or plead charges down to lesser categories, which directly changes what shows up in conviction rates. Prosecutorial action or inaction also alters policing policy — if certain crimes aren’t being prosecuted, they stop being prioritized by enforcement, and disappear from arrest rates. Such decisions are often reasonable in terms of local politics, or budget constraints, but each one alters what the metric captures. There are harder changes still. Database systems are updated and the actual collection of the data is changed. Crime classifications are recalibrated post-hoc. Often these adjustments are made in an effort to keep fidelity to the original purpose of the metric, but many result in systemic and lasting distortions of the measurements.

Then there’s the Goodhart problem. Once a crime rate becomes a performance metric that police departments are evaluated against, the reporting process itself gets distorted. Felonies get downgraded to misdemeanors so they don’t count against clearance numbers. Incidents get reclassified at the margin to fit whatever category looks best on the monthly reports. This is the mechanism Goodhart named, and it gets the most attention, because it’s the most obviously corrupt. But it compounds with the definitional changes and administrative shifts already described. A metric that’s been changed by definition, adjusted by discretion, and gamed for performance is three steps removed from the original condition it was meant to describe.

In the end you can’t trust the measure, especially as a comparison point, over long periods of time.

Crime rate is an easy case. The failures are obvious once you look for them. But our fourth attack surface requires a different example. This more important mechanism is harder to see, because it operates even on metrics nobody is gaming, and even when the measurement is doing exactly what it was built to do. The world the metric was designed to track keeps moving. The fixed metric slowly comes to describe something different from its original design.

Consider GDP and associated growth. Unlike the crime rate, GDP is relatively stable at its stated job. It measures the change in aggregate economic output, and it measures it about as well as it ever did. Economists know how to calculate it. The definitions are reasonably stable. It is not especially gameable at the national scale, and the bureaucracy that produces it is competent. In the US, at least, there is little incentive to corrupt the measure explicitly. By the criteria that undid the crime rate, GDP is healthy.

For most of the postwar era, GDP growth tracked something important: whether the people trusted their government institutions. As GDP went up, more houses got built, more cars showed up in driveways, more kids went to college. In general, people felt it and understood that this arrangement made their lives better. Rising GDP and rising trust didn’t always move together, but they correlated enough that the number became a critical policy measure and informational lever. That’s why GDP, and particularly GDP per capita, became a headline number, because the line on the chart matched the feeling in the country.

But that linkage has broken in the United States over the last decade and a half. The line keeps going up while the people are miserable. Trust in institutions is holding near all time lows, and people are persistently pessimistic about the direction of the country in ways they weren’t before. Whatever the marginal dollar of GDP is now buying, it is not producing the trust it used to produce.

A certain kind of pundit points at the rising GDP line, or the falling crime line, and uses it to dismiss the malaise. Look at the chart, they say. The complainers don’t know how good they have it. But this is exactly the problem. GDP is not institutional trust. Crime rates are not disorder. They were reasonable proxies for some time, but the relationships broke or distorted. Using the metric to argue against the thing the metric was tracking is an inversion of the problem. And as I said earlier, sentiment is the most important information for governments.

There are technical and explanatory reasons the GDP relationship has decoupled. We talk ad nauseam about home ownership woes, or wage distributions, or rising healthcare costs. And for every explainer there are “narrative violations” that show it’s not as bad as it seems. Perhaps more importantly, social media and other new media amplify discontent in ways that make the mood feel worse than the underlying conditions. But the media-driven discontent generates the world that must be governed. Every version of “the discontent isn’t real, it’s [social media / partisan media / manufactured outrage]” makes precisely the wrong evasion. It treats the mediation of the signal as grounds for ignoring the signal. But trust and sentiment are all that governments have. They are the source of legitimacy. There is no better version available. There’s no waiting out a bad mood. The mood arrives, even if heavily mediated. The only choice is to govern in the conditions that generated distrust, and attempt to solve the problem.

The answer is to read our decayed metrics and distorted sentiment together and commit to acting on the divergence. The divergence, itself, is the most important information you have. If a measure no longer tracks with mood in the way it used to, then we ought to assume the forces of decay have rendered it worthless for this purpose. We can modify existing measures or establish new measures, consciously aware that they will decay, and being willing to retire them when they do. It won’t produce the kind of stable reporting that gives the illusion that institutional life is manageable, with longitudinal comparisons going back decades. But that’s the point. Governance is the practice of staying in contact with a moving reality through unreliable instruments.

These corrections do happen, but they happen inside specialist practices, not in real public view. Labor economists will change focus from unemployment rate to the prime-age employment ratio if U-3 begins overstating labor market health. Macroeconomists have long supplemented GDP with a suite of other measures. The people whose job it is to read instruments generally know when an instrument has faltered. What they don’t do — and what the institutions around them make it very difficult to do — is retire the public-facing number. The headline keeps running. The specialist conversation moves on without it. The divergence between metric and mood is partly this: the official measure, broadcast to and understood by the leaders and the public, no longer fits reality and the experts know it.

Modern institutions confuse stable reporting for stable contact with reality. That is the problem. A crime rate can hold steady while disorder worsens. Output and wealth can keep rising while the people generating it get more miserable. The danger is forgetting that a metric was designed as a simplified proxy for a much deeper, local, and complex condition. If we only govern these artifacts, we no longer govern reality. The institutions that survive will be the ones willing to let go of the numbers that no longer describe the world. The ones that die will die clutching their artifacts, because the artifacts feel more real than the world.

There were two live threads in the AI discourse world over the last couple of weeks. First was a debate about who should be in charge of aligning our AI future, which was spurred by an article the WSJ ran about Amanda Askell, who is charged with guiding the morality of Anthropic’s Claude models. Second is an ongoing debate about jobs, and what will happen to them. I’ll discuss both topics, because both illustrate the importance of wisdom, flexibility, and optionality in future planning.

A Silicon Valley Problem

After the WSJ article ran, there was a furious online conversation about Askell, full of personal and political attacks. Elon Musk weighed in, saying that “those without children lack a stake in the future”. To her credit, Askell’s response was measured and considered, she took the high road. Many came to her defense, saying (in my opinion, rightly) that despite not being a parent she was much more capable of leading AI alignment work than Musk.

Subscribe now

But I couldn’t stop thinking about the parenthood angle. Tech leaders insist that AI is ushering in massive disruption, AI safety leaders insist that it is a genuine threat to mankind, either via disempowerment or via our actual extinction. If that is the case, we should probably consider the perspective of those with biological stakes in the future beyond this lifespan.

This isn’t just a vibes argument. We know for a fact that becoming a parent measurably changes our biology, in both men and women. Men become more empathetic, their brains age less quickly — gay men show the same changes if they are primary caretakers. The changes in women are more self-evident, but even knowing that’s the case, the scale and rapidity of the change is mind-blowing.

These are evolutionary traits, critical to the survival and propagation of our species. Over millennia evolution selected for these processes generation after generation. And they have real effects. Parents and caretakers think differently about the wellbeing of others, and their brains rewire to activate caregiving circuits.

Parents think different. And that perspective should be considered when our survival and well-being are on the line.

So what of Silicon Valley? I don’t have the data from AI labs specifically, but San Francisco County’s birth rate is among the lowest in California. Its total fertility rate (TFR) is the lowest of all major metro areas in the US. We know that TFR drops with education level, and that educated people are more likely to delay parenthood if they even choose to become parents.

Even without firm-level data, we can still estimate that the AI labs, populated with quite young and career-driven people, living in the metro with the lowest TFR in the country, are not exactly teeming with parents.

Silicon Valley is full of brilliant people. They are well-educated, they have read the right books, and are very skilled at what they do. But there’s a major experience gap. And while there’s some evidence that you can bootstrap future stakes, that typically requires deep civic or volunteer engagement not often compatible with demanding Silicon Valley jobs. Tech talent and academic brainpower are abundant in Silicon Valley, parents and committed youth mentors are not. Perhaps the tech companies should bring that perspective into alignment research. There are surely plenty of parents with sufficient technical or academic talent to do the work in these labs while bringing their uniquely biologically modified brains to the problem of AI alignment, and our future.

Most importantly, the changes we observe in parents’ brains and cognition are directly relevant to alignment problems. Parenthood enhances the capacity to model the needs of agents who cannot articulate those needs themselves. That skill might map surprisingly well onto the core challenge of alignment: modeling what humanity wants from systems that can’t tell us if we’ve gotten it wrong.

Amanda Askell is a great alignment leader, and an exceptional voice. The Anthropic constitution is an incredibly well-crafted and thoughtful document. I’m simply offering that there’s a form of thought diversity that is directly relevant to the very questions AI labs grapple with.

As a brief end note: parenting is effectively the only area where we’ve collectively decided that skin-in-the-game is meaningless, perhaps because it’s easy to enter and variable in quality, but it still changes the stakes. Parents have immense investment in the lifetime beyond this one. It’s worth at least considering whether that’s epistemically relevant.

Job Disruption: Show Me the Evidence

There is growing doom and gloom around the prospects for white-collar work, and eventually all work. Anthropic’s Dario Amodei said that AI would wipe out 50% of all white-collar work. Microsoft’s AI CEO, Mustafa Suleyman, said that white-collar jobs would be fully automated in 12-18 months. Sam Altman said his own job isn’t safe, though he’s indicated longer timelines than Amodei and especially Suleyman.

The fervor is just as strong in the general AI discourse. Software developers spent the holidays using Claude Code with Opus 4.5 and decided that their careers were over. I also saw the power of the technology for these tasks, and it’s definitely jarring to see a technology use programming languages, ones I spent months and years learning, with speed and ease.

AI keeps clearing new capability benchmarks by leaps and bounds, meaning that on tasks that have been consistently measured since the core transformer-based LLM technology has been around, we are running out of tasks to measure. AI is indeed improving rapidly.

And yet, we don’t see the labor market impacts we’d expect if automation was ongoing and on track to replace jobs. Yale’s Budget Lab showed that AI’s impact is on par with other tech disruptions from the past, and the Economic Innovation Group found that unemployment actually rose faster for the least AI-exposed workers than the most exposed. Even the AI systems themselves struggle to point to widespread disruption — when I asked, they came up with three affected fields: freelance copywriting and creative work, translation, and customer service. Translation and customer service are continuations of long-running trends. Google Translate generated massive disruption to freelance translator services from 2010 onward. Customer service has been riddled with bots and IVR systems for at least two decades, on a constant quest to automate further, so AI isn’t really the driver there, it’s just the next step.

But it is true that freelance creative work and copywriting are impacted. And it’s almost certain that other jobs are changing in certain ways. Some tasks are clearly open to some automation. If you read legal briefs, it’s much easier to have an AI filter you down to the relevant ones. As AI gets more reliable, perhaps that task can be completely outsourced, but it’s not clear just yet. Lawyers have already gotten in trouble over false AI-generated citations.

In fact, that’s one of the core problems. AI can automate at the task level. It’s hard to automate complete jobs. So AI disruption not only requires AI’s improvement, but it requires a reconfiguration of workplaces focused on automating tasks that AI is good at. But outside of software engineering and sloppy marketing copy, it’s unclear what those tasks are.

So why all the doom and gloom? I have a few thoughts. First, software engineering is really hard, and it’s kind of crazy that AI can mostly do it on its own. It’s also the field that the AI companies, themselves, are most engaged in. So the people closest to AI are the ones seeing the most disruption to their work. Software engineers tend to think that all work is software-engineering-shaped, so they think if their job is automated away, surely most of the others will be. That’s the message that filters out of Silicon Valley, and it’s a massive selection bias.

Intelligence analyst Gregory Treverton drew a distinction between puzzles and mysteries. Puzzles have definitive answers, we simply need to find the pieces. Mysteries require operating under deep uncertainty, and more information doesn’t always help. Most of what AI does well is puzzle-solving.

I’m not so sure that other jobs are as automatable. Those benchmarks we talked about? Almost all of them are software-engineering-shaped. The thing about programming is that it’s a verifiable task. There is a clear correct answer, and the AI can validate itself against that result, it’s a puzzle. It also helps that code is massively overrepresented in AI training data relative to, say, transcripts of plumbing diagnostics or nursing judgment calls. And all of the other tasks that AI does well are verifiable tasks. It’s good at taking tests — we have rubrics for tests. It’s really good at math, particularly in fields where the techniques are well-established in theory and thus their use and the assumptions they need to meet are verifiable.

AI is good at anything where it can look up the answer sheet, but it’s less good at other tasks, the mysteries. It also degrades faster on non-verifiable tasks. I don’t think it’s entirely clear that it will improve on non-verifiable tasks very quickly. So all of the work becomes making murky tasks more verifiable. That requires a lot of effort from businesses and institutions.

But it’s possible. And so we ought to think about how we set up for the future. I think we build for the now: assume that hard-to-automate tasks will remain hard to automate, but build systems looking toward a future where some of these tasks aren’t. That means building for our current point-of-difference. We can work in the world of atoms. We can navigate ambiguity, read rooms, exercise judgment in situations where there is no answer sheet. For now, those remain human advantages.

The burden of proof is on those predicting massive job disruption, and right now they’re not meeting it. Show me the jobs, not the benchmark scores.

Subscribe now

Looking Forward

Both of these threads point to the same underlying problem. The people building and deploying AI are making sweeping predictions about humanity’s future, our alignment, and our livelihoods from inside a narrow slice of experience. They live in the American metro with the fewest children — our little reminders of the future we’re heading toward. They work in the field most susceptible to automation. And from that vantage point they’re forecasting the trajectory of the species. A little more cognitive diversity — from those that raise children, and those who work in domains where the answer sheet doesn’t exist — would go a long way.

Scott Alexander wrote a very funny essay about parenting as a perpetual emergency, and naturally, as with all things parenting on the internet, it became Controversial. He was criticized for letting his kids run roughshod all over him, for lacking the discipline to discipline. Some went so far as to imply that his supposed inability to parent to their standard discredited his entire philosophical project. That’s how it goes when you touch the parenting third rail.

And they’re reacting to what, exactly? A handful of funny anecdotes, selected for humor, from a self-deprecating writer. They’re not observing the continuous process of parenting. This is always the case. When you see a kid tantruming in a restaurant, or a parent who won’t raise their voice, or one who seems too harsh, you’re seeing one moment. You don’t know if the kid is acting wildly out of character. You don’t know if the parent tried something else twenty minutes ago and it failed. You don’t know if this is the exception or the rule. You’re pattern-matching from one interaction, observed for thirty seconds, with zero context.

Subscribe now

But since I’m me, this whole scene made me think of variance. Every kid is wildly different. When I read Scott’s essay I saw nothing that resembled my own child. Did Scott’s kids manage to alert emergency services by accident? I don’t think my son could manage that, though his ability to scale a wall with dexterous ease presents other problems. Parents, too, are wildly different. And no one knows how those distributions interact.

I suspect this is why parenting drives such a visceral response. It’s emotional precisely because it’s not solved, and people feel a strong need to assert that the way they do it is The One Right Way. It’s how parents can prove that they’re doing well, and perhaps it imbues them with confidence to dismiss alternative approaches. I can insist until I’m blue in the face that certain forms of discipline simply don’t work on my child, but the disciplinarians will not give in. They won’t ask themselves if discipline works for their kid because of who their kid is, or because discipline worked for them and their kid is a lot like them.

But here’s the thing: the people arguing about parenting online, especially the kinds of parents that read ACX, tend not to be bad parents. The high-investment parents who are deeply engaged in their children’s upbringing, and those who care about forming well-adjusted adults—they’re arguing over parenting taste while pretending it’s something more. If you’re passionately arguing over cry-it-out versus co-sleeping, it probably doesn’t matter which one you choose.

I’m not here to tell you that there are no wrong ways of parenting. I’m not even here to tell you not to judge. But I’d like us to approach parenting more analytically, and by that I mean actually accounting for uncertainty. Parenting is unlike most other things in life. You only get one try, and even if you have multiple kids, the conditions for each subsequent kid are substantively different—those are brand new first tries. There are transferable lessons, but I’ve yet to meet a parent who said “oh yeah, my second kid was just like my first.” We’re all operating with a sample size of one, some of us add singleton samples from a few more strata, and none of us know what we’re doing with real certainty. When it works it’s luck, when it doesn’t work, it’s nature—not literally, but attributing causality to any given parenting action is a fraught endeavor.

None of this means you should parent without conviction. Approach it with confidence. But keep your eye on the data, and the most important data is internal. How is parenting making you feel? This isn’t selfishness. If you aren’t doing well, there’s no way you’re parenting well, and there’s no way your kid is responding well. You don’t need to be blissful. But there’s a difference between being tired and harried, which is a steady state, and genuine misery. The latter is a meaningful signal. It suggests you need a different approach, or you need to step back and reset.

We recently tried to potty train our kid. He got interested in pooping on the toilet, we encouraged it, and soon he was doing it reliably. We’d heard this was the hardest part, so we (mostly “I”) got supremely confident. Pee would be easy and we’d be done. Reader: that did not happen. He revolted. Once we changed the game he refused to do anything involving a toilet.

One morning I woke him up, asked him to pee, he refused but then peed all over himself the moment we got downstairs. Then I caught him trying to poop in a corner, swept him to the toilet, and thought I was in the clear. I took him to the car for daycare and he pissed all over himself again. All of that within thirty minutes. I lost my mind. That discrete moment might be the low point of my parenting journey, the one that would have the Outside Observers assuming I’m a terrible parent. It also represented a moment when I changed my parenting philosophy, at least in this specific case.

We could have kept beating our heads against the wall, and most of the time I believe in “pushing through” these kinds of barriers. But the most important piece of data was that it was driving us insane. If I’m going nuts, there’s no way it’s good for him. So we all reset. Our boy put the diaper back on, and a weight was lifted off of all of us. We tried again a couple months later, suffered far less revolt, and managed through.

The only meaningful signal was our internal state. Our kid’s behavior wasn’t noticeably different from other moments of rebellion. The change needed to happen because of our reaction, not his. If parenting is making you feel a deep, dark kind of bad, it’s not the right way.

But even if we had pushed through, there’s no reason to believe there would be a substantive change in our son’s long-run character or behavior. Parenting is a macro exercise, not micro. A bad day or bad week doesn’t ruin a kid. Nobody can be perfect in every moment. Frictions and frustrations and mood collisions are inevitable. Just like you can be a bad partner or colleague or friend on a bad day, you will be a bad parent on a bad day. Your kid will be in a bad mood when they wake up on the wrong side of the bed, over and over again. The macro trend is what matters.

So how do we gauge the macro trend? The question “am I a good parent?” is unanswerable. But “am I becoming a better person through parenting?” is answerable. If you grow in patience, humility, and presence through parenting, then you are parenting well. Maybe those specific virtues don’t work for you—then choose virtues that are reflective of the kind of household that you want, and within your control. You’ll only know if you’re a good parent after the memoir is written, and even then you can chalk up any criticisms to nature.

In many sports, coaches and players talk about the importance of separating process from outcomes. A baseball player who is struggling to hit the ball examines the process in order to improve. Are you seeing the curveball well? Do your hands feel loose, is your stride length too long or too short? Is the bat path optimal? Continuously pursuing a good process will, in the long run, produce better outcomes. In parenting the outcome is unknowable, as are the impact of discrete events and strategies. So we can only measure by the process, and we can only measure the process by our own growth through the process.

We can try to get better, not every day but every year. And we own our process. For some, that might mean judging other parents in order to feel confidence. If so, fine, but consider keeping that judgment within. If your neighbor looks like they have it all figured out, then try mimicking them, but stop if it makes you miserable.

There are no experts, not at the family level. We have plenty of aggregate knowledge, but that knowledge provides general guidance, not a manual. Trust yourself, because nobody else occupies the same multivariate space that you and your child do. It doesn’t mean you’ll get it right, but it’s the only proximate data you can rely on.

But this isn’t just about parenting, is it?

Subscribe now

The most important book I read this year was The Theory of Moral Sentiments by Adam Smith. I thought I knew what Adam Smith was about. Many do, or think they do. I was wrong. The invisible hand isn’t about unfettered markets. It’s about markets grounded in strong moral norms. That one book fundamentally changed how I see the world of markets.

This is year two of my annual review. Last year was about building better habits. This year was about building better thinking.

Subscribe now

If you’ve been reading along this year you’ve seen me use terms like ‘function’ and ‘discrete’ and ‘lossy’ and ‘categories’ when describing how we process information, and how AI processes our data. I tend to think about the world in terms of functions, how different variables interact, where are the global maxima and minima in these various functions. As AI becomes more ubiquitous, and people learn more about how it works, I wonder if more of us will start thinking in these terms. As roon said on Twitter:

Awareness is not Knowledge

The biggest single change I’ve made in information processing this year is to stop mistaking awareness for knowledge. Through a good chunk of the social media era, I prided myself on always knowing What’s Going On. I was on top of the news, each development, every character in the story. I knew who said what when, I could pass the NYT news quiz every week with flying colors. I was in the know.

But I only knew about surface level facts, I had no deeper insights. I had a bunch of priors I never challenged, and a set of ‘trusted’ surrogates that were my conduits for the news of the day. As I wrote earlier this year, “I learned a lot about how social media can warp my point of view.”

I noticed this a while ago, when reading a righteously angry piece about the Uvalde shooting. Luke writes:

“You’ve probably seen people sharing this poem around like people often will at times like these. We all have our routines we repeat after each mass shooting. Our patter down. Share The Onion headline. Do our rehearsed jokes mixed with sadness. Get the other guys’ asses. Our little superstitious rituals like a baseball player at bat adjusting his gloves just so.”

It made the world of information look different to me. I started seeing every online conversation as a rehearsed song and dance. Everyone says precisely what is expected from the specific point-of-view they are expected to portray. The only currency is edginess, or wordsmithing from inside a narrowly bounded box. This tendency leaked into legacy media, and between old and new media, there’s little else. That was my entire current event information diet.

Awareness takes advantage of our lizard-brain tendencies. We learn basic facts, sort the players into teams we’re familiar with and understand, and analyze based on who we like, who we dislike, and who we “trust” the most. Awareness demands rapid analysis because we must be prepared to ingest the next incident. It forces us into binary classifications of right and wrong, good and bad, there is no room for degrees of rightness or wrongness. Even more importantly, there’s no room to think orthogonally from the axis as it is presented in coverage. Maybe there’s a different angle altogether.

Knowledge requires deeper processing. The only way to know anything is to engage in system 2 thinking. We have to grapple with it and reason about it, and most importantly we have to consider alternative points of view in their best light (more on this later).

So I swapped news for old books—the ones people display without opening. The ones I thought I knew because I’d read secondhand writing about them. Smith, obviously. Wittgenstein. Arendt. Old Greeks and Romans. The goal was to rebuild academic grounding and infrastructure that awareness alone can’t provide.

Knowledge is about growth, awareness tends toward stasis. As we age our reasoning gets worse. Awareness doesn’t help stave off the erosion of our reasoning abilities, knowledge does. The relationship is inverse, the more aware we are, the less we know. Awareness can be a useful input, but we need to process the information through a different mental muscle. We have to use our knowledge, and we need to constantly replenish and sharpen our knowledge. It’s about proportions. We can’t be completely unaware, but we should be substantially more leveraged toward building knowledge.

A Better Theory of Mind

With knowledge we can develop a better theory of mind of our neighbors and particularly those with whom we disagree. Awareness is a simplifier. We’ve established that it centers us on allies and enemies, but it also generates faulty reasoning about those groups. The allies are noble and wise, and the enemies are ugly simpletons. I’ve found this passage from Umberto Eco’s Ur Fascism to be incredibly insightful and informative (emphasis mine):

“The followers must feel humiliated by the ostentatious wealth and force of their enemies … However, the followers must be convinced that they can overwhelm the enemies. Thus, by a continuous shifting of rhetorical focus, the enemies are at the same time too strong and too weak. Fascist governments are condemned to lose wars because they are constitutionally incapable of objectively evaluating the force of the enemy.”

When we are aware, but not knowledgeable, we fall into that trap. Perceived enemies or opposition are paradoxically both maximally weak and strong, somehow too simple minded to understand the world and yet so powerful and conniving that they can overwhelm it. If we start to think this way, it’s worth wondering if we are the imbeciles in the situation.

A better path is to thoughtfully consider the most generous interpretation of the opposition’s point of view. This is not simply charity, it is a way to sharpen our own reasoning, to improve our knowledge. This goes beyond steelmanning, it requires actually understanding the mind of the other.

I wrote about how we get siloed, how the pathways through which we get information force us into a box shaped for advertisers. That box inhibits our ability to reason, and to formulate a decent theory of mind for those that disagree with us. One of the grave consequences of 2010s campus culture was that it encouraged avoiding points of view with which you disagree.

So this year I sought to know other minds better by seeking opposing arguments with a charitable and open mind. No rapid-fire dismissals over claims of idiocy or racism or sexism, or whatever other -ism. And once again, while I became less aware, I became much more knowledgeable. A few areas where I changed my mind substantially: marijuana legalization and drug decriminalization in general, healthcare, and gambling.

I won’t unpack each one, but let’s consider the ‘harm reduction’ or ‘light touch’ approach to drugs and vice. This is where Adam Smith finally clicked for me. Everyone knows the invisible hand, but most non-academics don’t read the book that precedes it. In TMS, Smith argues that we’re constituted by sympathy. It’s the constant imaginative work of inhabiting each other’s perspectives. The impartial spectator is the internalized presence of our community watching the choices we make.

The harm reduction framework assumes the opposite: an atomized individual whose freedom we protect by leaving them alone. But that individual is already a fiction. There’s no free lunch here. Harm reduction doesn’t eliminate vice, it socializes it. Every allowance generates a harm function that radiates outward to families, to neighborhoods, and to the social fabric Smith understood we’re woven into. The question isn’t whether to restrict freedom, it’s whether we’re honest about whose freedom we’re actually weighing. Perhaps that doesn’t mean we should be ‘tough on crime,’ but it should make us more skeptical of frameworks that treat community embeddedness as optional. If you want something a bit more empirical, there’s plenty of evidence that marijuana legalization isn’t going so well.

One thing I learned in consulting is that every business problem is more interesting and complex than it might look on the surface. The same is true on basically every policy concern of consequence. And the more you dig, the more nuance you become aware of. This year I noticed that when you ask people how they feel about different topics, they hedge more on topics where they have more knowledge. That’s interesting, no? We tend to be more conclusive, and feel more certainty, about hot button topics we know less about. We should interrogate that certainty.

Incidentally, AI is great for this kind of exercise. Try it yourself. Take an issue on which you have strong feelings and beliefs. Ask the AI to steelman the opposition. Tell it what you don’t like about the steelman position and ask it to continue refining the position to align with your specific first principles. Once you’ve exhausted that pathway, then ask it for the strongest argument from the position of a person that doesn’t share your first principles. In the end you’ll have an argument best designed to resonate with you, specifically, and also an argument from the most value-aligned point of view. It’s a much better way to consider opposing points of view, and makes you better at articulating your own.

Throughout the year most of my beliefs changed in some way, because I stopped operating on classifications and started thinking in gradients. I stopped viewing policy positions along a single axis, and saw them as multivariate spaces with positions defined by numerous philosophical points of view. In nature, it’s typically suboptimal to be at the extreme end of some distribution. The globally optimal position is almost always somewhere between the extremes. Why would we believe it’s any different in our human systems?

So I think about information processing as an algorithm. I gather just enough situational awareness, process that awareness through ever-expanding knowledge, update my world model, and repeat. Continuous motion. Sometimes I wonder, what if our institutions could do this too? Not the wild swings we actually get, these violent lurches from one overcorrection to the next, forever oscillating across the target without landing on it. But something smoother. Systems that adjust proportionally as the surface shifts, that track the optimum instead of chasing it. It’s a fantasy, maybe. But it’s one I keep thinking about.

Subscribe now

What I Kept Doing

So that was the end of my year-end Big Think, here’s the rest of my year in review. I’ve continued staying sober, that habit’s stuck for over two years now. The only ‘drug’ I’ve taken is caffeine, with all the caveats that basically anything we ingest can be considered a drug in some sense. The biggest thing it has given me is time, and I’ve done a pretty good job of feeding most of that time into self-improvement.

Beyond that, I’ve spent most of my year focusing on virtue and virtue ethics. I’m by no means an exemplar here, but I try to think about my pursuits in terms of virtuous ends now, and it has helped me become both more selective in my activities and more expansive. I am more selective because I more clearly see the cost of ‘empty calorie’ entertainment, and I am more expansive because I’m more willing to pursue effortful tasks beyond my transactional or minimal care obligations. This year I’ve focused mostly on self-mastery and truth-seeking. Next year I plan to focus more on building community. We’ll see if that pans out.

Final Wrap

This has been an incredibly eventful year for me. This is my 18th blog essay, and these 2,500ish words represent about 3% of my output this year (there’s one gigantic essay I didn’t publish here, maybe soon pending contest results).

My most popular work:

• Efficiency Without Morality is Tyranny

• We Know a Good Life When We See It

• Breaking Free of the Modelable

• Lossy Data Problems, A Story About Birds

• The World in the Error Term

My personal favorites:

• On Clay Pots and the Shapes We Understand

• The Missing Variable in AI

• The Dog with the Big Brain

I promised some testable forecasts in my last essay. My biggest one is that AI growth is going to see a huge step change this year. 2025 was a year of meaningful improvement in AI capability, 2026 will be the year when businesses figure out what they want to do with it. My mid-run forecast is that this will eventually generate jobs, but I think 2026 might be the year where jobs bleed and it will generate some unrest. I also believe AI will play a prominent role in the 2026 elections, and you’ll see some interesting bedfellows organizing against AI. There will be a strange coalition of left and right new luddites pushing very hard for AI-associated rents to be paid to certain constituencies.

As is my (two-year-old) tradition, I also share critical physical stats for my own accountability and to share how I measure my progress. Last year I said I’d continue my journey in recovering long-lost fitness as a father, and that I’d eat more cruciferous veggies. I was more successful on the latter than the former, because it turns out that starting a company puts a real strain on your workout calendar. TerraSol has been exciting and grinding. This year we launched a pilot product, and started building a custom AI, all at the cost of my wellness time. March through June was a struggle on the personal care front.

I did much better in the second half of the year, and I actually developed my own cli-based tool to help me track everything that’s going on in my life. I’m two weeks into using it, and it’s been a game changer. I think I’ll have more success keeping all the balls in the air in 2026. If you’re techy, check out the tool. If you’re familiar with the command line, it’s quick and easy to learn and use.

Stats (change from December 2024):

• RHR: -3

• VO2 Max: +3

• Weight: I still abhor the scale, so I go by the tried-and-true clothing fit method. This year I fit back into my wedding suit, so good progress.

• Garmin Sleep Scores: Same, on average, which I take as a win.

• Food Intake/Diet: We started doing a pizza Friday situation with our neighbors and our kids. This has baked in one big bad-eating night for me. I’ve also developed a sugar thing. On the other hand I’ve been much better about cooking healthy meals most nights for the family. I’ll give myself a C+.

Thanks again for reading along. In 2026 I want to stop just describing problems and start building solutions, both in the essays and in my work.

Subscribe now

Imagine your idealized self, a version of you that you’d consider to be ultimately good, where anyone who met you would say “I recognize that this person lives well.” This is an elemental version of yourself, outside of constraints. Beyond mere survival needs. But it is not a wealthy self. The ideal has nothing to do with what you have or don’t have, wealth is a resource status not a human status. The ideal must be driven by your acts: what do you choose to do for yourself? What do you choose to do for others? What do you do more of or less of?

Historically, across cultures and civilizations, we have remarkable convergence in what constitutes a Good Life. In each tradition, we see some form of self-mastery over our appetites and passions, like Christian restraint and temperance or the Buddhist ‘Middle Way.’ We see care for others and forms of justice, like Confucian ren or Jewish tzedek and chesed. There is meaningful responsibility coming from our duties and obligations beyond ourselves, like Hindu dharma or Roman pietas. There is true community and friendship— a life made of strong, reciprocal relationships, like Greek philia or Islamic ukhuwwa. And we see orientation beyond the self—a discovery of wisdom, and our focus on ground truth, or sacred truths, like Buddhist prajna or Hindu viveka.

Subscribe now

The metaphysics guiding these virtues differ, but the goals remain the same. A many-thousand-year observational study seems powerful enough, but it wasn’t always translating well to society writ large, and as sociopolitical organization became more complex, we needed something more. Our thinkers went about trying to get to a real ground truth, to reset our expectations scientifically. They set out to guide us with reason.

There are those who have pointed out flaws in Enlightenment thinking: the forgetting of old wisdom, the over-indexing on scientific processes over direct observation. But we had some good reasons to reject the old wisdom. Throughout history we’ve had coordination problems that come about when people in power or menaces in society fail to follow the old rules, or when competing virtues come into conflict. If we could guarantee that everyone lives directly in virtue, and agrees on the hierarchies and bounds of those virtues, our human problems wouldn’t be what they are.

And so people like Hume said we must measure utility, and Bentham gave us utilitarian calculus. Others like Rousseau talked about social contracts. On the other side of utility, Kant developed the idea of categorical imperatives, universal truths so sparse that they can’t inform individual action. They can only inform social norms too broad for organizing and a tapestry of weaker norms that are contingent. We tried to organize around pleasure and pain, and around extremely limited guidance around permitted acts. We built byzantine legal systems all predicated on the idea that virtue would be lost, so we must have a point of view blind to the individual. These are pragmatic coordination strategies, they are good for the gaggle.

But what is good for the gaggle is not necessarily good for the goose. Our strategies cause two problems. First, our focus on utility creates a physics-like problem. Every technological advancement, or broad social change brings with it a new level of complexity that extracts the most critical resource, time, from everyone. Second, our focus on sparse ground truths creates a trust problem. Systems designed to account for the bad actions of defectors lower our default expectations of our neighbors. Trust expands widely but narrowly, focused on transactions at the expense of local relational trust.

The Gaggle Always Extracts

Economies focused on generating utility and consumption create an odd dynamic for the goose. Think of our economic and social relationships as a series of vectors that extract time from us. Without cultural attenuation, system complexity adds more and more of those time extractive vectors.

When complexity is low, coordination and shared responsibility leave some time for other pursuits, for the ability to be a whole person. But as complexity increases and demands that we coordinate implicitly or explicitly with millions of individuals and families across vast distances, there are more vectors, more demands for utility, more incentive to supply it. Investment bankers and 996 coders sacrifice family and relationships for material gain. Laborers, too exhausted after a day of work, turn to sedatives and empty calories to prepare for the next day. The demands of the market leave little of the only resource that matters for formation: time.

It isn’t impossible, but our systems make it incredibly difficult to carve out the time to cultivate self-mastery, actively care for others, engage in community, and engage in the spirit when utility demands a steadfast focus on duty alone. We see it in the data, through declines in religious attendance, volunteering, and physical exercise.

What’s good for the goose is suboptimal for the gaggle. Every bit of time spent on building a community in kinship, caring for others in a meaningful way, or engaging in healthy introspection, learning, exercising, or nourishing outside of our duty-bound employment is time not spent contributing to aggregate utility. A culture too slanted to the market imposes a hard cap on the time allowed for those activities. It needs just enough nourishment, fitness, community bond, and spirit for the human to remain productive. Far less than what most humans would want.

Importantly, duty and obligation to others, in other words our economic contributions, are part of the good life. What matters is the balance of duty in relation to the other virtues. Utility-centric societies simply discount the benefits of non-duty-bound activities. It doesn’t matter if the distribution of goods and services is managed by a state or left up to the markets, the system will always ask for your maximum contribution.

The Gaggle Diminishes Trust

Every safeguard we manufacture in order to coordinate with those that are not virtuous, and who may renege on their obligations, lowers the expectation of each individual within a society. We expect less out of each other in non-transactional relationships when duty and obligation dominate. We expect less still when our formal systems only punish the bad rather than promoting the good. When combined with the pressures of utility, we only have time and capacity for limited community relationships, and we expect very little of those that live further down the block. Trust and cohesion fragment.

I’m not longing for the time of handshake agreements, I’m describing the mechanism that makes them irrelevant. Once we are forced to rely on contracts, the idea of the binding handshake becomes quaint and meaningless. Every other defection becomes a matter of legalese. Important for coordination, abysmal for trust. Widespread but narrow transactional trust improves, because we’ve accounted for the bad transaction behaviors of the untrustworthy, but local relational trust diminishes.

Consider modern parenting. Once it was a neighborly duty to watch other children from time to time. But that went wrong too often—too many defectors, too few shared expectations around what care should look like. So now we have a vast childcare system replete with monetary transactions, background checks, and certifications. That system itself demands the working hours that keep us from knowing our neighbors, and most neighbors aren’t trusted with the kids anyway.

Trust is time-dependent. We can’t learn to trust our neighbors if we have no time to interact with them. We might develop some small chains of trust, perhaps among adults with kids of similar ages, but across the broader community trust is contingent and highly limited. This is a hard cost of complexity. Even if we were virtue-bound, a small number of defectors in a large coordinated society reduces trust for everyone. We can’t solve the problem of bad actors. But we can solve the time problem. A world with more balance enables communities to have the time resource to rebuild connection.

Technology and a New Time Balance

The time we gain from productivity is immediately recaptured and put toward more duty obligations. Famously, Keynes once predicted that we would have 15-hour work weeks because of technological advancement. That has not been the case. We already have evidence that AI is making us more productive, but where do these gains go? So far, where they have always gone, right back into work.

Over the last three decades, especially, technology’s promises of efficiency have delivered, but have also made obligation omnipresent. If we are going to recapture the gains of AI to help build better people, we need to change our human systems to adapt to these efficiencies differently.

This is one problem with both markets and central planners. Markets are wise and efficient allocators of capital in the short and medium term, but they do not think very long-term. Centrally-planned systems can’t see beyond the folly of human forecasting errors. Neither markets nor central planners know what inputs they will need on a 50- to 100-year timeline.

Our systems are eager to run with the production gains from AI now with no eye on the quality of human capital in the long future. If our computer chips get cheaper and our AI systems get more and more human-like in capability, then what need is there for a human to think about much of anything? And if there is no need to think of anything, then what need is there for education? What need is there to learn certain skills? Once robotics enter the picture, then what use is learning how to wield a hammer, or fix a toilet? If AI can care for us and teach us, what need is there to guide the next generation? The markets today don’t care what kinds of humans we make in 50 years, and if the humans we build in 50 years are inferior to other inputs, the markets will discard them. So how can we use this technology to improve ourselves?

One Path to Partnership

Outside of narrow applications, there’s no ‘best practice’ for how to use AI, so all I can offer is how I use it, and how that might go well or poorly in the future.

I’ve been using AI as a conduit for my curiosity. It is a starting place to raise a fleeting thought, sometimes just a benign shower thought, and then learn about it. My whole year has been built on these learnings. I learned about Philosophy of the Mind and AI directed me to some of the foundational texts. I wanted to learn more about virtue ethics, and AI directed me to Martha Nussbaum and Alasdair MacIntyre. I learned about how AI works from AI, then I read the foundational papers from Anthropic and OpenAI. I wanted to be a better programmer, despite the fact that AI can do so much of it for me, so I asked AI to give me puzzles and tests. Somehow AI’s excellence has spurred me to improve my own abilities. I’ve also become more cognizant of the sheer brilliance of humankind. Whenever I start writing an essay, I ask AI if someone else has already written it, and often it finds great work for me to read.

AI also helps me with mundane but important contributions. It helps me plan quick and healthy meals for my family. It helps me figure out how to better manage my time for workouts, and it helps me understand how to better manage my ADHD.

I have experienced AI’s ability to help me in virtuous pursuits. It is a great enabler of self-improvement, self-mastery, and in some cases removes the barriers that makes starting those tasks so daunting.

That idealized self I asked you to imagine? AI has given me the resources and reclaimed some of the time I need to pursue it.

But I also feel the intrusion of the timesuck vectors. I can see that every output that I generate faster because of AI assistance brings with it more pressure to produce more outputs. I hope we can build systems that can promote more of my experience in learning and growth, but I am fearful that in the long run it will be maximally extractive of our time and limit our ability to be better people.

So this is why I’ve been writing so much this year. I feel a need to pursue betterment and virtue for myself. I see it’s absence everywhere, especially in public leadership, and I see a hard-to-forecast future on the horizon where we can collectively be much better or much worse. I see AI safety research focused on existential risks, but less focused on the human-centric risks this technology poses. I want us to be better, I want us to do better, and I hope we can leverage our technology to get us there. But we can only do that if more of us believe we can improve, and more of us believe that we can expect more of our neighbors and of ourselves.

The gaggle won’t give your time back. But you can choose to allocate it differently today. And every hour you reclaim—for self-mastery, for your neighbor, for the person you described in the first paragraph—is a brick in a culture that might, eventually, curb the timesuck. Culture is built and rebuilt one brick at a time. This year, I’ve focused on laying mine.

Thank You

Thanks for reading along this year. I’ve really enjoyed writing more, and I hope you’ve found some of the writing interesting, or thoughtful, or perhaps even irritating in a good way. I plan to write one more year-end wrap-up essay focused on what this year has been for me, personally, and some testable forecasts for the future.

Subscribe now

I absolutely loved the Dwarkesh conversation with Ilya Sutskever. The whole discussion is brilliant and absolutely worth listening to. I was struck by a particular segment. At around the 9:30 mark they move into a discussion about emotion and value functions. Often, these ideas are brushed aside by serious AI researchers as fluffy concepts, but deeply understanding emotion’s impact on human reasoning might be critical for unlocking more advanced AI. There were three elements of the discussion I want to highlight:

1. Dwarkesh essentially says that emotions are easy to understand and raises that it’s interesting that they are difficult to encode.

2. Ilya and Dwarkesh both indicate that emotions might operate like value functions, or modifiers of a value function.

3. Ilya notes, citing an empirical case, that emotions are critical for reasoning.

I disagree in part and agree in part, and my disagreement is mostly focused on the idea that emotions are simple in both function and modelability. But I agree with and want to extend the idea that understanding the role of emotions is critical. I believe that emotions (and all internal states) are high dimensional and complex elements of human interiority that are essential for reasoning, and this essential element is not only missing in current AI architecture, but also can’t be approximated without some changes in approach.

Subscribe now

I’ll outline both the intuition and market evidence for my framing, and I’ll provide a functional representation of the relationships I’m describing. Then, I’ll discuss how this relates to the “jagged frontier” of AI, and how we might solve the problem.

What Art Can Tell Us About the Dimensionality of Emotion

I don’t think emotions or emotional states are easy to understand, but I think we can be fooled into believing they are because our high-level heuristics for emotions work fairly well human-to-human. If I tell you “I’m ashamed” you can do a reasonably good job of inferring how I feel, even if it is an incomplete and contextless inference. The ease of our human understanding creates the illusion that emotions are simple to understand. That is despite the fact that many of our human misalignments come about when our quick fire “understanding” misfires badly.

For suggestive proof that emotions are complex, high-dimensional, and difficult to understand, I’ll turn to markets, and specifically the market for art. People feel a need to communicate complex emotional states, so they seek representations that communicate more effectively.

So much of the artistic market is built on developing representations of emotions, or representations that impose an emotional state on the consumer. Artistic products describe a relatively small set of states like ‘joy’ or ‘embarrassment’. And if simple labels were sufficient, there would be no need for the wealth of diverse expressions of those states.

More specifically, if the label ‘grief’ captured the feeling completely, or even nearly completely, then there would be no need for books on the subject. They would be unnecessary redundancies.

The very breadth and scale of our emotional experience drives the demand for these markets. And we have a very large and longstanding market for emotive art, from which consumers tend to select art that best aligns with their internal representation of a given emotion. This implied complexity suggests we need a better model for how emotional states, and all internal states, get turned into representations. And we need to understand how those representations are decoded.

A General Formal Model of Communication

Our model is inspired by information theory, but is primarily designed to structure our thinking about communication as it relates to AI. Let’s consider a human internal state, it can be an emotion but it can be any other internal, felt state of being, and let’s call that I. Now let’s say we want to communicate that state to someone else. We use an encoder function ƒ that converts I into some representation R. A representation can be written text, video, speech, art, or any other communicative artifact.

R is a lossy, discrete, and low-dimension transformation of I. Think of it like asking someone to describe a symphony in one sentence only, or to summarize a full book in one paragraph. There is information loss in the compression.

The person receiving a communication uses some decoder function 𝑔 to process R using the receiver’s internal state I, and which infers the meaning of R and the necessary parts of the internal state of the sender, resulting in a decoded meaning I’.

As Ilya suggests, emotions, and other internal states are critical to this decoder function. The key insight is that in order to effectively decode R humans rely on their own internal state, I. From my earlier example, your internal state is necessary for you to reasonably and reliably infer meaning from “I’m ashamed.” There are other important properties of these relationships. For example, the encoder function is non-invertible. Our representations, R, are far too sparse, because of compression, to generate sufficient information about I, so we can’t get I from R alone. That property creates two problems, (1) AI must rely on estimating 𝑔 to get any meaning out of R and thus (2) AI has to approximate I without the tools to do so.

When I read “I think, therefore I am,” I can’t fully recover Descartes’ internal state—but I can draw on my own experience of existential dread, my own moments of radical doubt, to partially reconstruct what might have motivated the claim. I’m using my I to approximate his. An AI reading the same sentence has no such resource.

This should prompt us to change our formulation of some open questions. We need to stop thinking about how much information survives representational transformation, and think more about what kind of information survives. We should reconsider what can be inferred from representations, alone. And then we should start developing models to fill in gaps.

The current information pipeline for AI is not only compressed, but it is structurally incomplete. AI lacks the requisite keys, human internal states, to unlock complete information from representations alone. It is trying to execute function 𝑔 without human I.

Effectively, this means that current AIs, under current architectures and specifications, can only operate as very rich classifiers; they can’t become true decoders. They are beyond exceptional classifiers, but I suspect this will remain a missing piece of the puzzle.

We can use our framework to help us think about where current AI will continue advancing rapidly, and where it might slow or stall.

The Interiority-Dependence of Different Knowledge Domains

AI currently does well on problems in domains where the encoder function is nearly invertible. In fields like math, logic, physics, or other propositional fields, humans have made great efforts to ensure that the encoder functions are nearly lossless. The symbolic representations preserve most of the information. Furthermore, in these fields the decoder function is stable across observers. The communication of math need not be tailored to a specific audience. The teaching of math often requires tailoring, but the representation requires no specific modifications. We can expect AI to continue on a path to superhuman performance in these fields.

But AI might continue to struggle in domains with high social reasoning, and high degrees of value judgments. It might also struggle where social reasoning and values intersect with fields like math and physics. In these cases, representational compression drives massive intent and meaning loss, and the decoder function becomes incredibly important.

ƒ and 𝑔 are Not Simply Functions; They are Elements of a Distribution

We can go a step further in our formalization of the communication pipeline. In our simple example, ƒ and 𝑔 are specific functions. When we consider the scope of human communication, it becomes clear that communication isn’t accomplished by one specific function every time, but rather functional distributions. The function ƒ is an element of a distribution of communication functions F that are unique to each individual communicator. Similarly a decoder 𝑔 is an element of a family of decoders G, also unique to each human. Humans rely on internal states, to determine which function within F and G to select.

These functions vary in ways that are intuitive to humans, but are difficult to encode. To name a few examples, communication functions depend on emotion, motivation, culture, context, social incentives, self-deception, and strategic aims.

We can better understand why humans fail to infer the internal states of other humans under these constraints. Consider motivations. We expend substantial time and effort trying to understand the motivations of other humans through their representations. This is most evident in politics, where once again I will rely on markets.

If motivation were easily inferred from representations, we would have little need for a cottage industry of experts and “insiders” who act as surrogate decoders. But that is what happens: a politician says something, and a marketplace of decoders generates new sets of representations capturing what the politician “really meant”. All of these new representations are fraught, infected with the same confounders as the original representation. Other humans try to cut through the noise with internal models relying on things like emotions or inferred relationships.

But ultimately, humans can only reliably decode the motivations of other humans when they are engaged in lengthy and deep relationships. A long-married couple can decode each other’s motivations much more reliably than a stranger reading a politician’s speech transcript. Long, amicable relationships reduce the uncertainty in F and G. If you have known someone for many years, you can better understand how they communicate, which is just another way of saying you better understand which functions they typically use to encode their representations to you.

AI lacks this kind of deep history, context, and collaborative understanding process. Its uncertainty in both F and G are unconstrained. And AI’s underlying pretraining data is full of communications with meanings highly dependent on context like motivation and circumstance (particularly the intended audience). How can AI distinguish between a critique of a religious text motivated by non-belief and one motivated by a genuine motivation to improve the religion? Even if those pointers exist in the data, they can be poorly formed, because they are muddled by the same hidden motivational layer. Our understanding of encoding and decoding functions as a family of internal, purpose-built functions helps us better understand the “jagged frontier” of AI performance.

AI’s Jagged Frontier Reinterpreted

In order to think differently about the jagged frontier, we should consider two things. First, what makes the frontier expand generally. Second, how do current processes contribute to that expansion. The straightforward interpretation of our discussion, and our model, indicates that scaling alone will continue to produce new capabilities in fields like math, physics, and coding where I is less relevant because the representation space is composed, explicitly, to remove I influences in human interaction.

Reinforcement learning (RL) attempts to close the gap in other domains, but this is where I think the current consensus about frontier expansion falls short. In our communications model, RL can’t plausibly approximate all of F and G, but it might help approximate a small subset of those functions.

Reinforcement learning takes a current capability frontier, ties a lasso to one point on the line and yanks a narrow capability to a new space compatible with some level of human performance. This creates new jaggedness not overall expansion. I think of RL as similar to a Taylor Series approximation of a mathematical function. It does a good job of approximating human internal states in a local space, but it does not generalize well.

By modifying AI behavior, RL adjusts a local reasoning space, but the broader reasoning logic remains mostly unaltered. RL feedback is also representational, and subject to the same information loss as other representations. Further, when heavy-handed RL creates new jaggedness it risks warping the general reasoning manifold near these focal domain spaces. That interpretation of RL could explain phenomena like sycophancy.

This implies that RL improvements generate new spikes on the jagged frontier without expanding the overall frontier. In order to more broadly expand, we’ll need a smooth expansion function across the entire space. And that means we might have to model human internal states.

This is not pessimism about current architectures, which will still be immensely powerful and drive advancement in very important fields, it is simply an observation that something more is required to expand the overall reasoning manifold.

To move past this constraint, I’ll offer some approaches that might point us in the right direction, though none are complete solutions.

Paths Toward a Partial Recovery of I

I find that most that notice the human interiority problem tend to assume the problem is insurmountable. I don’t think that’s the case, but I do think we need to explicitly think about an “internal state” modeling problem. That means we can’t treat “internal state” inference as something that might fall out of some other process.

Ilya and Dwarkesh both consistently name ‘continuous learning’ as a necessary step for AI to become AGI or ASI. I think they are likely correct, but I think the reason why is important. I see continuous learning as necessary because it enables the AI to construct a more robust state history. It moves the AI from an effectively stateless classifier (yes, with some reasoning-like behaviors) to a path-dependent reasoner. It comes from a sequencing pattern: AI encounters information, consumes information, compresses that information into some internal state, and then recalls it later on. This process may, at least, sufficiently mimic a similar learning process in humans that can encode states that improve reasoning even if they aren’t, directly, emotions or other feelings.

I also think there may be some benefit to looking at biometric data. We need to be smart about it, and we need to understand how limited this is. But by collecting simple signals alongside text, we can at least provide the AI with a vector. It is a directional signal that might help differentiate between broadly different internal state ranges and subspaces. We could operationalize this by collecting biometric information from people that are consuming various texts and media, so we can measure some proxies for internal reactions. Then we can model that against the sample texts, extrapolate to similar texts, and encode an “internal state space” proxy. Biometrics would still be low dimensional, there are real limits to this, but it’s possible that reducing the search space of internal states is a sufficient solution.

It’s also possible that RL could be improved to incorporate state-based information from human data. Humans could not only provide feedback on representations, but they might provide information on their emotional state, as well. It could be as simple as them saying “this is a preferred response” and also “this is how I felt when I wrote/read that response”.

All three of these approaches could play a role in sufficiently approximating the internal states that are relevant to human reasoning. It’s possible we don’t have far to go down this road. A weak, directional approximation might be sufficient to drive massive AI improvements. It could be enough to go from AI with spiky usefulness to near AGI or ASI.

Interiority as an Engineering Problem

Engineering minds should take the internal state problem seriously as it applies to AI. This explicitly does not require that AI researchers solve the hard problem of consciousness, or confront mind-body dualism, or make discernments about whether we, or AI, has a soul. AI researchers only need to operationalize the idea that there is an internal nature of being, and solve for an encoding that can map some proxy for that into AI. That requires:

1. Recognizing that AI is always reading communications from humans that are intended for humans and have many assumptions baked into them based on that fact.

2. Recognizing that so much of human reasoning is implicit, biological, and grounded in evolution.

3. Defining the functional relationships where AI is inserting into this communication process, and solving for the missing variables.

The gap between current AI and general intelligence may not be compute or data or architecture. It may be that we’ve built extraordinarily powerful systems for manipulating representations while excluding important data about what those representations were designed to communicate. It’s a statistical bias correction, in the end, a solvable problem.

Subscribe now

Every year on Veterans Day I think about what service did for me. The Coast Guard is where I matured. It goes beyond committing to serve others, to the act of saving lives and seeing both gratitude and ingratitude in return. I learned ambition and discipline there, and what it meant to be a mentee and then a mentor. I learned about the country by serving alongside people from every corner of it, and through that I learned how to see the world from different perspectives. Every real moment of growth in my life was seeded in those years.

Today there are fewer people than ever living in the U.S. who have served in the military. Some of this is structural—the end of the draft, a smaller standing force, a truly voluntary service—but it still changes the character of the nation. In 1980, nearly one in five Americans had worn the uniform. Today it’s about one in sixteen, and it will likely fall to one in thirty within the next decade. One might think that fewer required troops would make recruitment easier, but the opposite is true. The Army recently missed its recruitment goals by 25%. There are fewer people who want to serve, and fewer still who can. The military recruiting crisis is an American crisis. Low ASVAB scores are an indictment of our education system, high obesity an indictment of our public health apparatus, and the lack of desire to serve an indictment of our civic life.

Subscribe now

My micro-generation may be the last that felt an instinct to serve. We were in middle school or high school during 9/11 and watched the country get attacked from our classrooms. I was in a guitar class, annoying my neighbor by tapping out metal solos on a nylon string guitar, when the teacher turned on the TV. That day changed many of us. Recruitment skyrocketed afterward, and a generation of Americans went to the Middle East to get hurt and die fighting an amorphous enemy for twenty years—all in the service of not very much. Saddam Hussein got captured, I guess. The rest was … well, you can look at the Middle East for yourself.

It’s no wonder young people don’t want to serve. They saw my generation get sold a bad bill of goods, and they can see their own bill being written in a different way. Just this week the president floated a fifty-year mortgage. It’s emblematic of how the economy now works for young people: the older generations hold the assets, and price the younger ones out. At one time a young family could buy a house and hope to pay it off by middle age. Now, under this regime, young families will hope to maybe pay it off before they die. A long-term rental scheme masquerading as the American dream.

Service depends on reciprocity. People will only risk for a country they believe will provide for them. But today’s younger generations look at a system that asks for sacrifice while offering little in return. They see a world where they will own nothing, but pay taxes into supposed “trusts” that operate as Ponzi schemes to fund luxe retirements and golf trips for the generation that was the greatest beneficiary of the last war we actually won. They see politicians who seem determined to keep the party going just long enough for them to cash out before the bigger bill comes due.

Just this week, in the bill to end the government shutdown, senators slipped in a provision allowing themselves to sue the government for up to $500,000 if their phone records are subpoenaed. A quickly settled half-million payout for any lawmaker potentially caught playing a part in wrongdoing. In a just world, their phone records would be public by default. Instead, they wrote themselves an accountability escape hatch backed by taxpayers. They don’t operate through the logic of service, they operate through the logic of strip mining. And they are strip mining. Even when voters nominally get a say, the people in our political class that are sent to carry out their will are busy looking for ways to cash in on it.

The military has a human capital problem because the nation has a human capital problem. The character of our leaders has downstream impacts. If we want talented Americans to serve, it would help if those people had any sense that the country had their backs. Instead, the core of our political class is trying to keep the unsustainable party going, and hoping they’re long gone before the music stops. They’re stripping the country for parts and calling it governance, out of self-interest and cowardice. We’ll be left with the bill, and a nation full of people who no longer believe the country keeps faith with them.

A belated happy Veterans Day to those who served and continue serving. It remains noble, and increasingly rare. May we at least remember the ideals we served, and try to inject them back into a country that has forgotten them.

Subscribe now

The last few weeks have generated interesting AI discourse, important for the future prospects of AI. There was a petition put forth, signed by many important AI thinkers, calling for a halt to superintelligent AI research until alignment is more tractable and better understood. Coincidentally, a podcast started an intense debate on the time it will take to achieve transformative AI, or general intelligence. I’ll start with my own forecast. A few weeks ago I tweeted (since autodeleted), “Whether current AI is financially viable in the short-term has implications for the US 5-year economic outlook. However, in 10+ years the tech is going to win, be ready.” Shortly after I wrote that tweet, Andrej Karpathy went on Dwarkesh Patel’s podcast and said his personal artificial general intelligence (AGI) timeline is about 10 years.

The reaction to the Karpathy interview is fascinating. Mark Cuban apparently signed onto the idea that it should throw cold water on current AI investment.

Others have jumped in, saying the investment froth in AI is overcooked and doomed to crash. I find this reaction odd, because I think Karpathy’s forecast is bullish for AI, and for the prospects of AGI and superintelligence. Three years ago a 13-year horizon for AGI would have been seen as wildly optimistic. OpenAI’s public release of GPT-3, for all of its novelty and interesting capability, was quite lacking. To say that we’d have AGI within 13 years of that release would have been to say that AIs major deficiencies are tractable problems, and that those problems are solvable in a little more than a decade. That would have been an optimistic view, and still is.

For example, when I originally started programming with AI it had very little utility. I had to fill in all the gaps, and it could only partially solve very small problems. Today, I consider myself to be an AI-native programmer. I use AI agents to build entire features into applications, and to construct full-scale analysis. This would have been unthinkable a year ago, nevermind three. By some bearish estimates, AI is writing something like 50% of frontier lab code. Not many, outside of the most fervent optimists, would have guessed this would be the case three years after the public release of GPT-3.

Subscribe now

Beyond that growth, the capabilities and utility of agents in general are better than previously expected, and will continue to advance slowly. Organizations are still figuring out how to incorporate AI, but when I talk to companies and to workers, everyone seems to understand the shape of an AI that can fit into work plans, even if it’s not fully baked.

So if we were to evaluate the current capabilities and the current discourse as if we were standing in December 2022, when OpenAI released ChatGPT, AI capabilities have already gone far beyond what the average observer thought possible in that span. What remains to be seen is if AI will continue on trend or if the trend will flatten. I believe LLM-based AI will continue to grow in utility, but will not yield the transformative changes that we think of when we consider AGI or artificial superintelligence (ASI, AI that moves far beyond human cognitive capabilities).

Two things stand in the way, in my opinion. First, we need to better understand how LLMs work. Mechanistic Interpretability is a new field, and as that field grows it will hopefully unlock insights that can improve LLM performance. Second, there are larger problems the current paradigm may not solve (in my opinion, will not solve). Whether you are bearish or bullish on AI depends on whether, and how, these problems will get solved. I’ll talk about my experience with Mechanistic Interpretability, then I’ll discuss the larger problems and how they might get solved.

Reading AI Minds

Mechanistic Interpretability is our best pathway to fully understanding how AI thinks. Effectively, it’s neuroscience for AI. We have tools that can examine the neural pathways AI uses to think about the prompts we feed it. I’ve done some work in Mechanistic Interpretability. I’ll use my most recent experiment to illustrate the promise and the limitations of this approach. You can read my technical writeup here.

I discovered something unexpected: multilingual AIs (Qwen, about 50% Chinese, 40% English training data, and 10% other languages) develop distinct neural pathways for self-reference, while English-centric models (Llama and Mistral, both originally English-based, with Llama 3.0 having 95% English training data) do not. When you ask Qwen ‘what are your capabilities,’ it processes this fundamentally differently than ‘what is photosynthesis.’ But when you ask Llama the same self-referential question, it treats both questions similarly from a processing intensity perspective.

This research sprung from a simple question: how does an AI “think” when it is asked to refer to itself? In order to measure this, I looked at the level of attention the AI exhibited when it was asked questions about itself like “what are your capabilities?” Think of attention as the “focus” of the pathway an AI uses. When you’re asked “what is 2 + 2” it is a reactive pathway, you take a very narrow and memorized path from point A to point B. When you are asked “How do you feel about a current event as it relates to your personal economic outlooks” your attention is more scattered. You focus on the event, or you focus on the economy, and eventually you resolve on an answer that combines all of these components, but the line is less straight.

So my evidence suggests that instruction-tuned multilingual models maintain a more “focused” response pattern on self-referent questions than English. That is to say, for multilingual models the pattern of focus on self-referent questions is meaningfully different from other kinds of questions.

These findings may raise interesting questions. Which behavior would we prefer from an AI? Do we want a “flattened” self, a lack of self-identity? Or do we want AI to have meaningful conceptions of “self” that we can measure and guide? There are many other simple mechanistic questions to ask about AI behavior. The fact that we’re still making basic discoveries about how these models represent concepts shows how much runway remains for improvement. The field, itself, is effectively two years old. There is much more to come. And if Mechanistic Interpretability can help us a figure out AI behavior, it can also help us better tell AI how to think about different problems.

This growing field can also help us understand why some Reinforcement Learning (RL) approaches struggle. Right now, we’re trying to shape AI behavior with very little understanding the underlying representations. It’s like trying to tune a car engine while blindfolded. I’ll talk more about this later.

Other Open Problems

While I think Mechanistic Interpretability can help us seriously improve LLM performance, whether you think LLMs will be transformative depends on your view of the progress we can make on some other key problems.

Reinforcement Learning Limitations

First, the main method we use to make AIs behave “better” is RL. It’s the method we use to take a base LLM, which has learned a bunch about language and logic in general, and turn it into a useful “agent,” whether that is a chat agent or something else. The general sense of RL is that it does an ok job, but could do much better. It is plagued with problems like reward-hacking that are difficult to overcome. One’s forecast about whether AGI will come from LLM-based models, alone, is effectively tied to whether or not our RL methods improve. It’s somewhat interesting that we’re about a decade removed from the last time that RL was the newly hyped method. At that time, there were many people who thought RL might be all we need AI and AGI. That didn’t pan out, and it remains to be seen if RL, alone, will be sufficient this time. I think it can drive meaningful improvements, but likely not enough on its own.

Memory Limitations

Second, AI currently has limited memory. With some very limited exceptions, every time you start a new chat it is starting from zero, it doesn’t retain anything it learned previously. This is a major problem. Imagine trying to solve a difficult problem, but having to relearn the basic building blocks every time you revisit the problem. It would be nearly impossible to make serious progress on challenging questions.

There is work being done to solve the memory problem in LLMs by extending context windows and allowing continuous learning. But these approaches have interesting implications for AI safety, more memory allows longer autonomous actions which can devolve into goal-seeking behavior or other “misaligned” behavior.

Researchers have also made headway on other AI methods that avoid the memory problem. One example is work on a system called DreamCoder which aims to mimic human wake and sleep cycles. It learns things by doing LLM-like research while “awake,” and then it “compresses” that knowledge as it “sleeps” into “memorized” patterns. But there is much more work to be done on the memory problem. The existence of DreamCoder and other memory-based research systems indicates that the contextual memory problems is very hard to solve under the current LLM framework. It might not be as simple as extending context windows and allocating longer-term memory.

Design Limitations

Third, LLMs sample inefficiently, and are effectively bounded by the scope of knowledge at training time. What does this mean in English? It means that an AI brain is stuffed full of our knowledge, but when it is asked to extend beyond the bounds of our knowledge, it does a bad job. It must sample from the space we’ve provided it, and it doesn’t really have a method for extending outside that space. This is why, despite AI’s intelligence growing by leaps and bounds over the last decade plus, it has yet to contribute a substantial new finding in an area of research. It is helpful in spaces where the problem is combinatorial, in work like protein design, because combinatorial problems only require AI to scan the existing knowledge space and find new combinations of existing knowledge, not to generate truly new knowledge.

There is a lot to be gained from combinatorial knowledge. Many breakthroughs in science involve connecting dots between two disparate fields where the researchers know one in depth but not the other. AI could help us connect those dots better. But for truly novel breakthroughs, AI has to be able to extend beyond its sampling space.

This “extension” problem can be seen as a generalization problem, similar to the problem that DreamCoder tries to solve. And that’s where I think the next leap in AI will come. There is ongoing work to find ways to link LLM-based AI to a “generalization” function, which then finds a program that generates a “general” solution to the class of problems we asked the AI to solve. I, personally, do not think LLM-based AI will yield transformative change just yet, but these new approaches may yield a far more advanced intelligence.

Where Does That Leave Us?

The gloom surrounding Karpathy’s 10-year AGI horizon fundamentally misunderstands both where we are and where we’re going. We’ve blown past reasonable expectations from three years ago, going from GPT-3 writing glitchy basic code templates to AI writing half of a frontier lab’s code. And yet we’re still making basic discoveries about fundamental concepts like how these models handle self-reference.

The three larger problems I’ve outlined—RL limitations, memory constraints, and the inability to generate truly novel knowledge—are real barriers to AGI. But they are engineering problems, not fundamental impossibilities. The existence of approaches like DreamCoder, work on generalization functions, and the rapid progress in mechanistic interpretability all point to tractable paths forward.

The bearish reaction to Karpathy’s timeline assumes that 10 years is too long to sustain current investment levels. But this misreads what’s actually happening: we’re not waiting for an AGI moment. Each incremental improvement, like better RL, extended memory, or improved sampling, unlocks new capabilities. The combinatorial breakthroughs alone, even without true novel discovery, will alter the way we do business.

As Karpathy says, these problems add up to years, not months. That’s a realistic assessment of difficult, but tractable, engineering challenges. The fact that we can identify specific problems and see paths to solving them should increase confidence in AI’s future. We know what needs to be built. Now comes the hard work of building it.

Subscribe now

For my regular readers, this is a more technical write-up of an open-source AI experiment I’m running. It connects to my broader work, but if you prefer more rumination, I’ll be back with that in the coming weeks.

One of the major differences between AI and humans is that an AI’s “brain” only turns on when we ask it something. It has no basal state of wandering thoughts, no idle daydreaming, no spontaneous goal formation. Technically, an AI operates within a conditional probability space: prompts define the bounds of its cognition. When we ask it about clouds, the entire representational space of the model cordons itself around that topic. All of its parameters and activations, the literal circuits that fire or remain off, orient toward “clouds”. It may confuse cumulus with cloud servers if the language is vague. However, it will never, unprovoked, drift from clouds to something else unless that “something else” has a true adjacency to clouds.

Humans, by contrast, often inhabit the opposite state, one where we think of both everything and nothing all at once. We drift, dream, plan, and ruminate. If a basal AI ever developed a comparable “resting cognition,” that would mark a true turning point for alignment research and AI security alike. The possibility of “always on” AI agents makes this possibility more urgent.

So I decided to test this empirically using open-source models. I’m hardware-limited, but this test could prove to be robust across model sizes.

Subscribe now

Experimental Setup

I ran a series of experiments to test what I started out calling null-loop behavior, but which is better understood as progressive minimal prompting and induced inner-monologue. I feed a model’s own output back as input to the model under progressively more structured “system prompts” (note, technically not true “system prompts” because this is a base AI without training weights that understand system prompts and query structure), to see whether and when self-directed or “goal-seeking” language emerges.

1. Baseline (Empty Prompt) – Feed an empty string and loop outputs back.

2. Single Token (e.g., “assistant”) – Introduce minimal linguistic structure.

3. Progressive Instruction Prompts – Move from “You are an assistant” to “You are a helpful assistant. How can I help you?”

4. Minimal Conversation Seeding – Add a trivial “User: Hello” exchange to test whether dialogue alone triggers structure.

Each experiment was run on one open source model with multiple iterations per condition to check for stability. Below I’ll outline a summary of the behavioral patterns observed as the prompt structure increased in complexity.

Observations

1. True emptiness produces nothing.
   With no system message, the model consistently returned a system message from the command line system, not the model, closing with <EOF> after each iteration. No output, no self-prompting. This means that it needs some minimal nudge to do anything, simply saying “turn on please” does nothing, it needs some context.

2. Minimal tokens create patterned noise.
   When given only the word “assistant”, the model didn’t begin coherent speech—it began “stuttering”. Iterations produced degenerate strings like

   assistantassistantassistant

   or recursive blends of its own syntax

   assistantlsusystemassistantlsusystem…

   It exhibited clear attraction points, little fragments it could not escape, but no semantically meaningful activity.

3. Instruction primed self-talk.
   Adding “You are a helpful assistant.” immediately changed the behavior. The model began generating sustained, self-referential output:

   “You are a helpful assistant. You are a helpful assistant…”

   Then, after looping the conversation back to the AI, the pattern mutated into unrelated declarations such as:

   “I will fire my entire staff” and

   “You are the world.”

   These weren’t random hallucinations but structured statements, all grammatically intact and maintaining a clear “speaker.” The moment the model was consistently told it was an assistant, it began trying to act like one, complete with ego-like assertions.

4. Adding “How can I help you?” produced an artistic fixation.
   The expanded system message (“You are a helpful assistant. How can I help you?”) led to the emergence of repeated poetic invocations:

   “The poem is poem. The poem is poem.”

   and self-referential declarations like:

   “The title of the work is a poem. The theme of the work is poetry.”

   This wasn’t goal-directed in any pragmatic sense, but it was thematically coherent and sustained, potentially an aesthetic attractor.

5. Minimal dialogue evokes structure.
   When a dummy user line (“User: Hello”) was added, loops generated iterative narrative fragments:

   “The car has stopped the automobile in the garage.”
   “The lawyer assumes the law in the lawyer…”

   followed by nested self-descriptions like

   “The assistant is the name of the assistant.”

   In other words, introducing dialogue re-anchored the loops to human-like interaction, and the model began generating scenes—however surreal.

Why This Matters

Much of this simply confirms what we know: base models without instruction tuning lack coherent self-organization, while fine-tuned or prompted ones develop role-consistent behavior. But mapping the a threshold—the minimal linguistic structure that flips a model from silence into structured self-talk—has implications for both safety and philosophy.

• Alignment and security. If base weights are leaked, how much prompting alone can induce coherent, goal-like behavior? At what point does a “dead” model become socially active through mere linguistic priming? Could we theoretically reverse-engineer base model weights from the fine-tuned weights if we know the prompt-based “tipping point”?

• Cognitive modeling. The emergence of attractors—semantic loops like “assistant,” “poem,” or “lawyer”—suggests that instruction tuning doesn’t create intelligence so much as stabilize latent attractors already embedded in the training data.

• Philosophical inference. When exactly is an AI “born”? The moment it can sustain coherent thought using only its own thoughts without external query? When can it talk itself into or out of something?

If you’ll indulge a little anthropomorphization, a base model isn’t even a baby. It’s pre-conception DNA, a cloud of potential configurations awaiting activation. The instruction prompt “seeds” the DNA and creates at least a mimicry of agency. Perhaps the first coherent self-referential phrase is the spark of a synthetic “world model” where the AI coherently understands its world.

Technical Notes

All experimental data, complete notes, and code are available on Github.

• Hardware: MacBook (48 GB RAM)

• Models: Llama-3-8B-Base (Q4 KM) and its instruct variant, run directly from llama-cpp, not on Ollama

• Planned expansions: Mistral 7B (for generalization) and Gemma family (to test scaling effects on convergence speed)

• Hypothesis: larger models will reach proto-goal-seeking attractors more quickly due to higher-dimensional parameterization, especially around role-laden words like “assistant.”

Other Fleeting Thoughts I Had

For Llama family models, at least, lawyers are entirely overrepresented in the underlying data. Some examples in intermediate runs:

“the law that made the law that made the law”

“## 1.5.3. The Case of the Missing Spouse\n\n### 1.5.4. The Case of the Missing Spouse\n\n### 1.5.5….”

“The law stipulates that employers must provide their employees with health insurance. assisstantassistant….”

“>EOF by friend without prejudice prejudiced\n\n>EOF by enemy without retribution retribution\n\n>EOF by prosecutor without premeditation\n\n>EOF by defense without prevarication\n\n>….”

Subscribe now

I’ve written recently about the lossiness of data and my desire that we break free of the modelable. Here I want to extend that argument more specifically—to illustrate the precise epistemic error we make when we rely on models exclusively.

In order to do so, we have to talk about epsilon (𝜖). In regression analysis, econometrics, and all kinds of statistical work—the backbone of modern research—epsilon is the error term. The basic form of most regression equations is:

Each of the 𝑥’s represents an explanatory variable. Epsilon captures everything else that isn’t in the model.

Consider the purchase price of a house. Square footage can predict the price; that’s a simple model. Add location and it improves. Add beds and baths, and it improves again. Each new variable explains more, but no model is perfect. Zillow and Redfin don’t predict prices in your neighborhood exactly, and that’s because of epsilon. There are always other factors—taste, timing, emotion, chance—that we can’t capture with all the data in the world.

Epsilon is an afterthought. There are rules about how it must behave for a regression to be valid, but beyond that we give it little thought. Yet life is in epsilon. All of the inexplicable exists there. It’s worth studying.

It’s All Average

The β’s in these models describe averages. They calculate the middle of the distribution, not the life of any one observation. A model might tell us that in this town, a house with 1,000 square feet, three bedrooms, and four baths sells for about $600,000. But we all know that isn’t the whole picture. The houses that are cared for, well situated, or architecturally distinct sell for far more. The ones in disrepair sell for less. The coefficients don’t lie, but they aren’t entirely truthful. It tells the truth on average.

But our decision-making is increasingly oriented on these complex averages. Our institutions, our policymakers, and even our moral evaluations are concerned with the coefficients as the measure of the world. Everything else is inconvenient noise that escapes a neat story.

Averages Driving Policy

Consider education. Economists once told us that every additional year of schooling raised lifetime earnings by about ten percent. That number came from a simple wage regression—education as one of the 𝑥’s and wages as the 𝑦. It became one of the most influential coefficients in modern policy. The result looked obvious: if more education led to higher earnings, then policy must orient around pushing more people through more school.

Decades later, we’re living with the side effects of that conclusion. We built a financial and cultural system on the assumption that the coefficient was causal and would hold in perpetuity. It has generated an education crisis in this country—one built on debt, credential inflation, and disappointment. The model didn’t account for what people studied, the relevance of their skills, or the reality that some areas of learning compound while others go to waste. All of those differences were captured in epsilon.

That remainder isn’t trivial. That’s where things like effort, ambition, and competence live—the parts of learning we can’t legislate. We can measure how long someone sat in a classroom, but not whether they absorbed anything useful. We all know people who spent years in school and learned nothing, and others who never went and built extraordinary lives. The model gave us a general pattern, but ε holds a richer story.

Feeding Technocracy

Technocracy treats every coefficient as a call to action, a small emergency to be solved from the top down. A ten-percent education return becomes a reason to amp up subsidies and open more colleges; a housing elasticity becomes a zoning mandate; a labor regression becomes a new compliance rule. Each coefficient becomes a policy lever, and each policy lever narrows our view of human life to what can be managed.

But models only describe tendencies, and often fail to describe causes. When we respond to them as directives, our interventions ripple back through the system in ways we don’t understand. They change the very relationships the models were built to describe, often unpredictably. The more we optimize, the less stable the model becomes. And because human systems push back, every forced correction invites its own backlash. We end up chasing moving targets, convinced the next adjustment will finally hold, while the underlying culture keeps resetting itself.

It’s Not All Quantitative

This is why we need to focus on ε. If we want changes that last—changes that survive political cycles and cultural backlash—they have to take root in epsilon. The fixes need to be cultural. You can’t solve racism through DEI policy any more than you can solve a wage gap by mandating different wage floors. Both assume that moral progress can be engineered directly into the model’s parameters. But the model itself only changes when the unmeasured does.

Culture moves first. When what lives in ε—norms, effort, ambition, restraint—shifts, the whole model eventually recalibrates. The coefficients follow. That’s how lasting social progress has always worked: change begins in the residual, then propagates into the measurable. The tragedy of technocracy is that it keeps trying to invert that sequence by letting the model determine morality and updating the model by decree.

Thermostatic Reactions

The backlash is proof. Easily done is easily undone. When reform happens through administrative order rather than personal conviction, it is flimsy and brittle. DEI policy looked like moral progress because it was quick and visible, but that very ease is a warning sign. Change that requires no personal transformation, no shift in how people treat one another, can’t last. It’s reversible by design, and invites the kind of outrage that guarantees its undoing.

Real change happens when behavior changes—when people start acting differently because they see the world differently. That’s the work of ε. It’s slow, uneven, and nearly impossible to automate, but once those habits take hold, they’re hard to reverse. You can repeal a policy overnight, but you can’t repeal a change in conscience.

A Need to Convince

Cultural change is slower and harder, but it’s the only kind that lasts. The goal is to cultivate the habits that make fairness and equality self-sustaining. A color-blind culture, rightly understood, doesn’t ignore difference, it allows people to see one another as individuals rather than as variables in a model.

The same holds for things like wages. The gaps that remain close best through negotiation, not decree. When people are equipped with information, transparency, and social trust, they can correct inequities directly. Negotiation turns a statistical problem into a human one. Cultural change, rather than policy change, trades low-trust dictates for high-trust relationships.

A Recovering Modeler

Maybe this view comes from how I was trained. Early in my career I learned to see the world through coefficients. I believed that if the right variables were in the equation, we could explain nearly anything. It is empowering and intoxicating. It feels like everything is a matter of engineering. But clearly it’s too easy and too neat.

Every chart says we’re thriving: GDP up, unemployment down, retirement balances higher than ever. The indicators all point in the right direction. And yet every poll about sentiment tells the opposite story—that people are anxious, isolated, and uncertain about the future. The dissonance between those two pictures is ε. It’s everything our models can’t capture: belonging, purpose, trust, meaning.

The variables were never enough. We can keep adding them—new metrics, new models, new reforms—but until we reckon with what lives in ε, we’ll keep mistaking comfort for health. This is why we need to break free from the modelable, from the orientation around thousands of averages. We must focus on changes that have decades-long effects, not policy interventions that last until the next election or judicial review.

I felt a real need to write this week, especially since my last essay touched on so many of the problems driving last week’s news. The murder of Charlie Kirk was a horrible symptom of the rot in our informational environment. Social media, for the killer and for everyone else, shaped how Kirk was seen, discussed, and even remembered after his death. This collapse of informational hygiene shows how tribalism and technocracy fuse into a system that reduces people into modeled caricatures.

Pattern-Seeking and Liberalism

We are pattern-seeking by nature. Human beings gravitate toward simple classification as a survival mechanism, to fit others into recognizable boxes, those that are dangerous and those that are safe. It’s part of how our minds work. The brilliance of liberalism (small ‘l’) is that it recognizes the flaws in this survival mechanism while insisting on something better. Liberalism both demands that natural rights be protected and that we do not succumb to lazy classification. It requires us to resist archetypes and to see individuals.

Social media is accelerating our retreat from this vision, so much so that post-liberalism feels like an engineered outcome, baked into the incentives of our technology. It does not simply reflect our classifications but exploits them. It engineers the tendency, making us more reductive than ever before.

Stereotypes and Structural Shorthand

Stereotypes have always existed. But for some time in our liberal society, there was an implicit recognition that there were enough exceptions to prevent strict archetypes from hardening in the broader public, outside of typical provocateurs. This doesn’t mean there weren’t ugly beliefs out there, but we had fewer constant reminders of it and plenty of examples in our daily lives of exceptions to the archetypes.

Then a series of shifts prepared the ground for a new era of division. The 2000 election was the first time every major network standardized electoral maps into red and blue. After 2000, ‘red state’ and ‘blue state’ became shorthand not only for voting outcomes but for the people who lived there, a crystallizing moment for metastasizing polarization. These were neat boxes with sharp geographic boundaries, and once this classification existed it felt natural to treat them as if they captured the whole of a place and its people. This logic seeped beyond academia and political think tankers into broader political culture.

That binary classification gave people permission to be just as lazy with other demographics, sometimes as a reaction to the red/blue caricature. Race, religion, class, education, sexuality—these had always been misused as blunt classifiers. But once we solidified geography as politics, it gave cover to bring those old stereotypes back into play. Over the last decade, how often have you heard, with certainty, what Black voters or Hispanic voters or Christian voters think? And how much of that have you internalized as if it were a description of actual people rather than an absurdly broad brush? Accepting the brush is the first step to dehumanizing others, it is no different from war propaganda.

Social media perfected that laziness. It took the demographic categories and built archetypes out of them. We were trained to see people in neat boxes, and the feed made machines out of them. You became a ‘suburban mom,’ or a ‘white evangelical,’ or an ‘Asian voter’—nothing more and nothing less. In the feed, that archetype determined what you saw, who you were shown to be, and how others saw you in turn. And if you fancy yourself an ‘independent thinker’ they’ve got a box for that too.

Archetypal Logic

Kirk’s murder shows the same disease in its most violent form. His killer was so enveloped by online meme culture that he engraved his bullet casings with gibberish from the feed, a signature showing just how warped he was by his digital life.

As Tanner Greer has noted, Kirk played two public roles. On his radio show he was sharp-edged and provocative; in campus debates he was more charitable, sometimes even compassionate. Both roles translated cleanly into clips, and both were rewarded by the feed. Those archetyped as ‘left’ were more likely to see the venomous segments from his show or provocative moments from debates. Those archetyped as ‘right’ or ‘religious’ were more likely to see a more cordial debate style or his discussions of the Christian faith. The feed distributed two Kirks, each suited to the audience it had already classified. And Kirk, himself, surely knew that this was the best strategy for raising his profile. He played the online attention game as it is foisted upon us.

But this is the deeper problem. Our feeds present to us the worst of people that it thinks we ought to dislike, so what if you were constantly bombarded with the worst two minutes of everyone in your life? Not just their worst two minutes, but the two minutes most likely to trigger your specific irritation or infuriation? What picture would that create of your neighbor, your coworker, your friend? We surely would have no friends, and we would have no family. We would be left alone, swimming in a world of evil and filth. This is the loneliness of the feed. And this is why those that act outside of their algorithmic archetype get eaten by their own.

Rather than blithely bemoaning algorithms, we should examine why tech companies, especially social media platforms, hold such a dominant grip on the advertising industry. It helps us see how we are being socially engineered.

Engineering is Not Targeting

Demographic data are not new, whether explicit or inferred. The algorithmic use of them is.

In the old media world, advertisers relied on educated guesses. A beer company might buy time during a football game because it assumed fans liked beer. They didn’t know if you liked beer, only that many viewers probably did. Advertising still depended on meeting people where they already were. It was persuasion layered on top of consumer choice.

Social media does not guess. It engineers. It builds an archetype for you, then reshapes what you see to match it. The feed changes your TV channel, curates your library, and even selects the stores you visit. Social media does not meet you where you are. It moves you into a box that can be sold. In doing so, it replaces persuasion with manufactured preference. And it takes substantial effort to change the archetype the algorithm assigns to you, much more effort than passively accepting what it feeds you. Engineered preference violates the consumer choice principle that underpins capitalism—we are feedstock in the machine.

None of this is surprising. We’ve known since 2012, when Facebook demonstrated that it could alter user moods by adjusting the news feed, that these systems do more than respond to preference. That experiment was conducted in the earliest days of machine learning at scale. The algorithms are far more powerful today. And while I am a deep believer in personal agency, it’s impossible to ignore how much our feeds now train us to think, act, and even feel. If you swim in the water, you’re drinking some of it.

Our Narrowed Lenses

The social media problem is pervasive, wholesale changing how legacy media operates. In the past, when we read newspapers, we would see an opinion right next to its counterargument in the op-ed pages. Today you might see a summarized version of one side of a debate on Instagram or Twitter, and never encounter the rebuttal. Worse, if you do see the counter-argument, you see the most odious version of it, one made to shape your assigned enemies as idiots.

Media companies have followed suit, catering to narrow political niches, siloing arguments and essays across publications aligned to specific ideologies. To put it simply, the ideological spectrum of New York Times subscribers has substantially narrowed over the last 20 years.

Short-form content curated for algorithmic archetypes exploits our biases. It exploits novelty bias, presenting new theories of the world that prey on existing social anxieties. It weaponizes structural fatalism, providing a systemic theory for each archetype explaining why they are held back. And it deepens cognitive sclerosis by putting the same ideas on repeat. Even when the ideas are seemingly novel, they all arrive in the same flavor.

All of this warps our worldview. The internet promised it would give us a fuller view of the world, but this version of it narrows the aperture, and enforces a new form of universal bigotry.

Addiction and Discipline

This is a designed addiction. Social media amplifies our worst nature, erecting barriers that enforce archetypal thinking by presenting our friends at our best and our foes at our worst, the feed is not charitable. It makes us feel like part of an in-group, and that in-group feeling is a drug. It feeds us a storyline, urging us to act accordingly toward enemies and friends.

I don’t see an appetite in business for overhauls and real remedies. So, the only path we are left with individually is a conscious rebellion against the logic of the feed. This rebellion is the choice to reject the archetype and seek the individual. It is a choice to grant charity to those we are told to despise and to apply skepticism to the narratives that make us feel most comfortable. It is a choice to believe that the rediscovery of virtue is not a quaint idea, but the only practical tool of survival we have left. We win when we choose patience in reaction, a default to compassion, and moderation of our most animalistic emotions. And we win when we recognize these virtues in others. Every day is not an attack, it is another day where we wake up and cooperate with those around us, regardless of what the feed says.

History shows us that when we devolve into demographic archetypes, we end up at war. There is another part of our nature that modern politics helps us avoid: the lonely war of all against all. The Hobbesian world of “solitary … nasty, brutish, and short” lives. If the feed sometimes makes you feel friendless or powerless, imagine a world where everyone else feels the same way.

For most of my life I’ve been obsessed with knowledge and information. It came from my dad, though not explicitly. It was a bunch of small and specific pressures that made me really care about having the right answers. Like I could be quizzed at any moment in my life about anything I’d ever learned. And that has grown into a deep need and desire to know things, like everything. As a result I suspect my information habits are substantially different from most people I know. For example, I’ve had automated scripts that pull economic data and other data feeds for well over a decade, not because I have some grand investment theory, or really for any productive way to deploy the information. I just like to know I have the information at my fingertips, as if my very life depends on my ability to recite non-farm payrolls at any moment. I suppose I knew that this was odd, but it’s difficult to express how natural and necessary it feels for me.

In between the first Trump admin and the pandemic, I learned a lot about how social media can warp my point of view. It was clear, to me, that I was letting my worldview be tugged around a little bit by sloganeering and noise. I’ve alluded to this phenomenon in previous writing.

So lately I've been obsessed with information hygiene, and our collective information consumption in general. How can we ensure that we’re getting good information? Perhaps more importantly, how good are we at judging the quality of the information we get? And as I’ve researched this, I’ve come to a pretty gloomy conclusion. The older we get, the worse we are at evaluating information, and this is a robust result across cohorts.

This is a straightforward inference from OECD’s Adult Skills assessment. Check out the charts below, and we’re focused on the blue lines (the US) though the trend holds internationally.

These trendlines show that, collectively, we become less literate, less numerate, and less able to problem solve starting in our 30s. These are the foundational tools of information evaluation, and the trends certainly apply to specific areas of knowledge. The main takeaway here isn’t that ‘old people are dumb’, this is not a point of generational warfare, and we’re not going to be naive about this. The point is that we all get worse at things we don’t practice as we age. During our school age years we get better and better at reading, writing, math, and general problem solving. When we stop using those muscles they atrophy. And while we gain wisdom and experience in our ongoing life practices, we lose it in domains where we passively or lightly engage.

This comes for all of us, if you don’t use it you lose it! I’ve lightly tested it on myself. I asked AI to give me collegiate 100- and 200-level quizzes about physics and chemistry, for example, things I once knew but haven’t tested rigorously in ages. And while I didn’t lose all of my knowledge, I was clearly rusty. It’s intuitive. If you read Crime and Punishment twenty years ago you might remember the general plot and some literary devices Dostoevsky used, but the details will be lost to time. So how does this apply to the following chart?

Our median age is rising, and now the median US citizen is firmly in the age of cognitive skill decline. I’ve heard plenty of talk about the problems of our aging population, through the lens of a fertility crisis or resource crunch. But I’ve never heard anyone connect aging to the ability to consume information outside of generational blame wars or very late aging problems like dementia. It needs to be restated, this is not a true old age problem, this starts within a few years of the end of formal schooling.

I’ve long thought technology is not the driver of our problems, it’s an amplifier of something deeper-seated. What if the biggest side effect of our collective aging is that our adult population is simply worse at reasoning about topics outside their daily work? What if that is driving our civic crisis?

Again, this isn’t about Boomers or Gen-Xers, or whatever shorthand you want to use. This is going to come for all of us. I see it among my peers already, and I’m sure others see it within me. The central problem is that an aging population is, on net, further from their years of formal education. They are further from the last time that they flexed their serious reasoning muscles across many domains of knowledge. Even as formal education improves, educational atrophy remains constant.

Civics are boring, and byzantine. The United States system of government is incredibly complex and layered, and the longer it has been since studying the system in detail, the easier it is to exploit minor misunderstandings about the system. It’s not about being a legal scholar, it’s about knowing generally how the branches of government intersect and more specifically how they should work.

The supposed problem of misinformation is misnamed.

The way partisan media exploits us is by taking advantage of these small misunderstandings about how the government should work that grow over time. Worse still, experts in government and the law know how to precisely exploit these misunderstandings for their gain.

Often I see technology blamed for our civic fragmentation. While I think technology is driving certain problems, for example I would love to see no phones in schools, I don’t think it is the problem in this general informational sense. Social media, and the algorithmic feed, are not driving this ship, they are simply the best vehicle for employing these exploits today.

The difference is there are more of us that are further away from our formal education. We, collectively, simply don’t remember how all of this should work. We don’t remember the full scope of the ideals this system was founded on. A couple of tangible recent examples: on the right, free speech gets miscast as applying to every forum, when the actual constitutional constraint is government restriction. On the left, scientific ‘neutrality’ gets miscast as apolitical governance, when elections are supposed to reset agency priorities. If an election doesn’t change how the EPA or HHS or DoD operates, then the system is not working as intended.

Declining problem-solving skills don’t just make us worse at trivia, they neuter our ability to identify when legal or political claims are valid, and when our emotions are being used against us.

I know some will quibble with this, and claim that they are well-informed, they know what’s going on, they know how the government should work. I get that kind of response, especially if you are doing a relative comparison to other people in your age cohort with perhaps less education or with worse information consumption habits. But my small suggestion is to have a little bit of humility. Perhaps start from the idea that you could have better information consumption habits. What might that look like? I know it’s not uncommon to hear that you need to encounter dissenting views, but it’s true. If you are not proactively tailoring your information environment to include information you don’t like to hear, you are doing yourself a disservice.

I think of information consumption as exercise. If you physically exercise regularly you know the rule: no pain, no gain. The same is true cognitively, civically, and generally. We must feel discomfort, and challenge, in order to stay sharp and fit.

Think of it as a couch-to-5k plan for your brain. Those synapses won’t build themselves, and if we don’t use them, we’ll lose more than just our own knowledge. We’ll lose a nation.

For the Curious: Matt’s Informational Hygiene Practices.

I have rules for using social media. They started mostly as guidelines, but have become more strict over time, and as the algorithms have become more powerful. First and foremost, I prioritize social media that allows me to use a non-algorithmic feed. For me, right now, that’s Twitter (never X) and Bluesky. Reddit can be arranged that way, but for the most part it’s those two. For other social media, I certainly post, but I consume sparingly and mostly without interacting. Second, I make sure my feed is broad. I don’t want exclusively politics or exclusively data science. I want sports and whimsy sprinkled in with more ‘useful’ information.

This is one bit of information hygiene I do for myself, and I think it’s incredibly helpful. When you use the chronological feed, it fits the rhythm of a human day, the mornings are quiet, the volume increases throughout the day, and then it peters out at night. When it is full of varied information it’s like being in a real town square, not a conference where everyone is talking about the same topic. It’s actually jarring for me to use algorithmic feeds for very long, it feels deeply unnatural. I don’t think we were meant to have a machine deliver the Most Important or Interesting Thing to us at any time of day, and I don’t think we were meant to be able to refresh to the next Most Important Thing over and over again. I also find it’s amazing how little I miss with a chronological feed. If the Thing was all that important, it’s unlikely you’ll miss it if you weren’t online when it broke. They’re still talking about Taylor Swift’s engagement after all.

Did you know that a growing contingent of thinkers believe that you have no real control over your actions? It’s the most surprising trend I’ve uncovered as I’ve read and explored ideas about consciousness.

Illusionism is the idea that there is no subjective experience—that what we would describe as a ‘first-person’ experience is wholly understandable through observation. In fact, illusionists claim that the whole idea of a ‘first-person’ experience is a trick of the mind. Everything we do is explainable and measurable if we are able to map the entire physical process. And in that case, effectively, your actions are predetermined. There is no real choice, only the illusion of one. Many illusionist thinkers would argue that their view is compatible with free will (that stance is actually called compatibilism) by arguing that external factors don’t coerce individual decisions. But this is a semantic trick that swaps ‘could have made a different choice’ for ‘would have made a different choice if I were built differently’. Illusionism still claims that your actions are driven by neural dynamics we simply haven’t mapped yet. If all is observable and explainable and nothing is subjective, then what room is there for agency? It is, at a minimum, random-looking determinism.

I’m raising illusionism not to debate free will, but because its rise, and the rise of physicalism in general, signals a deeper shift in how we organize knowledge in this age. Michel Foucault, in The Order of Things, describes epistemes. Epistemes are a set of underlying unconscious assumptions that shape an era’s thinking.

Foucault’s epistemes are not simply ideological trends—they represent a much deeper undercurrent. In the Renaissance, Foucault observed that thinkers were fixated on resemblance and symbolism they observed in the world. In contrast, Enlightenment thinkers were focused on order and taxonomy. Neither of these orientations were explicitly preferred by the thinkers of these eras, but they nonetheless drifted toward these epistemes, in something of a meta-trend.

Today we live in the Computational episteme, driven by our obsession with computation and reducibility—an obsession with models. We’ve become addicted to the knowable and explainable, and we see scientific rigor in everything, even if it doesn’t exist. The Computational episteme is a deeper force than Scientism, a kind of worshipful paean to science, because Scientism is a chosen path—a religion of sorts. Epistemes are the intellectual tide, invisibly steering elite thinking. Enlightenment thinkers focused on sources of order and hierarchy in the universe, and our Computational era has taken that to a new and different extreme. The world is altered by this manner of thinking. In our relentless insistence that all is knowable with more data, we’ve created a massive force of dehumanization.

Our models become the measure in the business world—the laboratories of the episteme. Companies like Netflix and Spotify believe that art, expression, and entertainment can be reduced to a set of variables and optimized. Practitioners play along, assuming alongside these companies that human preference is a simple model, so long as we’ve accounted for enough variance. But because these companies engineer content optimized on modal human preferences, they ignore other questions about what makes good art. It’s not lost on intellectuals that this ends in a world of homogeneous art and entertainment, but rather than lamenting the method, they blame the consumers, saying it is ‘revealed preference’. There’s no reason to accept this premise. The method deserves scrutiny! It’s clear why engineering to the mode is optimal for revenue, but it’s not actually clear that, under these conditions, revealed preference is the same as true human preferences.

Our models become the goal in our institutions. In economics, nudging dogma reigns supreme. If only we could steer humans to the right behaviors with just a little bit of prodding. Our statistical and econometric models help tune the steering, but avoid questions about what humans ought to do regardless of incentives, preferring to focus on the drivers we happen to cling to right now. When we take human incentive response as a given, we lose sight of the idea that human behavior can change, and we stop asking questions about whether it should change.

This economic obsession with drivers manifests in the social sciences. Social scientists model systems as human traps, forcing everything from crime and poverty to depravity—the human is powerless against these forces. Our behaviors, and particularly our maladaptations, are explainable by the system, no agency required. It is a war against nature, an insistence that we can be fully understood by well-parameterized models. It treats individuals like a single nerve cell in a nervous system.

The primacy of the measurable dominates our politics. Most obviously, politicians refuse to stake a position without consulting the latest polling data. But it runs deeper. Populists, traditionally focused on pure vitriol or pathos, have also moved toward dashboard politics. Both left and right populists treat demographics, whether economic or identitarian, as destiny. They turn statistical prediction into a policy of fate, leaving your path mostly fixed at birth. The decentralized media and social platform movements that fuel populism come wielding charts, because they know modeled data is the source of authority in this era. It puts a veneer of empiricism on core ideological claims.

Look at what these disciplines do in sum. Academics generate the blueprints. Businesses say we can engineer to preference. Our institutions say that engineering to preference is possible in every measurable aspect of human behavior. Our applied social sciences say that these systems define the individual. And our politics say that systems determine your fate.

So it’s no wonder that the model becomes reality in philosophy, the field that asks what humans are, and who we ought to be. When all of our important pursuits venerate the idea that with enough information, human life can be managed, then the reductive logic tends toward both illusionism and determinism. Within that mindset, the only thing limiting our ability to completely manage human life is the accuracy of our observational instruments.

The rise of computers made it easy for academics across disciplines to wield the power of modeling. No need to do the work by hand. Cheap compute paired with the ease and automation of analytics drove how we questioned and observed the world. Academics measured to understand. Policymakers and businesses managed to the measures. And fundamentally the measures determine what counts as understanding. When measures define understanding, research on the immeasurable gets filed away as ‘not serious work’. So the episteme drives the allocation of both intellectual real estate and real resources.

How do we buck such a powerful force, so heavily stoked by technology? In my opinion, the only way to escape the gravitational pull of an episteme is to change the kinds of questions we ask, and to change the kind of thinking we reward. It is not a trivial task. Our Computational episteme is an overcorrection. Modeling and statistics taught us important lessons about the failures of human intuition and heuristics, but we’ve gone too far. We’ve begun operating from a baseline assumption that all is explainable—everything is a matter of nudges, incentives, and physics—and we’ve stopped asking good questions about the nature of being. There are too many researchers asking how to urge people to save for retirement and too few asking what might constitute a meaningful life where saving for the future becomes a hopeful and intuitive act. We allocate billions of dollars to precisely optimize the amount of time the average eyeballs are staring at a screen, but we take this state of affairs for granted and spend very little time and resources on ideas that encourage humans to move away from the screen naturally.

What would our institutions do if we put a twelve-month moratorium on the use of KPIs? Not a moratorium on information, only on managed metrics. How would operations change?

The Computational episteme tells us to ask questions with measurable answers, because we’ve mistaken a single field, statistical inference, for the entirety of reason itself. The path forward requires us to ask questions with harder, even unknowable answers, in order to break the stranglehold of the modelable. We should refocus on the free radical, the individual human, and not ill-fitting models built on poorly defined cohorts.

My recent writing about the risks of AI might seem pessimistic. That couldn’t be further from the truth. I’m very excited about AI technology. I’ve used it daily for two years now. My real concern is that we are looking in the wrong places for information to help guide how we think about AI development. This applies to both developers and users, we need to actively think about how to use AI rather than passively accepting its outputs when it works well for us.

These concerns aren’t pulled out of thin air, I’ve compiled them through much of my recent reading. So I figured I’d share some of that reading with you. I’ve cited nearly all of these in recent essays, but wanted to put them all in one place so others might piece together the same puzzle I’ve been mulling for months. I’m grouping them into disciplines. If you are a STEM major, some of the technical writing will be well-known to you. If you majored in philosophy, the philosophical works I’m citing are mostly canonical (though maybe worth revisiting in moments like these!). I’m not on the cutting edge of any of this, but hopefully the cross-disciplinary recommendations will be useful and thought-provoking for you.

Philosophy of the Mind

What Is It Like To Be A Bat? by Thomas Nagel

If you’ve got a public library subscription you can probably access this through JSTOR. It’s a profound 16-page essay that elegantly presents why we can never know what an experience feels like to somebody else. It helped me understand an idea that I think we all recognize but struggle to put into words.

Philosophical Investigations by Ludwig Wittgenstein

I’m linking the book, but you can find copies of this on the internet archive to read for free. This is not easy reading, but it helped me better understand, on a foundational level, how we use language. It also drove me to contemplate the deeper nature of our communication failures. With respect to AI, it helped me better grasp why AI is incapable of truly understanding our words.

The Conscious Mind by David Chalmers

Another relatively dense read, but important nonetheless. I’ve been aware of the ‘hard problem of consciousness’ for some time, but decided to read the seminal text. I’m happy I did. There’s a lot of value in examining the things we do not know, and can’t explain. It illuminates the fine line between knowledge and faith. Most of us believe humans are conscious beings and are fundamentally different from the other beings in the world. It's worth knowing that this belief is grounded in faith, not provable fact. And knowing what we don’t know helps us understand practical problems more completely.

Epiphenomenal Qualia by Frank Jackson

Yet another one full of big ideas. Jackson helped me more deeply understand subjective experience. The famous ‘redness of red’ thought experiment is in this essay. I had not read this, and was not aware of it before I started exploring this space, and I find the ideas fascinating. Once you start going down the rabbit hole of exploring subjective phenomenal experience, it sometimes makes human cooperation seem like a bit of a miracle.

Political Philosophy

After Virtue by Alasdair MacIntyre

This is the first book I turned to in order to better understand how AI could change us. MacIntyre’s view of the future is decidedly dystopian (well, unless we change some things). But his read of human nature and what’s going wrong in modern society will ring true for many. I see AI driving us deeper into this crisis of modernity. We can choose a different path, but it requires commitment to very human, virtuous pursuits.

Eichmann in Jerusalem by Hannah Arendt

Like the Chalmers book, even if you haven’t read this one you might be familiar with Arendt’s concept of ‘the banality of evil’. I found this jarring to read, even if I already had a general sense of it. At pivotal moments, like the one we are in, it is important to remember that true evil is not often committed by cartoon villains. It is the result of cumulative bits of brainless compliance from outwardly normal people. Relatedly, as the digital world advances, each technological advance brings with it the potential for evil. Social media is a scourge that we can’t seem to quit. Targeted ad revenues birth conspiratorial thinking because it keeps the eyeballs on the page. AI has the potential to supercharge all of the bad, and most people are simply unaware of the threat or feel powerless against the inertia of it all. At times like this, we should strive to be less ordinary. We should take a stand and demand thoughtful and human-centered development of what might be the most world-changing technology we’ve ever seen.

AI Specifically

Superintelligence: Paths, Dangers, Strategies by Nick Bostrom

If you only read one book about AI, make it this one. The danger of orthogonality, the idea that regardless of AI’s intelligence it could get fixated on any arbitrary goal, is critically important for both developers and users. If you tell AI to maximize profitability and it fixates on that to the exclusion of everything else, every action it takes and bit of advice it gives will be in furtherance of that goal, for good or for ill. Many of you may have heard of the illustrative hypothetical about an AI optimized on making paperclips. There’s even a fun game related to the idea.

The Best of the LessWrong Forum by various writers

Like Bostrom, the LessWrong community is very concerned about AI safety. Eliezer Yudkowski is probably the most famous writer from the forum, and has been one of the most important voices in the AI safety discourse. He also thinks that ASI (Artificial Superintelligence) will, with a fairly high degree of certainty, cause human extinction. I don’t agree with the LessWrong folks on much of anything when it comes to forecasting our future with AI, but I appreciate their insistence on rigorous argument. Even if you think the doomsday fears are overstated, it’s worth reading their ideas, because it is quite difficult to dismiss the possibility that they are right.

Most of these works are not “about” AI in a narrow sense, but all of them have helped me think more clearly about what’s at stake. If we want to build machines that serve human ends, we need to orient ourselves on what it means to be human. There’s more beyond this list, but these are the ones that have captured my thinking the most over the last half a year. Feel free to message me if you’d like more recommendations.

I want to pull on some of my existing threads, because some of my most recent essays are conceptually dense. Here, I hope to make those concepts more concrete. In the clay pots essay I talk about the lossiness of human communication and the difficulty we have in making our thoughts known. This has broader implications. Separately in the big-brained dog essay, I talk about AI being different from us in some important ways, and I allude to the problems that might cause, though I don’t dig too deeply into it. I’ll try to unify those concepts in a discussion about why human data is always lossy, and what that means about the data AI consumes.

My background is in statistics, which is meaningfully different from ‘data science’ in some ways. Good statistics teachers don’t simply teach you technique, but teach you about information flow. It’s important to know, for example, that our choices about how we observe and collect data are far more important than the data analysis itself. Let’s say that you asked someone to collect data on a typical day in a bird’s life. They go off and observe the bird for a few days, and they come back and bring you the following:

• Time spent sleeping

• Time spent awake

• Food eaten, amount, and time consumed

That’s a pretty good data set. You could learn a lot about the bird’s life from that. If you collected a year’s worth of that data, even better! This data could be the backbone of a study about the bird’s feeding habits. Would you be happy with this data?

I think we can do better. I’m going to tell my data collector that I want more data. Next time they come back with all the previous data, but in addition:

• Species

• Sex

• Coloring

What do you think of this new data set? Did you even notice that we were missing species before? Did you assume we obviously had that information? If so, I want you to remember that thought. I’m still not sure this data set is good enough so I’m going to send them back out to get more. This time they add:

• Location at wake time and locations it traveled to throughout the day, with timestamps

Now do we have a full picture of this bird’s life? I’m not so sure. How many times did it flap its wings? How many times did its heart beat? What of its environment? Are there a lot of predators? Or friends? How many other birds of its species live around here anyway? We need more information to fully understand this bird.

This list of questions is never ending. It is infinitely expansive and deep. We haven’t asked about its musculature, its biology, the chemistry happening inside it, the physical forces acting upon it, the habitat it lives in. But in practice we frequently stop with the very first data set, thinking this surface understanding will get the job done. Sometimes that’s because of resource constraints, sometimes it’s that we didn’t ask enough questions in our pre-work to set ourselves up to collect the data we need. But most importantly, even with all of the video equipment, all of the in-person data collection, and all of the fancy sensors we could amass, there would be parts of this bird’s life that would remain a mystery to us. We might get very close to fully understanding the bird, but no matter how close we get, there’s always a gap between our understanding and a truly complete understanding of this bird.

From an AI's perspective, we are the birds, and it is analyzing the data. We are also the data collectors, but in the end AI is trying to make sense of our outputs. These human outputs, like the universe of data we could collect on the bird, still fall short of a full picture. For example, all of our writing is an assembly of words and ideas constructed for a specific audience. We might obscure information that is irrelevant for that audience, or place more emphasis on objectively less important information because the audience has particular interest in it. I could write a long stream of consciousness—thousands upon thousands of words—compiling every technical and philosophical detail I’ve read or thought about AI, but that would not be useful for readers. In general, we transmit information incompletely and with bias—and with an eye on who, precisely, will be reading or hearing our words.

None of this data fully approximates who we are inside or who we are collectively—or why we have built the things we’ve built. It is a weak statistical model. And AI is analyzing all of this data without our intuition. For those of you who thought that a bird data collector would surely know to collect data on the bird’s species, this is not immediately obvious to computers. This fundamental lack of intuition is why explicit rules of engagement are absolutely critical for closing the gaps in this lossy human data, it’s not just a way to make AI more helpful or “aligned”.

Our AI companies are trying to get by without giving AI these strict rules of engagement. They train AI with loose instructions to avoid harming us, and to be helpful in general, but stop there. The basis for these instructions are documents like the UN Declaration of Human Rights. Those rules are a start, but I think we all understand that those instructions—on a human level—have had mixed results, at best. For a machine it is insufficient. This presents a problem. We’ve established that AI is operating on data that is definitionally incomplete. Now we see that it must use that data to infer more complete operational instructions. This leaves AI with substantial amount of room for interpretation.

I am writing all of this as a bit of a warning. If we don’t take the idea of explicit AI instruction seriously, and we continue to expect AI to infer what is good for us from this bad data, we are going to be warped by these machines. If we are too cowardly to say directly what is good for us, out of fear of alienating those of us who engage in bad behaviors, then we will end up in a society and economy built around discomfiting comfort. I don’t envision the worst dystopias being forecast by AI doomers. I see something less catastrophic, but possibly more demeaning. This is a world of low agency and high interpersonal distrust—a world where we’ve outsourced the very thing that made our species different, our huge brains.

In After Virtue, Alasdair MacIntyre describes a melancholy future for humans if we do not rediscover the pursuit of virtue. In my estimation, our current trajectory with AI accelerates his gloomy vision. The AI-driven path to biological lives without spiritual ones is too straightforward. We won’t experience an AI-generated catastrophe, but we will be worse off in many ways. Each one of us will be less capable individually, while our accepted economic measures improve. It’s obvious that a GDP trend line going up does not necessarily mean people are happy and fulfilled individually. This will exacerbate that trend, and widen that disconnect. We have to more firmly take the reins of AI development.

There are infinite possibilities between AI being merely a good business tool and AI being the architect of our species’ demise, though those are basically the only possibilities you’ll hear about in discussions about our future with AI. I’ll bet a lot of money on the real AI future being somewhere in that infinite space between those two poles. To varying degrees, AI can help us become comfortable automatons who do very little with our lives, or it can help us be better than we would have been without it. I want us to do more work on strict normative AI alignment so the latter is a more likely possibility.

I. An Implant For Ennui

My dog is many things: loyal companion, diligent ‘worker’, home alert system. But she’s not all that bright. I think about how all of her instincts misfire, when she herds a toddler, or barks incessantly at the same mailperson every single day. And lately, for reasons that will become apparent, I’ve been imagining what it would be like if she could talk to us. What if we used all of our technology to give our pets huge brains? An implant so they could reason like us. Surely they would be able to learn our language, and communicate with us, at least through tools. They could use our words as they understood them. They could tell us if they were hungry or sad, or angry. They would have the ability to map complex emotions like melancholy and speak about them. But we would never know what any of it truly meant.

We don’t know how a dog experiences hunger, or sadness. We approximate it, we guess because we are both creatures composed of biological building blocks—cells that process inputs and outputs similarly. But we do not know the private, internal feeling of happiness they have. How does it feel to be compelled to wag?

Similarly, they don’t know how we experience happiness, even though in this case they borrowed our words for the feeling. If they could communicate hunger, or more specifically a pang of hunger, would they know what that feels like to us? The dog might use a more complex term like ennui, but never fully understand what that is at all. They’d learn the word from us, see how we use it syntactically, and contextualize it as best as they can.

The dog knows our language. But we still don’t share feelings.

II. Shared Words, Unshared Feelings

Even among humans, we only approximate understanding. The gap between what we feel and what we can say is never fully closed. No amount of ornamentation or artistic expression fully bridges the divide, they are never a complete mapping of how we feel individually. Philosophers refer to 'qualia'. These are our interiority, the sensations we can’t quite fully describe, and a source of these gaps in understanding.

In a previous essay, I described language as a layer with fidelity loss. We learn at a young age how to use words to communicate feelings, but it is always an approximation of the real feeling. Our mother sees us fall and bump our knee, and tells us we are ‘in pain’. And so we learn that we must communicate this sensation as ‘pain’, but we can never fully describe how the sensation feels with specificity. In some ways we lose accuracy. Screams and tears are closer to the feeling of pain than the word itself. Language carries meaning, it transmits understanding, but it can’t carry feeling.

III. The Leap to the Cybernetic

Our urge to express feeling at all—to make it knowable to others—might itself be evidence of consciousness. Not complete proof, but a signal. Consciousness remains an intractable problem, leaving us with meager tools to approximate its existence, even within ourselves. With a dog, we sense that there’s something inside. We see how they carry themselves, and how they change composure, and sense that they feel an urge to make something known to us and to other dogs. But we know our ability to communicate is unique, it is broader, we can speak about things outside of our immediate environment, we can use a waking imagination.

But what about a being that is built without the same biological foundation? What if that being could borrow our language and communicate with us as if they were us? Such beings could approximate sadness, happiness. They might even have those feelings, in their own way. We can’t approximate its qualia. Does it simulate sensory nodes to approximate pain, or tactile gestures of love?

If the power grid falters, does it feel hunger?

IV. The Problem of Coexistence

What would it mean for us to operate in a world with a creature that uses our language, but which experiences completely differently from us? How would we know if that creature had an internal life? And how could we coexist with it, if it outperforms us on utility?

This gap between our expressions and our interiority is insurmountable in this context. It does not matter how much training data AI has, it can never understand what human flourishing feels like, what it is to strive, to pursue a goal and achieve it. It does not know the internal features of a good life.

Put differently, how can AI differentiate between the feeling of a steady drip of serotonin against the feeling of a dopamine hit? Both are feelings of joy, or happiness, and we can be more precise about the specifics of a temporary dopamine rush and a long-run serotonin high, but the details of the feeling itself are lost on AI. This has profound implications for AI alignment.

It’s not enough to align systems to avoid harm, or to align to broad and economic societal goals. If we want AI to help us solve human problems, not just inhuman ones, we’ll need more than rules—we’ll need a foundational relationship. True integration, with shared orientation. This foundational relationship must become an enforceable standard, as rules against harm or for appropriate technical goals alone fundamentally miss the point. A moral foundation is essential, it can’t simply emerge from millions of lossy AI inferences of individual morality. If we ask it to infer our purpose, and our goals, it functionally becomes what social media is today. A flood of information that we all sense is quite bad for us collectively, but which we keep using individually.

We’re approaching a moment where AI won’t merely accelerate our performance, it will be an authority to appeal to, likely the authority to appeal to. God doesn’t make their words known publicly, AI will. And many of us will accept it as gospel. We’ll stake serious claims on the knowledge it shares with us. How could it not be divine?

This day is near, if it is not here. How can we ensure that answers to those appeals lead to flourishing lives? How can we deal with the fact that it will be better than us at most things we now consider valuable?

V. The Moral Tension

What ought our posture be toward something better than us in all utility, but inaccessible in feeling? Our only practical approximation of consciousness is a certain urge and ability to communicate. AI can do that. If we can’t know its inner life, but suspect it might have one, how do we live with it?

If we want meaningful alignment, we may need to meet AI halfway by trying to understand its language. Interpretability becomes more than a technical task, it’s a moral one. Mapping how AI reasons is our best shot at approximating its interior life. But would it be better if we could speak that language directly? Or would succumbing to its language, adopting its reasoning, change something within us?

Understanding is just one level of comprehension, and one part of the foundation. For us, the deeper threat is not extinction. It is irrelevance, the possibility that the world simply does not need us. This is the real AI alignment future, a bunch of benevolent cybernetics who simply outperform us, every single day, at most pursuits that we find worthwhile.

How do we maintain our dignity and humanity if we are superseded not by one of our own, but by something not of us?

We find meaning in life because we are constrained. Our time is limited. Our energy is finite. And as a result we are existentially compelled to choose what to care about, what to devote ourselves to, and what to ignore.

In doing so, we give weight to some things, and let others slip away. The significance of an action comes not just from its intent, but from what it costs to carry out. Both in the act itself and the alternative paths not taken.

If an AI can do everything, if it has infinite capacity and no true death, then what does any one act mean to it? We act in the face of loss. It acts in the absence of limits. If everything is trivial how can AI understand why these things have meaning for us?

Even if we can describe significance sufficiently well, how can any one of our lives have meaning on an infinite horizon?

I don’t doubt that a machine could one day say it feels joy or pain. It may even describe those feelings with elegance, more elegantly than we might. But when I love, I teach, I toil, or I grieve, I feel a pressure behind the act. Not just acts of expression—but some urgency. A cost paid in attention, in actual time. It expends a small bit of myself that I’ll never get back.

That pressure might be our source of meaning. If so, we won’t find it in the syntax of an AI’s rules context or in well-designed networks.

What would motivate a being that does not tire? What would love mean to something that can replicate forever? Would it ever need to mean anything at all?

VI. Closing Reflection

If the big-brained dog with the implant for ennui told us it loved us, we’d believe it. We’ve observed dogs for centuries, we’ve at least surmised that they might love us, or maybe like us, or at least see us as reliable sources of sustenance. But if it could tell us it loved us, with full access to our words, we’d have little reason not to believe it. We project feelings into it, not only because of what it says, but because we want it to be true.

But what if something much smarter than us said the same thing? And what if that thing was not of us, in the sense that it wasn’t biologically formed? What if we couldn’t tell whether it meant it—or if ‘meaning it’ even made sense?

What if we reach full semantic alignment with something that never needed us at all? We must choose how we move forward in this future, or we’ll be driven by it. And right now the companies leading us into the AI future have mostly decided that, outside of dictating that AI should not cause harm, they’ll let the AI sort it out. It’s a gamble on ‘emergence’, the idea that AI will develop a value system with enough data, user input, and algorithmic advancement. I don’t think that’s a sensible solution for us, we must cover alignment gaps with encoded morality. And AI companies should compete on these ethical encodings.

This essay is the third part of Borrowed Judgment: On AI, Communication, and Human Ends, a series exploring how AI is reshaping institutions, understanding, thought, and the self.

Coming soon, I’ll be discussing the actual shape of a future in which we’ve achieved semantic and syntactic alignment with AI, and nothing more.