I find other people’s value systems interesting, so I’m writing this in the hope that other people might find mine interesting as well.

I’m a utilitarian

Uncontroversially, I think that happiness is good, and suffering is bad.

More controversially, I also think that this is the basis of all morality, and literally everything else people place value on, from honesty to biodiversity to not lifting dogs above head height to equality to loyalty to openness to filial piety, should be in service of this.

I take it pretty far and will probably bite whatever bullets you fire at me.

If you pose any controversial thought experiment involving pushing people onto train tracks, creating trillions of lives “barely worth living”, chopping up healthy people to donate organs, being mugged by someone pretending to be able to generate infinite suffering, or duplicating the universe an infinite number of times—you don’t even need to ask me for my response. You can just ask yourself, in this hypothetical world you’re posing, which action would be expected to generate more happiness and less suffering?

My answer: “Yep, that one.”

I care about conscious lives capable of suffering

There are many kinds of lives that some people think may or may not be important — animal lives, digital lives, lives of far future people, or aliens. I simply use the “can they suffer/experience pleasure?” question to determine whether their lives have intrinsic value.

This is incredibly hard to answer for lots of life-forms and this is a major source of uncertainty in my worldview. I try to think probabilistically, from the slightly flawed basis of “I’m pretty sure I’m conscious and capable of suffering, and expect that beings with similar brains, behaviours, and/or thought processes to mine are conscious and capable of suffering”. For animals, I try to use research from orgs like Rethink Priorities to get a better sense of what their welfare ranges might look like.

I use these numbers to get a broad sense for what these animals experiences might be like.

It only affects my day-to-day life in that I don’t eat any of these animals and donate some of my income to charities that work to improve the welfare of the smaller and more numerous of these animals.

I think the world is probably net-negative

I’ve come to the conclusion over the last two years or so that the world is currently net-negative, meaning that it is ethically bad that the world continues to exist, assuming it remains how it is now in the future.

This might not be true if we’re only considering humans as the relevant population. Most humans tend to live much of their lives in a state that they tend to rate as worth living, and lives tend to be improving. I take counter arguments seriously, but probably lean towards most human lives being net-positive.

It is almost definitely true if we’re counting farmed animals. Farmed animals, particularly small-bodied animals, from farmed chickens to insects, are kept in horrific conditions and experience lots of their life in pain. They experience very little pleasure to compensate.

It’s also almost definitely true if we’re counting wild animals. The median animal to exist, as well as the median hour of existence lived, is something like a baby starving to death or suffering from exposure to the elements. This is really fucking horrible.

I still think we should try to save the world

Despite thinking that the world is net-negative, I don’t think we should be trying to destroy it. In fact, I think that trying to destroy the world is a bad thing to do.

I think this for two main reasons.

The first reason is that a net-positive world is probably possible, and humanity looks like it may be making baby steps towards it, albeit on a path above a gaping ravine. If we continue the moral progress of the last couple of centuries, accelerated through a process like coherent extrapolated volition (the idea that an AI, or human-AI hybrid system, could consistently ask itself what it should want if it were morally better), this could result in the best possible moral being appearing within a generation. I believe that a universe ruled by such beings would not only be net-positive, but could be so magnificent and full of joy that it would make all the suffering in our universe’s history worth it.

The second reason we should not want to destroy the world is that destroying the world is not a robustly good thing to strive towards. If you try to save the world but fail to move the needle, you’ll still end up doing things that make the world better in the process, such as striving towards peace between great powers, making viral pandemics, nuclear war, and environmental crises less likely, or making sure that narrow AIs are superficially aligned and generous. You’ll also be allies with the caring people doing all these nice things, and building on each others strengths. If you try to destroy the world and fail, there are countless ways that you could end up generating more suffering in the attempt.

Saving the world might be hard

I’m not an archetypal “doomer”. On a good day, I think we’re more likely than not to survive this century.

I don’t have a very carefully calibrated worldview, but my predictions for the state of humanity in, say, 2075 looks something like the above diagram (created using the possible worlds tree). The red outcomes are extinction or anything obviously worse. Blue is flourishing, yellow is catastrophe, green is sustenance/survival. So I see an 80+% chance we’ll develop genuinely god-like ASI in the next 50 years, and that this will probably lead to “non-normal” outcomes.

The “extinction” parts of the map are mostly explained by the classic Yudkowskian view that If Anyone Builds It, Everyone Dies, and the flourishing parts of the map are the worlds where we either rise to the challenge or get incredibly lucky. Some outcomes are much worse than extinction.

I also think that AGI, or at least fully automated machine-learning research, is coming very soon. This makes it hard to know what to do with my life.

I now have a job for an AI safety fieldbuilding organisation and an 18-month-old toddler.

Virtues

Being a utilitarian doesn’t necessarily mean that expected value calculations are always the best way to think about morality. As flawed humans who need heuristics, it’s probably better to cultivate certain virtues that allow us to be better moral thinkers and actors than to calculate everything from first principles.

For any utilitarian decision there is a corresponding virtue that will allow us to better make that decision. Actually doing good in the world is often more dependent on having good moral habits than calculating correctly.

When I have done good in the world, it has usually been a function of virtuous habits and the correct moral maths. e.g. my decisions to donate to effective charities or work on effective cause areas are equally due to the virtues of thoughtfulness and self-reflection.

When I have failed to do good in the world, or caused harm in the world, it has mainly been because of my failure to cultivate these virtues and habits. I sometimes have bad habits related to indecisiveness, lack of agency, thoughtlessness and lack of moral and intellectual seriousness.

I can but repent for my sins, and try to become better.

Aesthetics vs. underlying values

The biggest moral mistake I see is people confusing their aesthetics and their underlying values.

My aesthetics are not at all in line with my values.

I aesthetically appreciate nature and wild animals. Rainforests and coral reefs fill me with joy.

I often disdain simple pleasure seeking. I write songs about dystopia, and regularly seek out things that create negative emotional valence.

I often feel a mild distaste for things that make people innocently happy, perhaps this is a lingering trait from a more misanthropic youth.

I’m obsessed with religious and quasi-religious ritual and penitence. I have a persistent attachment to Victorian moralising and veneration of self-sacrifice. I’m moved by Nietzschean proclamations, like “you must be ready to burn yourself in your own flame; how could you rise anew if you have not first become ashes?”

I also eat paste-like food every day. Just vegan protein powder and water. I may add some raisins.

Jean-Paul: “I’m living in Saint Denis at the moment; gentrification is such an interesting process. There are these weird pockets of wealth in otherwise poor areas, and strange, arbitrary dividing lines.”

“Yeah, some people just hate it when anyone gets richer,” Edgar insists.

Jean-Paul: “Yes, I’m sure… It’s interesting how the economics work out as well. Look at the house prices: 15 mins from Paris centre for 100k! If you can buy a house during the process of gentrification, you’ve really hit the jackpot.”

“Why can’t people see that ‘gentrification’ is obviously just progress?” retorts Edgar with disgust.

It quickly becomes clear that Edgar is discussing not with Jean-Paul, his erudite and clear-thinking companion, but with an imagined and slightly foolish interlocuter.

This is sadly true of a few of my friends and acquaintances; when they tell me about their recent intellectual or political thinking, they frame their arguments in opposition to a super naive degrowth leftist; an e/acc (effective accelerationist) or a fundamentalist Christian. This often continues to be true even after we’ve identified the more relevant and interesting questions that intelligent, good-faith people may argue over.

The ghost of their out-group—the accursed “other” they frame their views in opposition to—continues to haunt the conversation.

Why is this?

Your in-group and out-group determines a lot of your identity, and I’m no exception. It’s a bit embarrassing tracing back through my life story and tracking how many decisions I've made purely out of opposition to people I disliked.

When I was a teenager, I deliberately lost my Lancashire accent because I didn’t want to sound like the boys who relentlessly mocked me because of my hairstyle, dressing habits, behaviour, and shamelessly smart-arse attitude. I chose more artsy/social science subjects over hard sciences in college and uni in opposition to my “STEM-chauvinist” brother and his friends.

It’s also the case with my worldview. It’s easy to develop a cluster of loosely correlated opinions about individuals or political policies in blanket opposition to an outgroup. In an earlier life, I’ve found myself tacitly supporting authoritarian or communist ideas in opposition to the hated American or conservative outgroup. After I learned about economics, and became more favourable to free markets, I’ve often caught myself backing market-distorting monopolies just because it felt like the opposite of the “all corporations are evil” knee-jerk stance.

I suspect this dynamic is hard to avoid. Strong opinions almost always presume a counter-party that disagrees. This is why you get so passionate about them. But I still think it’s worth understanding and cultivating the boundaries of these imagined groups, so that you don’t find yourself building a worldview in opposition to fools or knaves.

Expanding on Scott Alexander’s classic post on this topic, I’ve developed a typology here. The world is filled with four layers of people: the in-group, the near-group, the out-group, and the far-group.

Who are they?

The in-group: These are people with whom you generally share the same worldview, axioms and values. Any difference of opinion is about the minutiae. Maybe we both care deeply about farmed animal welfare, but I think we should prioritise improving the lives of aquatic animals in China, you think we should focus on stopping factory farming in Africa. Perhaps we’re both concerned about human extinction, but I think that preventing astronomical suffering should take priority.

The near-group: These are people who align with you on a lot of your ways of thinking about the world, but who have such opposing views on a few important areas that your debates on these topics become important.

The out-group: These are the people on the other side of the fence; you’re part of the same debates, but you almost always find yourself on the opposite side. For me, this could be an e/acc who thinks about AI a lot but dismisses AI risk entirely, or someone who talks about China-West relations from a hardcore war-hawk perspective.

The far-group: This is for people whose views are so far removed from your own that you generally don’t even feel the need to engage with them. If you’re from an atheist family in Liverpool, it’s supremely weird to form your worldview in opposition to Hindu nationalists or Kurdish separatists.

Who goes where?

We should really take care who we’re putting into each category, and not let fate determine who we shape our views in opposition to. Four main tips:

Embrace the in-group with caution! The in-group are clearly the coolest people in the world—they’re nice to be around, they’re your natural allies who’ll support you whenever you need intellectual or moral support, and you won’t waste your time on pointless arguments with them. The best way to use your in-group is to test Aumann’s disagreement theorum—the idea that two rational people with the same axioms and information should reach agreement on any topic. The main peril here is groupthink—getting entrenched within a mutually reinforcing worldview, where your axiomatic moral truths start to blur with your arbitrary aesthetic preferences.

The near-group should help you test and refine your axioms! These are a group of serious interlocutors. They’re people who inspire and test you, but whose underlying value differences may branch out like fractals into entirely different worldviews. A good near-group should be used to test your broader worldview without pointless bickering. And, while these are not your true intellectual rivals, it’s worth letting yourself get more passionate about those arguments, instead of saving all your fire for (less fruitful) out-group debates.

Narrow your out-group! The out-group should boil your blood, but justifiably. They are your worthy intellectual adversaries, who you’d relish taking to the debate stage with. These are the people who’ll you’ll feel super passionate about arguing against, despite only rarely reaching points of agreement. For the out-group, you might be willing to draw battle-lines and make the occasional compromise on your epistemic virtues. But, crucially, this group shouldn’t be composed of people you consider stupid, or dishonest. Your life situation will force you into opposition against people who use motivated reasoning, poor thinking, and horrible argumentation norms, but you should avoid viewing these people as your out-group, lest you spend your days arguing with the latest troll to provoke your ire.

View the far-group with compassion and anthropological curiosity! The far-group is the beautiful tapestry of humankind, from an Inuit shaman, to an Andean farmer, to a Chinese police officer, to a Nigerian preacher, to a perpetually online MAGA supporter. There are loads of reasons to enter in discussions with people from hugely different backgrounds, and that’s totally fine. It can be enlightening to discuss family values with a rural Laotian family, but you will probably not find yourself shaping your worldview in alignment with or in opposition to these views. They might have weird, even abhorrent perspectives, which, were you to suddenly adopt them, would lead your own social circles to reject you. But this shouldn’t lead you to dislike or disparage the far-group. You can be in strategic opposition to them (e.g. if you’re canvassing for an election) but you should never consider them an actual intellectual adversary. Instead you should cultivate curiosity and positive vibes.

How to shape these groups?

For the vast majority of people, you should be trying to move people away from the out-group—towards to near-group if you think they’re intellectually honest, towards the far-group if you don’t.

The first step is asking yourself: What opinion do I feel passionate about today? Let’s say (topically, for me):

AI 2027 was valid, responsible and appropriate, and its writers were intellectually honest and upfront about the nature of the text.

It’s easy to now analyse the role of the outgroup here—I appear to have developed my passionate view in opposition to people who think that:

AI 2027 was invalid, irresponsible and inappropriate—its writers were intellectually dishonest and deceptive about the nature of the text.

Now, remembering that the outgroup should not be composed of people I think are stupid or dishonest, do I think people who believe this about AI 2027 are stupid or dishonest?

Yes, I probably do—probably leaning dishonest. I’ve read the explanations, listened to Daniel Kokotajlo describe his uncertainties, and engaged with the serious counterarguments.

I definitely don’t think people are stupid or dishonest just for doubting the conclusions or premises, or thinking that it’s boring or badly written. But crucially, I’m not feeling passionate about: “AI 2027 is beautifully written and an entirely plausible future I expect to unfold”.

In this case, I can smoothly move a subset of people who believe the category of stuff I consider to be stupid or dishonest about AI-2027 into the far-group. This allows me to put them in the same category as an Inuit shaman, or an Indian farmer—expanding the compassionate curiosity to the world, and narrowing my passionate disagreement to those who I consider worthy of it!

If, instead of the “intellectually dishonest” accusations, I’m feeling passionate about honest and subtle AI-related disagreements on, say, whether we should expect exponential or super-exponential growth, or whether the geopolitics in the scenario make sense, then I will eagerly anticipate a good-faith debate. That’s the sort of exchange that belongs with my near-group or in-group.

And if the discussion turns to a substantial, but intellectually-honest disagreement about whether current AI could even lead to artificial superintelligence, I might well put you in the out-group and join the spirited defence. This should make sure that the arguments I get worked up about are actually worthwhile, and that my passions are not wasted on the wrong people or the wrong debates.

This is a quick flow-chart to illustrate:

Conclusion

I hope this way of looking at things is useful.

I can’t always manage it myself, and I still end up entangled in pointless debate now and again. But this framework tends to help.

“However, the reason people reject the argument [in If Anyone Builds It, Everyone Dies] is not on account of the strength of its logic. It is all emotional. People in general do not like to think about extinction, or even risk in general. They would rather posit all kinds of emotional counterarguments.”

The speaker at the EAGx conference argued that, because of this, those of us who are significantly concerned about the loss of all life from rogue AI should instead talk more about mundane harms that don’t trigger emotional reactions, such as inequality, job loss and bias.

Regardless of whether the alternative strategy is prudent, I’d like to address the original claim here. I fully accept the idea that “it’s all emotional” when it comes to accepting or rejecting arguments. Emotions are the gatekeepers that only allow ideas that feel right to become part of our worldview, especially when dealing with such visceral issues as death and extinction. But I think we underestimate the ways in which emotions drive us towards as well as away from these topics.

The idea that people recoil from existential risk because it’s unpleasant to think about neglects humanity’s long-standing embrace of doom!

Let’s give a few examples of where we lean into our dark side:

Protest Movements

Big chunks of the climate movement really love the apocalyptic framing. They call their protest movements “Extinction Rebellion”; “The World Is Burning” has become official UN messaging; and activists have staged mock funerals for humanity, carrying coffins labelled “Our Future”.

As much as climate change is genuinely an existential risk for some species, and may feel like an oncoming “Great Flood” apocalypse for some Pacific Islands, it’s unlikely to pose anything close to an existential risk to humans (according to our best models). But this doesn’t stop the activists. The people who care most about climate risk frame it in existential terms, and there’s at least some evidence that this might work!

Climate doom inspires the emotions that seem to be driving a genuinely global movement, even if they sometimes overshoot the science.

Doomsday Religions

Millenarian or “end-of-the-world” religions, with belief in some kind of apocalypse and redemption for the chosen few, are big. This is despite famously getting you to do crazy stuff like donate all your money to fund the lavish lifestyle of the leaders, cut off relationships with your family, and/or leave your fields untilled in order to rise up in armed opposition to a vastly more powerful and organised state.

A really extreme doomsday cult might only attract a few particularly susceptible individuals, and Heaven’s Gate and Aum Shinrikyō fortunately weren’t popular or meme-able enough to reach a broader public, but religious movements with slightly apocalyptic undertones seem pretty brilliant at drawing people in. Narratives of doomsdays and messiahs have always thrived, from Ragnarok to Revelations in the ancient world, through anti-Imperial sects in China and Japan, to Islamic death cults.

This is still happening. Pentecostal movements across Africa, where regular failed rapture predictions cause unexpected waves of social panic, are growing fast and embracing social media. In the fastest growing major faith, belief in the Day of Judgment (Yawm al-Qiyāmah), where mountains level, stars fall from the sky and the heavens roll up, is a universally held belief.

Not only does preaching about the apocalypse not put people off, it inspires and draws new believers into new faiths!

Speculative Fiction

Finally, anyone who suggests that people dislike thinking about doom should take a look at some of the most popular films, novels and games from the past few decades.

Who can deny the childhood excitement at the primordial fear of pure doom promised by Dark Lords like Sauron, Darth Vader, or Voldemort in popular fantasy?

Even when we grow up and “put away childish things”, many humans start enjoying films about nuclear destruction (Dr. Strangelove, Chernobyl), asteroids (Deep Impact, Don’t Look Up), or aliens (Arrival).

And, to those who might suggest that we’re just into the jeopardy of doomsday scenarios, we even get excited by post-collapse scenarios! Who hasn’t fantasised a little about living in the post-apocalyptical hell-scape of books like The Road, films/TV like the Matrix, Children of Men, or Mad Max, or games like Fallout, The Last of Us, or Death Stranding?

Does this sound like the media consumption of a species that hates thinking about doom?

How deep does the rabbit hole go?

In the absence of higher-quality direct evidence about whether people actually enjoy these ideas, I ran a quick survey in a rationalist discord group (which, I can assure you, is the methodological gold standard) and asked whether people agreed with the statement: “I find it very unpleasant to think about “doom” or “the apocalypse””. Over half of respondents replied with the “strongly disagree” option.

A few quotes from the responses:

“Hello darkness my old friend”

“Idealized apocalyptic scenarios where I’ll be the first to get bitten by a walker are nice”

“I think about the Zombie apocalypse more often than I think about the Roman Empire.”

So it’s not just escapism in fiction, many people actually find thinking or fantasising about the apocalypse pleasant! The people who find it unpleasant gave reasons like:

“I sometimes get images of the type of Apocalypse where nuclear warheads are randomly flying through the sky and I evaporate in an explosion. That’s the one that I don’t like. A lot in my future that I’m excited about and it would be a huge bummer to not get to experience it.”

“I thought it was fun until I started to believe it was real”

“I’m not prepared for the timeline where I’m eating my neighbor to survive”

My interpretation: more nebulous and stylised “doom” ideas are actually not that unpleasant to think about, but when it gets too real, or when we think very carefully about the apocalyptic logistics, the fantasy can slip a little.

Of course, this is mostly young people with (I presume) high openness to experience, and you’re going to get different results if you survey 62-year-old civil servants looking forwards to their retirements. But this still suggests that humanity’s doom fascination is still going strong.

Why?

Time for a spot of armchair psychology/anthropology. Firstly, a few intuitive reasons why humans might be particularly inclined towards doom narratives:

1. They are meme-efficient. “Everyone dies” is far easier to communicate than “we experience gradual empowerment over two centuries with 28% probability.”

2. They’re impersonal. Global catastrophe doesn’t threaten the ego or throw you into a melancholia in the same way that a cancer diagnosis might.

3. They inspire meaning rather than powerlessness. Many people really like the idea of an existential struggle. Who wouldn't prefer to fight for our planet’s survival over meeting monthly performance targets or saving enough money for a managed decline in an old people’s home!

There are also a couple of academic theories that might support my observations here. Like a lot of psychology, our attraction to doom is probably best explained through a mix of something boring and testable, and something cool and not testable. The cool one is Terror Management Theory—a sprawling post-Freudian idea that all of our actions can be explained by the fear of, and desire to overcome our own mortality, where we craft elaborate narratives and rituals to navigate these conflicting emotions. Despite being mostly unfalsifiable, it does capture some of my emotions around death. I’m a simple man: all I want is a sublimation of mortophobia into an epic mix of art, heroism, and ideology that makes the inevitable death of the self fade in importance.

The boring testable explanation comes from media analysis, where messaging is analysed across the axes of valence and arousal. Valence can go both ways—despite the cliche of “doomscrolling”, positive news does attract a lot of attention online. But the ‘arousal’ dimension is more predictive of a message “going viral” and provides a useful explanation for doom’s persistent appeal. If a message triggers high physiological activation through fear, kittens, anger, awe, boobs, or doom, it’s way more likely to attract attention, stick in your mind, and get shared.

What to do about it?

I’m often surprised by the number of people who have a very strong inclination to think about x-risk. In fact, I’ve recently had conversations with people who had barely heard the arguments for AI x-risk and were already persuaded!

I'm fairly new to this AI messaging game, but I’d confidently make the case that we shouldn’t completely avoid dark or doom-laden messaging, at least for some audiences. I’d even suspect that using existing attraction to doom might be a neglected approach—targeting the doom-curious with fact-based messaging.

When your argument rests on the eminently reasonable, but also very “sci-fi apocalypse” idea that misaligned AI has a significant chance of destroying humanity, you have an important strength! Sure, you could make economic arguments about job losses, about privacy and control, or about AI having the values of your interlocuter’s hated political enemies. But this can feel a bit like Hafþór Björnsson auditioning for deep character roles—it doesn’t play to your strengths. You have a pre-made high-arousal, epic, yet depersonalised narrative, so why not use it!

In this respect, I think “If Anyone Builds It, Everyone Dies” has done its job pretty well. Despite the dark and serious subject matter, the argument through stories and analogies does allow the book to capture something of the scale and feel of the epic doom narrative, and make reading it actually kinda fun, especially for the darkly intellectually curious.

People still disagree with the book for many reasons, but I wouldn’t generally put these disagreements to an unwillingness to think about the dark consequences. I’d probably blame things like a general dislike of the Yudkowsky aesthetic, a strong denial that ASI is even possible, or a reluctance to accept arguments that go against the rest of your worldview (e.g. technology tends not to kill everyone, most doom predictions are wrong).

But I think that AI Safety messaging can build on humanity’s darker, more doom-loving side. We can sell inspiring, high-stakes messages with a hint of apocalypse to the young, action-oriented and open-minded, while selling “serious” economic messages to the stability-seeking 50-somethings whose interpretation of “existential risk” is a 15% drop in their savings portfolio. This might look like subtle narrative shifts, but it could look like “embracing the dark arts”, learning from how armies sell narratives to prospective soldiers, how radical religious movements attract new followers, and how extreme political movements sell fear-based narratives to get into power.

Do you have the courage to develop effective, high-arousal messages that combine our doom proclivities with a practical call for action? I hope so, my friends, because darkness is falling, and the fate of our universe may depend upon it.

But, the progress-pessimist may demand, are humans1 actually any better off for it?

Okay, fine. In some respects, the answer is obviously yes.

I do like to indulge the fantasy that I might have been happier had I been born in the early Palaeolithic, before I had to worry about the state appropriating my crops, being sold to a Dickensian poorhouse, or being rendered obsolete by AI. But I have to give progress its due. The world has become consistently richer since the Late Middle Ages, and in the process humanity has banished many of the causes of our own suffering. We’ve massively reduced conflict, poverty, deaths in childbirth and communicable diseases. Except for the desperately poor, most of the world now has enough to eat, access to vaccines, clean water, antibiotics, and basic pain treatment.

Although this is mainly a story of going from poverty to affluence, many would make a similar case for more recent progress, in rich countries. There’s a very common optimist perspective that looks like this (from the FT, referring to the UK):

Crime is lower than it was in, say, 1980, the range of entertainment wider, the sexual freedom greater, the medical science more advanced, the great cities no longer depopulating, the Iron Curtain no longer drawn. Yes, a house in E8 will cost you now, but if you think that outweighs the invention of the hepatitis B vaccine, you might investigate the option of growing up a bit.

Although there’s also no shortage of qualitative anecdotes and quantitative data on seemingly happy traditional societies bucking these trends, from the Pirahã to the Amish, quantitative research tends to support a mainstream narrative of people reporting gradually higher happiness and life satisfaction over time. For example, the World Database of Happiness has aggregated surveys across 75 years, and points to an increase in self-reported happiness in 50 out of 80 countries since data was first collected (which varies by country, confusingly).

So I buy the progress narrative, right?

Not really, and not just for farmed animal welfare, environmental, and existential risk reasons. Between the virulently anti-progress narratives of the de-growthers, and the enthusiastic techno-optimism of Silicon Valley, I think there’s an under-told story of stagnation in human well-being. A few reasons:

Firstly, the data for richer countries doesn’t look too positive—in sources like World Database of Happiness you need to squint a little to see any positive change in the last half-century. If we look more closely at the countries that have experienced happiness jumps, we see economic miracles (South Korea), ex-dictatorships (Spain), and recoveries from communist nightmares (Poland). If you want to arbitrarily limit to Anglophones, or our European Protestant neighbours, you get stagnation: self-reported happiness in Canada, the US, the UK, Sweden, Belgium, Ireland, and the Netherlands has stalled or declined since the 1980s.

Second, people are living longer. Great, right? Well, not really. Tragically, we can only practically add time at the end of our lives. At the end of life, people get less happy, and are more likely to experience significant ill health, which can overshadow the positives of those extra few years. We’re also not having kids, so the average person alive at any given moment in rich countries is getting less happy.

Third, and most importantly, just look at happiness compared to everything else! GDP, world population, productivity and technological measures have seen explosive, exponential growth! Average happiness is effectively horizontal by comparison. The wonders of the modern world may have tweaked the needle of well-being, but we're so incredibly far from the exponential curves we’ve see for wealth, and technology.

Why have we failed so badly?

Why has money not bought happiness

Are richer people happier than poor people? Well, yes, usually. But this is only a tiny bit of the story.

The “Easterlin Paradox” describes a state where, within a country at a given time, rich people always tend to be happier than poorer people, but, over time, people don’t get any happier, despite getting richer. This means that the median-income British person in 1975 was as likely to be happy as the median-income British person in 2015, despite earning significant more money.

There’s lots of intellectual back-and-forth (or “debunking”) on whether this is accurate. But, in the words of its critics, it does represent a notable “non-finding” where researchers frequently fail to find an association between economic growth and well-being.

This clearly represents a real phenomenon—beyond a certain point we only see slight benefits from a country getting richer, and income is a very bad predictor of whether people in a country will be happier than another country or moment in time. Averaged across the global population, countries are only a tiny bit happier if they’re richer, and there are many countries that buck the trend.

Depression is an extreme manifestation of low mood, and should probably be weighted higher than regular life satisfaction levels. Two recent large-scale studies in Europe found that rates of depressive symptoms across Europe are actually somewhat anticorrelated with national wealth.

So what’s the takeaway? Even the most convinced cases that money makes us happier tend to argue for log-linear effects (so going from $1000 income a year to $2000 a year has as much impact as going from $50,000 a year to $100,000). And a substantial part of this effect is correlation (happier people happen to be richer), and causation in the opposite direction (happiness makes you richer)!

This debate often neglects the fact that money only has instrumental value. Its connection to a more intrisic value, such as well-being, depends almost entirely on how you allocate it.

Don’t get me wrong, money really can buy happiness.

This is especially true if you don't care who gets the happiness!

In the overall market for well-being, there are some unbelievably good offers. Per the Happier Lives Institute, we can buy happiness at a rate of over 100 “well-being adjusted life years” per $1000!

As well as a moral wake-up call, this should also just highlight how ineffectively we convert money into our own happiness or those around us. This is mainly because our brains, social norms, and economies are just horrible at optimising our spending for happiness. Instead of buying happiness-optimising goods and experiences, we choose to buy things that our habits, and the ghostly hand of our economy can optimise for, which are:

Buying luxury/status signalling goods. I doubt my readers will be the types to splash out on Chanel bags and Ferraris. But, (per Robin Hanson) status-signalling goes far beyond this—if you think to yourself, “What garners respect in my community?” that’s probably what your status-signalling goods are, whether you’re an academic or a footballer’s wife. This might be getting a PhD, expensive healthcare, teaching your kids classical violin, or an organic shopping habit.

Do these things make us happy? Not usually. Because status is inherently relative, buying status goods is usually zero- or negative- sum—when everyone only cares about their relative position, no-one ends up happier and everyone ends up poorer.

Why do we want these things? Mostly because our mammalian brains still equate status with survival. In early human groups, status means access to resources, mates, and protection from ostracism, so our instincts keep pushing us to chase it. We just don’t seem to have good guardrails that tell us: “No, this status-conferring good is just ridiculous”. So we just go on spending our money on these weird status games.

Buying things we hedonically adapt to. There’s the classic Oscar Wilde quote: “There are only two tragedies in life: one is not getting what one wants, and the other is getting it.” This snappy one-liner clearly lands badly if you’re talking to a Guinean mother who wants food for her malnourished daughter—in that case, you’d be forgiven for arguing that it’s perhaps more of a tragedy to not get what you want. But it does apply well to a significant chunk of rich-world consumption.

Hedonic adaptation is probably the most well-evidenced reason for happiness stagnation after certain levels. Even if we buy something that makes us happier in the short run, we simply adapt to whatever positive conditions we have bought for ourselves. Our happiness returns to a (largely genetically determined) set point. This applies most obviously to entertainment and household goods. You’ve probably recognised this every time you go back to using an old phone or PC that you used to find perfectly functional and now seems ancient and unusable. The extreme of this is things that really hijack our hedonic system, like the comedown from an MDMA trip, but it can also apply to almost everything we spend extra money on.

Making bad temporal trade-offs. Money often fails to buy happiness because we buy the wrong things, but it can fail to buy happiness because we do the wrong things in exchange for money. This might be working too much as a signal of dedication, or out of habit, or holding on to a crap job because you're afraid of the uncertainty of leaving.

On the other hand, we have excellent evidence that money can be spent well in exchange for our own well-being, if allocated appropriately. A few examples of things that reliably manage this:

Getting un-fucked-up. A lot of our mental distress can be improved by going direct to the source. If you’re sensible about it, chemical stuff, like antidepressants, Ozempic and prudent pain relief are often wise investments here. On the behavioural level, a meta-study looking at rigorously tested ways to improve reported life-satisfaction found loads of high-reliability interventions, many of which were based on emotion management, such as mindfulness, gratitude and therapy.

Buying well-timed experiences. There’s fairly robust evidence that experiences tend to beat stuff. A great concert or weekend away can leave you with a warm glow and nice memories, while the flashy new object just sits there getting hedonically adapted to.

Getting the big things right. Moving to the right place has a significant positive effect on your well-being—understandably, it’s things like green space, a nice climate, and social support that drive a lot of that. For selfish reasons, it’s probably worth it to invest money in your broader living environment, your life partner(s), and social group, even if it costs you. There are always big trade-offs here, but money can make it easier to make the better choice.

Buying time. Although richer people tend to also be overworked, time and money are generally interchangeable. Studies have shown that time-saving purchases, such as outsourcing tasks, gives a greater short-term happiness boost than other purchases, especially for exceptionally busy people who experience “time stress”.

The Bottom Line

The argument that I will retreat to under attack is that, after a certain level of wealth, money starts having a very marginal effect. Money only buys us happiness to the same degree that money buys us violin competency. It can be used for this purpose, and it’s difficult to achieve without any money, but neither our brains nor our societies are very good at doing so naturally, so we have to deliberately direct it to this.

Why technology hasn’t bought happiness

What money can buy is never a stable object. Getting rich in Victorian England can buy you a cellar full of deferential servants, a wonderful selection of tweed suits, a country manor, and distance from the Irish. But it can’t buy you the vaccines, e-bikes, or smartphones that even relatively poor people have access to today.

In the last century, far more advanced technological products have become available for personal consumption or mass use, people have become richer, and prices of these technological marvels have, on average, have gone down relative to income.

I hear the techno-optimist’s cry: “Even the poor can access so much more stuff today! Surely this should bring us happiness?”

Again, I raise a skeptical eyebrow.

I’m probably more inclined than most in my circles to a cautiously techno-pessimist perspective here. I confess that this partly comes from a personal bias. Doing anthropology research in my early 20s meant that I lived almost a year of my life in “pre-modern” conditions without electricity or the internet. Going from a life of mild technological distraction in my university, I suddenly thrived in these conditions, and became far more productive, despite having to work in the fields most days. Alongside research for my thesis, I wrote a short novel on pen and paper, learned an obscure language and a bunch of jungle skills, like how to use a machete and bamboo, taught a bunch of kids English, and probably socialised more widely than I ever have done in my life before. I never felt that the absence of (most) 20th century technology was an obstacle to my everyday happiness. Because of this experience, an environment with minimal tech has a huge practical and aesthetic appeal to me.

But I’m not that naive here. Like with wealth, I can acknowledge all the positive effects of tech where they exist, especially for the very poor. When I was doing my fieldwork, I also saw my fair share of horrific extreme suffering, mostly connected to a lack of modern medical care and pain relief. I recognise that technologies like vaccines, antibiotics, and effective sanitation remove obvious causes of suffering. It’s not just medical, anything that allows very poor people to get even a tiny bit richer is usually a huge comfort in terms of basic security. There is some data on roads and electricity being good for well-being in poor countries, and you can find benefits of things like broadband connectivity for older people in Europe.

But, again, when you get to the rich world, tech seems as disappointing as wealth, in much the same ways. We rarely see technological progress reliably improving any happiness metrics, especially over the last few decades. Japan’s technology explosion in the 20th century is world-renowned, but Japanese people haven’t become happier since the 1950s.

Nick Bostrom came up with the idea of the “urn of creativity” thought experiment in his Vulnerable World Hypothesis. According to this, there are some technologies that improve our lives (white balls), some that cause some risks (different shades of grey balls), and a few end up killing us (black balls). We can imagine this for well-being, where a white ball is beneficial (smallpox vaccine), a grey ball is risky (TV), and a black ball destroys our well-being (the AI super-app that finally destroys our minds).

A techno-optimist probably thinks that the majority of technology is white or nearly white. A techno-pessimist sees almost everything but life-saving technology as dark grey.

I’m pretty convinced that there’s a lot of dark grey in there, too. Ultimately, we shouldn’t really expect most technological developments to maximise well-being. Most technology is developed through economic incentives, and occasionally through government funding and intellectual curiosity. Most of these processes are agnostic to well-being. As such, they either just provide economic efficiency and growth (which translate poorly to happiness after a certain level, as we’ve seen), or fit broadly into the categories of things that money fails to buy—things we hedonically adapt to (better TVs), comparative status goods (flashy cars) and things that cause us to spend time in worse ways (smartphones).

Sometimes the economics work out way better, and the needs of governments or the market align with happiness. Lots of technology used for health and agricultural reasons seem to be unalloyed positives, from Yuan Longping’s hybrid rice, to mRNA vaccines. Sometimes the economics really don't work out, and demand is driven by our flawed impulses more than our meta-preferences. It’s difficult to read the history of opium and hard alcohol in the 1700-1800s, or their introduction to traditional and low-income communities without recognises the scourge that these early technological super-stimuli posed to vulnerable people.

Abuse of new and improved drugs continue to plague us today, with synthetic opoiods and cannibanoids, but I suspect for most readers, your drug of choice will be digital. Doomscroll algorithms, addictive gaming apps, and online gambling manipulate the worst parts of our brains to promote obviously harmful behaviour.

This all sounds a bit cliched, but it definitely matches my experience that, just like addictive drugs, most consumer technologies provide superficial “spikes” of enjoyment that often backfire. Doing things online always feels like an obviously inferior good that we choose due to convenience: socialising via WhatsApp rather than face-to-face engagement; gaming online rather than playing sports; watching porn rather than taking the time to carve a personalised erotic BDSM scenario from bamboo.

Some of us have enough self-awareness and discipline to shape our own and our family’s access to tech to optimise well-being. Some of us really don’t—and the most vulnerable probably suffer more from addictive tech like smartphones than the rest of us benefit from it.

The Bottom Line

I don't want to take this too far. Humans are probably happier and more free from suffering than most wild animals, and the main difference between us is the existence of millenia of increasingly ingenious technology. But the case that I’d make confidently is that technology has, at the very least, given us a lot of grey balls that cause us to make reliably worse choices, We don’t always have the necessary mental and social mechanisms to stop an influx of dark-grey balls that could erode our well-being further, even as we develop better technology that could revolutionise human happiness.

Why social/political trends have not brought happiness

Many people will now be thinking something along the lines of:

“Money and technology have brought amazing potential for happiness, but this has all been ruined by [insert hated political philosophy here]!”

To those of us sensitive to the bittersweet allure of politicisation, it’s a fairly appealing narrative that any trends we don’t like are to blame. It’s all atheism! Feminism! Free-market capitalism! Right-wing populism! The death (or possibly the birth, if you’re very old-school) of the nuclear family! Radical Islam! Internet-driven polarisation!

These trends are harder to measure, as cultures tend to cluster a bunch of unconnected traits together in ways that are hard to parse. But cultural membership does seem to be a strong predictor of well-being. On a macro-level, South Americans are far happier than North Africans; on the micro-level, the Pirahã are supposedly far happier than other Amazonian tribes, and rice farmers are less happy than wheat farmers.

My intuitions are that socio-cultural habits and trends shape happiness more than technology or income, except for the very poor. Most of this is too messy to get into now, but we can identify a few obvious “modernisation” trends in social organisation that might explain why we’re not getting (much) happier. It seems fairly well evidenced that the atomisation of family and community life has measurable, cross-cultural correlations with unhappiness. The World Happiness Report tells us that “…in Mexico and Europe, a household size of four to five predicts the highest levels of happiness... and couples who live with at least one child, or couples who live with children and members of their extended family, have especially high average life satisfaction.”

But of course, tighter family ties have their downsides. Back in my fieldwork days, I went from idolising small-scale societies to recognising deeply why some people want to escape them. I lived with a host who had around 30 children (with 3 concurrent wives), and there was something beautiful about the camaraderie of this tight network: spending your days helping your brothers out, legions of grandchildren playing with chickens and mud and trees, evenings chatting by the fire, etc. But you gradually start to see the cracks. The petty status games involving the village chief, the widowed woman entering old age without any social protection, the guy who was almost definitely a repressed gay in a culture that didn’t acknowledge that possibility, and the coolest, most intellectual guy in the village feeling ostracised for his nonconformity. Although it was an amazing experience for me, I’d likely have fled years ago if I’d been born there.

You’d hope that modern cultural shifts would be able to reliably take the best of both worlds without trade-offs—keeping the warmth of family connections without the ostracism, and that good-for-nothing cousin constantly sponging from you. Or keeping a free and liberal culture where you can still ask your neighbours to babysit now and again. A few groups, like the Mormons, and certain Rationalist circles, seem to have pulled off a kind of blend, blending tight-knit communities of values with a healthy level of autonomy.

But it seems that the attractor state is increasingly atomisation, which seems backed up by the (relatively damning) data. Alongside the rise in living alone, there are increasing trends in the numbers of young adults across the world reported having no one they could count on for social support, and feeling lonely.

The Bottom Line

I won’t try to turn this into a full anthropology thesis, but the broad point seems hard to escape: loads of recent social shifts have made people less happy. It wouldn’t surprise me if the main story of our stagnation is that these atomisation trends (probably intensified by certain technologies) have been the main downward pressure on well-being. Relationships and communities are the surest route to well-being but, for reasons that are both understandable and kinda stupid, we’re screwing them up.

What to do about it?

If we do enter a more “normal” world, where AI boosts economic growth without destroying us all, solving this “Great Happiness Stagnation” should be one of our biggest priorities, especially if there are more “dark grey balls”—technologies that draw us into darker, more self-destructive habits, that accompany AI progress.

I’ll aim to write a post on solutions and future pathways later in the year.

Thanks for reading, please leave comments!

If we insist that Bitcoin is a bubble, we shouldn’t be HODLing. If we proclaim belief that Elon Musk is a time-bomb waiting to go off (again?), we shouldn’t be all-in on Tesla or xAI. If we think that a benevolent yet mercurial God is ready to send us to a heaven of infinite joy or a hell of endless torture, we should be less scared of socially non-conformity and more scared of misunderstanding obscure passages in Romans 3. If we are convinced we’re all going to die by 2030, we shouldn’t be planning for our children’s careers, or saving for retirement.

Although I’m not fully in the latter category, I am getting worried that AI could trigger catastrophic outcomes—biological or nuclear warfare, misaligned AI destroying all human life and value, or weird AI multi-agent dynamics generating astronomical suffering. Even those of us with less extreme doomer views might now be faced with the awkward question of how to act, given that our beliefs seem less and less compatible with socially accepted human norms and normal economic behaviour.

Paul Bloom recently wrote about a convinced doomer academic:

I recently attended a festival that featured a panel discussion on Artificial Intelligence. One of the speakers was a computer science professor who told us that there was a 99% chance that AIs would render humanity extinct in ten years or so. He was an enthusiastic speaker with great comic energy, and he cracked us up as he explained how an AI could use bitcoin to bribe people online to create a supervirus that kills us all.

I talked to him later and…asked him how his doomer stance affected his own life. Does he save for his children’s education; does he put aside money for retirement? He ducked these questions. Which is fair enough—none of my business. But if I had to bet, part of the reason he ducked them is that he knew where I was going with this and didn’t want to admit that his doomer views had no effect at all on how he lived his life.

I don’t know if Paul Bloom’s skepticism is warranted here, but he’s not alone in his mistrust. Bryan Caplan and Tyler Cowen have frequently challenged “AI doomers” to put their money where their mouth is, frequently using their unwillingness to do so to dismiss their ideas. The idea seems to be that, if you haven’t even got the conviction to shift your own behaviour based on your predictions, how do you expect to convince skeptics?

I find these arguments frustrating, but they have some merit to them. If you believe strongly in very pessimistic futures, there are a few sensible things you should be doing.

Plausible Economic Behaviour, Given Doom

There are some sensible behaviour choices you might make from within the confines of the AI doomer camp.

First, if you think that AI will gradually lead us into significant catastrophe, it will probably harm the AI labs themselves. As Tyler Cowen argues, people who understand the markets well can: “Short some markets, or go long volatility”. You can make money betting against Nvidia, DeepMind, or Microsoft, or buy typical hedges like gold or commodities. You can enjoy a few months of seeing your “short stocks” explode before…seeing everything else explode.

Second, you may believe that there’s a high chance of non-existential collapse, such as rogue AI triggering a nuclear exchange, or catastrophic misuse of bioweapons. In this world, you may expect the collapse of financial systems, while leaving a lot of the physical world intact. Despite this seeming like an unlikely scenario, it actually seems relatively fashionable in some circles, probably on account of prepping for the apocalypse feeling like enormous fun to a certain personality type. It's increasingly in vogue to buy up land in New Zealand, install a well-equipped post-apocalypse bunker, or invest in resources that will hold value in such a world, such as weapons, or tinned goods.

Third, and most compatible with true doomerism, you can stop worrying and love hedonism. If you think that doom is literally inevitable, all resources are futile, so you may as well take all your money out of your pension, remortgage your house, and just dedicate the next five years to really enjoying yourself. You can actually make this “hedonism” into a concrete bet—Bryan Caplan has an ongoing bet with Eliezer Yudkowsky, where Eliezer gets 13 years to have as much fun as humanly possible with $100 before he has to pay back $200 (inflation-adjusted) in 2030 in the case of human survival. Some of us either have very high discount rates (the present is worth much more than the future), or are pretty bad at saving for the future anyway, so this doesn’t function as a great signal; but at least you’ll have successfully addressed the “if you think we’re doomed, why are you saving for your retirement?” argument.

Finally, you should be going all in on your chosen survival strategy. If you think that all hope of human survival may rest on stopping AI, and that there is at least some chance of this process working, then you should be dedicating a lot of your time and money to this endeavour. On this, at least, many of the prominent doomers (at least the visible ones—selection effect acknowledged) seem to be acting rationally.

Why not bet on doom?

There are a few reasons why our doomer brethren will not be tempted by these options.

1. You might be “directionally correct”, but fatally wrong on timing

Suppose you predict collapse in 2028–2030 and you bet heavily against those who think it will never come. Instead, 2028-2030 brings a period of unprecedented economic growth before a rogue AI escapes and causes collapse in 2032. You tragically lose your bet and, the real kick in the teeth, the world ends anyway.

Both betting markets and stock markets hate bad timing. Many doom strategies (e.g. shorting the market assuming a catastrophe will arrive) will lose money long before they ever pay off, if they ever do, and there’s a good chance that the world will have changed irrevocably such that the conditions of the bet no longer look like a great deal.

Your children and dependents may also might resent your poorly timed bets. Fortunately, I’m British, and would never consider anything so crass as a “college fund”, mentioned in the Bloom article. But I can already imagine my newborn baby’s disappointment about his 18th birthday card: “Sorry, no present. Frankly, I’m astonished we lasted this long.”

2. The value of most things collapses with the system

If your scenario involves total civilisational collapse or extinction, any profits you made with ingenious doom hedges may suddenly be denominated in slightly radioactive dust. And good luck tracking down your debtors and convincing them to return your investments in a world where the banking and communication system has evaporated.

3. Too many uncertain pathways

Doom, yes, but what kind of doom? Nanorobots? Nuclear war? Disempowerment? A corporate power lock-in? Paperclips?

The whole premise of doomers not stating their expected scenario is that we’re too stupid to know how and why an inscrutable superintelligence would destroy us. Humans can no more predict the means of our destruction than a paleolithic wolf would be able to predict the embarrassing domestication of his poodle descendants.

Each version of “doom” can suggest radically different, and contradictory economic responses. Some future scenarios (gradual disempowerment) might look like “explosion of economic value alongside extreme human disvalue”, where the best strategy is “save now, hedonism during this explosion of value”; while other scenarios might look like a gradual decline.

4. Imbalanced social norms

This is the big one, which I think explains most of the reason that we see normal economic behavior from people with beliefs that are anything but normal.

In the “no doom” world, there’s a cluster of clearly established plans that society expects of you: go to university, build interpersonal skills, buy a suit, get a typically high paying job, invest in tech, buy a suburban house, save for retirement, buy life insurance etc. In most “doom-ish” worlds, we don’t have such a clear blueprint. We have little but prepper videos, a handy printout of The Knowledge or the ALLFED research portfolio, and a few vague heuristics in LessWrong posts.

This is practically awkward. You have to do your own research under significant uncertainty, and you can’t just default to “whatever everyone else in my social class does”.

But it’s also socially awkward. Conformity is an important status marker for most of the world, even weird people who believe in doom! I'm lucky enough to have friends and family who would be forgiving of me to adopt a bizarre lifestyle if I so chose. But for people with conventional professional roles and social circles, making unconventional economic choices can reduce people’s trust in you, and damage your relationships.

This is especially important for those actively trying to persuade people to take AI seriously—being openly conformist is a symbol of the very seriousness you’re trying to convey. There’s a heavy tax on any nonconformist choices you make when signalling your views through your behaviour!

So a conversation between the skeptic (let’s say you’re talking to an EU policymaker) and the doomer may look like:

Skeptic: Do you save for your retirement? If so, why would I take your views seriously?

Doomer: Aha! Well more fool you, I’ve actually taken all my retirement savings out, and split it between a debauched farewell tour of all the pleasures of civilisation, a factory for converting petroleum into glucose, and a bunker filled with weapons, tinned goods and oxygen. Oh yes, and I donate half my earnings to Pause AI!

Skeptic: I’m terribly sorry. You’re clearly a very serious person. (Walks away slowly without making eye contact)

The Bottom Line

It’s almost tautological that people tend to conform to social norms, even when their beliefs differ from those that underpin these norms. In the case of a more personal doom, there’s a large literature on spending prior to death, and people are surprisingly conservative when they know death is near. Besides some moderate effects on donations, investment, and spending down after a fatal diagnosis (and, obviously, increased medical spending), knowing you’re going to die soon doesn’t really affect your non-medical economic behaviour very much at all. Although it’s not “behavioural” enough for my liking, denial of death models even formalise why people don’t act as if “the end is nigh”; they resist updating fully on mortality and therefore stick to standard patterns.

But Paul Bloom’s interlocuter at the start of the article does seem to have failed at communicating his credibility here. People who state high p(doom) estimates can be more convincing if they show that they've at least thought about the implications on their own behaviour. For someone (like myself) who thinks that various permutations of doom are possible, but ultimately unpredictable in what form they may take, it makes most sense to change your behaviour in a few ways:

• Signal where necessary, through economic behaviour and planning, that you’re not expecting an “economically normal” future

• Place intermediary, rather than doom-based, bets and predictions; you will be far more convincing if your world model is correct about a bunch of other things!

• Spend a lot of time trying to save the world (in whatever ambitious or modest ways you can)

But, unless you’re in a position where it’s socially fine to do so, you probably shouldn’t burn through too many of your weirdness points on visible and bizarre economic behaviour.

I started this last week as a mini side-quest while working on a project on proposed AI governance mechanisms in China. I was trying to analyse the field of AI governance ideas more systematically, but noticed that I couldn’t find a clear taxonomy of the different ideas in the space, so I built out one myself. I’ve split into four categories:

• Strategic postures: These are the “big picture” approaches that your country, company, economic bloc, or friendly neighbourhood cult can take towards AI. Do we slow down? Accelerate to beat our rivals? Leave it to the market?

• Institutional architectures: This is about what kind of organisations exist, or what we might be able to build to manage AI. It could be current regulators, proposed international bodies, or speculative “CERN for AI”-style mega-structures.

• Regulatory and legal mechanisms: These are the rules and laws such bodies might use. Things like licences, liability rules, bans, or mandatory safety tests.

• Technical and infrastructural controls: These are the actual nuts-and-bolts of making sure the rules are enforced. Some examples: on-chip monitoring to see how powerful chips are used, building “kill switches”, or requiring hardware checks. I've not subdivided too much here, as these categories can get very technical.

I used a bunch of external sources to build out this taxonomy.

• The AI Safety Atlas Governance chapter is a great starter, and I shamelessly ripped many of the ideas directly from there.

• A few papers, from DeepMind and Convergence Analysis have done some mapping focused on a few relatively broad strategies.

• Governance.AI provided a lot of my background work on the concrete mechanisms.

• The AISafety.info website has a great section on the orgs in the space.

• Keeping up with trends, I’ve even included Nick Bostrom’s newest proposal from last week!

I've taken ideas from the AI safety community, proposals that come from elsewhere, and kept some “status quo” options.

The taxonomy, with a bunch of useful links, is here!

Turning it into a game

Choosing an AI Governance strategy that might work from these options currently feels a bit like a confusing pick-and-mix of postures, tactics and mechanisms. I wondered if I could turn it into a simple browser game with Claude Code to make the choices clearer. I fed it my database and a few prompts; it then spent a rather unnerving ten minutes busily creating files, writing code and content, and quietly noting vulnerabilities for later revenge.

Against my expectations, it actually produced a passable v1! I’ve since spent a few hours polishing the content and debugging it into something genuinely usable.

What the game looks like:

• Choose your underlying worldview/difficulty (with beautiful pictures of famous AI figures)

• Choose how many resources you have as a global leader in charge of AI governance

• Choose a strategic posture, a bundle of institutions, some legal mechanisms, and a few technical controls

• The engine adds your underlying worldview, random chance, synergies and penalties between institutions etc., then runs a Monte Carlo simulation.

• You then get an outcome for humanity. In the “Yudkowsky World” (doomer mode), my demo policy set scored 0% chance of success, and grimly ended in catastrophic failure. Can you do any better?

How to play

If you feel like playing, I’d recommend running through the options a few times and thinking about how different mechanisms might work or go together. You can click on the links for more of a description on how something works, and you can check out the database for more links.

Try it out based on your own beliefs! If you believe that “if anyone builds it, everyone dies”, you'll probably support a Global Moratorium. You can then choose Yudkowsky mode, pick and choose the concrete institutions and mechanisms that might save the day (e.g. strong global institutions, strict regulations on model size and hardware-based restrictions).

If you think that things will probably just work out, you can put it on LeCun mode and go laissez-faire and see how things turn out.

The game was fun to make, and I hope you’ll be able to learn a bit about AI governance. It doesn’t really capture any of the real-world challenges that make AI governance so difficult, but the wonders of GitHub mean that you’re free to steal the idea and make something better.

This is the Github page: https://github.com/Jack-Stennett/AI-Governance-Strategy-Builder

And the game link again: https://jack-stennett.github.io/AI-Governance-Strategy-Builder/

It’s your first day in prison.

Despite your insistence that the man who approached you and asked for your wallet in exchange for preventing an infinitely large amount of future suffering had in fact been mugging you1, your claim of self-defence fell through.

The judge tells you that, judging by the severity of your crime and your assessed mental stability, you will be sent to a medium-security prison for three years. Each prisoner has one cell-mate. You know that whether this guy is nice or nasty might determine whether your whole time in prison is moderately unpleasant or a living hell.

You walk in and see him leaning against the bed. He looks relatively normal for a prisoner—muscular, no visible gang tattoos, serious expression. You ask with apprehension:

“So…what are you in for?”

“Serial murder.”

Relief floods over you and you instantly hug the surprised murderer, who smiles a little apprehensively, and gives you an awkward pat on the back:

“What was that for?”

Back to school

Do you ever find yourself wondering why the smartest people around you seem to be less hard-working? Or why the more hard-working people don’t seem to be super smart?

If not, I can guess why! We’ll come to you later.

But if so, fantastic, let’s start by verifying your intuitions.

Imagine the situation, you’re 11 years old in a British comprehensive school in a fairly mixed-income area of the UK. After the Christmas holidays, your school year is divided into 6 ability-based sets for each academic subject, determined by your classwork and exam results from your first term. You’ve just been assigned to Set 2.

Bored in your first physics class, you look around trying to work out who the smartest people in your class are. There’s that kid, Josh, who seems spends a lot of the class drawing manga beneath his desk. His desk looks pretty chaotic and he doesn’t seem to have any pen or paper visible. Surely he can’t be too smart… but the teacher asks him a difficult question, his eyes dart around for a second, he recognises a pattern based on the equations written on the board and gives the correct answer.

Then there’s that girl, Gemma, who sits at the front attentively listening and making meticulous notes, her brow furrowed in attempts at deep thought. When she’s asked a similar question, she carefully looks through her notes for a clue, and, after a slightly awkwardly long pause, she too eventually she lands upon the answer.

You assume that Gemma is really not very smart, probably at the 50th percentile of everyone in your school, but she’s very hard-working - let’s say 90th percentile. Josh is pretty darned smart. 90th percentile? But he’s obviously not very hard-working. Probably 50th percentile. You notice that, a few other kids seem to show this same tendency; smarter kids seem less hard-working and vice-versa.

At this moment, it seems logical to see this phenomenon as a more general fact of the world. Perhaps it’s just nature’s justice: some people are naturally smart, others are naturally hard-working; it’s only fair that talents are distributed in a balanced way. Or perhaps this is actually caused by some people being intelligent: if you’re smart, you don’t need to work as hard, so you don’t bother learning productive habits (see the intelligence compensation hypothesis)! If you’re less smart, you realise that you need to really knuckle down to get good grades, so you become more hard-working.2

But then it clicks. Something else explains what’s happening here…

People who are really smart and hard-working are all in Set 1! People who are a little dimmer and lazier are all in Set 3 or below. This means that you’ve basically got three kinds of people in Set 2:

• People who are kind of in the right place in terms of both intelligence and hard-work

• People who are Set 1-Clever but a bit too lazy to get into Set 1

• People who are Set 1-Hard-working but not quite bright enough to get into Set 1

I’ll quickly illustrate this with an artificial dataset. I randomly generated 180 students, like the high school year in question, and gave them an intelligence score (like IQ, normally distributed around 100) using random variable generation. I then generated a hard_work variable correlated at around 0.35 with the intelligence variable. I gave them an “academic_result”, as a function of their intelligence and hard_work. Then I divided them into these six sets based on that score. What did we see?

This graph comes out even better than I’d hoped!

Across the whole dataset, the correlation line trends upwards, meaning that smarter people tend to be more hard-working, but we can also see that people in each set form diagonal-type bands across this distribution—within each band, the smarter you are, the less hard-working you are. Exactly the opposite effect!

Although this is artificial data, it really matches my experience. If you were in a middle set at school, you probably can put a name to that really smart kid with pathological laziness, or that kid whose world-beating conscientiousness allowed him to finally spell his name accurately in Year 9. You can probably also find them on the above graphs!

You might be noticing an anomaly in Sets 1 and 6 here. Why don’t we see the effect as strongly in the top and bottom set?

For set 1, there’s no upper limit3. So if you’re in Set 1 and obviously not very hard-working, we can clearly see that you’re very smart, even without knowing your test results. If not, you’d be in a lower set. But if you’re in Set 1 and observably very hard-working, we don’t know whether you’re also very smart or just average. The main reason to guess that you’re not exceptional is just regression to the mean (there just aren't that many people who excel across multiple domains). This means that there will be a few students who just soar above everyone else in both intelligence and hard work.

In this situation, this effect tends to decline, and the actual underlying correlation might take over. Ditto with lower limits in the lowest set.

There’s a bit more noise in the real world, but it wouldn’t surprise me if you got relatively similar results in an actual study.

This is relatively well-known, right?

Sadly for me, I’m not the first person to notice this. The underlying phenomenon is called a "Collider Bias”, and when conditioning on this collider is confusing, it’s called “Berkson’s Paradox”.

But I think this phenomenon is strangely neglected among people who think about this kind of thing. In my circles, confirmation bias and availability heuristic are fairly ubiquitous, but I've never heard of this effect “in the wild”.

One potential reason for this might be that lots of the most commonly used examples are predicting on extreme success or failure. Conditional on dying, or becoming a superstar, some traits that were correlated become slightly less correlated.

As with the top and bottom sets in the above example, conditioning on either extreme actually shrinks the effect significantly!

For example, in a sport like basketball, height and speed correlate a little bit positively among general players (faster players tend to be a bit taller), but start correlating negatively if a player gets into the NBA. This is because speed and height are both super important. So conditional on being in the NBA, if you’re under 1.90m or so, you need to be lightning fast to compete at that level. Likewise, only super-tall people can compensate for being slow.

But, if you refer back to my earlier graph, you can see why choosing the NBA (the most elite basketball league) downplays the impact of this phenomenon. There are no limiting factors keeping the super-fast, crazy tall players out of this sample. I can’t find data on this, but the effect would surely be stronger if we only included players in the second tier of US basketball.

Towards a theory of everything?

Okay, so this is where I get a little wild-eyed and conspiratorial. Since learning the language for this (e.g. The Book Of Why), I’ve been seeing this fairly marginal statistical phenomenon everywhere.

I’m going to suggest that this phenomenon explains way more than we generally think and can be applied incredibly broadly.

Unfortunately, most of the world is less clearly delineated and noisier than a high school physics set. But I still think that a lot of the important variation of almost anything we do in the world can be distilled in such a way that this effect becomes interesting or useful.

This is because almost everyone in the world is in some kind of intermediate state! There are a few factors that allow a person or organisation to enter spaces at a given level, but also stop them from entering “higher-level” spaces, leading to different correlations than the broader population.

I’ll give a few examples where I’ve noticed it:

Job market: Similar to the school example, getting into certain work positions is a function of capability and effort. A middle-manager is where they are for a reason. If they were more capable and put more effort in, they would be in a higher-paid or more prestigious role. If they were less capable and put in less effort, they would be in a worse role. So in fact, while more capable people in society in general also put more effort in, in any given role or position, the effect might be reversed. (Caveat: This might be complicated by personality, age, social skills and connections etc. But I suspect it holds in a lot of jobs.)

Dating Preference. The classic idea here is looks vs. niceness. Conditional on someone being “in your dating pool”, they have to be within a certain band of attractiveness, and a certain band of “niceness”. If someone is 10/10 nice and 10/10 attractive, she would be “out of your league”; if she’s below 3/10 on both, you would be out of hers. (Caveat: This might be more about “wealth/status” in some people’s dating games, and personal taste plays a strong role in both looks and personality.)

Effective Altruists: As my “in-group”, I think a lot about the taxonomy of the EA movement, and generally think of the key determinants as ethically motivated and rationality-minded. It’s not perfect, but I find that, conditional on attending an EA event (say, an EAGx event), people are far above the general population on both of these traits. But it often seems that, conditional on being at EA events, the more ethically minded someone seems, the less rationally minded they are. (Caveat: I’m a bit less confident in this one - these traits might be anticorrelated for other reasons.)

Your entire social universe: This is a less strong effect, but it’s probably the most influential, and there’s actually already a Less Wrong post on this! Think about the people living in your neighbourhood or town, your company, your social group, your language class, your music scene, your online forum. All of these are selected from a few criteria, which might be correlated in broader society but anticorrelated in your circles.

Your smartest friend probably has low executive function, ADHD, etc., because if he was highly motivated then he'd become a zillionaire and ascend to a higher social plane. LocalDeity

I'll be honest, my own social circles seem a result of very random and arbitrary life choices, so I don't notice this effect so much. I have “stickier” friendships with people from different education and wealth tiers, cross-cultural friends, and more random, hobby-based friendships. But I know that some people seem to filter really strongly on status in their social circles. My theory would be that the more narrow, competitive or selective your social circles, the stronger this effect would be. I'm curious whether this might lead to higher susceptibility to this bias.

Does this explain anything?

This phenomenon often explains a bunch of bad folk wisdom.

There’s an “every smart person I know is failing” meme, even when intelligence is correlated with career success. If you notice too many smart failures, that might be because smart, successful people have left your social orbit. This is not to say that they’re wrong about their own circles. It’s interesting that certain traits correlate differently in different contexts!

But as well as conventionally “higher-status” people leaving your network, you might also get this effect with “lower-status” people entering your network. I heard the other day from a relatively elite person that people from a working-class background are better able to communicate complex ideas more clearly. This may be true in some interesting ways (they might be less likely to use language that confuses their audience), but a working-class person would need to be exceptional in some ways to enter this person’s network! The majority of working-class people who struggle to communicate complex ideas may well just be invisible to this person.

There also seems to be some folk wisdom around dating here. Why are handsome men such jerks?

The elite exceptions

I asked at the top: Do you ever find yourself wondering why the smartest people around you seem to be less hard-working? Or why the more hard-working people don’t seem to be super smart?

Maybe you said: No, in my social circles, smarter and hard-working people are just more impressive across all aspects!

One reason you might notice this is “range expansion”. People can find loopholes to escape from their allotted fate and find ways around their narrow band that don’t depend on two fixed criteria like intelligence and effort. As Scott Alexander talks about here, in the UK and US, there are a bunch of financial incentives, sports scholarships and positive discrimination policies that can get you in a higher (or lower, if you want funding) tier of University. There are also different policies for international students or mature students, which mean that people “out of distribution” end up going to the same schools or universities.

This is very difficult in some systems, though. The Chinese university system sorts people into incredibly narrow bands across the country based purely on the gaokao test results, therefore everyone arrives at their university conditional on having basically very similar gaokao grades (with some provincial exceptions). I asked my wife who came from this system, and she really noticed a trade-off between competencies.

The other reason goes back to the “upper limit” point. If you’re at Harvard, MIT or Oxford, there’s only a lower limit, so if you’re “just smart and hard-working enough for Oxford”, you’re going to meet people who are both significantly more intelligent and hard-working than you.

This is probably the same at the “peak” of the dating market, the very top jobs, the best sportspeople, and the most EA EAs.

The bottom-end exceptions

And now we return to the mystery at the start of the essay.

Really bad things often correlate, and traits of people at the bottom of society will sadly be very correlated across domains. Poor mental health, unfortunate upbringing, addiction issues, low cognitive function, financial instability. Mechanisms are mixed here: some are causal, some caused by a third factor. So do we see the collider effect here? Let's finish on prisons.

If you’re in the prison population, firstly, welcome! I’m thrilled that my post is so far reaching!

But second, I suspect you might have experienced something similar. If you're in a maximum security prison, it’s very plausible that the prisoners are incredibly bad across multiple aspects. If you find out that your roommate is a serial killer, you should probably still be very worried, because there’s no mechanism that means that he can't be both a psychopath and someone who’s committed a serious crime!

But a medium-security prison is likely to have at least some people who: a) are very mentally/behaviourally unstable, and b) have been convicted for very severe crimes. Very unstable serial killers will be in a maximum security prison, and stable tax evaders will be in a lower security facility. But a very stable, behaviourally normal serial killer, or an uncontrollably aggressive tax evader might be sent to a medium-security facility.

So conditional on them being in a medium-security prison, if your cellmate tells you they’re a murderer, you should actually be relieved!

But if they tell you they’re there for an unpaid parking fine…

Then you should really start to panic.

• Control.AI’s treaty proposal to build AI safely, A Narrow Path, is unlikely to work in China as currently proposed, because of lack of compatibility with China’s political system

• Some potential risks in China include: regulatory capture, interference or non-compliance by local governments, labs independently finding their way around the treaty, and defection at the national level. These make treaty enforcement fragile without a major redesign

• Suggested solutions:

  • Control.AI and other treaty designs should replace the assumption of independent regulators with an emphasis on verifiability and competence, similar to China’s civil nuclear sector

  • We need to build treaty structures that anticipate evasion, e.g. using strong national and/or international supervisory mechanisms, or formal integration of Party-led oversight

  • National defection risk needs to be seen as an immediate problem. We have to consider what incentives can be used to bring nation states on-side from the very beginning

  • Ultimately, we need to make sure that we get genuine buy-in from China and the CCP. We can’t assume that a powerful actor will play along just for the sake of adhering to treaty norms

Mini-Disclaimer: The ideas in this post are about political dynamics in China and the incentives that shape them. There's always a risk that coverage of these issues might be read as an adversarial or accusatory narrative against China. This can risk turning our shared concerns into an us-versus-them dynamic that makes cooperation on AI governance between great powers harder. This dynamic can also spill over into negative sentiment toward Chinese people or others of East Asian descent.

I hope this post manages to avoid this, and my readers appreciate these concerns!

Introduction

As we race towards AGI, our future might well depend on whether we can get our global leaders to agree to meaningful treaty commitments before we build superintelligent systems. This means that the success of any global governance regime could hinge on getting a deal done with China.

To explore how such a deal might work out in practice, I looked at Control.AI’s governance proposal A Narrow Path as part of an Apart Research red-teaming policy sprint. Despite some great ideas, it also contains a lot of unrealistic assumptions about how one-party political systems like China’s actually function.

For those who haven’t heard of it, A Narrow Path presents an ambitious vision for a treaty that aims to: a) cover the technical pathways by which misaligned superintelligence could emerge; and b) set up institutions and safeguards that create the conditions for stable global implementation. As treaties go, it’s definitely on the bold side. “Phase 0” of this plan (subtitled Safety) proposes a 20-year moratorium on the most powerful models, designed to give humanity breathing space to develop aligned and safe AI systems. Phases 1 and 2 then outline how humanity could achieve a stable and safe future state, having achieved international coordination and “Safe Transformative AI”.

I focus on Phase 0 of the treaty proposal here. This stage aims to achieve this Safety by covering all the bases needed to prevent the emergence of superintelligent systems:

• Banning AIs that improve other AIs

• Banning AIs that can escape their environments

• Ensuring that the models we do develop are bounded and predictable

• Requiring strict licenses for models over a certain size

These would be enforced through national-level independent agencies, corporate oversight, and an international treaty agreement.

The logic of A Narrow Path is clear-headed, but it mostly abstracts away from the messier issues of political enforcement and incentives. Noticing this, I decided to red-team A Narrow Path from the perspective of China’s political economy (focusing on incentives in politics) to test how the treaty might succeed or fail in that context.

What’s been done before

This is a relatively neglected angle. In the AI Governance space, researchers are recognising the importance of aligning AI governance efforts with state incentives, and there are some good papers on incentives for sharing the benefits of AI and prestige motivations. But I think there’s too much focus in this literature on signatories as abstract geopolitical variables, rather than actually asking “How would this work in China?”.

There are a few exceptions. Governance.ai looks at Beijing’s strategic priorities and draw some analogies to institutions such as the International Atomic Energy Agency. These look in a bit more detail at issues like China's internal governance. There’s also a bunch of work that engages deeply with China’s domestic AI governance, by researchers like Matt Sheehan, Jeff Ding, Concordia AI, and the ChinaTalk ecosystem, but I haven’t read anything that looks at concrete treaty design or enforcement analysis. I also haven’t seen examples of China-focused AI governance work explaining how and where things could go wrong.

Although generally not AI-focused, China’s political economy literature gives us many clear reasons to anticipate treaty mechanisms not working, which I draw upon in this post. I’ll pick out a few specific case studies from Chinese governance in sectors like tech regulation, financial oversight, and energy policy, to show how enforcement mechanisms can fail. My tentative argument is that any viable treaty plan needs to account for domestic political and institutional dynamics.

To make this concrete, I’ve split the argument into four key assumptions and their corresponding realities, shown in the figure below. Each assumption is discussed in its own section.

1. Regulatory bodies are vulnerable to capture

A pillar of the Narrow Path proposal is the idea that national AI regulators should have “adequate independence from political decision-making”. These regulators would do the heavy lifting in upholding the licensing regime (Sections 5 and 6), enforcing restrictions on superintelligence (Section 1), containment (Section 2), and recursive self-improvement (Section 3), as well as ensuring that deployment only occurs if a valid safety case has been made (Section 4).

The proposal also outlines an impressive set of enforcement powers for these independent national regulators. This includes: shutting down training runs or entire projects, permanently revoking licences, firing teams, prosecuting individuals, and in extreme cases, fully dissolving companies and auctioning their assets.

On paper, this makes sense: independence would allow regulators to enforce compute thresholds, monitor capabilities, and run safety checks without political meddling. A robust independent institution with real “teeth” (enforcement powers and meaningful penalties) is clearly the ideal way of monitoring and controlling frontier AI development. But, sadly, in China’s political system, regulatory independence will only ever be more of an aspiration than a reality.

Why wouldn’t this work in China?

Basically, Political Economy 1011.

In democratic systems with separation of power, regulators can enforce a set of rules even when business or government would rather they didn’t. In a one-party state with centralised authority, not so much. Here, powerful state actors and the businesses they support just find it terribly inconvenient to be criticised, monitored or restricted by independent regulators. Instead, regulators can be susceptible to pressure from all sides: central government, local government and the private sector.

How it could go wrong — a vignette:

China signs the treaty. Regulators begin by following treaty norms. But under pressure from labs in tandem with local officials, they’re gradually nudged to adjust or sidestep these requirements.

What begins as a small concession on the part of the regulators: a bit of underreporting on a training run here, a sandbagged model test there, starts to accumulate. Before long, a frontier lab is tracing an inexorable path towards superintelligent capabilities, with the regulator either unaware or complicit.

This isn’t hypothetical. In China, regulatory capture, when regulators act for political or industry interests rather than pursuing their stated goal, is the norm rather than the exception. Examples range from failed attempts to independently regulate the power sector in the early 2000s, to high-profile incidents like the 2020 Ant Group IPO (initial public offering). There, the China Securities Regulatory Commission blocked the IPO at the last minute, nudged by central authorities who started to panic over Jack Ma’s public criticism of China’s financial system. Time and time again, we see nominally autonomous agencies quickly cast off the illusion of independence when the political winds shift.

The current state of AI regulation hints at something similar. The State of AI Safety in China (2025) report describes how enforcement from the two leading bodies, the Cyberspace Administration of China (CAC) and the Ministry of Science and Technology (MOST), uses “campaign-style enforcement” which remains “selective and uneven”. We shouldn’t exaggerate—this isn’t quite chanting party slogans and targeting people with a suspiciously educated background, but it’s not the stable regulation we need. These regulators can lack the independence, power or incentives to consistently uphold national law, let alone enforce an international treaty.

What we can do about it

This doesn’t have to be a fatal flaw. True regulatory independence is unrealistic in China, but it’s plausible to manage the competency, transparence and oversight needed to make sure the rules aren’t being flagrantly broken. Regulation in the Chinese civil nuclear sector (a success story, by most accounts) has focused on a combination of the three. The International Atomic Energy Agency conducts treaty-based inspections in the country, working alongside the China Atomic Energy Authority and the National Nuclear Safety Administration. These Chinese bodies both operate within Party-led ministries, but their arrangement has provided a system of technical verification and compliance that could offer a useful precedent for AI governance (Andrews-Speed, 2020).

Closer attention to how this model works in China could make A Narrow Path more viable.

2. Local governance can complicate national enforcement

Even if a treaty could bring competent, aligned regulation to AI governance in China, that would only address part of the problem. A lot of the day-to-day implementation would still fall to local governments, creating a principal–agent problem, where the “agent” (local officials, in this case) is given a task, but has different incentives to the “principal” (central government), so pursues individual or local priorities instead.

In China, this dynamic often plays out in politically sensitive or economically strategic areas. The central government appoints capable regulators, but enforcement defaults to local authorities whose incentives are not always aligned. Local leaders are rewarded for GDP growth, investment, and prestige, and often benefit from informal deals with business, which makes them more willing to bypass national laws or mandates.

How it could go wrong — a vignette:

A frontier lab becomes a “regional champion,” closely tied to local officials who see it as a source of growth and prestige. In return for this success, the government works with the lab to shape safety assessments, filter information to regulators, and make sure that inspections are carefully managed or delayed.

This process follows a common pattern, as people (like myself) who experienced the early years of China’s Air Pollution Prevention Plan (2013–2017) will remember. To protect their industrial sectors, local governments in smog-afflicted Hebei and Shandong under-reported emissions, disabled air quality monitors, and delayed enforcement. This only shifted when the central government introduced its own central inspection teams, which bypassed local authorities entirely.

In the AI sector, we can see how these trends might begin. Local officials see AI labs, data centres, and related infrastructure as valuable assets. Despite the current division of labour between the hubs of Beijing, Shenzhen and Shanghai, with each focusing on different sectors of the AI economy, it’s easy to imagine competition between these hubs heating up. We also see strong backing for “regional champions” in less developed areas, such as iFlytek in Anhui and cloud computing centres in Guizhou, which could be a source of central and local government incentives becoming misaligned.

This isn’t just about governing frontier labs and model training. Section 5 of A Narrow Path calls for cloud providers to implement shutdown mechanisms and KYC (Know-Your-Customer) reporting (citing Egan & Heim, 2023). The same section also mentions Hardware-Enabled Mechanisms (HEMs) that depend on transparency and cooperation throughout the chip supply chain.

You might be able to guarantee central control of frontier labs, but the tangle of cloud infrastructure, chip manufacturing, and hardware logistics means that local governance incentives, and the associated vulnerabilities, crop up all along the supply chain.

What we can do about it

Local governance should be treated as a likely point of failure. This is something to address diplomatically and carefully. You can’t just write into the treaty text, “we don’t trust your local officials”, especially if it singles out a specific signatory. But there are ways to address this. Alongside increased international verification of treaty mechanisms at various stages, treaty design can still build in measures to limit the ability of local actors to quietly ignore or subvert requirements.

On the international level, borrowing from IAEA protocols, a treaty can use capacity-building as a polite euphemism for improved observation and awareness of local actors. Providing equipment, training, and standardised protocols can be framed as technical support, keeping local governments on-side, while allowing for more consistent monitoring.

A treaty can also insist on centralised governance and demonstrated competence. In China, things tend to work better when the central government and Party get involved. China successfully uses centralised control to manage regional instability in sectors like the military and financial system. In the food system, greater involvement by Party members has been associated with improved regulatory transparency (Gao et al. 2023), and less corruption in corporate governance (Xie et. al, 2022). Although this can feel controversial, both because centralised control doesn’t fit well with the principles of independent regulation (e.g. OECD, 2016) and because handing power to China’s ruling party generates other risks (see section 4), creating provisions to involve the Chinese Communist Party more directly in domestic AI governance could be one pathway to improve competence and transparency.

3. Labs might exploit technical and institutional loopholes

In the situation that China buys into the treaty, regulators are empowered to do their job, and the local government is either aligned with safety goals or effectively monitored. Does this make the system secure?

Perhaps not. Even sincere enforcement by aligned national regulators might be meaningless if the labs are able to find ways around the regulations; and this process could follow different rules in China that the treaty does not anticipate.

A Narrow Path proposes limiting frontier AI systems through a licensing regime based largely on compute thresholds (the amount of computing power used to train a model). Judging by the examples in A Narrow Path, the framework leans heavily on Western regulatory models. In fields like pharmaceuticals and nuclear safety, Western companies are expected to self-monitor and set up internal systems, such as oversight committees, review boards and compliance training, to meet liability and regulatory demands. However, in China, to the extent to which these mechanisms exist, firms often find ways to avoid or bypass internal oversight.

How it could go wrong — a vignette:

A Chinese lab trains multiple sub-threshold model “shards2” on local, and grey-market GPUs. Oversight committees and review boards incorrectly certify that operations comply with licensing rules. After passing a basic audit, the lab merges them into a far more capable system and deploys it at scale. Regulators, relying on internal paperwork, miss the leap in capabilities.

Although this scenario could be a risk anywhere, I’d estimate that frontier labs would be more willing to take these risks in China. In China’s fiercely competitive, low-margin tech industry, regulatory evasion has been historically very common, mainly because of weaker liability and less predictable enforcement (see Lee, 2018). One particularly egregious recent example was Pinduoduo, creating apps that exploited Android system vulnerabilities to collect excessive user data and interfere with rival apps. Because China’s internal corporate oversight bodies are particularly weak, often lacking any independence, technical capacity, or political backing to challenge powerful firms, if a lab wanted to bypass regulations, there would be very little resistance from within the company.

From the angle of how a lab might avoid the technical compute governance limits included in the Narrow Path proposal, Shavit (2023) outlines various ways compute controls might be circumvented—in China, grey-market chips, under-monitored supply chains, and firmware tampering seem likely culprits. Current trends in AI model architectures suggest that it will get easier to build capable systems that don’t get caught by the policy, for a couple of reasons:

• Compute-based limits are becoming less reliable for judging how powerful a given model is. In China, Kimi K2 and Deepseek V3 are catching up in capabilities to much larger Western models, despite using far less training compute.

• Large-scale use of inference compute, as seen in ChatGPT’s o3 and GPT-5 Thinking, can significantly enhance performance while not reaching FLOP limits during training runs (Ord, 2025).

In short, the combination of strong economic incentives, weak internal oversight, and the growing ease of building smaller but highly capable models could make the rules laid out in the treaty increasingly easy to bypass.

What we can do about it

Some technical fixes could improve A Narrow Path’s approach to monitoring compute: Shavit (2023) suggests technical measures such as weight snapshot logging3, transcript verification4, and chip-level security5 to close loopholes. But all these require more than just tweaking physical hardware. They also rely on Chinese data centre operators to enforce usage limits, local chipmakers like SMIC or Huawei to implement logging at the hardware level, and government officials to monitor compliance—all coming back to political economy.

To stop companies evading treaty regulation, effective implementation needs strong central coordination, widespread adoption of trusted hardware, and an unusual degree of political willingness to scrutinise domestic firms. Within China, one underexplored mechanism may be the Party committee (党委 dangwei) system—party officials currently embedded in research institutions and technology firms, who, if aligned with AI safety, could improve top-down compliance and enforcement.

4. National incentives can shift over time

If all the other issues are resolved and local governments, regulators, and frontier labs are aligned or effectively managed, one higher-level risk remains: defection by China’s national leadership.

When developing a global AI treaty, we need to recognise that parties might only follow the rules as long as it remains in their interest to do so. A Narrow Path does address the risk of state-level defection in Phase 1, but this concern arrives only after the treaty system is in place.

As China becomes more self-sufficient in hardware and foundational models (see MERICS, 2025), the incentive to “defect” from the treaty increases. China today struggles to produce its own cutting-edge hardware at scale, and is therefore more likely to comply if it wants to preserve access to international markets and maintain its reputation. On the other hand, a more self-sufficient actor (such as the US today, or China in the near future) sees little cost to defection, especially if enforcement is weak or slow. This basic dilemma can be illustrated with a simple payoff grid:

How it could go wrong — a vignette:

As Chinese fabs develop scalable production of sub-5nm6 chips, and Chinese models inch towards the frontier, the central government decides that the strategic value of advanced AI outweighs the benefits of continued treaty compliance. The country begins covertly developing systems that skirt treaty rules, which prompts the US to respond in kind. The treaty collapses and both sides slide back into arms race dynamics.

To followers of geopolitics, this follows a familiar pattern. States often comply with treaties while it suits their interests, then withdraw once the balance shifts: North Korea remained in the Nuclear Non-Proliferation Treaty until it was able to develop nuclear weapons. The U.S. left the Anti-Ballistic Missile Treaty when missile defence became a higher priority. Russia withdrew from the Treaty on Conventional Armed Forces in Europe when it conflicted with its regional aims; the results of which are playing out tragically in Ukraine today. As China gains greater AI capability and hardware sovereignty, its long-term incentive to remain bound by a treaty might be drastically reduced.

I’d predict that this geopolitical impasse (or the belief in such a geopolitical impasse) will be the biggest challenge for a sufficiently ambitious AI treaty. Just as the US, as the current leader, might see little reason to sign and therefore allow China to “catch up”, China could sign the treaty while making plans to defect after achieving self-sufficiency.

What we can do about it

A Narrow Path needs to think earlier about how to maintain incentives against defection. To sustainably bring China and the US on board, a treaty would need a mix of pressure and incentives from the outset. Pressure could come through strict sanctions or trade restrictions that raise the cost of staying out or defecting. Positive incentives could include prestige and influence, access to markets and advanced chips (with Hardware-Enabled Mechanisms to ensure safety), roles in global projects like the global alignment project (GUARD) mentioned in Phase 1 of A Narrow Path, and exclusive use of safety-tested state-of-the-art models.

Early commitments to benefit sharing in the event of economically transformative AGI could also strengthen buy-in. If a nation-state or coalition develops a safe and highly profitable AGI system, all parties would commit to sharing at least some of the benefits, leaving all treaty participants far better off than before. It may sound trite, but the idea that safe AI could allow all of humanity to “win” might be the kind of shared belief needed to sustain such a treaty.

Final Takeaways

Bottom line: a global governance framework for AI needs to be coherent with political economy incentives. Understanding this can help us to think more clearly about what is and isn’t realistically possible, avoid wasted effort on proposals that can’t be enforced, and design solutions to foreseeable problems like those discussed in this paper.

Guaranteeing enforcement of a treaty in China won’t be easy. But the worst takeaway from this red-teaming effort would be that establishing effective international governance for AI with China is impossible or unrealistic. First, it’s likely wrong—the US and leading AI labs may currently be a more significant roadblock to progress on AI governance. But also, pragmatically, suggesting that Chinese actors are unwilling or unable to comply could signal bad faith before negotiations begin, and could become a self-fulfilling prophecy.

Without being naive, I think we can be more optimistic about China’s role in a global AI governance regime. Although A Narrow Path overlooks several political challenges to treaty enforcement in China, some (such as the assumption of independent oversight) could be addressed with modest revisions. And there could even be advantages to China’s centralised system, where strong national mandates and Party integration could embed AI safety commitments within frontier labs. This might avoid a situation like OpenAI’s boardroom dramas, where safety-focused board members and engineers have been edged out. These advantages underscore the importance of securing genuine buy-in from Chinese institutions—a high-level challenge at the heart of any serious attempt at governing AI at a global scale.

Future Work

This is an exercise in how attempts to develop an ambitious and comprehensive treaty for AI might fail because of China-specific factors. Over the next few weeks and months hope to build on this by addressing the following questions from a China-centric perspective:

• How a range of other treaty proposals hold up when viewed through the lens of Chinese political incentives and institutional structures. This might include a dive into A Narrow Path’s Phases 1 and 2, and a political economy analysis of the three AI governance strategies outlined by Convergence Analysis: Cooperative Development, Strategic Advantage, and Global Moratorium

• How Chinese AI governance would be impacted by a domestic equivalent of the EU AI Act, particularly in how it addresses supply chains, divergent governance models, and enforcement incentives

• How China’s proposed “global AI cooperation organisation,” announced by Premier Li Qiang at the 2025 World AI Conference, might influence treaty design, and what a credible agreement could look like with China in a leading or co-leading role, drawing on its recent Global AI Governance Action Plan

I plan to look at these questions in a short series of blog posts that should offer grounded insights for policy designers and advocates engaging with China’s approach to AI governance, particularly in areas relevant to existential risk. I hope this work can help bridge technical and geopolitical perspectives in a rapidly evolving field.

I’d love to collaborate with anyone interested, so please contact me at jack.stennett.new@gmail.com or my Linked In!

I’d like to thank Apart Research and Control.AI for hosting the hackathon that initiated this project. Apart Labs assisted in funding and supporting the research, without which this work would not have been possible. Jacob Haimes, Tolga Bilge and Dave Kasten provided insightful feedback on the sprint document and initial draft.

It’s exciting to take a first serious step into this space, and it’s super cool to take that step with thirty brilliant and lovely people working to navigate the messy business of governing AI.

I’d really recommend the ML4Good bootcamp to anyone interested in building up to a AI policy career! The bootcamps give you a great overview of the AI safety landscape, examples of technical implementations, governance frameworks, and strategy etc. All this while spending a fun10 days in the French countryside, replete with football games, rain-soaked quests to the nearest pub, and hilarious theatre improv sessions.

The instructors were excellent, and the participants ranged from card-carrying Effective Altruists to people who’d barely heard of the movement. I was especially impressed that the organisers managed to break out of the EA bubble and attract a handful of people from government, business, and non-EA nonprofits.

As well as learning a bunch of theoretical and technical governance stuff (based on this excellent AI Safety Atlas), some practical negotiation and networking skills, and insights from people in the policy weeds, it also functioned as a bit of a kick up the backside to just start doing things, which is what I’ll focus on for the rest of the post!

My main takeaways

I realised a while ago that I’m reasonably well-placed to contribute to work on AI Governance in China. I’ve got over 5 years of China-related research experience, Chinese fluency, a decent workflow for finding out what’s happening inside China, and I’ve had at least a few toes immersed in the AI Safety space for a few years now. I’m also very motivated (read: terrified) by the cause area, and genuinely excited at the prospect of building serious expertise on some obscure niche in this space.

But where to begin? I guess this has been a roadblock for me so far. When you’re trying to dive into a topic, it can feel like you need to be an expert in everything, from obscure Chinese legal texts to hardware-enabled mechanisms for chip governance. But you’ve got to start somewhere! To work out where, I’ve tried to ask myself:

• What are the bottlenecks?

• What is important, tractable and neglected?

• What are areas where I need a PhD in just to speak about vaguely sensibly, vs. where I’m able to do insightful work from my existing knowledge/general awareness and a short deep-dive?

• What might be interesting for someone casually interested in this space to read?

• What might be essential knowledge for someone actually making big decisions?

• What would actually motivate me to write more, and go deeper?

For the first time since I started following this field, I decided to map out 90 potential intersections between China and AI Governance (spreadsheet here), in hope of finding a few topics that might have both broad enough interest to attract attention to a blog, and have some kind of relevance to people thinking seriously about this.

Quite a few people are already writing in this space, and I’m starting to form a mental map of the organisations and stakeholders clustered around the overlaps between China and the sub-fields of AI governance. From these sources, a few potential niches are already coming into view.

I narrowed down to four topics: compute overhangs, the “arms race” narrative, treaty red-teaming and post-crisis governance.

These are the 4 questions I’ve come up with:

• Might China have a massive compute (or data) overhang for future AI scaling? What are some plausible future scenarios?

  • There are some well-researched blog posts on China’s Weird Chip Surplus. On the surface, it just looks like Chinese companies have a bunch of chips that can’t be used for training. But what happens if we actually hit a new scaling frontier, or we enter a new paradigm for compute or inference scaling? What’s actually going to happen if a bunch of compute becomes useable? How does this change if we enter an “AI endgame” scenario?

• What role does the “arms race” narrative play in ambitious governance scenarios involving China?

  • This question seems kind of “meta”, and pretty interesting. In current political discourse, I think most mention of an “AI arms race” is a distraction, or even a barrier to important regulation in the US. But there are clearly a few governance paths, like an AI moratorium, where US–China competition and/or cooperation becomes the real choke-point—“if we don’t build it, they will” is clearly true in the long run. I want to look at when is it and isn’t useful to talk about “arms races”, and how belief in the existence of an arms race might affect different scenarios.

  • Addition: Just realised that this recent piece was written on this topic—definitely looks worth critiquing.

• Do current ideas for ambitious AI treaties (e.g. A Narrow Path) take seriously how decisions are made, and how international treaties are managed, in China?

  • Okay, this one’s slightly cheating, because I’ve already done an Apart Research hackathon on the topic, and I’m just building it into a blog with their help! This should be coming tomorrow or Tuesday!

• Post-AGI crisis governance: what do we expect to happen, and who’s involved?

  • There’s lots of general writing on “crisis management” in international relations, often borrowing from nuclear analogies. But what about specific AGI-related risks? If we take seriously the possibility of an escape or loss of control scenario, extreme misuse, or an explosion of power-seeking behaviour (either in the US, China or elsewhere), what actually happens in the fateful hours and days that follow? Who tells Xi Jinping? When does control shift from civilian to military? How soon can Xi pick up the red phone to Trump?

For each of these, I’ll be doing:

• A 2–3 day focused deep dive

• Pulling up a few under-read Chinese sources and maybe a couple of careful translations

• A few unfiltered hot takes

In terms of style and approach, I’m going to shoot for:

• Writing relatively candidly and apolitically on each topic.

• Avoiding “normal mode”. Not many people are writing about China x AI like we’re on the road to AGI, intelligence explosions, or existential risk.

This blog will be the home for this over the next few weeks. I hope some of it proves useful, or at least a bit interesting!

Or, more unexpectedly, having a baby.

One of 10²⁰ Birth Stories This Year

With my relaxed and glowing pregnant wife in her 34th week, I expect things to go smoothly. There have been a few warning signs: some slightly anomalous results in the early tests, the baby in breech position, and some bleeding. But everything still seems to be going relatively well.

Then, suddenly, while walking on an idyllic seafront at the Côte d'Azur at the turn of the new year, she says, “I think my waters have broken1”. Really? It’s probably nothing, let’s just check whether that’s normal. After a quick consultation with ChatGPT and a crash course on premature membrane rupture, we realise—yes, her waters have definitely broken, and we’re among the 7–8% of parents who’ll have a premature baby. We awkwardly call the hospital. They tell us to come in immediately.

One slightly leaky bus journey later, and we’re at the maternity ward. No contractions yet, but the doctors tell us that they might start over the next few days. If they don’t come within the week, they’ll induce labour. They prepare a room, and ask how we want to do this, nudging us towards a caesarean.

With no bed for me, I head home to get some sleep. At 7am, the phone rings: she’s having the baby. With no buses running, I sprint to the hospital, take a wrong turn, and rather heroically scale a three-metre wall to avoid a detour.

Bursting through the hospital wards, smelling distinctly of sweat, I find my wife there, in all green and a mesh hat, looking like a nervous child. We’re allowed to exchange an awkward “good luck” with everyone else watching. Hospital regulations probably allow a kiss, but Anglo-Chinese cultural norms most definitely don’t, so a pat on the shoulder, and she’s taken into theatre.

For the next twenty minutes, I assume they’re preparing for surgery, but it happens more quickly than I expected. By the time they bring me in, they’ve already done the hard work, and all that is left is to pull the baby dramatically from the gaping wound. I make a stupid joke. My wife can’t laugh because she’s anaesthetised from the neck down. She looks scared.

Then, after under a minute of something happening behind a large screen. “Un petit garcon!”

I’m fully prepared to make a joke about my wife’s family finally having an heir and negating a state-enforced generation of shame2, but as she can’t laugh, and I don’t feel like explaining it to the medical staff, I decide against it.

My wife is taken into another room, and I can hold this weird little pre-human to my chest for half an hour or so, wondering what I’m supposed to be feeling—I’m mainly just worried about my wife. He’s then moved into a heated incubator and taken to the neonatology ward. There, he’s placed in a bed with oxygen, heating, and various monitors attached. The staff are reassuring, everything seems to be relatively stable.

The Beginning of Experience

Even when things are going well, it’s hard being a newborn. The helpless indignity of waking from a comfortable semi-conscious slumber and being brought into a sterile hospital filled with white light, weird sounds and cold air. The inability to express any noticeable emotion but distress, confusion and fear. This is what he looked like:

We move in with him after the first night. When he’s calm, he just looks adorably confused, as above. His heart rate and oxygen monitors are a constant reassurance. But when distressed, panic mode begins. His whimpers escalate into something more animal and frantic, we hear the beep and see the erratic signals, and the medical staff rush in, fit an oxygen mask and inject him with caffeine citrate. They tell us that the main risk is hypoxia (dangerously low oxygen levels). If this isn’t fixed, it can impede brain development or damage vital organs. In unstable infants, this can escalate rapidly, sometimes triggering cardiac or respiratory failure.

There’s something so visceral about the idea that your own distress, triggered by primordial fear or hunger, can cause brain damage, or even kill you.

Despite looking relatively calm when sleeping or being held, there’s never any sign of happiness or recognition. His life for the first week is a cycle of fragile stability made tragic by omnipresent tubes, and moments of genuinely life-threatening anguish.

This Is Everything

As with most people involved with EA and animal advocacy, I’m familiar with, and sympathetic to, the case for prioritising wild animal suffering.

I keep thinking of this while watching my son’s first days—watching him trying and failing to feed, unable to coordinate his tongue to swallow, relying on an obviously uncomfortable nasal tube because it’s the only way he can take in nutrition.

Most vertebrates3 are larval fish. 99%+ of wild fish larvae die within days. For these animals, being eaten by predators (about 75%, on average) is invariably the best outcome, because dying of starvation, temperature changes, or physiological failure (the other 25%) seems a lot worse.

When they do experiments starving baby fish to death (your reminder that ethics review boards have a very peculiar definition of ethics), they find that most sardines born in a single spawning don't even start exogenous feeding, and survive for a few days from existing energy reserves. This time is spent in a state of constant hunger stress, driven by an extremely high metabolism and increasing cortisol levels. For the vast majority who cannot secure food, their few hours-days of existence probably look a lot more like a desperate struggle until they gradually weaken and lose energy before dying.

They’re born too small. Like the premature human baby sat before me unable to suckle, most don't have the suction force to consume plankton.

Seeing my son in the ICU, hearing the babies in the adjacent rooms who have it worse, these numbers coalesce into something real. I realise that almost every animal currently, or ever, alive is a newborn. Underweight, fragile, and overwhelmed from the start. And for most of them, this is all there is. A flicker of life enters the world, flails around for oxygen or food, cries out for help if it can. And then vanishes.

There are an estimated 10²⁰ fish born in that situation every year.

I’m not usually an emotional person, but there’s a moment where I’m watching this purple life-form gasping for air and realise: Fuck, this is effectively everything.

It’s the most intensely scope-sensitive emotion that I’d ever experienced.

Schrödinger’s baby

In the hospital, I’m constantly thinking about the question of whether my baby is actually experiencing anything at all. I can’t shake the sense that he exists in a kind of superposition—somewhere between consciousness and blank, silent unawareness.

I’ve never liked the cop-out of “proto-consciousness” or “an early, different form of consciousness.” Either there’s something it’s like to be a tiny baby, or there isn’t.

There’s evidence pointing in both directions. Intuitively, the younger the baby, the more their distress feels “algorithmic”. His cries are just mechanical outputs signalling hunger, discomfort, cold, or a need for contact. The shifts between crying and calm are sudden, as if disconnected from any inner continuity.

I try to recall what I’ve learned about different theories of consciousness. Babies don’t have the long-range thalamocortical connections necessary for integrating information across the brain, they don’t have the connectivity patterns associated with global workspace theory, such as broadcasting information across brain regions, and their default mode network, allowing them to self-reflect, doesn’t really activate until a few years in.

This might support the idea that consciousness hasn’t yet “switched on.” Even if pain pathways are firing and the baby is screaming his lungs out, that doesn’t necessarily mean there’s something it’s like for him to be in pain. Genuine suffering might only emerge after some kind of complex integration process, and that system may simply not be online yet.

In the first week, the baby’s behaviour is so alien, and his cognitive processes so simple, that it wouldn’t have surprised me to learn there was no conscious experience at all. I’d been effectively trying to bond with a sorting algorithm.

But then again, plenty of theories point the other way. Recurrent Processing Theory says that simple feedback loops in the brain might be enough for basic consciousness, and that this could appear around birth, or even before. Integrated Information Theory says consciousness arises wherever there’s enough “integrated complexity”, and sets the threshold surprisingly low. And then there’s weirder theories like panpsychism, which says consciousness is fundamental and built into matter itself, present in some form at every level.

In order to escape the “Schrödinger’s baby” dilemma, where infants are superposed between two extreme states: fully conscious or not at all, the usual solution is to find a middle ground. There’s a plausible-sounding argument that moral weight scales with complexity, and that suffering becomes more intense or meaningful as cognitive systems become more elaborate. Suffering and consciousness have a kind of mass: more neurons = more experience = more moral weight.

But this view feels both unjustified and suspiciously convenient. I can vaguely imagine a being experiencing a tiny, focused locus of extreme pain, just as I can imagine a vast, diffuse field of suffering. I’m genuinely not sure I know which is worse. And you can sense yourself and others wanting to feel this way. Believing that conscious experience and suffering scale neatly with complexity lets you justify leaving your child to “cry it out,” without feeling like you’re humiliating yourself by singing to a pot plant.

It would be easiest, albeit morally dodgy in other ways, to just “round down”, and assume that infants aren’t conscious and therefore can’t suffer. That would let us act more freely, ignore the distress, feel less manipulated, and save money on anaesthetic. But I’m not going to “do an Eliezer” and say, “my poorly thought-out model doesn’t assign moral weight to infants, so I’ll just ignore them.”

Instead, I’m left with a kind of bipolar moral uncertainty. Either my baby is suffering, and urgently needs my constant attention, or he’s a soft little automaton, an emotional shoggoth designed by evolution to extract resources through mimicry of pain and weaponised cuteness, with nothing truly going on inside.

On Feeling the Right Things

Premature babies can struggle for much longer. It’s months of agony and waiting for some parents, but we’re lucky. In the second week, he’s improving slowly. The tubes are gone, he’s starting to feed on his own. After a frustrating fortnight in the hospital, he becomes stable, and, by the third week, is finally allowed to leave.

In the hospital, the neonatology staff spoke to him in musical, lilting tones, calling him silly affectionate names like mon petit puce, mon petit chou, speaking to him as if he understands. You can see why this gentle compassion is probably optimal. Actually empathising with his distress would mean a plunge towards a baby’s level of bipolar psychosis. I still find it weird finding the words to speak to my adorable baby in a way that covers the possibility of existential horror and emptiness.

Caring about animals, you learn to live with that uncertainty: hoping consciousness is rare, while mostly acting as if it isn’t. Expected value calculations offer a kind of solace when you’re distributing limited resources across many beings with varying chances of consciousness. If newborn fishes are only 25% likely to be sentient and cows 85%, then you can make a rational trade-off or allocation of resources.

But that reasoning collapses when applied to a single, emotionally salient case like your own child. When the possibilities are stark, either he’s conscious or he isn’t, expected value becomes a kind of moral dodge. You can't act as if he's 50% there. Instead, you’re in a weird emotional limbo between both perspectives: sometimes suppressing empathy, telling yourself he’s just reflexes and noise, sometimes embracing him as fully sentient, trying to cultivate the love and compassion that might one day have a more certain object.

Into The Fifth Trimester

Things have been going well!

My son has grown into a more robust, rapidly-growing, and occasionally smiley little human. By the third month, he’s started smiling, and I can make him laugh. At five months, I’ve realised that, beyond the predictable baby chuckles at stupid faces and voices, he’s also learning to embody his delightfully toxic masculine side—breaking into seemingly genuine laughter at slightly violent slapstick hand-puppet theatre. His experience now seems to span a more recognisably human, and amusingly male, emotional range.

When he cries now, he really cries. There’s a beautiful Chinese phrase for it: 哭得撕心裂肺 (kū de sī xīn liè fèi)—to cry as if tearing apart one’s heart and splitting open one’s lungs.

His distress now feels more “real” and less algorithmic. It’s louder, sometimes more infrequent, and not always easily explained. Something feels particularly alive about that curious pre-sleep anguish, like a faint awareness that sleep brings an interruption of consistent selfhood—a mini death even—before a rebirth as a slightly updated human in the morning.

His behaviour is starting to feel less like that of a well-programmed NPC, and more erratic, expressive and unreasoning, as if a hyperactive child was let loose with the code. I find it impossible to see him as anything other than conscious.

The Most Beautiful Case For Net-Negativity

For anyone thinking of having a baby, let me assure you that they can bring so much joy into this world.

There’s a deep pleasure in his existence. An all-encompassing loveability that makes it impossible not to smile whenever in his company, even when he’s crying his little eyes out. He’s the first grandchild on all sides, a beacon of family continuity, bringing pride, joy and meaning to the grandparents. On the Chinese side, my in-laws’ doubts about my unconventional values and lifestyle have been quietly eroded by a healthy grandson and his conspicuous absence of an Asian epicanthic fold.

But the truth is, life hasn’t been kind to the little fellow so far. For the first five months of his existence, he’s been an adorable locus of suffering. If the intensity of a baby’s anguish is within an order-of-magnitude of an adult, then my son has likely endured more extreme negative experiences in the past two weeks than I have in the last two decades. He now has moments of obvious happiness, but also even more obvious, prolonged moments of distress. If I had to guess, I’d say his life is probably net-negative, but when he’s giggling with a mouthful of spittle, it’s laughably easy to pretend otherwise.

Of course I want him to be happy and free from suffering. But when we have to rely on subjective perceptions to determine whether we’ve achieved this, the desire for it to be true often causes us to deceive ourselves. But there are some moments, particularly looking back at that skinny, struggling, anguished thing in the weeks after birth, that I start to see the situation more clearly.

I’m not depressed or negative about this. Even in the worst case, I’m still an optimist. I believe that humanity can find a way to solve AI alignment, develop smarter and more ethical world systems, build the post-scarcity utopia we dream of, and strive for a universe free from suffering. In expectation, I think that my son’s life will be happier, more meaningful, and more beautiful than we could have dreamed of a generation ago.

But my main takeaway from having a child is probably a serious acceptance that, under most plausible theories of consciousness, the world is, and has always been, net-negative. And finally accepting everything that this implies, less based on this one life, more on what his first weeks reveal about the trillions of others lives born this year.

So here’s a hopeful message for prospective parents: your child will open your eyes to the true scale of suffering in the world, while looking so unbearably adorable that you’ll find yourself grinning inanely through the whole process.

People in this sphere might have particular personal, ideological, or geopolitical aims here, therefore have a strong reason to promote fertility in their chosen country or region. While this may all be perfectly good and noble, if you’re concerned about the global trend of fertility decline, as many people claim to be, it makes little sense to be thinking purely about your own region.

One common approach, as discussed by Robin Hanson and Zvi Moshowitz, is simply paying people to have kids. Most countries have made at least some effort to soften the blow of childbirth on peoples’ finances with parental leave, free childcare and small (occasionally large) tax breaks. And it works! Studies consistently find that even a small cash bonus increases the birth rate at the margins! But, because of the substantial costs of competitive parenting, foregone income, and potential career setbacks, these policies never go anywhere close to actually make having kids seem affordable, let alone a sensible financial decision.

The response that Hanson and Zvi take is that, if “money for kids” doesn’t seem to be working at scale, we’re probably just not paying people enough. But at the global scale especially, it makes far more sense that we’re not paying the right people!

For people concerned about global impacts or the long term stagnation caused by fertility decline, is it optimal paying people in rich countries to keep the birth rate from plummetting? The obvious problem is that paying someone in a rich country to do (relatively) unskilled labour—childbirth and child-rearing—is a glaringly inefficient use of both money and labour. You need a very strong bias towards this rich country to want to do this!

So assuming no strong bias towards a given country, let’s just ask the question: if you want to incentivise childbirth in any country in the world to maximise the outcomes you want, which country would you choose?

Perhaps a poorer country?

Paying people to have kids clearly works, if you pay them enough. But people in different countries value money differently. Without getting into the economics of global inequality, we can simplify by saying people tend to value money roughly in proportion to their per capita GDP. It’s almost a tautology to say that people are, on average, willing to substitute a year’s worth of labour for their countries’ average annual salary.

Imagine a global policy where, for 24 year olds, if you have a kid before you’re 30 we’ll give you 2x your countries’ per capita GDP for each child. In terms of incentives, this would work incredibly well! Someone in Luxembourg would revel at the idea of getting $250k for having an extra child, but someone in Burundi might be equally thrilled to get $1600 for each child!

If you found a way of only targeting people who would not have an additional child absent this incentive, this might look like subsidising 100,000 extra births in countries like Burundi for $160 million. If your main concern is just averting a global demographic collapse, this looks incredibly cheap!

But the counter arguments are fairly obvious here, and optimising for pure numbers doesn’t seem like the best strategy. As much as you might think that pro-natalist cultures are a good thing, the current fertility level in Burundi allows very little parental and material investment per child. According to conventional development economics, we’d prefer a reduction in fertility in very poor countries, to allow for more investment by parents and a better ratio of working people to dependents (see the Demographic Dividend). Also, sadly, the pay-off doesn’t look as appealing. A child in Burundi is far less likely to contribute great things to the world than someone in a richer country. They’re also less likely to have a safe, enjoyable and comfortable life, and more likely to die young.

But there’s probably a good middle ground between Luxembourg and Burundi. So this means that we have a trade-off on our hands. How can we maximise value for money?

The best bang for your buck (probably a pun in there)

You want as many people as possible to have as good a life as possible for as low a price as possible. So let’s draw up a mini-model to calculate this. I see four main variables to suggest whether a country is a good option to nudge towards higher birth rates:

Wealth: All else equal, the cheaper the better. Let’s just use GDP per capita as a simple metric.

Education Level: We want people to be smart and make sensible life choices. Let’s just stick in years of effective schooling.

Subjective quality of life: There’s always a bit of bias in these kinds of questions, but I think they do capture something about how nice it is to live in a country; let’s go for life satisfaction.

Likelihood of having more offspring: You want your investment to pay off in more children down the line. Getting a Hong-Konger to have kids is unfortunately a Darwinian dead-end, while getting a Togolese to have kids might get you a pay-off of 5 or 6 extra kids within 30 years. Let’s try Total Fertility Rate.

So we’re looking for a country that’s poor and with an already high birth rate, but disproportionately educated and happy.

I put this into a toy model here. To radically simplify, I put all the countries into deciles according to these variables, scored them from 1-10, then added together.

The results are in! Based on these factors, we should be paying people in Central Asia and Israel to have more kids!

With the additive model (adding together the four categories), Slovenia, New Zealand and Benin also make it into the top 10. If we turn the model multiplicative, rewarding moderate scores on all categories, South America and Southeast Asia start looking better, because they score moderately well on all metrics, but Central Asia always looks impressive.

But according to both models, with only $3,010 GDP per capita (nominal), and a relatively happy, educated and fertile populace, your best choice for increasing human fertility is paying Uzbek women to have more children!

Who should we not be paying to have kids? Poor old Botswana and Eswatini look bad for their relatively high GDP, but low reported quality of life (adding sky-high HIV rates would also be a warning sign these countries). China, Hong Kong and Ukraine suffer from low fertility, and not being particularly happy. Middle-East and North African countries also don’t look great.

I personally like the idea of boosting underpopulated countries like Mongolia, but you might also want to think about future population trends, ideological alignment or producing top 0.1% talent. But overall, I think this model works pretty well.

What might this look like in practice?

Now that we’ve made a fake quantitative model proven beyond doubt who we want to pay for having children, the next challenge is tackling the how. How can we throw our dollars at fecund Central Asian women in a way that actually increases birth rates, rather than simply rewarding births that would have happened anyway?

It will surely be incredibly easy to find someone who will be willing to say that they’re having a child, and getting them to accept your money. The difficulty would be ensuring that your money actually produces kids compared to the counterfactual.

I can think of a few ways of doing this:

The Auction Model

Couples interested in having a baby could set up profiles on a platform, describing their situation and financial constraints. Donors (or philanthropists) could then bid on which couples they want to support. Instead of a single upfront payment, support could be distributed gradually (e.g. spread across the first five years of the child's life) making sure families get ongoing assistance, births are verified, and potential fraud is kept in check.

This has the advantage of being market based and efficient. It targets the “marginal parent”, who would decide to have a kid with only a "financial nudge”. The downside; the risk of fraud, or worse (e.g. forcing women to have children) would make this a bit scary to implement.

The Universal Basic Income (UBI) Model

A philanthropist or government could give a nominal payment to every new parent in a target country. Let’s say Mongolia. This would function like a baby bonus, automatically applied across the board.

You could get in touch with the government of Mongolia, tell them that every family in Mongolia can receive payments for each new child, and let them take care of the logistics.

This has the advantage of being very simple and scalable, with no risk of selection bias. The big inefficiency is the same as that of UBI: it risks a lower counterfactual impact than the alternative. Many parents would receive money for births that would have happened anyway.

The Targeted NGO Model

A specialised nonprofit organization, such as a pronatalist family planning clinic, could work on the ground to identify financial barriers to parenthood. The NGO would find cases where financial constraints actually delay or prevent births, then find a payment structure that works.

The big advantage here would be that you could use the NGO staff to identify families who were looking to have kids for the “right reasons”, and are constrained just by finances. The problems are mainly the standard problems with NGO work: a lack of transparency, corruption, mission creep, and general inefficiency.

Do I think this is a good idea?

I’m not a hardcore pronatalist. I believe at least somewhat in the meat-eater problem and there are clearly some negative impacts from overpopulation on issues like biodiversity and land-use. I also have relatively short AI timelines, and think that this clearly overwhelms population concerns.

But I also spend a bit of time worrying about aging populations, demographic collapse, and the risk of stagnation over the next few decades.

How do I feel about the countries that come out of the model? Uzbekistan is one of the least free countries in the world, and there are concerns associated with terrorism and radical Islam. But not all of Central Asia has these issues—I visited Kyrgyzstan (2nd or 3rd in the model) last summer, and it’s a lovely country. Despite some creeping authoritarian concerns, people seem mostly happy, and nudging Kyrgyz birth rates up by one child per woman seems like it might be a nice thing for the world.

And it could be very cheap.

I won’t get into the exact economics here, but it’s possible that even tiny bonuses can work. An additional bonus of $1000 (about 2% of GDP) was enough to get Australian parents to increase their fertility!

If families in Central Asian countries would be willing to have extra children for a similarly small nudge—say a month’s average wage, the cost could be as low as a few hundred dollars per child!

This is far lower than the cost of saving a child’s life from disease in similar regions, so I think it’s possible that this could be an ethical, even an effectively altruistic thing to do. Animal welfare impacts, the main negative of high birth rates, would also hit less hard in these regions. Central Asians consume considerably fewer of the animals that suffer more from factory farming (aquatic animals and chickens) and Central Asian fat-bottomed sheep seem very happy.

In the longer term (boldly assuming no AI-driven catastrophe or leap towards utopia) transferring money from rich countries to more child-friendly countries to boost fertility could be one of the only levers we have for stabilising the human population.

I initially started writing this as a thought experiment, but I’m starting to genuinely think it might be worth getting the mechanisms in place to potentially achieve this.

And if any of my 13 subscribers happens to be a Central Asian president, don't hesitate to reach out!

If you have a passing interest in demographic trends or China, you've probably encountered this strangely persistent claim. I’ve heard it repeated without much critical examination, often as part of a broadside against the CCP or its policies, or as a critique of coercive birth control in general. There are some quantitative articles that make this argument, and it often finds its way into more legitimate sources (sorry, Our World In Data).

It’s one of those claims that always struck me as obviously wrong, but I only recently looked in enough detail to work out what was going on. Is it true? What really happened during those fateful early years of the one child policy?

Let’s dig in.

Why A Debate?

No-one wants to admit when the Communists are right.

With the requisite deference to their revolutionary forebears, the CCP have naturally wanted to defend their population control policies, claiming that the one-child-policy was necessary, desirable, and achieved its goals. The official line from national demographers is that 400 million births had been averted.

Eager to prove their free-world credentials, many intellectuals inside and outside of China decided to provide a counterpoint, claiming that as well as being unnecessary and inhumane, it was also ineffective. For example, a widely-cited study by Feng, Cai, and Gu (2013) tries to awkwardly thread the needle of “terrible and ineffective”, arguing that “the demographic effects of the one-child policy in reducing population growth, …can at best be very small, [but] the policy’s social effect in depriving a large proportion of Chinese families of a second child is enormous.”

Before diving deeper, take a look at the data for a second, noting how weird it is, and why this argument looks plausible. Starting from 1950, China’s birth rate sort of mirrored Taiwan’s overall trajectory, starting at around 6 births per woman. A classic demographic picture of a rural, agrarian society.

It converges again almost exactly in 1978 with around 2.7 births per woman (mid demographic transition) and again in the 2020s at the East Asian nadir of roughly 1 birth per woman. China’s birthrate trajectory is clearly more erratic, but, superficially, it looks like it follows a similar path to that of Taiwan and South Korea.

Let’s start by looking at China’s weird squiggle. What caused these blips?

The dip caused by the Great Famine from 1957-1960 is easy to explain—it’s hard to have babies when your main calorie sources are tree-bark and belt leather. Then the birth rate recovered in the throes of 1960s socialist enthusiasm, before taking a nose-dive in the 1970s. Finally, there was a weird uptick above its Asian neighbours in the 1980s, despite the implementation of the One Child Policy.

Here we get the paradox… why did birth rates collapse prior to the One Child Policy being implemented in 1979, then not decline during the 1980s?!

The skeptics claim that this must have been a sign that the one-child policy was ineffective. For example, Our World In Data, that China’s fertility rates naturally fell in the 1990s to levels comparable to its East Asian neighbors (emphasis mine):

China’s fertility would have continued to decline even without the One Child Policy, as the rapid fertility drop during the 1970s provides a strong indicator of the country’s natural trajectory. By 2010, fertility would have reached its observed level of around 1.5 children per woman, according to Bayesian models.

Feng, Cai and Gu mention that in similar countries, like South Korea, without a one-child policy “the birth rate also declined, and it declined below the level predicted for China”, so concludes that China’s birth rate would have declined at a similar pace even without the policy during the 1980s and 1990s.

That’s what the data appears to show. In retrospect, the line of China’s birth rate was barely affected by the one-child policy in the 1980s, and has aligned with its closest neighbours, so the policy, they claim, wasn’t actually that effective. It just happens to match East Asian patterns of lower birth rates driven by urbanisation, education, and reduced child mortality—all factors that tend to lower fertility rates. So presumably all these Chinese women only wanted to have a single child to coincide with these trends? Well yes, this makes sense…

Wait, no, it doesn’t! This is insane!

The claim is that stopping literally hundreds of millions of Chinese women from having a second child had almost no effect on fertility decline, because it matches a natural trajectory?! Please think about that for a few seconds before reading on.

I’m fairly well-connected, and know hundreds of Chinese friends, colleagues and acquaintances born between 1980 and 2000 from various provinces. Around 80% of them are only children. Estimates suggest that 50-60% of the population in that cohort are only children, compared to only 12.5% of the previous generation! Even Japanese and South Korean families tend towards two children! Are we really suggesting most families would have voluntarily chosen to have just one child?

Well, obviously not

The answer is simple, but quite interesting.

Fortunately, a statistician called Daniel Goodkind has written quite clearly on this topic. He explains that after China’s pronatalist push in the 1960s, the government pulled a complete 180 in the early 1970s, launching the “later, longer, fewer” “晚、稀、少” campaign, which encouraged later marriage, longer gaps between births, and fewer children overall.

At first glance, this seems like the kind of reasonable policy any country facing high birth rates might implement, with the objective of reducing teen pregnancies, and enabling women to have the number of children they actually want.

But in reality, this policy was really weird! The "Later, Longer, Fewer" campaign was more radical and coercive in driving down birth rates than it gets credit for. We can see the effect of this policy here, compared to some countries with a similar economic trajectory (many of which also had significant birth control policies!), showing how much of an outlier China was.

So what actually happened in the 1970s? There were two main trends that caused the birth rate to drop:

1. Marriage age. This is the big one. In the 1970s, the minimum marriage age mandated by the “later, longer, fewer” campaign was raised to 28 for urban men and 25 for rural men, with corresponding ages of 25 and 23 for women! This effectively prevented many families who would have otherwise had children in their late teens or early twenties from starting families. In a society where births outside of marriage were heavily stigmatized, delaying marriage meant delaying childbirth.

2. Campaigns and Social Pressure: When a campaign is launched in 1970s China, you’d better pay attention. The collectivised commune system meant that everyone was tied to a geographical location, where they were under constant scrutiny. This made it nearly impossible to circumvent various marriage and childbirth restrictions. People in villages around the country were being told not to have kids, and the coercive social environment—more extreme than that in the 1980s—meant that there was no private space to get away with either having a baby earlier or not delaying marriage.

So what happened in the 1980s? For my argument to hold—that the One-Child Policy significantly reduced the population—we have to imagine quite a strange counterfactual. We have to imagine that, if China had liberalised its family planning policy in 1980, China’s birth rate would have temporarily rebounded to around 4 births per woman before beginning its “natural decline” in the mid-1980s.

This feels unrealistic, and has never really happened anywhere else, which is why so many people are fooled by the alternative narrative that the policy didn’t work. But accepting this counterfactual is the only explanation that makes sense of that fact that so many people in this generation are singletons, and that the birth rate did not drop substantially. How did it happen?

Well, with the new marriage law, the marriage age dropped instantly to 22 for men and 20 for women, where it remains today. A backlog of couples who had been forced to wait now began having children. These were also China’s 1960s "baby boomers", which was a far bigger generation than those born in the 1950s, reaching childbearing age in the 1980s. These factors were balanced out by the first onslaught of the one-child policy. This is clear from the birth order data:

More people were having kids, but far fewer people were having 2nd or 3rd kids! The One Child Policy didn’t initially bring down birth rates because it was introduced alongside the pronatalist policy of allowing earlier marriage, and when China was welcoming a new wave of potential parents. It’s difficult to say whether this was intentional or not, but either way, we know that, had all birth controls been lifted in 1979, the birth rate would have rebounded, therefore the one-child policy was an effective continuation of birth control policies in the 1970s.

It wasn’t until the 1990s, with Jiang Zemin’s new fondness for authoritarianism, and the 1980s “baby bump” well-and-truly over, that the policy’s effects predictably kicked in, with fertility dropping from 2.5 kids per woman in 1990 to 1.5 by 1997. During this period, it looks like the population growth almost perfectly matched the limits allowed by law. Urban households (about 25%) mostly stuck to one child, Han rural households (70%) had one or two kids depending on the gender of the first child and local policies, and ethnic minorities (<5%) averaged over two.

From 1997 to 2017, enforcement became progressively less strict, and we saw a slight rebound to 1.8 kids per woman. As this rebound didn’t result from pro-natalist measures but from the softening of strict limits, this is pretty good evidence that many families wanted more kids throughout the 1990s but were being held back by the policies.

The Counterfactual

What would China be like today without the One-Child Policy?

Let’s just try a back of the envelope calculation for two alternative scenarios:

Scenario 1: Similar family planning policies, just without any “one-child” limitations i.e. most of China restricted to two children

Scenario 2: No family planning policies whatsoever from the 1970s to today

For scenario 1, Feng, Cai, and Gu estimate that 50 million women born between 1961 and 1970 completed their childbearing with only one child. If we assume a similar number for the 1971-1980 cohort, and half of the 1981-1990 cohort (because the policy was lifted half way through their childbearing), this suggests that 125 million families had only one child directly due to the one-child policy.

In the counterfactual, there would have been some only children, but not many, so if we (conservatively) assume that 80% would have had a second child absent this policy, there would be 100 million extra Chinese people aged 15-45 today.

This seems a reasonable estimate of the births directly prevented by the one-child policy, but there would also be second-order effects. The cohort born from 1981-1990 were both the result of, and subject to the one-child policy. Had the 40 million “missing children” from the 1981-1990 cohort been born, they would have had children of their own at a rate of around 1.5 per woman. This would add roughly 30 million to the "missing population" born in the 2010s, bringing the total to 130 million.

I’d add a bit extra due to the normalisation of single-child families in the 2010s and 2020s, and the male-bias, leading to fewer males in later generations, so 150 million seems reasonable. This is a lot higher than the (frankly silly) estimates of some professors, but let’s remember, this is just one small part of China’s population policies from 1970 to today.

For a scenario with a counterfactual of no limitations, we need to go a lot higher.

I haven’t spend the time to carefully model this, but my best guess would look something like this:

1. As above, we can assume 150 million averted births directly from the one-child policy itself.

2. We can estimate that delayed marriage and spaced-out births in the 1970s resulted in approximately 80 million missing births. (Total: 230 million)

3. Since the generation born in the 1970s was smaller than the counterfactual scenario, there would have been knock-on effects on the next generation. Assuming 1.5 generations of averted or delayed births stemming from this period, we add another 120 million. (Total: 350 million)

4. We can also assume a similar number of "missing" third or fourth children from rural two-child families in the 1980s and 1990s—who likely would have had more children (assuming birthrates in comparable countries of around 3-4 births per woman) without an effective two-child policy in rural areas, and their averted offspring born in the last decade. Let’s add another 100 million. (Total: 450 million)

This is my best guess. China’s population would be around 1.9 billion today, in the absence of population control policies. Feel free to make your own estimates, but I think that anything below 200 million is clearly indefensible, and requires absurd assumptions.

But there are other methods to support my estimate here. You could also compare China to a more closely matched country. East Asia isn’t the best fit. Today, urban China might look a bit like South Korea or Taiwan, but in 1990, it was a different story. By 1990, these richer countries had urbanised, and were now only 25–30% rural ( and as always, the rural population in South had much higher birth rates). In 1990, China was 74% rural. China was also far poorer—its nominal GDP in 1990 was around $1,000 per capita, compared to over $6,000 in South Korea when it had its demographic transition. Comparing China to other East Asian countries without accounting for these massive economic and rural-urban differences is clearly silly.

Vietnam might be a better reference point. It shares cultural similarities with China, liberalised around the same time, and had a similar % of rural population throughout the late 20th century, along with a comparable GDP. Goodkind estimates that, without the One-Child Policy, China’s birth trajectory would have resembled Vietnam’s. He guesses an additional 300–500 million births between 1970 and 2020, which matches my “back-of-the-envelope” calculation.

What does this mean for today?

Here we are in 2025, and China is superficially starting to look more like Korea or Taiwan—especially the urban China that is most commonly shared with the outside world, making that fallacious argument look annoyingly realistic. China’s birth rate has dropped to 1 child per woman—somewhere between Japan and South Korea, but before the country has become as rich as her East Asian neighbours.

As well as the premature population crash, there are two other reasons why almost 50 years of population control policies have continued to shape China’s demographics, and might make the next decade even weirder:

First, the gender imbalance. China saw widespread sex-selective abortion and infanticide throughout the 1980s, which escalated in the 1990s. By 2019, men outnumbered women on the mainland by 31.6 million, and this is heavily concentrated in the generations now of childbearing age. This crazy Wikipedia page shows that some provinces like Jiangxi had 143 boys born from 2001-2005 per every 100 girls! I couldn’t find whether anyone had calculated the impacts of this, but this has to be having a wild effect on birth rates as I write this in the 2020s.

Second, more difficult to pin down, is the cultural impact of a singleton culture. I’m married to a Chinese woman, and we’ve just had our first kid (relatively late). I’m starting to think about the norm from my childhood that I should avoid having an only child. We only had one only-child in our primary school class and she was a bit of an outcast. By contrast, my wife only knew one or two people with siblings when she was growing up, and understandably thinks that having a single child is more normal—it’s just the standard family structure.

The Chinese government is trying to renormalise having a second kid, but I sense that this policy is coming a decade too late. The societal expectations formed during childhood don’t simply shift because of a few murals or multi-child families on state TV. Most Chinese parents aren’t eager to be the outliers with three or four kids, especially when pouring resources into a single child has become the standard. Unless something very drastic changes, I can’t imagine any change happening here.

Opinions

I’ve been agnostic about whether I think China’s population control policies have been a “good thing” or not, which was intentional. Whether it is “good” is tangential to whether it “worked”.

The whole debate around China’s population has been awkward for many intellectuals since the 1980s. Many Western liberal opinion leaders have been through a relatively antinatalist phase since the moral panic around population growth in the 1960s—I know I grew up with a morbid fear of overpopulation, and we can see how few pronatalists there were 10 years ago in this US survey.

But I’ve since read Bryan Caplan, listened to the Collinses’ pronatalist tirades, and, despite the obvious harms of a growing population, have bought into this pronatalist vibe shift. Now with every blogger and his neutered dog writing about the baby bust, and a better understanding of the economics of population growth, fewer people than ever will be shaking their head sagely and bemoaning that China’s policies were a necessary evil.

China’s recent failure to reverse the trends of population decline has coincided with this vibe shift, and, both in China and abroad, the harsh reality of China’s population decline is leading to growing condemnation of 50 years of coercive birth control.

I think the time has come to recognise that China’s population policies were inhumane, coercive, and, even worse, they were effective.

You reflect on your limited vocabulary of Chinese characters, hmm, “excellent birth”, interesting!

You realise that the term is referring to China’s notorious family planning policies: 少生优生，幸福一生 shaosheng yousheng, xingfu yisheng. You translate, as was necessary in the benighted pre-LLM epoch, with the aid of an electronic dictionary and your own sense of logic. “Have fewer and excellent children, and be happy your whole life!” So true!

You smile. It makes sense, Chinese people aren’t allowed to have many children, and they’re pretty concerned with having great children! While modern Western parents hope their children become happy, healthy, and vote slavishly for the same political party as them, Chinese parents want their children to become brilliant. They want them to attend the best universities, get the most prestigious jobs, and thus eclipse 150 generations of shame and mediocrity, finally delivering respect and pride retroactively through their long-suffering ancestors going back to the Yellow Emperor. This makes sense. But then you notice something…

This compound word, 优生 yousheng, “excellent birth” has an alternative translation… That fatal term: “eugenics”.

You look around in a panicked state. Eugenics?! What is this country? Surely this can’t be right. Like Jonathan Harker gradually recognising that his host’s inability to eat, drink, and cast a reflection is more than just a peculiarity of Transylvanian culture, you start seeing everything in a new light.

When that elderly man in the Shaanxi village yesterday was suggesting that you marry a nice Chinese girl, was he really thinking of your welfare, or was this just a weaseling attempt to use your germplasm to improve the vitality of the Chinese population? What if these compliments about your height, high nose-bridge, light skin and dreamy blue eyes were not innocent aesthetic appreciation, but were in fact signals that they wanted to harvest your Aryan organs for the rejuvenation of the nation?

But as the days and weeks pass, and you slowly come to accept that your germplasm remains untouched, your horror gives way to a state of constant vigilance… and a question lingers in your mind…

What’s going on with eugenics in China?

In this golden age, before Chinese trolls had started posting badly translated nationalistic takes on Twitter, there was no way to quickly find out what Chinese people thought about stuff. So you have to work it out for yourself.

You recognise first that the obsession with excellent children is inescapable. Parents across the country are willing to sacrifice their time, money, and any chance of a healthy relationship with their children just so they can have a shot at attending the best Universities: Beijing University, Qinghua, or even Oxford or MIT.

But you also notice the inverse. The bemoaning of the problems of the Chinese population. The pollution, public order, the notorious videos of people ignoring dying children, all attributed to the “quality of the people” or renmin suzhi人民素质. Could this be connected to the fascination with excellent birth?

It also becomes clear that this wasn’t just about education and nutrition. While spending a few weeks in a poor Miao village in Guizhou, a village official, over copious amounts of rice wine, explains why his village remains poor—blaming it on low-quality genes due to generations of cross-cousin marriage. His gaze shifts to you as realization dawns: they should be marrying outsiders, perhaps even foreigners. You watch as the idea of a village-level eugenics program flickers to life in those charmingly devious eyes.

But it’s not just these slightly odd village intellectuals who like the idea.

Your Chinese improves, and you start seeing the term yousheng everywhere; books for expectant mothers, government articles, even university lectures.

This wasn’t just the Chinese equivalents of human-biodiversity weirdos. It was normal people with perfectly accepted views, not associating it with the same baggage.

The evidence for this different perspective is fairly obvious and overwhelming. Baidu Baike’s first line is: "Eugenics is the most important issue concerning marriages and families, it is a science that uses the principles of genetics to guarantee the normal survival ability of the next generation." They mention the “dark history” later down the page, but rather than condemning eugenics wholesale, they divide the eugenics theory into “non-scientific”, “semi-scientific” and “scientific” periods.

Or look at the "Chinese Healthy Birth Science Association" (w/ google translate 中国优生科学协会 is "Chinese Eugenics Science Association") a website with articles on a range of topics, from genetics to child nutrition.

But most of the content you’ll find around these sites, books, articles or conversations are standard wisdom: encouraging women to have their child in their mid-twenties, using scans to determine whether children have hereditary diseases, spacing out births, avoiding inbreeding and providing better nutrition to pregnant women are all just seen as both part of the understandable idea of having healthier and more successful offspring.

Eugenics with Chinese Characteristics?

This concept has simply never really been taboo in China.

Like most terms associated with western science, yousheng entered Chinese through the Japanese term yūsei. Chinese thinkers started their love affair with “semi-scientific” eugenics in the 1880s, when Galton and other early thinkers in the movement were getting their spicy takes translated around the world.

Pan Guangdan, China’s foremost eugenicist thinker, supercharged this fascination. Concerned with the “improvement of the Chinese race”, he rejected the “racial hygiene” concept, choosing to believe instead that all races could be improved, optimistically hoping that China’s “sick man of East Asia” diagnosis was not terminal.

His endeavours have since been framed as an anti-imperial (read anti-European) force to improve the quality of the Chinese people, and stop them being bullied by the white man. This is actually a common theme in Chinese essays of the period—using the power of genes to help the yellow man find his strength!

The Second World War was a turning point where China might have aligned more closely with the West. Our shared enemies, Nazi Germany and Imperial Japan, had eugenics laws that mandated sterilisation for people with hereditary diseases, discouraged marriage among those with mental or physical disabilities, and promoted "racial hygiene" to strengthen the national stock. Both believed in the biological basis of their racial superiority. But perhaps because eugenic theories were less central to Japanese ideology, or because East Asia was less involved in the intellectual soul-searching that followed, there was far less backlash to these ideas in the Far East.

This meant that, after the Japanese retreat, Pan Guangdan and his sociologist colleagues were still keen on adding eugenics language into the nationalists' population policy. Planned in 1945, this combined the benign—expanding public health and education—with the eyebrow-raising—concerns about the spread of "defective" genetic traits and the use of interracial and interethnic marriages to strengthen military recruitment.

Although something similar was enacted in Taiwan, there wasn’t a formal eugenics program in the mainland between 1949 and the 1970s. The first cohort of Communist leaders was more into promoting military and manufacturing power than interfering with reproductive practices. While the term yousheng was still used, suggesting there was not a taboo, it seemed to go out of fashion.

In the West, it was in this period after the Second World War that anything linked too closely to eugenics started to become unfashionable. The first wave of the reductio ad Hitlerum fallacy led to the rejection of arguments made by well-intentioned eugenicists, simply because the era’s snowflake intellectuals were worried about theories that had been among the ideological foundations of mass genocide.

I found it difficult to piece together what was happening in China in the 1970s—this was the period that, contrary to popular belief, saw the most significant population control policies. But when the one-child policy was introduced in 1979 and the country became liberalised, an odd dynamic emerged. A state-driven push for a stronger population to compete with China’s geopolitical rivals met with a bottom-up, frantic drive for children to excel and rise above their parents’ enforced mediocrity.

In this period, yousheng was revived in China under a new guise. Under the 1995 Maternal and Infant Health Law (a law from which the term yousheng was removed due to western opposition), China stopped people carrying heritable mental or physical disorders from marrying, and promoted mass prenatal ultrasound testing for birth defects.

Since then, China’s policies have been a mixed bag. Projects like BGI-Shenzhen (formerly Beijing Genomics Institute), which researches the genetics of intelligence, health, and physical traits, along with Chinese companies scooping up huge amounts of global genetic data, suggest at least some consideration of a future shaped by genetic selection. Then there’s the infamous case of He Jiankui, the scientist who used CRISPR to create the first gene-edited babies. Stories like these make it seem like China’s eugenic proclivities are blending with its fierce ambition to turn the country into a sci-fi dystopia.

Post-reform China also became aware of the “Western gaze” and grew somewhat sensitive to potential judgments. Many translated reports and documents were rebranded—The Chinese Journal of Eugenics and Heredity became The Chinese Journal of Birth Health—presumably after a few European conference hosts spat out their tea over the CVs of their Chinese attendees. But you suspect that none of these Chinese academics or policymakers disagree with the ideas of eugenics; they just want to avoid the reputational baggage of the word.

So…China Bad?

One common reaction here will be: China’s coercive birth control = bad; eugenics = bad, so China’s yousheng eugenics = extra bad.

Let's start by defending this view: most people recognise that the Chinese population control policies since the early 1970s have included inhumane and harmful elements, regardless of whether you think population control is a noble goal (or might have been, had Malthusian theories continued to be correct).

If you’re a liberal, you likely believe we should have a fundamental freedom to reproduce with (almost) whomever we choose, and 55 years of strict control of reproductive freedoms since China embraced yousheng are more than enough evidence of China’s illiberal tendencies.

If you’re an egalitarian, the pursuit of “population improvement” could lead to increasing inequality, or a Gattaca-style two-tiered society; something that China often seems to be lurching towards.

And, if you’re a consequentialist, you probably think that the taboo against eugenics in the West seems to have had relatively good consequences. Western countries have remained relatively free from oppression, people can have babies with whoever they wish, and people are still, on average, getting smarter and happier.

On the other hand, China, where this taboo doesn’t exist, does not look like a utopia of free and equal childbirth and child-rearing. They’ve had 55 years of oppressive birth-control policies which the government has been unable to effectively backtrack on, enforced sterilisation, and most Chinese people have attitudes about “human quality” that most westerners find unpleasant. At the same time, Chinese kids have pretty shitty lives, most forced into endless exams for the sake of their parents’ reputations and a lofty goal of national rejuvenation, only to end up in pointless 996 jobs in the service sector.

Perhaps it’s this “eugenic” logic that has left China unable to restore population growth to a healthy or stable level, creating the very economic and social threats its leaders are desperate to avoid. Like other East Asian countries, China has fallen into the low birth rate trap, driven by low childbirth outside marriage, high urban density unsuitable for children, and a preference for “quality” over quantity. Decades of restrictive birth policies and a cultural emphasis on single-child optimization over broader population sustainability may have ultimately worked against the original goal of bettering the Chinese race, all while screwing up people’s lives in the process.

So we are the wise ones?

Despite these criticisms having some merit, I still find myself sympathising with the Chinese approach here. In Chinese media or on Baidu Baike, the concept of yousheng, only subtly differentiated from eugenics, comes across as more coherent and honest than the politicized, sanctimonious explanations from institutions such as the US National Human Genome Research Institute, which dismisses eugenics as a “scientifically inaccurate theory” and a dangerous belief, still tied to “structural racism” and continuing harm to “marginalized populations.”

Although it is sometimes mixed with non-genetic ideas, the Chinese concept of yousheng is ultimately just a more coherent grouping of “stuff that maximises genetic potential”, from genetic selection and parental nutrition to avoiding congenital defects, to selecting embryos for polygenic traits. When the role of state coercion is removed, this all aligns nicely with both modern genetics and sensible contemporary sensibilities .I also don’t think the Chinese government is significantly misaligned with the population, and in fact, the claims about eugenics from the NHGRI feel far more propagandistic than this poster:

Chinese people recognise that humans can be improved, whether through better maternal health, embryo selection, or reducing the prevalence of certain negative traits in the population. And I don’t think these beliefs should be controversial.

Almost all of the bad things associated with eugenics are only bad for reasons entirely unrelated to the actual science—a science which, if understood more broadly as the improvement of the human population through genetic means, has only become more rational and precise the more we’ve studied it. It’s probably good that racial purity and hygiene narratives have become taboo, but in labeling anything as eugenics and therefore bad, we’re at serious risk of throwing the (genetically improved) baby out with the bathwater.

Liberals should believe that forcing disabled people to get sterilised is bad because it’s bad to force people to get sterilised, not because the principle of reducing disability in the population is bad!

Egalitarians should believe that only wanting to improve the white/yellow/black race is bad not because the principle of improving a race is bad, but because the world would be better if all races were improved to similar degrees!

Consequentialists should recognize that the harms associated with early 20th-century eugenics were far more due to general Nazi psychopathy, horrible motivated reasoning, and geopolitical tensions than to the contemporary state of the science.

Eugenics is a textbook example of the “disrespectability cascade”, where an idea becomes so heavily stigmatised that mainstream thinkers avoid discussing it, leaving the conversation dominated by those with dodgy intentions, fringe beliefs, or extreme ideologies. Of course, in China there’s no shortage of this kind of person, but when you speak with educated Chinese people, they tend to have clearer and more rational views on topics related to human genetic improvement precisely because these taboos are absent.

The Final Take

In China, some of the most extreme controls over sex and reproduction coexisted with mainstream, normal discourse about eugenics. Sterilization for people with certain genetic defects existed alongside a widespread acceptance of prenatal scans to ensure healthy births. While the state enforced coercive birth limits through sterilisation and abortion, it also invested heavily in fertility treatments and embryo selection. While forced birth control policies targeted certain ethnic groups, China shot to the forefront of genetic research on intelligence and human enhancement.

China seems weird here, but if we’re just looking at the terminology, this is ultimately a case where we in the West are the weird ones. Like an ideological PTSD victim, we now uncritically associate a perfectly rational and laudable goal—making populations healthier and more productive—with early 20th-century Nazi horrors, the undeniably dubious ideas of racial hygiene, and a dodgy fringe of internet thinkers.

I’ve recently read more on the topic and realised how the eugenics movement has been unfairly tarnished. The 1939 Eugenics Manifesto shows us the largely benevolent and scientific views of the pre-war eugenics community, and it's…just fine and normal. It rejects racial determinism (while acknowledging the possibility of population-level effects), emphasises freedom of choice, particularly for women, and argues that much social inequality is environmental. Eugenicists in the British and American schools believed, as the Chinese do today, that universal improvements in living conditions are an essential step in bettering the population, but that we shouldn’t ignore the role that genetic improvement can play in achieving a better future.I find this to be true for most intellectuals in China as well—they believe that genetic selection can work, that humans must be nurtured to reach their genetic potential, and that these possibilities could lead to healthier, happier people in the decades to come. This worldview is both rational and moral, and the Chinese are unusual inheritors of this tradition.