This is THE most painful lesson I have learned since starting PauseAI US.

I used to think that courage was an ingredient that a lot of people were lacking, but I could give it to them by just taking the hits myself. So I thought there were a lot of people who just needed to be topped off a little to contribute to the solution.

Actually, courage is more like a process or orientation, and people who aren’t making their own courage are refraining from doing so for a reason. They will resent and attack any courage you display on their behalf.

Courage might threaten their belonging or their image or their bag. A huge dynamic that completely missed me most of my time in the EA/rationalist world was that courage would require a break in social conformity, and most of those people valued fitting in far more than the content of all their moral talk. The moral talk was a game, especially all the talk about agency and independence— a way of enacting their tribal membership. I was blindly stomping all over their social tissue by calling out that they didn’t really act with agency for the good. These people didn’t need a little top-off of courage from me to join my cause. They talked my talk about AI danger for a completely different reason, and they needed agreed upon excuses NOT to take courageous action because doing so would break the bonds of their community. Me offering courage was calling their bluff.

If they don’t want their own courage then they definitely won’t want yours. They might say they want courage. They might even ask you to do courageous things for them. But if you implicate them in actions they are unwilling to take for themselves, they will attack you like they autoimmune attack their own voice of conscience for threatening their belonging to the group.

So, such people are not one missing ingredient away from being part of the solution. They are constitutionally unable to be part of the solution for whatever reason they suppress their own courage. And they will become your enemies and seek revenge on you for exposing them wherever they feel they can get away with it (they are, after all, cowards).

You can inspire genuine courage in others by example, but you can never have courage for them. Courage has to come from the heart and it has to serve the person’s overall intent, or else it will be rejected.

It may surprise you to know, reader, that I think it’s true that the genie is out of the bottle. Thanks to the recklessness of the AI industry, we will be fighting the genie for the rest of humanity’s existence.

We can’t undo the knowledge of transformers and that machine learning scales, as far as we know, indefinitely. Just as we have to monitor and enforce against the proliferation of nuclear weapons now that we know how to make them, we will have to fight to suppress the destructive power of the knowledge of making AI forever.

We still have the chance to shape that fight. We can fight the genie with an international treaty to pause and determine how we proceed from there through diplomacy.

Or we can put ourselves at further and further disadvantage by making AI stronger without being able to control it.

Or we could just lose the fight by becoming subjugated or going extinct. It happens all the time to other species.

This dream that, because the genie is so strong, we must simply wish that it will be a benevolent genie, reflects the way people want to behave toward powerful bullies, like the AI labs. They want to believe they have gained favor with the bully, that they can be part of his spoils. They DO NOT want to believe the bully is an enemy, because that would mean they have to oppose him.

Well, I hate to be burdened with telling you this, but all we’ve done so far with this appeasement strategy is create powerful rivals for control of earth: the AI oligarchs and the AI itself. We will never appease the genie and, if the stories tell us anything, we will not outsmart him at his own game. There are no shortcuts when dealing with wannabe usurpers.

If we get serious about pausing AI, we will win. We will be able to beat the genie back with vigilant enforcement, just as we deal with the other ills that plague humanity like disease and war.

There is no easy way out. We will be dealing with threat of AI forever— our only choice is how.

Here’s part 1.

tl;dr: Even though I’ve always seen the callouts I do on my personal twitter as pretty extracurricular to PauseAI US’s main programming— which is now connecting constituents to their representatives to talk about their concerns about AI danger— they punch above their weight and they do represent “the tip of the spear” of our Overton window shifting mission.

Well, I’m proud of my callouts and I’m tired of entertaining doubts from enemies and allies alike who can’t handle them. I’ve taken seriously that I might be going about them wrong, and they have never been my priority, but at the end of the day, this may be my legacy and, by breaking a path for moral pause within the AI industry, it’s conceivable they will be an important contribution. It’s wrong to work for the AI companies and it’s wrong to defend them. It’s wrong to have friends who have a stake in disempowering or destroying humanity. This is not something we can be nice about and discuss over tea. There is no “right” way to deliver this vital critique— the reason they don’t want to hear it is that they are doing wrong.

I’ve learned more about the real world in the last three years pursuing PauseAI than I thought there was to know. I was an exceptionally idealistic and naive person to start out with. I’ve always been able to follow high ideals which, although I knew it was unusual, blinded me to how to the world really worked quite often. Through the lens of animal welfare as a young ethical vegetarian, I learned from a young age that adults can be very wrong and that they will often be extremely defensive because society allows them to. If society improves, they generally forget that they ever acted any different than they do as upstanding citizens today.

A legal scholar, John Bliss, has interviewed me extensively and served as an observer of protests I led. He describes me in his work as a sociological “zero”(referring to Mark Granovetter’s Threshold Models of Collective Behavior (1978)), which can sound kind of funny, but refers to me not needing the social proof or peer pressure of others to make social change. “Ones” won’t change on their own, but will if activated by a zero, “twos” want to see ones involved first, etc. and this is why social movements either cascade or fizzle out.

There is a lot that sucks about being a zero. From my perspective, I’m just doing what’s right, exactly as I have been my entire life, and as I thought everyone else was trying to do. But I don’t get my ideas of what’s right from other people or social groups in the same way that apparently almost everyone else secretly does. Everyone pretends that they do what they do because they individually reasoned that it was morally right. They never say “I started supporting gay marriage because I noticed a lot of people around me were”, and that’s usually not how they think of it in their own heads. I only recently realized that many, probably most, people are not genuinely confused but waiting for a consensus to form before they declare a side on a moral issue. I want to know what’s right; lots of people just want to know who’s going to win. So I get in these situations where I know I am right, and I know I am acting to protect vulnerable people, and I’m outraged that almost no one else either wants to or is ready to accept what’s happening.

I do callouts when I think the person should know better. I’m hoping that more eyes on the situation will create that self-awareness or peer pressure. I think they should apologize for not having gotten the moral issue right in the first place. And I feel betrayed because I thought the whole Effective Altruism (EA)/Rationality community were zeros trying to protect the world from AI danger. All this “be nice to our friends” and “but maybe, conceivably, they don’t think they are doing bad, even though maybe they should…” shit is coming completely out of left field. Like, I thought all our of interactions and community were predicated on this higher loyalty to the good, even when it was socially hard, so I see all this “it’s not nice to put your friends on blast” stuff as a complete copout.

It’s easy for onlookers to feel scared that they could be judged for not being morally good enough more than what they should fear: being sold out to the AI industry. I promise you, the people I call out were able to get the right answer. For someone who hasn’t been immersed in the issue of AI Safety for a decade, I get that it could seem like I’m asking a lot— I’m not. Every single one of the people I have called out has already made a conscious deal with the devil to get where they are today siding with an AI company. They took the money. They signed the NDA. They accepted the favors. By paying off enough people with money and community and cool research opportunities and promises of (eventually) using their power for good, the AI companies have cultivated a community that will back them up and provide the social safety that most people crave. And they did it using the community of people I thought were righteous zeroes standing up for what’s right.

So, if you look at my callouts and think my standards are too high, then you’re not the audience I would call out. I don’t expect the issue of AI danger to be easy and decipherable for everyone. I expected the people who knew the issue well not to allow themselves to join the project of literally building a superintelligence, which, I promise you, is wrong and they knew it, even if you don’t yet understand why.

I call those people out because I don’t give a shit that they think their cravenness should be protected by decorum. I’m going to do whatever I can to expose them. They have hurt my feelings in return and made me doubt myself, because it is hard to trespass politeness norms so much. But they are the ones that are hiding their actual trespasses behind social norms. They are the ones who have done wrong, and the world has a right to interrogate that.

It’s pathetic how much my callouts have taken their world by storm. And that’s what keeps me coming back to callouts, despite trying to leave that world behind— there is a chance to make inroads here for the rare person who is willing to do it. The bad guys depend on an extremely fragile veneer of plausible deniability. And every day the public is gaining the ability to see through it.

I know many AI Safety dudes who feel more kinship with the bad guys building AI and trying to take over the world with it than they do with people protesting data centers. How could that be possible?

Because it’s a culture war thing. AI Safety should not be on the side of the tech world, but culturally it yearns to be. The dirty little secret reason AI Safety people like talking about the libs’ denial is because it’s a topic on which they get to be aligned with the AI industry and other AI stans. It follows the same script as many identity-laden culture war issues before, like libs being “anti-science” for not liking GMOs, and it makes the AI world virtuous and smart and scientific in comparison.

The Grey tribe likes this excuse to fight leftists in familiar ways rather than come together in the unprecedented way the moment allows and calls for (#PauseAI). They would rather reject our allies against this dangerous technology for not meeting their “epistemic” standards than deal with the fact that their preferred company has a disqualifying lack of moral standards.

Even though they are right about AI’s abilities, I think the “AI is real” crowd does tend to be importantly spiritually wrong here. The libs are trying to deny those making AI power by denying what AI can and will do. That’s just false, but I appreciate the moxie. The safety bro move here is often to use the power of the technology as a form of evidence that it’s right to accept their authority. At least the leftists have fight in them.

I predict that when you stop presenting AI Safety as requiring they kiss the tech culture ring, the libs will stop doing the mental gymnastics to deny dangerous capabilities. They won’t have to if it doesn’t come with the baggage of accepting cultural defeat. It is often a much higher priority for the safety bros to make people admit that tech has the true worldview because it can build great and terrible things than it is to talk about the danger itself, let alone doing anything to directly address the danger. Leftist capabilities-deniers can and do support safety from dozens of other valid frames, like distrusting industry self-regulation and de facto compute governance via opposing data center construction. There’s plenty we can do together to make the world safer even if some people want to bury their heads in the sand about just how dangerous AI could be. No need to hold up our extremely urgent work to make the libs kneel and admit the nerd tribe is superior.

And here we are demonstrating peacefully on the Capitol lawn:

And we’ve gotten great media coverage for the cause:

But what people in the know about the pre-Pause AI Safety community inevitably want to talk about is what I say on Twitter in my free time. I’ve even been covered in the news for it. I just filmed an episode of Doom Debates where I ended up spending the entire 3 hours answering for callouts on Twitter. Because I say stuff like this:

(Replying to Joe Carlsmith’s announcement that he’ll be joining Anthropic and a long post explaining his reasoning)": Sellout. And the post is grade A cope.

Amanda Askell is a bad person. What she should do if she cares about us or Claude is put a stop to building Claude, but she’d rather be the star of this little PR narrative that she’s singlehandedly designing the morals of a machine she thinks could replace us.

So [Zvi Mowshowitz] is just bad now. If you were ever listening to him to learn about AI Safety then forget about him— he’s too enchanted with a recursively self-improving agent! The kind that could make dangerous, unaligned, out-of-control, ever more powerful superintelligence!

Well, I’m proud of my callouts and I’m tired of entertaining doubts from enemies and allies alike who can’t handle them. I’ve taken seriously that I might be going about them wrong, and they have never been my priority, but at the end of the day, this may be my legacy and, by breaking a path for moral pause within the AI industry, it’s conceivable they will be an important contribution. It’s wrong to work for the AI companies and it’s wrong to defend them. It’s wrong to have friends who have a stake in disempowering or destroying humanity. This is not something we can be nice about and discuss over tea. There is no “right” way to deliver this vital critique— the reason they don’t want to hear it is that they are doing wrong.

Until recently I would have said PauseAI US is separate from my callouts. Because, like, your donations don’t fund callouts and I deliver them in my personal voice. I never instruct anyone else to do callouts, because they come from deep personal familiarity with the person I’m talking to/about from my decade in the EA/rationality AI Safety world. They are not part of PauseAI US’s programming because the org does not do insider influence— it’s entirely about educating the public and elected officials for a simple policy of coordinating to pause frontier AI.

This summer, over 100 people have signed up to meet in-person with their representatives in their home districts with our training and support. To support programs like this, you can donate to PauseAI US 👈

But the org’s theory of change is pushing the Overton window, and my callouts are arguably the tip of that spear. I’ve always resisted equating the org with me as a figurehead, but I’ve grown to accept over three years of this that “the founder is the startup”, as they say. Those tweets drive a lot of donations and ultimately introduce a lot of volunteers to the cause. I haven’t really owned that impact, and just tried to treat it as something separate, something I felt I had to do as my duty, not strategically for the org. But my tweets do represent the worldview PauseAI US is trying to evince— that it is wrong to take part in building AI that could kill us, disempower us, and undermine our democratic self-determination of our future. Actually wrong. Not hypothetically wrong, or wrong at some point in the future. Actually wrong right now. Such that you will be treated as wrong for doing it in public.

People complain that I don’t phrase the callouts like a debate claim and according to debate decorum. But that’s not really the point. I say it in public, but I phrase it for the person it’s intended for to know in their own heart whether it’s true. I know how to cover my ass, if that’s what I wanted. I know how not to stick my neck out by sticking to publicly verifiable claims, using passive voice, making little understated nods, sandwiching the real point between compliments, offering face-saving outs, etc. But that’s actually not how to be more persuasive; it’s how to be liked. And you can get very far if you’re willing to be disliked.

I come in like a radical flank. The person I’m targeting doesn’t have to like me for my comment to blaze a trail in their head toward Pause and holding AI enablers personally responsible. Maybe they can’t yet walk down the trail I bushwhacked, but the next person who takes them down that path is going to have an easier time. In 5 years, they may remember that they hated me, but believe they supported Pause the entire time, and that they were early to disentangle themselves from supporting the AI companies. From experience with shifting the Overton window, I believe that many people need to have the chance to lose their shit or duke it out at least once before they’re emotionally ready to change their mind on something big. A moral growth spurt tantrum. I provide the opportunity, what could be the first step toward redeeming themselves.

The reason that less confrontational behavior is considered more polite is that it is more ignorable. It allows the person more ways not to have to take the criticism to heart. Direct confrontation strips away strategic ambiguity and plausible deniability. AI danger is not an issue where I can just lie back and wait to be proven right without having to push the matter— we can’t afford that. I need to share what I know about conflicted motives and weak moral backbones now because it’s our only chance to intervene before disaster. Pride is not more important than lives. Really consider, if a bioweapon was released tomorrow, would you still feel I had been too harsh?

I don’t know for a fact if I’m right about some of the things I say— I couldn’t, because I’m not in that person’s head— but I’m pretty confident, and it’s worth trying to hold up a mirror so that maybe they don’t like what they see (and what everyone else can infer) and stop. Many EAs and rationalists consider it anathema to “discourse” to notice intentions and motives if that person doesn’t explicitly agree that they have them. I call this “toxic charity” and it makes it impossible to call out liars and abusers if they don’t want to confess. Toxic charity is often employed to avoid confrontation. But most often I see toxic charity used as a pretext for believing one’s own bullshit, which is why I think directly attacking a person’s cope and excuses with a callout can be effective. On some level, they know.

This may seem like a confusing issue when you’re in the midst of it, with a conflict of interest and afraid to upset the powerful people around you. But anyone looking in on this conversation from any other point in history would see the obvious foolishness and corruption of trying to regulate dangerous AI while being dependent on money or benefits or social connections from the literal AI companies. I see the opportunity to cut through the layers of excuses and bullshit and obfuscation with the hot knife of what I know is really going on in my former cult— something not many people have the background to do— so how could I demur? People complain that I can’t do it with aplomb, without the people who were doing bad resenting and hating me for pointing it out. But that doesn’t change what needs to be done. I know I can make forward progress if I’m willing to take their lumps. To me it feels like, What other choice could there possibly be?

To hesitate to correct the people building dangerous AI would be to sell out the entire world for their comfort. Many of the people I’ve called out have millions of dollars in compensation and equity from the AI companies to dry their eyes with and a bevy of fans and hangers-on to lean on, versus a few criticisms from me. The fact that I’ve been able bother them so much is damning in itself. And it makes me think they still care about the truth and what’s right somewhere deep inside.

Since I started the organizing that became PauseAI, I’ve encountered a wall of paranoid fear from EAs and rationalists that the slightest amount of wrongthink or willingness to use persuasive speech as an intervention will taint a person’s mind for life with self-deception— that “politics” will kill their mind. I saw people shake in fear to join a protest of an industry they thought would destroy the world if unchecked because they didn’t want to be photographed next to an “unnuanced” sign. They were afraid of sinning by saying something wrong. They were afraid of sinning by even trying to talk persuasively!

The worry about destroying one’s objectivity was often phrased to me as “being a scout/not being a soldier”, referring to Julia Galef’s book The Scout Mindset. The “scout mindset” is described in this book as really wanting to know what’s true. It is contrasted to the soldier mindset:

I call it soldier mindset because it's very much like being a soldier on a battlefield, trying to defend the fortress of your beliefs against any evidence that could threaten to weaken or undermine it.

I think we have all the info we need to contradict the fear of not being a scout in Julia’s choice of metaphor. Scouts are important for success in battle because accurate information is important to draw up a good battle plan. But those battle plans are worthless without soldiers to fight the battle! “Everyone Should be a Mapmaker and Fear that Using the Map to Actually Do Something Could Make Them a Worse Mapmaker” would be a much less rousing title, but this is how many EAs and rationalists have chosen to interpret the book.

Even a scout can’t be only a scout. If a scout reports what they found to a superior officer, and the officer wants to pretend they didn’t hear it, a good scout doesn’t just stay curious about the situation or note that the superior officer has chosen a narrative. They fight to be heard! Because the truth of what they saw matters to the war effort. The success of the scout and the officer and the soldier is all ultimately measured in the outcome of the war. Accurate intel is important for something larger than the map— for the battle.

Imagine if an insecticide-treated bednets hemmed and hawed about the slight chance of harm from their use in anti-malaria interventions. Would that help one bit? No! What helps is working through foreseeable issues ahead of time at the war table, then actually trying the intervention with each component fully committed. Bednets are soldiers, and all our thinking about the best interventions would be useless if there were no soldiers to actually carry the interventions out. Advocating for the PauseAI proposal and opposing companies who are building AGI through protests and by contacting your representatives is an intervention, much like spreading insecticide-treated bednets. But, instead of bednets, the soldiers are people armed with facts and arguments that we hope will persuade the public and government officials.

Interventions that involve talking, thinking, persuasion, and winning hearts and minds require commitment to carrying out the intervention, and not simply to the accuracy of your map or your reputation for accurate predictions. To be a soldier in this intervention, you have to be willing to be part of the action itself and not just part of the zoomed out thinking. This is very scary for a contingent of EAs and rationalists today who treat thinking and talking as sacred activities that must follow the rules of science or lesswrong and not be used for anything else. Some of them would like to entirely forbid “politics” (by which they generally mean trying to persuade people of your position and get them on your side) or “being a [rhetorical] soldier” out of the fear that people cannot compartmentalize persuasive speech acts from scout thinking and will lose their ability to earnestly truth-seek.

I think these concerns are wildly overblown. To the extent that people become biased by associations in AI Safety, it’s generally association with the AI industry! (I’ve never heard this referred to as “soldier-y”, perhaps because the AI industry only asks that they stay out of their way.) I developed the PauseAI US interventions with my scout hat on. When planning a protest, I’m an officer. At the protest, I’m a soldier. Lo and behold, I am not mindkilled. In fact, it’s illuminating to serve in all of those roles— I feel I have a better and more accurate map because of it. Even if I didn’t, a highly accurate map simply isn’t necessary for all interventions. Advocating for more time for proper safety and governance of frontier AI to be established is kind of a no-brainer.

It’s noble to serve as a soldier when we need humans as bednets, to carry out the interventions that the scouts have identified and the officers have chosen to execute. Soldiers win wars. The most accurate map made by the most virtuous scout is worth nothing without soldiers to do something with it.

First, let me say that I learned this the hard way. I am an excellent decoupler. I like thinking abstractly. I can easily imagine a different context or no context for whatever focal topic. And I decoupled too close to the sun by actually missing huge parts of what I was talking about (namely, context) because I had determined ahead of time they didn’t count.

Decoupling is the ability to mentally disentangle the parts in a situation from each other and from the particular context they’re in right now. Like being able to answer a hypothetical question with an answer you would never give in real life, such as “If you had to have sex with a cow, would you rather it be dead or alive? Why?”

This is actually a real question from an application to a hype house from creators who famously prize “high decoupling”.

A high decoupler, according to those who value this trait, should be able to engage the hypothetical on its own terms without saying something like “I would never, ever, ever do that, so I’d find a way out.” Because it says in the hypothetical that you HAVE to, a high decoupler would move past taking issue with the premise and consider which would actually be worse and why. Decouplers bite bullets. Some people value decoupling extremely highly, and they enjoy testing their ability to isolate taboo topics from the context that would “technically” make them bad. Generally they say they value open-mindedness and critical thinking. “Edgelords” is often a fair description.

It is an act of interpretation to determine what can be decoupled from what, what is essential and what isn’t. It has to be done to have any kind of gears-level understanding and prediction in your head. But a goal of “high decoupling” is wrong because it fetishizes the stripping away of parts and allows them to be stripped selectively. And, by making it your goal to think in a high decoupling way, you can easily be taken advantage of by people who just want to put you in simulated situations of debasement or suffering and make you take part. The cow question has an info button beneath it, which explains “we need to make sure that you’re ready for anything”, which might as well be a DANGER siren saying “we will violate your boundaries”.

Taking pride in an identity as a “high decoupler” is lowkey a way to assert that context isn’t real, that everyone should be concerned only with getting at the platonic ideal. And so it can become a license to dismiss inconvenient components of a real world system as contingent context and strip them away. I’ve heard things like “It’s only because of cultural conditioning that women don’t like catcalling. If you decouple the compliment from our culture’s negative ideas about men, then it’s actually nice.” Cultural context can just be stripped away, while the essential (and, therefore, true) nature of the interaction is asserted to have an intrinsic positive quality. In fact, one issue with catcalling is it is sometimes intended in an inherently, genuinely threatening way, where the woman is demeaned or rationally fears for her physical safety. That possibility is being sneakily dismissed by simply being labeled as expendable context, but it’s being presented as if the counterintuitive true nature of the interaction has been revealed by the decoupling analysis.

Saying that you’re decoupling can be a smokescreen for smuggling in your preferred context or asserting that your contingent preference is objectively right. Or it can be a reason to express socially unacceptable or aggressive views, like which humans are worth more (“Say you HAD to choose between two people to kill and all you know is one is white and one is Asian…”) or making it known what you think with plausible deniability (“If I HAD to cheat on my wife, it would be with Sonia. But that’s only if I had to!”). In these examples, the person is clearly yearning for an acceptable way to express an unacceptable view, and they choose the pretense of intellectual fearlessness, even suggesting that if others wouldn’t be willing to go there themselves, it’s because they can’t face the truth.

Things are correlated and connected. They are, in fact, coupled. If you allow yourself to think in a decoupled way without eventually recoupling before taking your insight to the real, contextualized world, you will be missing real information that is in fact part of the whole. If you allow un-recoupled high decoupling from others, you may be missing why people really want a reason to give “decoupled” answers.

I’ll describe the video above for those who don’t watch. It’s a “degenerate” wizard in Harry Potter world watching the Triwizard tournament and telling someone offscreen that he bet his last $3k 20,000:1 on a 5-part parlay bet where the conditions are the unlikely ending events of the end of Harry Potter 4. One is the death of Cedric Diggory, for which he cheers. “This is the most fucked up bet of all time,” his friend says, “you’re cheering for a dead child!” The last condition of the parlay is Voldemort’s return. “Tell me Voldemort’s back…” he encourages at the screen as Harry cries over Cedric’s body. “Tell me he’s back.” Harry looks up at Dumbledore and gets the words out as tears stream down his face: “He’s back.” “He’s back! HAAAHaaHaaahaaa!!!” The degenerate celebrates. “This is the greatest day of my life!”

This is a joke about how unlikely the events of the narrative are. But it’s not even really a joke about how perverse bet hedging can be. When you have money riding on a bad outcome, part of you wants it to happen. Hedging is a moral hazard. Even just trying to call it right can be corrosive.

Robin Hanson has done more than anyone else to promote the idea of prediction markets as a way of surfacing and aggregating information from crowds. He argued that real money and track records would keep individual bettors properly motivated to get the answer right, like futures markets. But the descriptive answer is not the only question. There is also a normative question. Prediction markets are about events that haven’t happened yet. What will happen crucially depends on what the people involved want to happen. What people want to happen depends in part on what they think should happen, and this can be contaminated by trying to call it right. What should happen is often not the default scenario if no one tries to make it happen.

In the early days of Manifold, a prediction market platform that uses play money and cultivates a very playful vibe, it was common for my spicier rationalist/effective altruist friends to bet on the probability of success of their relationships. The easiest way to control the outcome and be “right” about your own relationship is to sabotage it. You may protest that surely someone wouldn’t let the results of a prediction market make relationship decisions for them, and maybe they wouldn’t when things are going well. But when things are tough, maybe it enters their mind that the market thinks the relationship is doomed. The market may be right, but it was unnecessary to add an element of pride at having listened to “the truth” before the relationship’s truth had come to pass on their terms.

Having an incentive to make the right prediction competes with the incentive to make the right thing happen. Betting on a bad outcome, and either creating a personal financial return to it happening or bragging rights that you called it right, gives you a stake in the bad thing happening. You might think it won’t, but it will. Something that is objectively terrible for those it directly affects suddenly has a silver lining for you. This is corrupting.

I have seen this actually happen to people I know who try to hedge their bets so they will profit even in the event that plan A: “save the world from misaligned superintelligence” doesn’t work out. They do this either by making pessimistic bets on prediction markets or by buying stocks in NVIDIA and the AI companies. In some cases I tried to talk them out of this beforehand. (Not my point here, but buying stock in an AI company is literally taking part in what they are doing. It is raising their value and giving them more power.) They swore up and down it was just the rational thing to do to sign up to get a little consolation prize, that it wouldn’t affect their commitment to the cause at all. But afterwards I would catch them saying things like, “well, as a stockholder it was good for me…” almost apologetically, revealing that their investment in the bad future was clearly on their mind. The final stage of this corruption is becoming conveniently convinced that the future where you get the money or you get to be right is the good future.

This same basic phenomenon can occur without any fancy platforms. Watch enough football games and you’ll encounter a fan who sulks the entire game saying that their team is going to lose, maybe even reaming them out for their poor performance, like they can’t bear to just root for them in earnest if they might not win. At least they won’t have been wrong if they’re disappointed. This same logic is behind “it’s over” and similar declarations that it’s “inevitable” that the superintelligence will replace us. People usually reach for them because they are overwhelmed and the bad future being a foregone conclusion excuses inaction, but I’ve seen many times how painfully exposing it seems to be for people to imagine standing up for Pause when we might not win.

Pause happens when enough people join and make it happen. The easiest way to “call it right” on AI is to be a pessimist and do nothing. If you’re using “calling it” as your cope and your shield, you might as well be laying down and dying. And winning the “laying down and dying” market is going to be pretty cold comfort then.

In Scientology, they have a word for someone who is free of delusion. When a Scientologist has overcome “reactive mind”, they are Clear. A Clear may also have superhuman intelligence and self-control.

There’s a word for this in the circles I used to run in, too. In Effective Altruism/Rationality/“This Part Of Twitter”, a clear has “good epistemics”. Like Clear, the idea of it is not bad in principle. Just like it would be good not to be reactive, it would be good to be rigorous and thoughtful about how you know what you know, which is the real meaning of epistemics— pertaining to epistemology.

Some people are surprised to learn what epistemology actually is when they’ve only heard the EA/rat/TPOT use of the word epistemics. That’s because, there, Good Epistemics is a dog whistle and it means “coming from the right background and sharing our frame”. Anthropic employees were hired not on the basis of being EAs, they’ll claim, but on Epistemics. AI Safety orgs would love to hire outside of the EA/Rationality community, they claim, but those candidates simply don’t have The Epistemics.

Someone who isn’t Clear might still be in delusion based on unconscious traumas from their past lives, which can manifest as not accepting the truth of Scientology. The corresponding state in EA/Rationality/TPOT is “bad epistemics”. Interestingly, in that space it is believed that many activists and AI Ethics people who don’t share EA/rat/TPOT culture have Bad Epistemics 😠, but most people at AI companies who do share their culture are found to have Good Epistemics 😁, which overrides them being part of the AI problem.

An adversary of the faith is called a “suppressive person” in Scientology. This is simply referred to as an “adversarial” relationship in EA/Rationality/TPOT. Adversarial people never have good epistemics, because if they did then they would all be friends, or so the EAs/rats/TPOTters believe.

I chose this title following the excellent essay Fuck Nuance. It argues that nuance is a noble-sounding excuse to be unclear and not try to arrive at meaning from data. I think The Epistemics are a lofty excuse to screen for ideological conformity while pretending it’s about merit or virtue.

It bothers me that a lot of what people like about another person’s character is basically wealth. “Kindness” often means “giving a lot of emotional labor” or emotional services like “making me feel good about myself”.

Often I’m expected to pay all of the emotional costs the other person feels when I point out that they are part of something really bad by building AI. I’m supposed to cushion the blow, tell them it’s not their fault, make it really easy for them to change, congratulate them ad nauseum, not make anyone else feel bad in a way that might trigger their own personal fault, etc. I am supposed to pay for all of this when they are the ones putting others in danger. If I don’t, others will try to punish me by saying they’ll turn against the cause of PauseAI because I made them feel bad. And they’ll say that retaliation and closing their ears to an important claim are all okay because I’m “mean”.

Being “mean” here is just not being able to foot the bill for their feelings. This is a cheap notion of kindness, like telling a child they are “good” for being easy for the parent. Surely kindness means more than being convenient or handing out favors?

Real kindness means caring about the welfare of others. And that is not the same as being polite or pleasant. Sometimes it means being brave. It usually means being spread pretty thin across the people that need help or protection. Don’t let real kindness be priced out of the market.

Stage 1: Awareness of the situation

You know that there is artificial intelligence being built and it is improving.

✨ You are stuck at this stage if: you “tried AI in 2024 and it was bad” ✨

Stage 2: Belief in the problem

You know that artificial intelligence could be dangerous.

✨ You are stuck at this stage if: you think AI is “just a stochastic parrot” or “not sentient” and that means you’re “not worried yet”. ✨

Stage 3: Awareness that the problem is happening NOW

You know the problem is not years away, but could be manifesting now.

✨ You are stuck here if: you think that there is plenty of time for others to solve the problem, or plenty of room for as many tries as it takes to get the solution right. ✨

Stage 4: Empowerment to act

You are prepared and empowered to act. You understand the actions it is your place to perform.

✨ You are stuck here if: you see harm caused by AI in the world today and vulnerability to much more, but in response you say “I can’t do anything about it” or “it is what it is” ✨

This is where most educated people reading a blogpost about AI danger tend to be stuck. It takes some time to struggle through the feeling of inertia, the deference to the very people putting us in danger because they are “experts”, the excuses that it’s not your job to speak out or demand more from our leaders. It’s confusing to find yourself intellectually convinced that we need to take serious corrective action as a society but to look around and not see others taking action.

You, reader are probably in this stage. What’s that? You say you’re not? Well have you ever taken action?

Stage 5: Taking Action

You take appropriate action.

✨ You are stuck here if: you feel worried about AI and you think you are seeing AI danger in the world, but you struggle with the motivation to act, believing you can take action, or believing that it’s your place to take action. ✨

If you have ever contacted your representatives, taken part in a boycott, attended a protest, volunteered or donated money to AI Safety charity, then you have taken action! CONGRATULATIONS!! 🥳

In all seriousness, many people who spend their days talking about AI Safety on forums or on twitter have never taken 2 minutes to write their congress member.

You can ascend to stage 5 right now by answering a PauseAI US call to action: https://www.pauseai-us.org/takeaction/

If you feel resistance to taking a 2-minute action like sending a message to Congress, why? What’s coming up? What stage are you stopping at?

I’m opening comments to everyone on this one. Give me your answers below.

This is the Inti Watana in Ollantaytambo, Peru, and it used to be the spiritual center of the Inca world. It was the most sacred site in the entire empire because this was where the highest priests bound the sun to the Earth for another year, keeping the rhythm of the cosmos intact. It looks like that now because the Spanish destroyed it in an attempt to destroy the Inca religion and culture, leaving the ruins as a reminder of their dominance.

The Inca did not at first recognize the Spanish invasion. In fact, when Pizarro landed in 1528, many high-ranking Inca saw it as a chance to gain a new ally against their rivals at court. Emperor Atahualpa underestimated Pizarro and was captured and killed. Atahualpa’s brother Manco Inca cooperated happily with the Spanish as their puppet emperor for a few years, until Spanish brigands ransacked his house and nothing was done, making it clear he was not in control. Manco Inca then became a resistance leader, falling back to the outpost Vilkabamba where the jungle was too thick for the Spanish to reach. In 1572, the last Inca emperor is captured and beheaded— Spain was completely in control of Peru.

All of which is to say, the AI industry are Pizarro’s ships pulling up to humanity’s shore. If you think the benefits of AI company success are going to trickle down to you, think again. You may need Claude for your work now, but Anthropic does not need you. When GPT-7 can do every human job and more, OpenAI will not need human customers. Gen AI is not the foundation of an ecosystem. Or, at least, not our ecosystem. It’s a conquerer.

Many people who had hoped to work with our AI conquerers have already been left behind:

Prompt engineering?

Soon after their releases in 2022, many netizens had high hopes that they could specialize in chatGPT and Midjourney. Getting good images was very hit or miss in those days, and not everyone had instant API access to all the models, so there was an opening, it seemed, for people who might specialize in using the tools. “Prompt engineering” was going to be the “learn to code” of the future, they said. Similarly, small-time developers were at first hopeful they could do things like “make a wrapper for GPT-4”. Like a whale carcass fallen to the bottom of the ocean, they thought genAI would be a new source of nutrients they could scavenge.

But it soon became clear that there was no alpha left for anyone but the AI companies. No usable work remained to be done as each rapidly oncoming model release was more and more capable of doing all that scavenging work on its own, and better. When OpenAI announced the GPT Store in 2023, I recall seeing many devastated comments on twitter and facebook saying “there goes my startup” after it was revealed that chatGPT would spin off its own apps now based on prompts.

Custom prompts and B2B SaaS?

Even midsize startups were robbed of the jobs they expected to perform. I knew a guy whose startup was working on an AI medical transcription product. They spent nearly a year developing their prompts and workflow for Gemini. And then last year, in one new release, a single prompt to Gemini worked better than the workflow they had built. That company has a job AI can’t do, which is obtaining regulatory compliance, but those that were banking on software add-ons to frontier AI have been left with little to do.

AI agent workflows are cringey now but I only expect the trend of autonomy to continue.

AI as normal business technology?

The business class initially liked the promise of increased productivity from AI and lower labor costs. But now it’s getting to the point that those people are uneasy about all the instability AI development is causing. Businesses can’t plan if they don’t know what the lay of the land will be month to month. Anthropic’s Chief Product Officer Mike Krieger sat on the board of Figma, who had been an early partner with Anthropic, while Anthropic was designing Claude Design, a direct competitor, completely blindsiding Figma. Large companies that attributed layoffs to AI efficiency looked like assholes and, now that the teaser pricing has raised, they are on the hook for a lot of tokens, some saying they spend more on AI bills than they did on employees.

Technical AI Safety?

Up until about now, it was a bizarre fact about the AI Safety world that legitimacy came through association with the AI companies. Indeed, according to Dario Amodei’s empirical paradigm, you could only do technical safety inside a company making cutting age AI models. Increasingly, the entirety of frontier machine learning and technical safety discourse is happening within closed labs, with nothing for anyone else to do. Would-be independent technical safety researchers face an uphill battle with so much of what they study being the property of a corporation.

Academics working on already released models have a hard time doing relevant work because it takes so long to get published relatively to AI development time. Third party model evaluator orgs depended on good diplomatic relations with the AI companies to be allowed to evaluate new releases. As you can see on the METR chart above, they say they are beyond their ability to evaluate Claude Mythos, and Apollo made a similar admission when evaluating Claude Opus 4.6 because they weren’t able to evaluate eval awareness in the few days Anthropic threw them a bone and gave them.

There’s no place for the rest of us in the world the AI industry is trying to build. We are not working together with the AI industry— the AI industry is using us, our data, our art, our writing, our subscriptions, our misguided attempts to profit off of each other by sucking up to the big new player on the scene, and our compliance… until they have destroyed our jobs, social lives, and politics and they don’t need us anymore. They are not weaving together with humanity and building a thriving future ecosystem for us all. They are conquistadors. And when they get what they want, they won’t care if the native population lives or dies.

My blessing and my curse is remembering how I and others used to feel and think. I have less Overton-window-shift amnesia. This is where most of my social change insight comes from.

But collective amnesia is how we cement Overton window shifts, so it puts me in a weird position.

— me, in the tweet above from Sept 2025, for those who may be listening

Sometimes I feel like the only person in the world who remembers that almost no one older than me around me supported gay marriage in 2000. I vividly remember the consensus position in church and among my parents’ friend being annoyance at gay people for bringing it up in front of the children. They would mention that someone was gay the same way that my great grandmother who grew up having black servants for slave wages in the segregated South would mention that someone was black. “I saw that girl who works at the bait shop,” she’d say, then she’d pause and lean in to whisper “she’s black” before continuing with a story completely unrelated to the woman’s race. But it was the way she said it that was so similar to the 2000s parents— she clearly felt she was enlightened for noticing the person’s race and being so cool about it.

Ask any of these people what they were doing during the fight for civil rights that marked their era, and they’ll tell you they were right there on the front lines! Most of them aren’t lying or in denial— they (sort of ) attended a protest once, and they always had one or two out gay friends… or acquaintances!— they just truly don’t remember how they felt weird and threatened seeing two men hold hands in public or how they thought civil unions should be enough because marriage is a religious sacrament. This is Overton window shift amnesia.

The Overton window is the range of thinkable thoughts in society— what’s okay— at any given time. What’s in the center of the Overton window is seen as obviously correct and taken for granted. What’s outside of the center is radical or even literally unthinkable. Fights like civil rights for all races and sexualities require nudging that window so the idea of desegregation or marriage equality goes from unthinkable to merely radical to acceptable to mandatory. There is Overton window inertia— at first it doesn’t want to move because there are entrenched interests in the way things are right then. But after it has moved, most people quickly forget they ever felt much different about social issues than they do today.

My model of Overton window shift amnesia is basically that most people don’t think about what’s okay from first principles alone. We take the temperature of the culture around us and measure ourselves relative to that. What we store in our heads is something like “I’ve always been a more progressive person” or “I’m kinder than average” (and everybody thinks they are better-than-average at most things). “What’s okay” on things like civil rights changes on the order of years and decades, so when we retrieve memories about the past regarding what was okay, we think “I’ve always been a more progressive person” or “I was kinder than average” and apply it to today’s standards. Basically people have a bias for believing they are ahead of the curve as a personal trait, so it checks out that they would have always been on the historically winning side of big shifts.

This may sound bad, and there are a lot of bad things about it. But one really good thing about it is that it allows society to move forward. I really felt that LGBTQ rights had won when I saw a tweet (which for the life of me I could not track down) from a teenager who didn’t realize gay marriage had been illegal in his lifetime and why everyone had been making a big deal about LGBTQ rights. That’s the world we want, where progress is settled and even the very idea of discriminating on the basis of an arbitrary characteristic belongs in the past.

The bad part of Overton window shift amnesia is being doomed to repeat history. We so easily forget how much resistance we have overcome before for a just cause because those victories seem like foregone conclusions when you have retrograde amnesia. We forget why we can’t take shortcuts or listen to that charismatic man who has everything figured out for us.

The idea of pausing AI is zooming through the Overton window and I still see the amnesia after just three years!

Political salience of AI issues. When I started organizing PauseAI US, the AI Safety experts around me in Berkeley told me a pause was all but impossible, technically and politically. I was asking them what they thought of Pause precisely because the idea had polled so well after the Future of Life open letter for a 6-month pause. Didn’t that show the public sees the danger? Sure, they said, but the issue isn’t salient to them. Well, now, barely three years later, Americans rank AI danger as more important than abortion or guns.

Mainstream acceptance of x-risk. The Berkeley experts said existential risk is just too “weird” for mass movement-building. Outside the Overton window. And politicians weren’t really any better, in their estimation. They would bounce off the weirdness of the topic and see doing anything about it as sticking their neck out. Well, just three years later, literally this week former Senator and Presidential candidate Mitt Romney said extinction risk from AI was “our highest and most urgent national priority”. He also authored some pretty decent legislation (despite its goofy name “Preserving American Dominance in Artificial Intelligence Act of 2024”) on AI Safety before leaving office. You don’t get more mainstream than this endorsement.

Government capacity and right to regulate AI. The Berkeley experts and much of the tech world talked about Moore’s law style trends in developments as if they were laws of nature, saying that superintelligence was inevitable and tech progress cannot be stopped. Just yesterday the White House used export controls to effectively shut down all access to Anthropic’s Claude Fable. Because Anthropic is the leading AI company and Fable/Mythos their most frontier model, this is a partial pause in one swoop.

The moral argument against AI development. Quite a few characters on twitter and, again, the Berkeley experts would have told you it was not only futile (because tech progress is “inevitable”) to regulate AI, but deeply wrong. Just last year, former White House AI and Crypto czar said, in an utterly typical remark on AI regulation, “We’ve got to let the private sector cook”. He was one of the architects of the state AI regulation preemption scheme to shield the AI industry from as much regulation as possible. Yesterday he shared this New York Times op-ed, screenshotting a section saying that developing AI should be immediately stopped if any of the dire warnings CEOs like Dario Amodei give are true, that any other response would be “monstrous”, and captioned it “Yes”.

Zooooooooom goes the Overton window!

Like many fights before, pausing AI will very likely feel impossible until it feels inevitable. There is a long way to go to an international treaty to pause AI until it’s safe and consensual. It looks a lot longer if you forget how far we’ve come, and how fast. The most effective way to fight Overton window shift amnesia is to remember that we have a bias to believe it has always and will always be like it is now. But what’s unthinkable today can shift into obviously true before you even know it. It has happened before and we must make it happen again to save the world from AI danger. #PauseAI

If you went back to a random war in, say, Europe, 200 years ago, it would remind you of football. War used to have bright colors, fun costumes, bands and drum lines. And there was audience participation. Common folk used to come out to send armies off with fanfare. You would hear dispatches about troop movements and battles like we hear news coverage of scores and substitutions today. Local pride depended on how well soldiers were doing and hills won.

Does this energy remind you of anything else? If you were a computer science or tech nerd in 2023, you probably went through a similar atmosphere related to rapid progress in machine learning and LLMs. I criticized third party evaluations of LLM models many times as being like ESPN analysis instead of like regulatory reports. Now there’s an actual ESPN clone for tech news, just bought by OpenAI, TBPN.

My Dad was a sports journalist. He did it for the love of the game, but his favorite altruistic lens on sports was that they are a controlled alternative to war. Sports are one answer to the question the 19th century “Father of Psychology” William James poses in the essay The Moral Equivalent of War, which seeks to find a constructive way to capture the uplifting and purpose-imbuing psychological effects of war. As armies got less flashy and war got more destructive and less fun to root for (sometime after the Civil War and definitely after WWI), football in the US took on more of the local pride, fanfare, gambling, and commentariat.

Frontier AI today is war/sports for nerds, and it’s bringing out the worst in them. The hardcore fans are treating “the AI race” like a cold war, and the companies are playing with real weapons. Increasingly, AI companies are getting involved with actual militaries. Anthropic’s Claude was already involved in killing 150 Iranian civilians, 120 of them children, when the US mistakenly bombed a school Claude had identified as a target— one of the worst war crimes against civilians since Vietnam.

AI fans who identify with AI progress are disinclined to think seriously about the harms and risks from AI. They protect the AI developers from scrutiny and pushback. Today we would say it’s depraved to hope a war would happen because it would be fun, but my grandfather who grew in the aftermath of WWII in Mississippi told me that every boy back then was itching to prove himself in battle. Some AI fans talk seem willing to take any risk— up to an including “every man, woman, and child on Earth dies”— to keep the drumbeat of war going so they can drum along at home.

We need ways to channel that competitive urge that ennoble the human spirit instead of driving it to destructive and callous ends. I want to help the AI fan ecosystem transition to AI dev sports instead of running on reckless and potentially harmful AI development. If we had AI competitions that have all the fun of evaluations, the news and commentary, the starmaking, and the gambling of AI development without pushing the frontier of capabilities, they could be more fulfilling for the fans AND not serve to protect greedy, unscrupulous developers.

The football field is a manicured battlefield where the teams can enter to fanfare, take formation, and where you can precisely measure yards won and lost. It is enclosed and safe. The real challenge is coming up with an arena for AI development sports that’s contained. Since all knowledge is interconnected and dual use— to the point that technical AI safety research often has bigger implications for AI capabilities than for safety— I don’t know how to do this for frontier machine learning. But I bet finding the AI equivalent of a football field and designing AI challenges that are fun and self-contained is a solvable problem. And I bet there are a lot of readers out there who would have wholesome fun trying to solve it instead of spectating on actual AI warfare.

I see a growing number of people not just using AI, but identifying with AI— as who this prosthetic allows them to be. Many addictions boil down to how the substance or behavior lets you see yourself. “AI as identity” could be described as an addiction to the kind of person you feel you are when using AI. Not everyone is as online and surrounded by AI power users as I am, so this may sound histrionic, but you will see it soon if you haven’t already.

I’m not talking about indulging in too much AI use. There’s over-reliance on AI and then there’s AI as a prosthetic identity. Over-reliance is bad enough. I had a PauseAI university organizer tell me that some of his classmates almost couldn’t finish their own thoughts or answer questions about themselves in conversation without “asking chat”. I think this is a light version of AI as identity. A notable difference between this kind of AI dependence and the kind of AI as identity I’m talking about is that what the college students were doing is taking a shortcut that’s normalized.

AI as an identity is about becoming a better person with AI than the person feels they could be as themselves. The run-of-the-mill AI identifier doesn’t simply think it’s okay to be lazy by using AI. They want to be radically enhanced by essentially becoming part AI. Transhuman, even. Generally, such people are desperate to be seen by others as early adopters and mavens of the technology, maybe even as deputy AI developers (based on their performative public exasperation about a problem with Claude you’d only encounter after using $10k worth of tokens). AI-powered striving often forms the basis of their social group. These are the people who need it to be known they have agents running all night and have 10x’d their productivity. The ones who go to the OpenClaw conference and dress up as lobsters looking for a wife.

The most extreme version is what I call wanting an “AI brain transplant”. These are the people that paste chatGPT output into twitter replies, for example. It’s one thing to have AI do your work for you in school so you can get ahead; it’s another thing entirely to have the robot to socialize in your place. I half expect such people to answer the mocking rhetorical question “do you need chatGPT to fuck your wife for you?” with “As an AI assistant, I have no body and can only assist...”

The problem is, as Dan Dennett observed, you want to be the brain donor, not the brain recipient. Having your brain “upgraded” is being killed and zombified. But, as with anorexia and thinness, there is a fatal attraction to the idea of enhancing oneself in an impossible way. AI as identity comes from a similar spiritual sickness. The person wearing AI as identity doesn’t want to be who they are without the latest model.

The feeling of being enhanced by AI is a drug, and it makes people identify with and excuse the drug dealers— the AI companies. There’s a belief that to be credible in AI Safety, you have to be a power user. But actually, using AI this much is a cognitive and moral hazard. It’s like thinking you have to try every new strain of heroin on the street to be an excellent DEA agent. It’s good for a DEA agent to be educated about heroin, disastrous to be addicted, and delusional for him to think he won’t get addicted if he’s shooting up.

The drug isn’t AI per se— it’s the latest model. And you can see even AI Safety experts today on Twitter are itchy for access to Claude Fable for some reason. Their goal is supposedly to prevent existential risk, but they can’t stop talking about how productive they were with Fable, and questioning the technical basis for the White House’s decision to restrict its use. Why can’t they be slightly less productive with Opus 4.8? Because impossibly productive is who they want to be.

The lust to be on that next shit builds a dangerous tolerance for taking risks with frontier models. There has always been this assumption with AI scaling that the goal is to get as close as possible to the edge without driving over the cliff. That has always seemed like a plan to fail to me. Staying well away from the cliff is a much more robust plan. But if you get a high from being on the edge, and it feels like withdrawal to go back to Opus 4.8, you’re not going to be able to be objective about how much danger you’re in. Or how much danger you’re putting everyone else in.

A lot of people, including a lot of the people who are supposed to be keeping us safe from AI, are becoming addicted to a product that the AI companies control. They are letting their addiction whisper in their ear that it’s okay to keep pushing the frontier, or that the right thing to do for safety is somehow to cheer on developing stronger and stronger models for them to use. And it motivates them to find reasons not to take on the AI companies and their rollercoaster.

But even if AI as identity weren’t playing into the interests of AI companies, I would still find it very sad. The recipient of a brain transplant dies.

I’ve written about the concept of overresponsibility before (here and here) and those posts were received gratefully by many readers. The burden of believing you are responsible for things outside of your control is soul-crushing, and learning to explicitly reject it is freeing and life-changing.

But we mustn’t forget there are legitimate moral responsibilities. “Underresponsibility” is rife in the AI world today— it is the engine of the “AI race” logic.

I remember talking to a guy (can’t say who) who ran an effective altruist charity about this around 3 years ago, when I was just starting up PauseAI organizing. He was trying, gently at first but then more insistently, to get me to agree that I was wrong for naming the AI company CEOs in protests I organized. “It’s not about the CEOs,” he insisted, “it’s the race.” He was basically saying that the CEOs were completely interchangeable and anyone in their position would behave exactly the same because they were forced to by “race dynamics”, basically by game theory and logic itself. He thought it was harmful to name the CEOs individually because it created the idea that they had individual responsibility for what was happening instead of the real villain, what the well-known essay calls The Incentives:

The core idea is that the mere existence of perverse incentives is a valid and sufficient reason to knowingly behave in an antisocial way, just as long as one first acknowledges the existence of those perverse incentives.

I think Dario Amodei created and exported this narrative for AI scaling more than anyone else. He could not have scored bigger with propaganda.

The Nuremberg defense is: “I was just following orders.”

The Anthropic defense is: “We are better than the counterfactual.”

The Anthropic defense gave Dario complete moral license with his employees, the AI insiders, and a lot of early onlookers to do exactly what all the other AI companies were doing with no accountability while claiming to be the good guy.

It should be pretty simple to see what’s wrong with this— Anthropic is just positing that the counterfactual is whatever they say. They can do bad things because whatever evil they do doesn’t count next to this hypothetical world that they get to design as a foil. They describe AGI as “arriving” and speak of “navigating” the upcoming “transformation” as if it is predestined and not something they are working hard to make happen. Since this whole AI thing is just happening anyway, in their telling, whether or not that’s safe or consensual isn’t really on their ledger, whereas being better than OpenAI is. In this “counterfactual” world, OpenAI or China could do ANYTHING to win, so Anthropic is the good guy for doing ANYTHING minus 1. Anything is permissible as long as they can imagine something worse. It’s basically Dr. Pangloss but for evil instead of for cope.

True, the repast was mournful, and the company moistened their bread with tears; but Pangloss consoled them, assuring them that things could not be otherwise.

“For,” said he, “all that is is for the best. If there is a volcano at Lisbon it cannot be elsewhere. It is impossible that things should be other than they are; for everything is right.”

— Candide, by Voltaire

Another frequent implication of the Anthropic defense is that they shouldn’t have to give up their chance at winning the race to do the right thing, even if the world would be better off because of it, because why should they have to sacrifice if OpenAI doesn’t? They think they should only be required to stop making the problem worse by dropping out of the race if that would single-handedly solve the whole AI problem.

They spell it out pretty explicitly in their latest pamphlet:

A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions. It would also require that each can verify that the others have actually stopped. […]

None of this is necessarily impossible in principle—the world has built verification regimes for other complex technologies (e.g., the Intermediate-Range Nuclear Forces Treaty)—but those regimes took decades to build both the infrastructure and the trust. We don’t have that long. A unilateral pause by one lab, by contrast, is achievable immediately, but accomplishes much less: it would change who the front-runner is, but it would not create the wider deliberative process that is currently missing.

Okay, then, accomplish “less” and stop doing active harm. It would be doing an awful lot of good. But because they can imagine a scenario whereby they do even more good by holding on to their power and literally being the problem advancing the AI frontier, according to the Anthropic defense, they have to!

The name “Anthropic” comes from a class of philosophical dilemma in which it matters who is the person in the dilemma. Basically Anthropic named themselves “main character”. The idea that they would simply do their part and stop making the situation worse by pushing the AI frontier (something they know is bad because the company promised it’s initial investors that it would not advance the frontier) is treated as out of the question. They feel entitled to be the ones to either save the world or transform it. But that’s crazy— no moral duty includes the proviso “but only as long as you get to be the star”.

This is underresponsibility. Anthropic and all the other companies that have helped themselves to Anthropic’s groundbreaking spin here are criminally underresponsible. By framing everything bad they do as “necessary” in the Panglossian sense and believing really hard that they are the good guys, Anthropic can justify anything.

One of the emerging themes of my June blogpost-a-day is the reality of conflict. The world is different than we wish. Other people are different than we wish. There would be conflict in the Singularity. There is legitimate, irreducible conflict between the Judean People’s Front and the People’s Front of Judea.

This is not a new theme for me. One of my most famous blogposts of yore, written 10 years ago, is about the reality of tradeoffs. We are in triage every second of every day was about the denial that we should ever have to withhold medical care from one to save two, even though this is obviously the case in some tragic situations and refusing to countenance that only reduces the number of lives that can be saved.

Nowhere in the episode were the beneficiaries of the triage discussed. There was no attempt to determine how many more people were saved because hospital staff took difficult, decisive action. There is no discussion of who should have died in that situation if not Nathalie — someone with many healthy years ahead of them? Two people who could have been saved with the same amount of oxygen? There is only denial that anyone had to die at all. There is no gratitude for the extra lives saved — only loss aversion. There is no acknowledgement that Fink would very likely not have wanted any other patient to die, either, had she met them, much less an acknowledgement that people matter whether you have personally met them or not.

Making better choices through conscious triage is no more “playing God” than blithely abdicating responsibility for the effects of our actions. Both choices are choices to let some live and others die. The only difference is that the person who embraces triage has a chance to use their brain to improve the outcome. The suffering of the person who doesn’t receive the scarce resource is no less because you, personally, haven’t witnessed it. When Fink saw Nathalie’s suffering, it should only have informed her as to the gravity of the situation — both for Nathalie and for those who did receive the oxygen.

There’s a popular triage-denying morality and ideology right now through which many influential people look at AI development. If I had to summarize it in one word, it would be “abundance” for the wonks and “win/win” for the nerds. Being a “techno-optimist” is related, and so is reviling “Moloch”. What these all have in common is taking pride and identity in having faith that we will transcend the reality of tradeoffs and limitations. A way to show off today— pre-post-scarcity— is to insist against all evidence that a conflict is an illusion, because in your mind you believe in “mistake theory”. It’s like prosperity gospel or manifesting over on instagram. Techno-manifestation.

Real moral muscle comes from navigating actual tradeoffs and limitations. In the same way that being so gifted that you don’t learn to deal with failure in school is actually a weakness later on, always positing out of the box solutions that avoid a conflict in values is dodging actual moral judgment.

The biggest pitfall of this kind of, frankly, magical thinking, is that believers project their luxury values onto AI. To be seen in fear of abundance or technology is to be petty in spirit. To pause would be to back down from the challenge of greatness and the telos that progress has been leading up to— solving every problem by changing the game so everything is possible and we never have to choose. It leads to a belief that AI will discover a sort of moral order that isn’t really there and bring it into being.

Accepting limitations is disappointing and it will require grieving the fantasy of all conflicts being solved or avoided. Nobody wants to embrace triage. It’s a horrible, brute fact up there with the reality of death. It’s harrowing that we live in a world we aren’t able fully solve, but have to be strong to help those that we can. We won’t be able to help them if we’re stuck in a fantasy about never having to choose.

There’s this joke from The Life of Brian, and it goes like this:

First, yes, this is hilarious. It’s hilarious on its face and it’s hilarious because it really be like this in social movements.

But, this joke irritates me because when people say it to me about the AI Safety world, they are often asserting that we should all just get along. That we are all the same. And that’s not true. Seemingly small differences can be extremely important.

Consider first that social movements are about raising awareness and changing society. Often they are about bringing up entirely new classes of concerns. People think that, for instance, PauseAI should be the same as Stop AI because this is a new issue at the edge of the Overton window— they think it’s all “anti-AI”. And this represents progress. We’re lucky most people don’t think of “AI Safety” and “AI” as the same thing anymore!

So most people don’t have room in their concept of AI Safety as a cause for the important disagreements that PauseAI has with Stop AI. PauseAI doesn’t do anything illegal. Stop AI was founded in part because I removed two of the founders from PauseAI for pushing to do illegal civil disobedience actions. This is an important disagreement that requires us to be different. Given that we are different, I also resist being identified with a different group that does different stuff and being treated as if we are all the same. Or, most annoying, being treated as if we are contradicting ourselves because of things a different group does. Sometimes people will judge the very ideas of pausing or stopping AI negatively by the fact that they see what they interpret as “not having our house in order” because there are multiple AI Safety organizations disagreeing with each other. But this is like saying the United States need to get its act together because we have multiple political parties. The principle is not different because AI Safety is smaller and they know less about it.

A lot of people hear “PauseAI” and “Stop AI” and think I just repeated myself. I understand— I also would not be attentive to every distinction in a domain I don’t know. But this, too, reflects an important difference. PauseAI is officially agnostic about what happens after the Pause. We could get it together and have safe AI with equitable and wise governance. We could find out with more study that it’s actually not as dangerous as experts feared. We could end up having to stay paused forever as it became increasingly clear that no form of superintelligence was safe. But Stop AI is anti-superintelligence in principle, and anti-transhumanism is an important value of theirs. Stop AI does not like PauseAI leaving the door open to some possibility of a safe superintelligence future. The two groups have different priorities and values, and we are fighting for different things, even though we overlap on opposing further frontier AI development today.

So we start out with this idea that all things in a category you don’t have a lot of room in your head for are the same. Then there’s this further idea that, if two things that are “the same”, they should be one. This is not so and the misconception goes pretty deep.

My grad school advisor was David Haig, the greatest genius I have ever known. He studied literally everything (our lab meeting was called “the Fundamental Interconnectedness of All Things”) and his great works were in math-lite, more conceptual evolutionary theory. He saw many things clearly that confounded traditional lenses. And his key insight, if I may attempt to distill his huge and varied corpus, was that conflict goes deep. Even systems that we think of all one thing, like a human body, and even a genome, are actually systems of multiple parts that reproduce together to the extent that have shared incentives. It’s actually fundamentally extremely difficult to maintain a situation where all the pieces of a system have the exact same incentives, and there is no skipping the nitty gritty of aligning incentives, because, if a replicator can have higher reproductive success by cheating the whole, then by definition it will be more highly represented in the next generation.

(This is an evolutionary theory restatement of the alignment problem. The solution to the genetic alignment problem in organisms is called meiosis, and you can read my explanation of that here.)

It’s not foolish to be in conflict with your siblings because you’re so closely related. You have a lot of genetic interests in common with your siblings, but you also have the most dire competition with them for your parents’ resources. There really might not be enough to go around, which is why many species of bird will push weaker siblings out of the nest. It would be in the stronger sibling’s reproductive fitness interests to have an extra sibling, but not if the cost of that sibling is dying in the nest or reducing its own vigor to have its own offspring.

Likewise, there is a very serious threat from the other orgs in your space. They compete to control the tiny amount of space dedicated to your entire cause in most people’s heads. They compete for money. They are potential liabilities to the cause area and to you. They could make serious mistakes or contain genuinely bad or dangerous people. John Cleese wouldn’t sound like such an asshole if the People’s Front of Judea are terrorists.

All of which is to say, for our purposes, conflict goes deep. Conflict is not an aberration from how things are supposed to be. Lack of alignment is deeply to be expected and can’t necessarily be waved away and smoothed over because “we’re all on the same side”. The “same side” contains multitudes. Many activists repeat the truism that “the number one threat to social movements is infighting”, but I think this is like saying “if you want to stay married, don’t get divorced” and it ignores when the right answer is to be separate and simply disagree rather than live in an unhappy marriage. Aligned movements can afford not to infight— unaligned movements will be deeply unwell if they try to proceed with a conflicted vision. Unfortunately this is just more of the fallacy that there should never be conflict or that conflict is not natural to the enterprise of working together.

The entire concept of Moloch is an assertion that minorly misaligned incentives that lead to, for example, a race to the bottom, are somehow irrational to have. But if the misalignment is genuine, it’s not easy to fix, because solutions that work for some may genuinely not be in the interests of all parties. The solution that’s usually offered to Moloch is actually just dictatorship.

So, yes, it’s funny that the Judean People’s Front and the People’s Front of Judea sound the same, and it’s funny that they hate each other. But that doesn’t make it fallacious. You don’t have to be a narcissist care about small differences, but also the differences aren’t small if that’s your world. Problems with AI are rapidly becoming everyone’s world. Very soon it’s going to seem crazy that there were ever so few organizations resisting AI danger, and PauseAI is going to feel as broad and big tent as it is. We’ll have many extremely niche organizations to deal with all the highly specific harms and impacts from AI, as we should. And we will have a diversity of opinion and interests about how to handle problems from AI. All of which is good.

Jokes are truths you’re not allowed to say directly. Marc Andreessen is slipping you a truth claim you wouldn’t accept if it were said in seriousness.

That’s why any disgusting sentiment like this has legions of “omg it’s a JOKE ffs” and “embarrassing, they can’t even tell it’s a joke 😬” on twitter. Each commenter is afraid of being the one left out, the one who doesn’t understand that this utterance exists in some scratch space outside of the real world. Duh.

But of course the reason Marc Andreessen said this was to affect reality. He’s hiding in plain sight by saying something so outrageous we’d all turn against him if he were deadass: “I have no moral responsibility. I will make technology that harms the human spirit and no one, not even the Pope, will do anything about it.” Now the audience, each in fear that they are the only one not seeing the Emperor’s New Clothes, search for ways to exonerate the man and therefore excuse themselves from standing against him. Oh, I get it. He’s saying something so offensive it couldn’t be true.

With this kind of manipulative “joke”, there’s a power gap between Andreessen and the audience, and he is preying on the audience’s insecurity. Jokes like this are a dare to challenge the speaker’s power to dictate reality when each member of the audience knows that the safest bet for themselves is to excuse the speaker— and every time the jokester gets away with it is another precedent that makes it harder to stand up the next time.

Jokes like this are how people like Andreessen test the waters, figure out how far they can go, and nudge the Overton window. The fact that he would say “I wasn’t being serious” if pressed doesn’t diminish those effects in the slightest. Sometimes, if that release valve allows people not to process uncomfortable implications of what was said, it can enhance the inoculation effect of the joke against future offenses.

Speak in crass or evil “jokes” long enough and one’s history alone will cloak one. “What do you expect? That’s how he always talks.” The audience will say, irritated at YOU for not getting that this is established rather than the offender. The same thing has happened to me when I didn’t like someone’s friend of a friend sexually harassing me at a party— I was the one being weird by not knowing that that guy “doesn’t really mean” anything he says.

The solution is not to be hip enough to magically know the speaker’s true intentions. Their deepest intentions are usually, consciously or unconsciously, to create ambiguity that they can use to their advantage. Regardless, it is not your responsibility to figure them out.

The solution is that all jokes are true. When somebody says something, either they mean it, or they are responsible for meaning it.

The more you practice this policy, the easier it is. People will try to make you feel stupid for not “getting it”, but actually you have the power when you don’t let the speaker cast their spell on you. All it takes is one person to point out the Emperor is naked.