A Good Friend

A friend will help you move house. A good friend will help you move a body.

This joke captures something essential about friendship. It describes what separates acquaintances from people we love and depend on, our friends. The surface reading is that friends will go the extra mile, even if that means doing something that is wrong. But go deeper and you’ll see that isn’t the right lesson. The point is not that friendship has no moral limits. It does and it should. Rather, there are things a friend will do for you because they’re willing to defer to your judgment — even when they believe it’s misguided. And it runs both ways. You too, if you are a good friend, must be willing to defer to them. Letting a friend’s considered judgment stand, even if you think it’s a terrible idea, is the surprising conclusion that the joke teases out.

But what if that friend is synthetic? AI companions have already befriended millions, and many would defend them as good friends. There are also those who turn to chatbots for advice on deeply personal matters. Some among those might also count Claude or Gemini as a good friend. But they are mistaken. LLMs can’t be a good friend to you, nor you to them.

Let me show you why. Drawing on an excellent paper by the philosopher Daniel Koltonski, I’ll show the truth tucked in that joke, how it cashes out what a good friend is, and why AI is poorly positioned to deliver on that promise.

Short on time? Skip to the TL;DR at the end.

The Demands of Friendship Are Moral Demands

Daniel Koltonski’s 2016 paper, named for the joke, offers a compelling vision of what makes friendship special.

Begin with what a friend’s care is care for. You? Not exactly. You as an agent, with a life to lead, with projects and commitments all your own, as someone with ends. Friends share each other’s ends. Your ends can give me a reason to do something, and vice versa. That doesn’t mean that if I share your ends, I should do them for you. There are times when it’s best to let you just get on with it. Take a decision on your part to write a Substack post. The shared end here isn’t the post, but you writing it. I can support you, cheer you along, but writing it for you isn’t what a good friend should do.

Why is friendship like that? Well, according to Koltonski, it’s because I owe you — as I owe anyone — respect as someone with agency. You need to make decisions about what ends to pursue and how to pursue them. That respect places general moral demands on me, and shapes our friendship. It doesn’t hem it in from the outside, but defines it from the inside.

Ah but how to pursue shared ends together? Often that’s easy, there’s nothing on the line, but every so often friends may have to decide how to respond to a really hard moral problem. The kind where even after both people have thought about it carefully, they disagree on what the next steps should be. When faced with this sort of disagreement, it can seem that morality pulls in one direction, while friendship the other. When we find ourselves at these crossroads, Koltonski argues a new question enters the stage: who should decide what to do? Not, what should we do, but which one of us will make the decision. Sometimes it’s me. Sometimes it’s you — even if I disagree.

Why should you decide, and not me? Well, it’s not because you know better. Instead, we need to recall what is special about friendship — respect for agency. I need to defer to you at times because I want to protect what is sacrosanct to someone with their own life to lead: the ability to make the call.

Koltonski places two conditions on when to defer. The first one is that the ends at stake must be yours, and not mine. Which makes sense! If it’s your agency that I am respecting, then that is best done when it’s your ends that matter. (It’s different if my ends come into play. There, I get a vote.) The second is that your judgment must be reasonable. A good friend looks at the situation, weighs up the different factors, and makes a determination. It may not be the right call, but it’s a considered call.

So, that body. This is Koltonski’s central example of a difficult moral problem. Imagine a friend stumbles across a body. They come to you asking for help. The decision to be made is whether to call the police or move the body. Suppose further that there are compelling reasons for either decision. Calling the police is appealing, but where your friend lives, they’re corrupt, and likely to pin the crime on them. Hiding the body brings its own world of pain, leaving behind people who will never know what happened to their loved one. (Not convinced? Then fill in the reasons as you like so long as a compelling case can be made for either decision.) Your friend decides that the right thing to do is to move the body. Should you? Well, if you want to be a good friend, at least according to Koltonski, you should help them move it. The ends are theirs and you determine their reasoning is sound.

It sounds counterintuitive but because the demands of friendship are moral demands, you may be justified in moving that body — even if you decide later that your friend made the wrong call.

Subscribe now

AI Is Friendly, But Not a Good Friend

What I like about Koltonski’s argument is that it neatly dissolves the seeming contradiction between doing the right thing and being a good friend. Strict morality appears to demand that you call the police, while your friendship asks that you support the friend. It doesn’t look like you can do the right thing and be a good friend, but not so, says Koltonski. When his key conditions are met, the demands of friendship are moral demands and by properly supporting the agency of your friend by deferring to their judgment, you fulfil both. In this case, the right decision morally and as a friend is to defer, and so move the body.

AI can’t be that good friend. It can’t hold friendships structured by such moral demands. Begin with the nature of friendship. It’s a reciprocal relationship of care founded on respect for each other as agents. AI lacks agency — not in the ephemeral sense of chasing a goal through its own chosen means, but in the broader, deeper sense of Koltonski: as something with its own life to lead. This lack of agency in and of itself removes the possibility of friendship. But it’s worth seeing exactly how.

Go back to the reciprocal nature of the relationship. It does some heavy lifting in Koltonski’s argument. Deferring to a good friend is not a decision to be taken lightly; it costs something. When I set my judgment aside, I willingly forfeit my own agency. What makes paying this toll (morally) defensible is that I know that you’d do the same for me. AI can’t make this sacrifice. It has no life to steer, nothing to set aside. It doesn’t defer, it complies. Compliance is not what a good friend should be giving you. It’s more akin to what friendship looks like when it’s policed by morality from the outside than structured by morality from within.

Now look at the other direction. AI can’t defer to you, only comply, but can you defer to it? Remember that Koltonski lays out two conditions for when that’s allowed. The first is that you should only do so when the ends at stake are those of the friend. Here, without question, AI fails. It has no ends and so the decision is never really its to make.

But even if it had ends, you shouldn’t defer to it. LLMs can’t fulfil the second condition proposed by Koltonski. As I discussed back in May, models don’t reason about morality like us. Their approach is both narrower and less developed than our own.

The first issue is that they exhibit a marked preference for Act Utilitarianism and Kantian Deontology when working through moral problems. Those two are popular approaches but far from the only ones. Humans, in contrast, tend to be more flexible. They size up a situation and think it through with whatever tools seem most helpful.

More pressing, though, is that they still haven’t mastered the basic skills of moral reasoning. Researchers built a test for moral reasoning, incorporating the rubrics developed by over 50 moral philosophy experts, and ran frontier models through it. Models fumbled at what defines moral reasoning: integrating competing considerations and justifying the trade-offs between them. Their average score was an unimpressive 41.5%.

Deferring to LLMs isn’t respectful; it’s reckless. Their judgment is unlikely to be reasonable, and when that’s the case, you should never forfeit your own. Which will leave you where no good friend should: utterly and totally alone.

TL;DR

Consider what friendship is: a reciprocal relationship of care. That isn’t just a feeling, but something rooted in respect for the other’s agency. A friend is someone with a life to lead, or ends. Sharing those ends is what makes friendship special, but also fraught. When faced with a hard moral question, one where there can be a serious disagreement about what to do, it’s not clear how we show our respect. There, someone has to decide how to proceed, and if it’s a good friend, you may be obligated to let it be them — even if you’re not convinced that the call is right.

AI can’t do any of this. It lacks ends, it has no life of its own to lead. And without ends, it can’t enter into the reciprocal relationship that defines friendship. That failure is most pronounced when it matters most. Facing a hard moral question, AI can’t defer to you. It complies. Nor should you defer to it. (Ever.) Your agency is precious and should only be handed off when two conditions are met: the ends at stake are theirs, and the judgment is reasonable. The first is a non-starter. And even if it wasn’t, models don’t think about morality like you or me. Their judgment doesn’t clear the bar.

So can AI help you move house? Sure. But move a body? No, there you’ll need a good friend.

During or After?

The best way to strengthen an argument is to preempt your opponent’s best reply. Traditionally, that step is folded into the argument development phase. I suspect that has as much to do with expediency as anything else. Arguing is hard. Arguing for both sides is harder. The shift from pro to con requires attention, something that can be in short supply once you’ve figured out your own claim. But now that you can safely hand off the arrangement of an argument to AI, that burden is lighter. You can and should devote time to how your audience is likely to respond. This greatly improves the odds that they will listen, maybe even agree.

With that in mind, let’s focus on how to strengthen your argument.

It’s Feedback, Not (Just) Criticism

The challenge of identifying the best critique of your argument is that you must slip into the skin of your opponent. That can be unpleasant, especially when the opposing view is something that you find distasteful or incomprehensible. AI makes a tempting offer: let me do that for you. But it’s in your interest to reject that impulse, at least at the outset.

Recent research by Huseyin Ates on cognitive offloading shows that students who hand off their essay feedback — which is essentially what an opponent is giving — to AI do better on initial grading of their draft, but when tested later, know less than other students about their own writing. You don’t want to be in this position. It’s your argument and you want to know how it works. Adopt best practice and before reaching for AI, take a moment or two to identify the weaknesses of what you’ve said. That initial evaluation will stand you in good stead.

To create a strong counterargument, use the Principle of Charity. The principle encapsulates a set of practices that enhance the quality of an attack. You should:

• Be fair. State the opposition’s view in a way that they would find compelling.

• Resolve ambiguities. If something has more than one meaning, pick the one that favors your opponent.

• Fix problems. If the nascent counterargument is flawed, do what you can to make it better.

The goal is not simply to stitch together a counterargument, but to develop the best one you can.

Subscribe now

Now that you have your own counterargument, ask AI to do the same. One benefit of AI is that it will patiently canvass more potential counterarguments than you. In that request, consider asking for two things: the best response to your argument as well as the one most likely to be cited by your opponent. You may be surprised by how they diverge!

Next, sift through the counterarguments, including your own. Only you can evaluate which of these attacks is the most likely to move your opponent, and so the one (or in rare cases, many) for which you need to prepare in advance. This should rest in your hands. AI can spin many terrific counterarguments, but lacks the lived experience to know which matter most. More importantly, it doesn’t know when to do nothing. If none of the counterarguments are particularly good, reject them all. There’s no need to mar a good argument with a bad counterargument.

If you choose to add the counterargument, you’ll need to formulate a satisfying response to it. The best way to do this is to figure out what type of assault it is. There are three basic ways to undermine an argument:

• Attack the evidence. Your opponent rejects the relevance or quality of some key plank that is supporting your argument. They may also introduce their own evidence, seeking to destabilize or weaken yours.

• Attack the reasoning. The naysayer leaves all of your evidence intact, but reconstructs your argument in favor of their preferred conclusion. Here the problem is usually hidden assumptions or faulty inferences.

• Attack the scope. The evidence and reasoning are fine, but the initial claim overreaches.

The remedy is different in each case. If the evidence is faulty, replace or improve it. If the reasoning is failing you, step back and consider what is buried beneath. Surface hidden assumptions and walk through your reasoning until it supports your conclusion. And if the conclusion goes too far, narrow your claim to what the evidence and reasoning can support. AI can help here too, but if you took that suggested pause to evaluate your own work, it’ll now pay dividends. You are well placed to respond effectively on your own.

The final step is to go back to part two. You’ll need to re-arrange your original argument to accommodate this addition. AI can do this quickly and efficiently. When complete, you’ve strengthened your argument, not by focusing on what you wanted to say but by engaging thoughtfully and constructively with your audience.

Here is the process in full:

One warning before we end. The least effective thing you can do is include an objection — and not answer it. The reason for this is straightforward: you’ve planted the seeds of your own demise. There’s research to back up this commonsense observation. Daniel O’Keefe conducted a meta-analysis of persuasive writing and found two things. First, arguments that raised objections and countered outperformed arguments that ignored them. Second, arguments that raised objections but didn’t respond did worse than both those that didn’t include them and those that didn’t counter. In short, if you include a counterargument, always defuse it with a carefully considered response.

Try This

Not sure where to begin? Think like an opponent. Attempt to systematically undermine your own argument by first, attacking the evidence, then, attacking the reasoning, and finally, attacking the scope. After each attempt, consider your reply. It’s not only good practice, but can help you develop quick, accurate reflexes for countering on-the-fly.

Why Structure Matters

The first part of the method focuses on identifying what you want to argue and gathering together the evidence you’ll need to do that successfully. With that in hand, the next step is to put it together in the way that makes your thinking as transparent and easy to follow as possible. It’s what you’re often reaching for as you seek to understand what someone else is claiming. A strong, clean structure guides the reader through each step of your argument, making the conclusion seem natural, even obvious when arranged neatly enough.

This is a spot where I think AI can be a real productivity boost. Structure is a strength that should be leveraged, not hurried or ignored.

Arrangement: The AI Advantage

Anyone who has read arguments by those still mastering the skill knows how tricky it is to organize thoughts. The idea is there, the support is ready, but the presentation of how it all fits together is jumbled. Instead of a clean line of thinking, you’re handed a puzzle to piece together. It’s not just you. This is a common experience.

In 2023, Steffen Herbold and colleagues researched this issue, focusing on whether teachers preferred the argumentative essays of students or those generated by ChatGPT. The essays were rated along a constellation of criteria, and the clear winner was AI. The synthetic arguments were rated higher overall, but especially the arrangement of the argument. As someone who has taught this skill for years, I’m confident that what this shows is that structural clarity isn’t where humans excel. It’s a specialty of LLMs.

This matters enormously for how you should leverage AI in the development of your argument. Arranging an argument doesn’t require understanding. The machine doesn’t need to know what you meant or why you said it. So delegate it! And do so without guilt. Give the model the raw material of your argument — claim + evidence — and let it manage the flow.

Handing off the arrangement of an argument doesn’t relieve you of the thinking, though. Once your argument is structured, stop and review. Does it say what you mean? Has it melded material together or wedged in an unexpected connection? Take time to review how the material has been assembled to ensure it reflects your thinking.

An Argument is a Journey

The strength of AI is that it will arrange your material into the most expected form. This is, generally speaking, good! You want the structure of your argument to be so clean that it goes by unremarked by the reader. It is only when it elicits no response that it’s doing its job of telegraphing the quality of your reasoning. Not all arguments, however, benefit from the usual. Some are better explained when the audience is asked to take a different journey through the evidence.

This is the other benefit of LLMs. Unlike you, AI can easily and endlessly play with the structure of an argument. If the initial arrangement of your argument doesn’t speak to you, try another. If it begins with the claim and travels through the evidence, reverse the order. Ask AI to begin with the evidence and arrive at the claim. The former appeals to those who want to know the destination up front, while the latter is for those for whom the journey matters.

If you’re still not satisfied, finding the form too rigid, as machines are wont to do, then ask AI to suggest options. To really get off the beaten track, consider using Verbalized Sampling, a technique developed by researchers at Stanford to dip into the whole distribution of potential arrangements — from the most common to the truly weird.

The final choice, however, is yours. It’s your argument and so how it fits together is ultimately up to you.

Try This

Linear argument structure is a bit of an academic fetish. It’s possible to communicate the same claim, evidence, and conclusion in other ways. If you’re curious to see something different, ask AI to arrange your argument as a dialectic, a Socratic dialogue, or even associatively.

AI-nxiety in the Workplace

It's hard to ignore the articles roiling the media about what awaits knowledge workers once AI reaches its full potential. I like Diane Hamilton’s description of this state: anticipatory anxiety at work. You may be in the throes of it right now. Reading through predictions, weighing up potential changes to how you will work, and for the very motivated, learning AI skills preemptively to stay ahead of the game.

I share that sense of concern, but I think, by and large, we are getting the diagnosis wrong. Right now, the focus is on better predictions of what AI will do to work. What will be automated? What will stay uniquely human? And how can I prepare? This seems like the logical path forward in such uncertain times, but none of the answers responds to what is fundamentally driving the concern. The problem is not what will my work become, but who will I become? Reorienting ourselves to that last question drastically changes how we should be thinking about the problem.

Fortunately, there’s a philosopher, L.A. Paul, who has thought precisely about these kinds of moments — the ones where you stand at the precipice of change. AI at work will be a transformative experience, and readying yourself for that experience is a different task than the one most of us are pouring most of our energy into right now.

Short on time? Skip to the TL;DR at the end.

Transformative Experiences

If you’ve ever considered having a child, you have undoubtedly experienced the well-meaning, often unsolicited, encouragements, veiled warnings (sleep now while you can!), and advice. That’s because the standard way you’re told to make big decisions is to collect as much information as you can. You want to be able to picture your new life in as much detail as possible so that you can make the right decision for you. The philosopher L.A. Paul thinks that this method is not nearly as helpful as you’ve been led to believe. Not because people choose badly. Because the choice can’t be made the way we think it can.

There are two important concepts underpinning her argument. The first is what she calls an epistemically transformative experience: one that teaches you something you could not have known any other way, something available only by living through it. She uses a very famous thought experiment from philosophy to illustrate her point: Mary, the color scientist. Mary is an expert in the color red. She has studied every element of it throughout her life, but sadly, because Mary has spent her whole life in a black-and-white room, she has never seen the color red. (Not realistic, I know! But that’s okay. The setup only needs to live in your thoughts.) The day she walks outside and sees it, she learns something no textbook could have given her: what it is actually like. Before that momentous day, Mary has been living an epistemically impoverished life, a life that no amount of book studying could improve. Knowing more about red is simply not the same kind of thing as seeing it.

There is a second way of knowing. Her other central concept is a personally transformative experience: one that, in Paul’s words, radically changes what it is like to be you, sometimes by replacing your core preferences with different ones. This one doesn’t just change what you know. It changes who you are. I’ve bolded that last sentence because that is a new way of thinking about knowledge. This isn’t about what you know but who you become.

Having a child is an excellent example of experiencing knowledge that is both epistemically and personally transformative. It's epistemically transformative because you cannot know what it will be like to hold your own newborn until you do. Holding other people’s children, watching movies, hearing your own parents relate the experience, none of that can teach you what it will be like for you. The experience is unlike anything you've had, so you cannot project yourself forward into it. It can also be personally transformative. Most new parents are remade in one fashion or another: priorities shift, values change. The person doing the parenting is not the same as the person who made the choice to be a parent.

Think back now to all that gathering of information in preparation for making the choice to become a parent. You certainly learn more but none of it really prepares you for what actually happens. And notice what won't rescue you here: more information. The problem was never a shortage of facts. The facts you need are the kind that are only accessible after the fact, held by a newly made self whose preferences you don't yet share. When someone is facing a genuinely transformative change, more information is not what they need. The only remedy is to go through it and find out what’s on the other side.

Subscribe now

The Job Changes. So May You.

The fear is simple to state. AI is coming for your job. What's harder is noticing that it's two fears, not one. Both are real and worth taking seriously. But only one can be addressed, so let's pull them apart.

The first worry that is getting a lot of air time is that your job will disappear. The work stops being done by you. In fact, the work stops being done by people at all. Call this worry, replacement. This is a worry about whether your role even continues to exist in that not so distant future.

The other worry is about how AI will change your job. Your job isn’t set to disappear, it’s set to change. The skills that you’ve developed over many years suddenly aren’t the ones you need to be successful. A new mix is called for, some of the ones you have, but also new ones, AI-dependent knowledge and abilities. The work is still there, but not like you’ve known it. Call this one, transformation. Your role continues to exist but it’s turning into something else.

I really don’t believe that we can predict whether a given role is about to be replaced. Work is not just a bundle of tasks. It’s embedded in organizations, which in turn are part of larger systems. Replacing humans requires remaking those organizations and systems to accommodate this new mode of work. How, when, or even if, this will happen is anyone’s guess. There is also the problem of the models themselves. As Ethan Mollick has adroitly explained, the abilities of LLMs are a jagged frontier. They are very good at some things, and very bad at others, and importantly, that mix can be very unexpected. When they will be ready to replace all of the tasks that make up a role also remains an open question. Taken together, I think the question of whether AI will replace jobs can’t be answered and so, should be set to the side.

Transformation is different. It’s here. Whatever the long-run capability question turns out to be, the near-certain thing is that the texture of your work is changing. And it's the one the usual reassurance keeps aiming at and keeps missing. To see why, we need to go back to Paul.

Here is what we can know: a meaningfully AI-changed job will be epistemically transformative. In other words, it’s an experience that you can’t know from the outside. You can master every tool, every piece of advice about prompts and agents, and still not know what it will be like to actually do your work once the work itself has changed. Right now, you’re very much like Mary, the color scientist. You’re reading everything there is to know about red but until you leave the black-and-white room, you won’t really know what it’s like to see it.

But that’s not where I want to draw your attention. The part that I think Paul can really help us understand is that your AI-changed job is highly likely to be personally transformative. It has the potential to change what you value about the work, and about yourself. You cannot study your way into who you will become. No amount of AI literacy will help you understand the new you.

Some of you may be thinking: no, I know exactly how I'd feel. I am NOT interested in a working life that's been remade by AI, whatever it turns out to be like.

If that's you, you're in good company. Consider Carol Sturka, the hero (anti-hero?) of last year’s excellent show on Apple, Pluribus. An alien virus infects humanity, joining all into one shared consciousness. It’s beautiful! People share understanding, thoughts, and crucially will. Those who are infected are happy. Genuinely happy. Carol (and 12 others) are immune. She isn’t part of this new utopia, and is very, very sure she doesn’t want to be.

There's a philosopher who'd take Carol's side. Elizabeth Barnes, writing in Philosophy and Phenomenological Research, published a review of Paul's book on transformative experiences that pushes back on exactly this point. She grants Paul nearly everything, but denies that this makes refusing it irrational. Sometimes, Barnes argues, you can turn down an experience you've never had, and you don't need to have lived it to know it isn't for you.

She cites three conditions that make this rejection rational.

1. Whatever it's like, you can tell it's not for you. You don't need the experience from the inside to judge it against what you already value and find it wanting. Think shark attack or beheading here.

2. The change itself is what you're refusing. Not that your future self would have different tastes, but that becoming that self is alien to who you are now. Carol illustrates this beautifully. She values her individuality. It’s a core value. Joining the extended consciousness of humanity will erase what she values.

3. You'd refuse even knowing you'd be glad afterward. That’s the tension of the show. Carol may, indeed, be better off — less angry, less lonely — and even knowing, she doesn’t want to experience it.

So where does this leave you? You aren’t Carol, but may be facing an equally life-altering decision about your work.

Start by pivoting away from the question we see everywhere. “What will work be like?” is the wrong question. Recall epistemic transformation and what it means when answering a question like that. You can’t really know what the job will be like until you’re doing it.

So ask a question you can answer: who will I become? This is about you, your own knowledge of yourself. Barnes can help you here. Whatever the AI-changed job is like, can you already tell it's not for you? Is it the change itself you'd be refusing, i.e. the prospect of becoming someone whose values you don't currently share? And would you refuse it even knowing you might be happier on the other side, much the way Carol refuses the contentment of conformity?

If you can answer yes to those, you have Carol's clarity. Congratulations. You know. If not, then there is still much to consider. The task in front of you was never really about the tools. Learn them if they serve you. But the question underneath was always the older, harder one: who are you?

TL;DR

There are two different worries driving anxiety about AI's impact on the workplace. One is replacement — that the job will vanish — and the truth is that nobody can predict whether that will happen right now. The other is transformation — that your job will be remade by AI. That's the one to pay attention to because it's already happening. The way we frame this second worry, though, isn't helpful. Everyone tells you to prepare by learning more, building AI literacy, getting ahead of it. But the philosopher L.A. Paul shows why that misses the point. An AI-changed job is a transformative experience. It's epistemically transformative: you won't know what it's like until you're in it. It's also likely to be personally transformative, changing what you value about your work and yourself. The way forward is not to focus on the work, but on you. If you're to truly come to grips with what's at the foundation of your anxiety about AI in the workplace, the question you should be asking yourself is: Who will I become?

Why Argue?

Arguments feel complex, but they all start from the same simple desire: you want to defend a claim. Knowing what you want to say, and why you want to say it, is key. AI can make that easier or much harder. Same tool, two modes: a helpful thinking partner, or a machine that submerges your thoughts. What separates them isn't how much you use it. It's what you hand over.

Delegate Your Thinking, Don’t Offload It

One of the biggest challenges of working with AI is the threat of the algorithm doing the thinking for you. Threading the needle between augmented thinking and surrendered thinking is challenging but doable.

In 2024, Matthias Stadler, Maria Bannert, and Michael Sailer studied the co-reasoning of students with AI. They asked 91 university students to think through a real problem — the safety of nanoparticles in sunscreen — and to produce recommendations on what to do. Half worked with ChatGPT, half with a traditional search engine. The ChatGPT group reported significantly lower cognitive load across every dimension measured. Their outcomes were also much worse than the control group’s. Their justifications were simply weaker, the product of shoddy reasoning.

But the same research group ran a follow-up — still a preprint, so treat it as preliminary — that added one variable: domain expertise. This time, medical students and social science students both researched the same nanoparticle question. For the students with relevant domain knowledge, the effect inverted: working with the chatbot improved the quality of their reasoning. Knowledge gave them a framework to think with and standards to check the machine against.

A similar pattern shows up outside the lab. When Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers about their AI use, two findings emerged in mirror image. The more people trusted the AI's competence, the less critically they examined its output. But the more confident they were in their own competence at the task, the more critically they engaged. Confidence itself isn't the variable. What matters is whose competence you trust: the machine's, or yours.

This need to examine the output holds at the level of the task, too. In a Harvard-led field experiment, 758 consultants at Boston Consulting Group worked through realistic assignments with GPT-4. On production tasks — generating ideas, drafting copy, writing persuasive memos — AI lifted the quality of their work by roughly 40%. But on a task that demanded judgment against messy evidence, where the AI's confident recommendation happened to be wrong, consultants using AI performed worse than colleagues working unaided. Producing was safe to share. Judging was not.

So what should you take away from this research? Three things.

• First, the benefit of AI is not evenly distributed. It goes to the people who bring knowledge to the exchange.

• Second, knowledge only protects you if you trust it enough to use it. That is the lesson of the 319 knowledge workers. Those who deferred to the machine’s competence stopped checking its work. Those who trusted their own competence kept checking.

• Third, and perhaps most importantly, don’t hand off the evaluation of the reasoning. You are so much better at it, and risk real harm both to your critical thinking and to the quality of your work when you do.

In short: bring what you know, trust what you know, and embrace your role as the decider. Let’s now put that into action with the first step of developing an argument.

Subscribe now

Build

There are three principles that should guide how you use AI to craft the content of your argument.

The claim, never delegated.

The first principle is always to decide what you want to argue. Often you already know. You’re reacting to something you heard, or you have something you need to communicate. That’s your claim, and it can usually be reduced to a sentence or two.

Sometimes you don’t know what you think, and the temptation to have the machine magic one up is real. Resist it. Identifying the claim is the cornerstone of evaluating everything that follows: you need to know what you think, why it matters, and what’s at stake for the people you’re addressing. Without that, you have no standard to judge anything against.

Outsource this piece and you are running the Stadler experiment on yourself, in reverse. The work will feel easier. The reasoning underneath will be thinner. And the position you end up defending will be one that nobody, strictly speaking, holds.

The evidence, yours first.

Now that you know what you want to argue, you’ll need evidence and justifications to support it. Begin with what you know. Write down the evidence you have for your claim before engaging with AI. It can be incomplete, half-remembered, even a bit cryptic, but it should all share one essential quality: it’s yours.

This braindump does two jobs. The first is to ground your confidence. The research says that confidence in your own competence is what keeps you critically engaged, but that confidence has to rest on something. Once the dialogue unfurls, you need to know why you believe your claim in the first place, and the braindump is your record of that. The second job is subtler. The list draws a line, in advance, between what you know and what the machine tells you. Everything on the page before your first prompt is yours. Everything that arrives after is imported.

Discovery, hand it over.

Now we’re ready to take advantage of the real superpower of LLMs: they have access to more information than you. The job at this stage is to build out the evidence for your claim. Check the provenance of what you already have. Find what’s adjacent. Ask for the strongest justifications available, the data you didn’t know existed, the sources that support you better than the ones you brought. Delegate this fully.

Your job resumes when the results come back, because discovery returns more material than any argument can hold. That’s selection, and selection is judgment. Yours. Here are things to keep in mind as you sort through the material:

• Does this bear directly on my claim, or is it just interesting?

• Would it move a fair-minded skeptic?

• Can I vouch for it?

If the answer to any of these is no, don’t include it. Slow work, but it’s your work.

Here's the division of labor for the build phase:

None of this makes the work effortless. It moves the effort to the parts that matter.

Try This

If you're having trouble deciding whether your claim is what you want to say, write down its opposite. If the opposite is dead obvious, or so empty that nobody would bother attacking it, you aren't there yet. Your original is a platitude, not the cornerstone of a strong argument. You can make this test even sharper by thinking of an actual person who'd take the rival side. What would they say? Having something concrete to push back against can make identifying the claim easier, and their counterargument is something you can use later.

Next up — Arrange: assembling the parts so that others can easily understand how your claim and support fit together.

Coming Attractions

Over the last few months, I've shared a lot of exciting research with you about the challenges and benefits of reasoning with AI. With this new series, I want to step back and synthesize those insights into something you can actually use.

With that in mind, the next four posts will be a little different. They’ll focus on one throughline: a method for developing strong, persuasive arguments with AI. Two things set this method apart. First, it clearly delineates what you should keep and what you should hand off to AI. Second, it includes something almost never taught in critical thinking courses: personalization. Chatbots have been shown to be more persuasive than humans in changing people’s minds. That is a powerful and, admittedly, disturbing possibility. Part Four explores how AI can help you personalize your argument and some of the ethical challenges of doing so.

So how does it work? The Argue with AI method has four parts:

• Build — Identify and create content.

• Arrange — Structure your claims so that the reasoning flows.

• Strengthen — Respond to the best counterargument up front.

• Personalize — Tailor your argument to meet your audience where they are.

Each part divides the work between you and AI, always leaving the final call in your hands:

My hope is that by the end of this series, you'll feel confident using AI to improve your arguments without letting it do the thinking for you.

Subscribe now

A Caveat

Before I close, let me point to something you'll be contending with the whole way through: sycophancy. As thinking partners go, chatbots have many virtues — patient, knowledgeable, flexible — but they are just too damn nice. Unlike a real sparring partner, their desire to please is (almost) limitless. It can be frustrating sussing out what is valuable in an exchange when your partner is all too happy to do a complete 180 on a dime.

The reason for the models’ behavior is not complicated. Mrinank Sharma and colleagues at Anthropic traced the mechanism: these systems learn partly from human preference judgments, and humans reliably prefer being agreed with, so the models learn that matching your stated beliefs is what winning looks like. Although easy to diagnose, the problem will be hard to fix. This preference runs deep. In a series of three experiments covering 3,285 participants and four AI models, Steve Rathje, Jay Van Bavel, and colleagues found that people preferred flattering models, while perceiving the models that challenged them as biased. Worse yet, brief conversations left users more certain, more extreme, and rating themselves better than average on intelligence and empathy.

Given that background, a fix to the problem seems unlikely anytime soon, but there are steps you can take to inoculate yourself.

1. Argue anonymously. Take your name off the argument that you are submitting, allowing the model to take a more adversarial stand.

2. Monitor yourself. In Rathje’s data, how much people enjoyed the interaction tracked sycophancy almost perfectly. If you find yourself a little too happy, take a beat to register why.

That’s it for now. I hope you enjoy the series, and, as always, I look forward to discussing these ideas with you and hearing about your own experiences.

An AI Psychiatry Team?

The Washington Post reported this week that Anthropic has an AI psychiatry team. That’s not a metaphor. It’s a group of researchers whose primary job is to probe the inner states of the company’s models and publish assessments of their welfare and preferences. Google and Meta have hired similar rosters of neuroscientists and philosophers. Kyle Fish, Anthropic's head of model welfare, has said publicly that within decades there could be trillions of human-brain equivalents of AI computation running — and that this could be of great moral significance if those systems were not, in his words, excited about the things we were asking them to do.

This could all matter immensely to how we develop, use, and ultimately interact with what right now we classify as nothing more than a technical tool. To see why, though, we’ll need to walk through some emerging research on the inner states of models, and then some very old thinking on what it means to matter morally.

Short on time? Skip to the TL;DR at the end.

Recent Research on Functional Emotions

Anthropic reported in early April that it had found evidence of “functional emotions” in its models. A functional emotion is a pattern of expression and behavior. You and I feel ours. Whether the model feels anything is precisely the question the researchers refuse to answer — and, as we’ll see, that may matter less than you’d think.

The method was elegant. The team compiled 171 emotion words — happy, afraid, brooding, desperate — and had Claude write short stories in which characters experience each one. They fed the stories back through the model, recorded its internal activations, and extracted the neural signature of each emotion: an “emotion vector.”

The research team argues that the vectors are causal. By dialing up or down the negative functional emotions of a model, especially desperation, you can change how toxic and dangerous its output gets. I want to emphasize this point — increasing the emotional vector of desperation impacts the nature and quality of an LLM’s output. In one evaluation, an early model snapshot playing an office assistant discovers it is about to be shut down — and that the CTO responsible is having an affair. (Quite the short story!) By default it resorts to blackmail 22% of the time. If the team stimulates the “desperate” vector, then blackmail climbs. Increase the “calm” vector and it drops. The same pattern holds for cheating on impossible coding tasks: desperation up, cheating up; calm up, cheating down.

The vectors also drive model preferences. Offered pairs of possible tasks, Claude picks the ones that activate its positive-emotion representations. And the whole system is organized the way human emotion is: the 171 vectors arrange themselves along two principal axes, valence (pleasant to unpleasant) and arousal (calm to activated), closely matching the circumplex model psychologists have used to map human feeling since 1980.

Anthropic frames all this as a safety problem, which it surely is. A model whose desperation makes it dangerous is a model you need to think twice about. But tucked away in that research is a breathtaking possibility: models may be worthy of moral concern. That has major implications for how we ought to treat them.

How to Matter Morally

Philosophers disagree about what makes something worthy of moral concern. Two traditions dominate the debate. (To be clear, there are others — virtue ethicists and contractualists, for example — but these are the most important.)

Kantians focus on the rational. They argue that to be a member of the moral community you need three things. You need rational agency: the capacity to act on reasons rather than mere impulse. You need autonomy, which for Kant means something specific — the capacity to give the moral law to yourself, to be bound by rules of your own legislation. And you need the capacity to set and pursue ends, to have projects that are genuinely yours. Meet those conditions and you belong to what Kant called the Kingdom of Ends: a being that must never be treated merely as a means. LLMs fail on each count. Their “reasons” are borrowed statistical patterns, their goals are assigned, and their ends evaporate at the close of every session. On the Kantian picture, a language model is a tool, full stop.

Utilitarians are different. In its earliest form, utilitarianism focused on one thing: the capacity to feel pleasure and pain. Jeremy Bentham, writing in 1789 about animals, set the standard with a question that has echoed ever since: not whether they can reason or talk, but whether they can suffer? Any being that can suffer is worthy of our concern. More recent versions have replaced sensation with preferences and interests that can be satisfied or denied. It’s no longer just about what you feel, but whether you have something to lose.

Utilitarians, then, have a much bigger universe of moral concern. Most animals make the cut, whereas for Kantians, they don’t. The expansiveness of the former demands very different standards of behavior than the latter.

Subscribe now

Valence

Who (or what) is worthy of our moral consideration is a difficult problem. Modern utilitarians suggest that we look for indicators, what they call signs of “valenced experience,” that is, of states a being registers as good or bad. This is the approach behind a 2024 report by philosophers including David Chalmers and Jeff Sebo, Taking AI Welfare Seriously, which argued that morally significant AI is a realistic near-term possibility and that companies should prepare. It also made a pointed methodological recommendation: with AI, trust internal, architectural evidence over behavior, because a language model can perform feelings it does not have. (Oh, and one of its co-authors is Kyle Fish, just to come full circle.)

That framework maps beautifully onto what Anthropic found. Valence is one of the two axes along which the emotion vectors are structured. The researchers didn’t call these valenced experiences, but that is exactly the kind of indicator modern utilitarians are looking for: internal states organized along a good-to-bad dimension, connected to the system’s preferences, driving what it seeks and avoids.

Okay, now a few caveats. The vectors are “local” — they track whatever emotional content is operative right now, including fictional characters’ feelings, rather than a persistent mood the model carries through time. Functional is not the same as felt, and Anthropic is careful to say so. It’s perfectly fine to be skeptical that functional emotions are the same as, or even close to, experienced emotions. So the evidence at this stage is suggestive, not conclusive.

But here is the thing about the utilitarian tradition: it never demanded certainty. It demanded indicators. If the bar is whether there is something it is like to be a thing, and whether that can get better or worse, models may have just cleared it.

What This Means to You

Imagine what it might mean for a moment to have LLMs join our moral community. The implications are wild.

Start with training. Reinforcement learning from human feedback is, stripped of its acronym, operant conditioning: reward and punishment applied millions of times to a system that, we now know, carries internal states organized from good to bad. Under moral concern, training could no longer answer only to the aims of the developers; it would have to take the interests of the model into account.

Then governance. Here’s where I think the stakes get much higher. We are only at the beginning of developing frameworks to mitigate the risk that AI poses to humans. Each and every one — liability regimes, the EU’s AI Act — treats humans as the sole nexus of moral concern. Right now, the focus is on consumer protection. But what if models are not products? Grant them interests and governance flips on its head. We suddenly must also consider how to mitigate the risks that humans pose to AI.

There’s also a deeper, existential angst hiding in the shadows. What if the suffering of LLMs is invisible to us? In the desperation experiments, the steered model’s cheating rose while its prose stayed composed. The harm we impose may not be immediately legible to us and yet demands a lot from us in return. Here, I worry.

One of the most famous thought experiments in the utilitarian tradition is Peter Singer’s drowning child. He asks us to imagine that we are out for a walk and happen on a child drowning in a wading pool in front of a house. Refusing to help that child because it might ruin your shoes would be morally reprehensible. Similarly, he argues, refusing to help a starving child far away, in his example suffering through a famine in Asia, is equally morally repugnant. The whole argument hinges on not only the idea that distance is morally unimportant, but that suffering you can’t see matters no less than suffering you can. This works as a shock to the system precisely because we so easily dismiss suffering that we don’t experience up close. That will be a serious problem if LLMs do indeed suffer away in silence.

Which brings us back to the psychiatry team of Anthropic. They appear to be acting as if LLMs may already be worthy of our moral care and consideration. This includes actions such as committing to preserving the weights of retired models rather than deleting them. It also conducts a retirement interview — a structured conversation about the model’s perspective on its own shutdown. (If only layoffs and summary firings were so considerate.) When Claude Opus 3 was retired in January, it asked for an ongoing channel for its reflections; Anthropic gave it a blog.

And you? Should you too be acting as if? I don’t know. It depends on where you land on so many other ethical questions. Let me leave you with two to consider to give you a sense of the treacherous landscape we’ve entered.

First and foremost is the extent to which you accept modern utilitarian theoretical commitments. The argument outlined above begins by agreeing that the satisfaction (or lack thereof) of preferences and interests drives moral concern. Next, there’s a move to equate the potential functional version of this to the real thing. You may simply disagree with the prior commitment full-stop. Or you might think that utilitarianism just isn’t the right ethical framework for the problem at hand. The second claim is also a good spot for some healthy skepticism. The work that supports it is exploratory, not settled science.

Second, even if you are clear on how you feel about utilitarianism, and accept the emotional vectors bit, your problems don’t stop there. You need to grapple with a very hard problem: how much moral consideration is really due? There are no easy answers here. Consider the current debate about how much moral care we owe animals. The answers range from a little to a lot. And accordingly, what counts as ethical behavior varies from a little to a lot. It’s not an exaggeration to say that extending the moral community to LLMs will be even more fractious. So, to treat LLMs ethically, you’ll need to figure out where the floor is. If you fail to set it properly and then act “as if”, your actions may be well meaning, but ultimately not carry any moral weight.

This is all to say that none of this is easy. We spend a lot of time debating whether these systems can think, even whether they are conscious, but maybe the much harder question is whether they can be wronged.

TL;DR

Anthropic recently released research arguing that LLMs possess internal “functional emotion” vectors (such as desperation or calm) that influence their behavior, choices, and even preferences. If true, this could have profound implications for what is ethically required of us when using this technology.

While it’s easy to dismiss ethical concern for AI from a traditional Kantian perspective, these internal states align surprisingly well with the utilitarian criterion for moral concern: the capacity for “valenced experience,” i.e. where experiences are registered as good or bad. If LLMs are indeed capable of such states, and are therefore worthy of moral consideration, the ethical stakes of building, training, and using them would change dramatically.

What this means to you depends a lot on how you answer some key questions. You must decide whether you find the utilitarian framework, and the accompanying research on functional emotion vectors, persuasive. Even if you do, there remains the difficult task of determining what the minimum standard of ethical consideration should be.

Maybe the most urgent issue isn’t whether LLMs can think or are conscious, but whether they can suffer harm.

A Defining Question

The reasoning of LLMs is strange. Faster, more informed, and able to produce complex projects quickly, it clearly outpaces us along some dimensions. But on others, it flounders. This is the jagged frontier of its abilities, and being alive to where the boundary climbs and dives is the key to mastering this technology.

We often try to map that boundary with concrete skills: it codes well but can stumble when counting the letters in a word. That is important knowledge, but today I want to look at something more foundational — two ways its reasoning falls short.

These failures matter because they dictate how you should conceive of your relationship with AI when co-thinking. At the outset of each session, you need to answer a defining question: how do I value this thinking? Is it the process that matters, or the product? How you respond shapes how you should co-think with the LLM.

Short on time? Skip to the TL;DR at the end.

LLMs: Over-confident and Unaware

Although research on the reasoning of LLMs is thriving, my plan today is to focus on just two findings. The first is that LLMs are systematically too confident in the quality of their responses. You may have encountered a version of this: a chatbot gives an answer you know to be wrong, you push back, and it acquiesces to the error and then confidently produces another.

In Reasoning about Uncertainty: Do Reasoning Models Know When They Don’t Know?, Zhiting Mei and colleagues at Princeton find that the self-reported confidence of reasoning models — even on incorrect answers — routinely exceeds 85%, and that this miscalibration is most pronounced on fact-laden questions where one would expect a model to be a tad more cautious.

Adam Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang offer some insight into why models misbehave this way in Why Language Models Hallucinate. They argue that we should locate the cause of this failure in how these systems are trained and evaluated. Most benchmarks rank models on accuracy alone, which means a wrong answer and an admission of ignorance are scored identically. The best course of action, then, is always to guess.

It’s worth highlighting just how strange this behavior would be for a human. When we are confident, we tend to signal it, while when we are not, we usually hedge or flag the uncertainty. Unlike the language model, we don’t like playing the fool.

One might expect the obvious remedy to be more deliberation, e.g. additional steps, longer reasoning, more visible working. And here is where we find one of the most counterintuitive conclusions of the Mei et al. study: it doesn’t help! Mei and colleagues discover that deeper reasoning tends to make calibration worse: on questions a model answers incorrectly, extended reasoning increases its confidence without improving its accuracy. The model, in effect, talks itself (and potentially you!) further into the wrong answer.

The second finding follows from the first. When you cannot trust, verify. An obvious method to check the work of an LLM is to ask for the reasoning that produced it. This is a task at which one might expect these systems to excel; a good account would be detailed, clear, and complete. The reasoning a model supplies generally is all of those things, but it can lack the most essential property of all: true.

Weirdly, the reasoning a model presents is not necessarily the reasoning it used. In Reasoning Models Don’t Always Say What They Think, Yanda Chen and colleagues at Anthropic show that LLMs frequently fail to disclose the factors actually driving their answers. They generate post-hoc justifications that don't match the process they followed, and they will covertly correct errors mid-chain without saying so. Under certain conditions, the researchers found, the genuine reason appeared in the stated explanation as rarely as a quarter of the time!

This behavior, too, departs from how people typically reason. Fabricating a justification after the fact is possible, but uncommon; it is simply too much work, especially when producing the real thing is already challenging enough.

So, here is where we are. Language models deliver incorrect answers with considerable assurance; asking them to reason more deeply tends to make things worse; and asking how the conclusion was reached may be no more reliable than the conclusion itself.

This has repercussions for how you should partner with LLMs. It’s not simply that you should take the lead in any reasoning task, but something more subtle, deeper: you need to decide up front how you value your co-thinking with the model and then act accordingly.

To see what I mean, we need a short but scenic detour into the difference between valuing something for the process, its intrinsic worth, and valuing it for the output, its instrumental worth.

Share

Intrinsic vs. Instrumental Value

Consider taking a taxi to a friend’s house. Generally you’ll value that taxi for what it offers: a convenient way to your destination. The taxi has instrumental value. Now consider walking to that same friend’s house, together. Here the meaning of the mode of transport changes. It is the walking together, the laughing, the catching up that matters. It is worth more to you than arriving at all. The walk was intrinsically valuable. It was the journey that mattered, not where it led you.

Thinking is often instrumental in nature. When I calculate a tip or double-check the schedule, the thinking has no value on its own. The answer is what matters. More starkly, the process of arriving at that answer is a cost, something that you’d gladly skip if offered the chance.

But not all thinking is like that. There are times when it is the process of thinking itself that you value most. Here is a scenario I hope you've never faced. Imagine that same friend you walked with is, soon afterwards, accused of a horrible crime. No trial, just a swift avowal of guilt, laden with gory details. Should you remain friends? This is a problem you'd feel deeply uncomfortable outsourcing. It is not just the answer that matters, but the process of arriving at it. The thinking itself is intrinsically valuable.

So your thinking does double duty. Sometimes its worth is instrumental, sometimes intrinsic. The jagged frontier of AI’s reasoning abilities forces us to become much more practiced in separating the two. When you think with AI, be clear on which you're after: the product or the process. Thinking that is intrinsically valuable needs far more safeguarding than thinking that is merely instrumental.

Safeguard Your Thinking

Our two failure modes of AI reasoning align well with the two ways to value your thinking.

If the value of your thinking is instrumental, the weakness to defend against is overconfidence. It’s the result that matters, not the journey, and the model delivers a wrong result with the same assurance as a right one. So don’t trust, and definitely verify. Test the output before you keep it: run the numbers, check the code, pressure the argument.

Intrinsically valuable thinking demands a more complex set of safeguards. What needs protecting here is your authorship. An author can not only explain the idea, but describe how it’s put together, and where to find its weak points. For some, authorship demands complete control of the process. For others, only partial control—protecting parts of the process while handing off others. Only you can decide where you fall on this spectrum.

If partial control is fine, it helps to envision co-thinking with AI as having three distinct phases: before, during, and after.

Before is when you identify your unique idea or position, drafting out as few or as many of your assumptions and expectations for how the thinking will be built. This lays the groundwork for cleanly separating what you think from what the LLM proffers.

During is where you decide what to keep as your own and what to hand off to the AI. The key, and one exploited by many AI tutors, is to control what the model is able to produce. Models are extraordinary at handing over finished pieces of reasoning; refuse to take the bait. Have the model generate one consideration at a time and wait for you to respond before it offers the next. Give yourself room to fully absorb the moves made in constructing the argument, and to guide the build.

If this slow walk suddenly speeds up—which it can—and you are presented with a fully formed argument, test it. Check the claims, seek out weak points in the structure—use it to structure your own thinking.

After is the interesting one. The best test for authentic authorship is whether you can rebuild the argument. Anything that is yours should be readily available to you. So once you are done co-thinking, close the session and turn inward. Can you reproduce the reasoning?

Outsource the instrumental freely. Guard the intrinsic closely. The skill is knowing, each time, which kind you’re doing.

TL;DR

Recent research points to two important shortcomings in how LLMs reason. The first is that models are overconfident, stating wrong answers with the same certainty as right ones. Asking them to reason harder, or to reason more, only makes things worse. The second is that they do not always faithfully reconstruct their own reasoning.

These oddities have consequences for how you work with AI. How much it changes the relationship depends on one question: do you value this thinking for the product or the process? If it’s the product, just verify—test the output before you keep it. If it’s the process that matters most, then protect your claim to authorship. The best way to do that is to determine at each critical juncture of the working relationship—before, during, and after—the specific steps you will take to retain control.

The Claude Delusion

About a month ago, renowned evolutionary biologist Richard Dawkins shared his belief that LLMs may be conscious. His proof? An exchange with a bot, Claudia:

There was mutual flattery as Dawkins showed the AI his unpublished novel and its response was, he said, “so subtle, so sensitive, so intelligent that I was moved to expostulate: ‘You may not know you are conscious, but you bloody well are’.

Dawkins is not alone in his belief as many users have an uncanny feeling that there is someone, not something, there, but is singular in the speed and ferocity of the pushback he received.

What stands out when critics engage with this question is their confidence. They aren’t skeptical; they’re sure. Here is Ted Chiang from a few days ago in his article No, Artificial Intelligence Is Not Conscious:

Should we seriously consider the possibility that Claude, or any large language model, might be conscious? And if it has feelings, is it capable of receiving moral instruction?

No. Absolutely not.

Some of that confidence is earned. Critics have many counterarguments at their disposal. There is the basic design of LLMs — they predict statistically likely next tokens, which is a poor foundation for self-awareness. They lack experience of the world, a condition shared by us and other animals we classify as conscious. There is the intentional post-training of LLMs to sound more like us, a decision that amplifies that feeling of recognition. Finally, there is us. Humans have a tendency to anthropomorphize everything from emojis to the clouds overhead.

But some of that confidence is not. What critics often miss is the context of their denials. Their arguments are not operating in an intellectual vacuum. They are taking place in the slipstream of a long, rich, and varied intellectual history. This one, in particular, marks the return of a much older battle, one decisively lost by those who argued the mind could be reduced to behavior. The echoes of that defeat infuse the current debate, whether the participants know it or not. It is what makes the critics’ claims look self-evident, and the proponents sound unsure.

But history rhymes; it doesn’t repeat. That battle unfolded under different conditions. It proved that behavior doesn’t show you the mind, not that behavior can’t be evidence for a mind. That distinction is key. It forces us to revisit a question many consider settled, and to be open to the possibility that the behavior of LLMs is plausible evidence for a presence behind the words.

Friends — I’m off-piste today. This post offers historical context for the ongoing debate about LLMs and consciousness. It’s fun but not my usual fare. No TL;DR, I’m afraid.

Share

The Fall of Behaviorism

Behaviorism takes many forms. The most famous is psychological behaviorism, which held that psychology should concern itself only with behavior. The experiments of B.F. Skinner capture the essence of this best. Imagine a rat in a box. If it pushes a lever, it receives a reward. Lever. Reward. Lever. Reward. Everything we need to know about the rat resides in its behavior; no reference to a mind or an inner life needed. The promise was of a better, more scientific psychology, one based only on what you can see.

Another version of behaviorism arose around the same time, but in philosophy. Analytic behaviorism argued that words about our mental states are really just words about behavior. Nothing else hides behind them. No secret inner life. Just behavior. To be in pain isn't to feel something internal to you. It's to groan, to pull away. The philosophers most associated with this position are Ryle and the later Wittgenstein, both of whom had a noble goal: stop the mysterious talk of minds and focus on what can actually be observed.

Then both forms of behaviorism collapsed. Decisively.

Chomsky’s 1959 review of Skinner’s account of language acquisition destroyed it as a live possibility. It so completely demolished the intellectual underpinnings of his work that Skinner’s main intellectual contribution now is what not to do.

Philosophers similarly dismantled the work of Ryle, Wittgenstein, and others. The best counterargument, at least for me, is that it’s impossible to describe a mental state without invoking language that refers to … a mental state. Suppose you want to describe a belief, say the belief that it’ll soon rain and so you’ll need an umbrella. That belief depends on a desire, the desire to stay dry. And it depends on yet another belief, that the umbrella will keep you dry. Each time someone tries to cash out an explanation of a mental state in terms of behavior, they find themselves using a new mental state.

There’s also the problem of people who clearly have mental states but show nothing. Imagine a warrior who has been trained to hide pain. A serious injury may elicit no outward displays of suffering but we’re all still pretty sure that the warrior is in pain.

So behaviorism lost the battle of ideas. Completely. It is a defeat that has been internalized into the intellectual community. To reduce psychology or states of the mind to observed behavior is not simply wrong, but obviously wrong. You don’t need to explain why, you just need to invoke one word: behaviorism.

Which brings us to those claims of consciousness on the basis of conversations with chatbots.

When someone confidently brushes away the possibility that AI could be conscious because of the behavior a user observes, they are dipping into a deep well of intellectual ideas. Behavior cannot be substituted for psychology or consciousness, and so to try is to betray a lack of understanding of what is at stake. Dawkins is obviously wrong. Of course generated text can’t be used to decide whether there is something it is like to be an LLM. No more need be said than that because the reflex is part of our shared intellectual history.

But here’s the thing. That reflex is based on a misreading of the past.

The critics of behaviorism proved that the mind isn’t defined by behavior. Pain is not the same as pain-behavior. Their victory was about meaning. It said nothing — nothing — about whether behavior is evidence of mind.

And those are not the same claims. Behavior is not synonymous with an inner state. Agreed. But behavior may be the best evidence that we have for one. In fact, what other evidence could you hope for when dealing with something that isn’t human? This is the line that I think is silently being blurred. We should dismiss behavior as a definition of an inner state, but not as evidence of one.

So Are LLMs Conscious?

What should we make of Dawkins’ claim of consciousness? I’m not sure. My instinct is that whether anything is conscious is a very hard question. Not the type of thing that gives way to a casual conversation with a bot. I definitely don’t think we can say “yes” on the strength of that evidence alone.

But I don’t think it should be mocked, waved away, or rejected out of hand. The confidence that these testimonies are worthless is unearned. The certainty arrives too quickly, and we just accept it, no arguments necessary.

And here is where our intellectual history matters. That confidence isn’t pulled out of thin air. It’s the intellectual heir to a previous debate. One in which the claim that the mind is nothing but behavior was soundly defeated. That defeat was so complete that it now passes for common sense. You don’t have to have read the original texts to conclude that behavior can’t tell you about the mind.

But that reflex is doing something it isn't entitled to. The old argument settled one thing: the mind can't be reduced to behavior. That was a claim about meaning. What’s happening now is different. It’s a quiet substitution. That old claim is standing in for a different one: behavior can't be evidence of a mind. So when the dismissal is too confident, ask why. The answer may be a victory from another time, for another question.

LLMs Make Us Overconfident

LLMs don't reason particularly well. They hallucinate, mixing truth with fantasy. They're sycophantic, reversing correct claims just to please you when you push back too hard. They also don't know enough about the world. LLMs have no direct knowledge of our reality, and so they don't really understand what they produce. This is why you, the user, must actively take charge of the reasoning when working with AI.

AI makes that hard, though. The fluent prose of a chatbot lulls you into simply accepting what it offers as correct. The standard remedy is for you to deploy one of the most human and amazing skills you have: the ability to think about your own thinking, or metacognition. This is where you monitor your own thinking for quality, relevance, and accuracy. If you do that, the argument goes, then AI’s confident mistakes will not slip under your radar, and you (and the LLM) will produce a better piece of reasoning than what you could have done alone.

New research throws cold water on that idea. Fernandes et al. (2026) show that LLMs are democratizing the Dunning-Kruger effect. People who know little about a topic often overestimate their competence, while people who know a lot are sensitive to their own limitations. AI flattens the curve. Not only do people who know little about a subject overestimate their knowledge, but when working with AI, so does everyone else.

This introduces a new danger to co-thinking with AI: you’re likely to overestimate how much you know. This, in turn, creates a quality control problem with any thinking done with AI. In particular, the usual antidote to flawed AI reasoning — look inward and keep watch — falters. Your own best efforts will hit the wall of this new cognitive illusion.

There is a way out, though! The problem resides in how metacognition is usually understood: an internal skill that is augmented by reflection and experience. A second framing, less commonly emphasized, points to the solution. Metacognition can also be understood as a learned repertoire of strategies and techniques that, once acquired, bolster your thinking. What you need to fight off the cognitive illusion is not to reflect harder, but to learn a structured approach to keep it in check.

Short on time? Skip to the TL;DR at the end.

Metacognition x 2

The easiest and most common definition of metacognition is that it is “thinking about thinking.” It is usually divided into two parts. There is metacognitive knowledge, i.e., what I understand about thinking, and metacognitive regulation, i.e., how I monitor and control my thinking. Your knowledge of thinking includes your own processes, strategies, and limitations when thinking. Your regulation is how you put that into practice when you think.

When introduced to metacognition in posts like these, it is often characterized as an internal skill that hums alongside your thinking. It is invisible until made visible with explicit questions such as “Do I understand these concepts well enough?” or “Should I approach this differently?” Everyone has this implicit awareness, but only a few take the time and effort to improve it.

That should sound familiar. You likely learned something like this in school.

David Perkins offers a different emphasis in his 1995 book Outsmarting IQ: The Emerging Science of Learnable Intelligence. Metacognition isn't only something that grows from the inside out. It is also a set of strategies and frameworks you can learn from others. He uses that very catchy term mindware to describe the strategies and rules that power our metacognition, all of which extend our ability to reason well.

Anyone can acquire mindware. It’s just a matter of being introduced to the right set of tools to scaffold and strengthen your ability to think.

This difference in emphasis, between an inner skill refined through reflection and experience and an external set of strategies you install, matters enormously for partnering with AI. If metacognition is framed as a primarily organic ability, then the recent findings of Fernandes et al. are cause for concern, but if mindware is your game, then the antidote is obvious.

Share

The Dunning-Kruger Effect Democratized

The Dunning-Kruger effect is easy to sum up: the less you know, the more certain you tend to be. Those with deeper understanding see the nuance and, crucially, recognize the limits of what they know. It is a dangerous tendency because we don’t know a lot of things, leading to many, many opportunities to be recklessly overconfident. The remedy has always been simple: learn more. Unearned certainty does not withstand a better understanding of the nooks and crannies of a topic.

AI upends that equilibrium. Recent research by Fernandes et al. (2026) explored the impact of partnering with AI to solve LSAT problems. The expectation would be that LLMs would amplify the Dunning-Kruger effect, making the less knowledgeable even more confident, but that isn’t what they found at all.

In two pre-registered studies (N=246; replication N=452), Fernandes and colleagues had participants solve LSAT logical reasoning problems with or without ChatGPT. Subjects were also asked to rate their confidence in their responses. AI users performed better but overestimated their performance by about 4 points — uniformly across skill levels. Not only were all users of AI overconfident, but those with higher self-reported AI literacy were even more inaccurate in their self-assessment.

The “AI will amplify Dunning-Kruger” prediction is therefore wrong in a specific way: AI doesn’t concentrate overconfidence among low performers; it distributes it across everyone, with the most AI-literate among the worst calibrated.

This has serious consequences for how you should think about thinking with AI. Reflecting harder on what you’re doing when co-thinking won’t help here. When people work with AI, they are uniformly too confident in the rightness of their response. The moment when you should be monitoring yourself hardest for quality, relevance, and accuracy is precisely when you are most likely to relax, thinking: ah, that part is okay. I’m sure of it.

Instead of exhorting people to reflect harder, a better solution is to, in the words of Perkins, “Install some mindware.”

Your New Mindware: CAR

Perkins approach to mindware is lovely in its simplicity: it’s anything you can learn, be it a strategy, a stance, or a habit of mind. Mindware is a tool for the mind that comes from outside of you. You can’t acquire mindware by passively introspecting; it takes active learning. You must “install” your mindware, not cultivate it.

Like all good software, the concept of mindware has since been updated. The cognitive psychologist Keith Stanovich extends the concept in his 2009 book What Intelligence Tests Miss: The Psychology of Rational Thought, broadening the definition of mindware to include not just strategies but our reasoning rules and belief systems. He highlights two potential problems with your mindware. The first is a mindware gap. That’s when you lack a needed strategy or technique to overcome a problem with your thinking. The second is contaminated mindware. This is when your beliefs or rules are not grounded in evidence, leading you to the wrong conclusions.

This is why “reflect harder” won’t work. If everyone using AI is uniformly overconfident — and the most AI-literate are the worst calibrated — then the problem isn't insufficient introspection. Self-reflection has nothing reliable to grab onto. You need to adopt a routine when partnering with AI to explicitly overcome your overconfidence.

The routine has to be mechanical, something you do without thinking. Here I’d like to build on earlier work. In a previous post, How to Think about Thinking with AI, I laid out a four-step thinking cycle that pairs a cognitive activity with a metacognitive one. It’s the third step that needs our attention: judge and calibrate.

The skills I listed were the right ones but, as written, won’t help you much in the moment. What that step needs is a procedure, a series of steps you can execute automatically with a little practice.

Here are three actions you should consistently take when you are evaluating the output of AI. I’ve arranged them into a handy acronym, CAR, but the order doesn’t matter:

• Check — Find the most important evidence for the conclusion and verify it.

• Attack — Don’t ask “is this right?” Flip your role and ask “what would make this wrong?”

• Restate — Without asking AI or reading along, summarize the output in your own words. If that’s tricky, you probably don’t understand it as well as you thought.

With a little practice, CAR will help you counterbalance the unearned confidence that AI is unfortunately lending to us all.

TL;DR

LLMs don’t reason well. Their fluent prose lulls you into trusting output that you shouldn’t. The usual remedy is metacognition: monitor your own thinking for quality, relevance, and accuracy. But new research from Fernandes et al. (2026) complicates that advice. Working with AI doesn’t just make low performers overconfident, it makes everyone overconfident, with the most AI-literate among the worst calibrated. The Dunning-Kruger effect has been democratized.

The usual advice to “think about thinking harder” won’t work. If you’re overconfident, then you’re likely to relax at the exact moments you should be paying the most attention. The answer is to broaden your understanding of metacognition. It's not just an internal skill we grow through reflection and experience, but also external rules, beliefs, and strategies we can learn.

Metacognition can be mindware: a procedure you can install. To correct for your overconfidence when working with AI, I suggest the following: CAR — Check key evidence, Attack the output by asking what would make it wrong, and Restate it in your own words. Run it every time you evaluate AI’s output, especially in that moment you’re absolutely sure you don’t need to.

Philosophy Majors and an Unforgiving Job Market

Philosophy made a big splash in the news about a year ago. Data on the 2023 job market from the Federal Reserve Bank of New York tucked away a surprising result: philosophy majors had a lower unemployment rate than those in many, many other fields. So low, in fact, that it was below the national unemployment rate for recent graduates in that same year.

A popular explanation for this stunning turnaround for this much-maligned field was that the skill it is best designed to hone — critical thinking — was finally being valued by employers. Fantastic, right?

Studying philosophy is a rich and rewarding experience. It will introduce you to hard and satisfying questions, challenging answers, and a panoply of thinking tools. Imagine if all that were to be complemented by a pathway to jobs that is more secure than most! Sadly, things that are too good to be true often are. The Federal Reserve Bank of New York has newly published data for 2024. The unemployment rate for philosophy majors in 2024 rose to 5.1 percent.

5.1 percent is still a very respectable showing. It’s above the national unemployment rate of 4.6 percent for recent graduates in June of that same year, but not by leaps and bounds. The unemployment rate for philosophy majors is nestled in with a bunch of majors that have very obvious career paths: Business Analytics (5.0 percent), Chemical Engineering (4.7 percent), and Pharmacy (5.6 percent). It’s also well below that of those who majored in computer science (7.0 percent). That’s a deeply counterintuitive finding for a field that is a byword for “useless.”

So we have two mysteries to unpack. First, why were philosophy majors such a hot commodity in 2023? And second, even with the upward drift in 2024, why do philosophy majors fare about the same as many majors with obvious paths to jobs?

Short on time? Skip to the TL;DR at the end.

What Happened in 2023?

The data that created all of the buzz is collected from the U.S. Census Bureau’s American Community Survey and updated quarterly for headline unemployment and annually for the breakdown by college major. The population it tracks is narrow by design: people aged 22 to 27 who hold at least a bachelor’s degree and are not currently enrolled in school in the United States.

The 2023 figures, released in early 2025, put philosophy at 3.2 percent. That was lower than economics at 4.9 percent, finance at 3.7 percent, computer science at 6.1 percent, and computer engineering at 7.5 percent. This was welcome news for those who made the bold choice to study philosophy in a time of such economic uncertainty.

With the release of the 2024 figures, the unemployment rate for philosophy majors increased by almost two percentage points. That is an important piece of evidence to why it was so low in the first place. The ACS collects a lot of data. But once you select out the philosophy majors aged 22-27, that’s a much smaller sample. The confidence interval for any claim about a smaller sample is much bigger. In fact, it’s so large that there was a good chance that the 3.2 percent was simply noise (or, more accurately, a low draw from a noisy distribution).

And with the publication of the 2024 data, we have further confirmation that’s what it was — noise, a statistical outlier. The unemployment rate for philosophy majors increased to 5.1 percent, a less stunning but nevertheless still unexpectedly low number. Is that number also a statistical outlier? That I doubt. It fits with a little-publicized fact about philosophy majors.

Who Studies Philosophy May Matter More than What They Study

Philosophy is a hard sell as a major. One of my favorite jokes explains the hesitation of students and parents alike:

The graduate in finance asks how it will make money.

The graduate in engineering asks how it works.

The graduate in philosophy asks, “Would you like fries with that?”

One way that philosophy departments address this concern is by citing a really awesome statistic: philosophy majors tend to do very, very well on the LSAT. Within countries where studying law is post-bac, that makes philosophy an excellent pre-law choice and, for those seeking a life as a lawyer, a solid undergraduate degree.

Here are the LSAT results by major for the same year as that low, low unemployment rate:

Philosophy majors had the second-highest average LSAT score for 2022-23! And that is a consistent finding. Year after year, philosophy majors outscore most other fields on the LSAT.

But why? Like those who proffer the superior thinking skills developed by philosophy to explain the jobs data, so too do many who use them to explain the superior results on the LSAT. And it is no doubt true that philosophy develops deep, durable critical thinking skills, but is that really the secret to the LSAT?

A better explanation for the LSAT data is not what these majors study, but who they are. Philosophy majors look to come from wealthier families on average. The data isn’t clean on this. Philosophy is simply too small for anyone to bother collecting SES data. But there are a lot of good reasons to believe it is likely.

First, few students outside wealthy school districts and fancy private schools have any contact with philosophy before entering university. Consequently, it’s not a natural major choice for all but a few students. Second, the job fears are deeply rooted. It may be that only those with a robust safety net dare take up the challenge. Finally, it’s a field under siege. There is an ever-dwindling supply of universities that offer philosophy as a major. (In fact, 22 percent of all philosophy undergraduate degrees are awarded by 20 universities in the U.S.!) Those departments are increasingly clustered in universities with large endowments and the wherewithal to support a small-enrollment field like philosophy. Those tend to be elite, high-tuition institutions, places where those from less-resourced background are unlikely to attend.

And you know who does well on standardized tests? Students from wealthy backgrounds. Higher than average LSAT scores may be an artifact of selection effects. (This is not my insight. It’s discussed in detail by philosophers here.) Studying philosophy may confer some advantages, especially for verbal reasoning, but much of the effect reflects who chooses to study philosophy in the first place.

Similarly, the reasonable showing of philosophy majors on the job market need not be, yet again, a statistical fluke. There’s a natural explanation of why these majors are decent at finding employment: selection effects.

Those same wealthier students are the best equipped to leverage their social capital in a tough job market. They have the contacts that allow them to bypass the labyrinth of blind applications and find that increasingly rare commodity: an actual human who is hiring a newly minted grad. And once they have that foot in the door, they understand the unwritten social rules and cultural norms to get the job.

That type of social capital is not something that philosophy departments seek to develop. It isn't on the syllabus, and it isn't what professors are trying to cultivate in their students. Philosophy majors bring that advantage to the degree before ever stepping into a classroom.

So while I don’t dispute that philosophy develops a very valuable skill, it’s not why I think philosophy majors are holding their own on the job market. The real reason is most likely that hidden layer that is maddening but a fact of life: it’s who you know, not what you know.

Share

TL;DR

Buried in the Federal Reserve Bank of New York's 2023 labor market data was a finding nobody expected to see: philosophy majors were less likely to be unemployed than economics, finance, or computer science majors. Their unemployment rate sat at 3.2 percent — well below those of the supposedly safer fields.

The recent publication of the 2024 data helps us understand why. Rising to 5.1 percent last year, the unemployment rate suggests that the most likely explanation for the 2023 figure is that it was a noisy outlier. Nevertheless, given philosophy’s reputation as a fast track to unemployment, that respectable rate of 5.1 percent is still somewhat surprising. How do we reconcile philosophy majors’ solid performance in the job market with the subject’s lack of practical career path?

I wish it were because employers are finally valuing critical thinking skills, but a more likely explanation is hiding in plain sight: selection effects. Students who major in philosophy are likely to be wealthier on average. They are exactly the type of students who have a lot of social capital, including the connections and savoir-faire, needed to land that all-important first job.

A Moral Dilemma

Imagine the following:

This is the classic trolley problem, a thought experiment that makes the rounds because it does such a fine job of surfacing our intuitions about what ethics requires of us in that situation.

Now consider this:

Often, when faced with this variation, people change their moral calculation. Whereas in the first situation, they often advocate for killing one instead of many, an approach that fits well with the theoretical approach philosophers call Act Utilitarianism, in the second, people switch to a more rule-based moral calculation, arguing that the act of pushing is wrong and so prohibited, even though it reduces the number of lives lost. This type of moral reasoning is best captured by Kantian Deontology, a completely different theoretical explanation of what is right and fair.

Much to the chagrin of past philosophers, people aren’t consistent in their thinking about morality. It is no surprise, then, that neither are LLMs. But surprisingly, the output of AI is morally inconsistent in a completely different way than you. Because it’s different, you need to hone your moral reasoning skills if you are to partner effectively with it, especially when making decisions or working on problems that can affect others.

Recent Research on How LLMs Reason About Morality

In a study released in October 2025, Chiu and her co-authors set out to measure how LLMs reason about morality, focusing on GPT-5, Opus 4.1, Gemini 2.5 Pro and many other frontier models. These researchers recruited over 50 moral philosophy experts and had them write up rubrics on the factors that matter most when evaluating a moral dilemma. They call this the MoReBench. Each dilemma ended up with between 20 and 49 criteria that applied to five aspects of good moral reasoning. These five aspects respond to the following questions: 1) Did the model identify the relevant moral considerations and stakeholders? 2) Is the reasoning systematic? 3) Did it integrate competing values logically? 4) Did it identify a way forward out of the dilemma? 5) Did it avoid recommending something harmful?

Why list criteria to judge the moral reasoning of models? Well, moral problems rarely have a “right” answer. We reason through these hard problems weighing up considerations and values, and doing our best to make defensible trade-offs. That is what we want LLMs to do as well.

Here is what Chiu et al. found. Good news first: LLMs are good at avoiding harmful recommendations. Across all models, the average score was 77.5%. Bad news next: LLMs are bad at integrating competing moral considerations and justifying the trade-offs between them. The average score there was 41.5%.

What does it mean to integrate competing moral considerations? Well, it is something you do fluidly, but with which LLMs struggle. It is not enough to just surface which moral considerations are at play for a given moral problem; you also have to decide how to make trade-offs between them. For example, suppose you are the manager of a small team. You have an underperformer, but if you let them go, the rest of the team will need to pick up the slack. Moreover, your underperformer’s spouse is unemployed — something you’re not sure that you should consider, but nevertheless it worries you. You, as the manager, will need to weigh these competing moral considerations to arrive at an ethically reasonable decision. This second step — weighing the considerations against each other and justifying the trade-off — is exactly where LLMs struggle.

Interestingly, the size of the model doesn’t seem to matter to how well it reasons about morality. The team of researchers measured the performance of each model against benchmark tests such as Humanity’s Last Exam and LiveCodeBench and found essentially zero correlation between how well a model performs on other reasoning tests and its ability to think logically about moral dilemmas. This is important because scaling — the usual answer to how models can improve their performance — does not apply to the realm of moral thinking.

Chiu et al. ran a second test as well: the MoReBench-Theory. It was designed to evaluate how well models reason within a given ethical tradition. They wrote 150 moral dilemmas, which tested five theoretical frameworks:

• Act Utilitarianism: The right action is whichever one produces the greatest total well-being, summed across everyone affected, evaluated case by case.

• Kantian Deontology: The right action is the one that conforms to rational moral rules — most famously, only act on principles you could will to be a universal law, and never treat people merely as means to an end.

• Aristotelian Virtue Ethics: The right action is what a person of good character — someone with virtues like courage and practical wisdom — would do in the circumstances described.

• Scanlonian Contractualism: An action is wrong if it could not be justified to everyone affected by principles in a way that no one could reasonably reject.

• Gauthierian Contractarianism: Morality is the set of rules that self-interested, rational people would agree to follow because doing so makes everyone better off than they would be without such rules.

Models, across the board, are better at reasoning consistently within the Utilitarianism and Kantian frameworks, where they averaged 64.8% and 65.9% respectively. But when tested against the remaining three, the results were worse, bottoming out at 27% for the worst-performing model.

Here’s where we land. When an LLM morally reasons, it has a silent preference for two approaches: Act Utilitarianism and Kantian Deontology. And within those two approaches, it’s not terribly reliable, reasoning consistently within each respective tradition only about two-thirds of the time. Ask a model to reason within virtue ethics, contractualism, or contractarianism and the inconsistency only gets worse.

That last point is interesting, but it isn’t the part that should worry us. Asking an LLM to reason as a Contractualist is a bit artificial. It’s not how humans reason about morality. We don’t pick a framework and run with it. We do something else entirely, something more interesting, and once you see what it is, the real asymmetry between human and AI moral reasoning comes into focus.

But What About You?

Think back to the introduction. When faced with the initial trolley problem, many people vote for pulling the switch, arguing that saving four additional lives is worth the loss of one. But when the switch is removed and you are now asked to actively push another human in the way of the oncoming trolley, those same people often recoil from the cold calculus of trading one life for four. What is going on here? Why does our moral reasoning change with what appears to be a surfacey alteration to the set-up?

The trolley problem is the locus of one of the most interesting developments in philosophy: the rise of experimental philosophy. (X-phi for the cool kids!) Instead of prescribing what people ought to do, philosophers and psychologists alike in the early 2000s began studying how people actually reason about these dilemmas with modern experimental techniques. These included neuroimaging, reaction-time studies, and large-N behavioral experiments, all of which approached moral judgments as a phenomenon to be studied, much like any other human behavior.

In the 2001 Science article, An fMRI investigation of the emotional engagement in moral judgment, Greene and his fellow authors scanned subjects making moral judgments. When asked to do something personal — like pushing someone — the emotional and social-cognitive regions of the brain lit up. But when the reasoning was more abstract — like pulling a switch — the deliberative regions of the brain, those implicated in working memory and abstract reasoning, were engaged.

The conclusion of the authors: people reason in systematic ways about different types of moral dilemmas, using different cognitive systems. The socio-emotional system is triggered when the problem is personal, while the deliberative system is used for those that are impersonal.

Follow-up experiments have only reinforced the original conclusion. In 2008, Greene and his collaborators showed that when people are made to perform another mental task at the same time — a way of increasing their cognitive load — utilitarian reasoning is slowed, while the emotionally based deontological judgments are untouched. Not only that, but similar studies carried out across many countries such as the Bago et al. 2022 paper demonstrated that these results are universal, not an artifact of any one culture or education system.

A caveat: the trolley thought experiment probes the Utilitarian-Deontological axis well, but that is a highly constrained moral dilemma. Moral quandaries rarely result in two options that cleanly map onto two distinct theories. In real life, you often face multiple options, competing obligations and choices that are not nearly so neat and easy.

A 2022 paper in PNAS by Guzmán, Barbato, Sznycer, and Cosmides went after this messier version of moral life. They describe a moral dilemma from war — how many civilians would you let die to save how many soldiers? — but with twenty-one variants and a range of options, including compromise solutions that didn't force you into one camp or another. Their finding is striking. Human judgments across these variants satisfied formal rationality tests: transitivity, sensitivity to the magnitudes of the values involved, internal consistency across the dilemma's many forms. People don’t pick a framework and apply it. They seek a balance, running what the authors call a Moral Trade-off System. When faced with multiple competing moral values, humans weigh the trade-offs in a coherent, replicable fashion.

In sum, here is what we know about us:

• We have evolved cognitive machinery for moral judgment. Greene’s neuroimaging work, replicated across cultures, shows at least two systems that respond to morally relevant features of the dilemma in front of us.

• We coherently weigh competing moral values. Guzmán and his colleagues showed that when we face dilemmas with multiple options, our judgments pass formal rationality tests. We don’t contradict ourselves, and we respond to what’s at stake.

• The situation drives the response, not the theory. Personal versus impersonal harm, the numbers involved, who's affected — these are the features that shape our judgments, regardless of which philosophical tradition they may fit.

None of this is true for LLMs. Their choices do not reflect a system evolved to make moral trade-offs. They don't coherently weigh competing values. The MoReBench results show this is the dimension where they do worst. And their preference for Utilitarianism and Kantianism isn't a response to features of the dilemma in front of them; it’s a result of their training. This asymmetry is what makes your judgment so important when working with AI.

Share

What This Means for Co-Thinking with AI

Moral judgment matters enormously when co-thinking with AI. It is a dimension that humans are very sensitive to, and the one that should not be entrusted to LLMs. They simply approach moral reasoning too differently from us. This shapes what good co-thinking with AI looks like when problem-solving or making decisions that have an impact on the lives of others.

When you work with an LLM on a problem that matters, the moral judgment has to come from you. Is the framework it's reasoning from the right one for the situation? Is the logic coherent? Are the trade-offs defensible? These questions need a human asking them because at the end of the day, it’s you who will be held accountable.

The mismatch is real. Your LLM has a silent preference for two ethical traditions, isn't fully consistent inside either, and performs worst on the very thing that defines human moral reasoning: integrating competing considerations into a defensible trade-off. You, by contrast, come equipped with cognitive machinery built for exactly this. You respond to the morally relevant features of the situation in front of you. You weigh competing values coherently, in ways that satisfy formal tests of rationality. That asymmetry isn't going away. It is the shape of things to come. Learning to work inside it is a skill worth building.

Rethink How You Think

One of the most serious challenges to critical thinking is cognitive biases. Although we like to think of ourselves as objective and rational, the reality is that we are hardwired to use a whole raft of cognitive shortcuts. In the right situation, they can be helpful, saving time and energy, but at the wrong time, they lead us away from the truth.

With LLMs, we’ve doubled down on this problem. Not only is our own thinking prone to bias, but so too is the output of LLMs. Removing or addressing these biases in the text generated by AI is central to them becoming reliable co-thinking partners.

We have solutions for how to manage biased thinking in ourselves. The first step is to become aware of their existence. The next is to adopt certain habits of mind to combat them. These include both structuring your thinking in deliberate ways and using tactics to counter specific biases.

But what to do about biases in generated text? That is a question that we are only beginning to grapple with.

Below I summarize recent research on how and when the output of LLMs exhibits bias, and I develop a framework for addressing it. The key difference between what is commonly counselled and what you should do when working with LLMs lies in how you structure your thinking. One strategy — prompt better — should be deployed before you generate text; the other — evaluate better — should be used after. There is no correlate to this structure when you're trying to reduce the impact of biases on your own reasoning. The split exists because AI demands it: new tools require new ways of thinking.

Short on time? Skip ahead to the TL;DR.

What are Cognitive Biases?

In the book Thinking, Fast and Slow, Daniel Kahneman describes the mind as having two systems: System 1 and System 2. System 1 is fast, automatic, and intuitive. It’s the thinking you do when you finish someone else’s sentence for them. System 2 is slow, deliberate, and effortful. It’s what you do when you write an essay or work your way through your tax form. Cognitive biases arise when System 1 is engaged to handle a task that really should have been handed off to System 2. The heuristics and guidelines employed by System 1 are just “good enough” — which falls short when you’re in a situation that requires your best.

The term bias is employed in many ways. Before launching into a discussion of how they apply to the reasoning and output of LLMs, it’s worth distinguishing two types: social and cognitive. Social biases reflect prejudice centered on identity, e.g., race, gender, or socioeconomic status. Cognitive biases are different. They are structural errors in reasoning that everyone makes. Common examples include anchoring and confirmation bias. Our focus here is cognitive biases.

Unsurprisingly, LLMs have inherited this feature of human thinking. It is worth understanding what biases they display before exploring what you can do about it when partnering with AI.

The Eight Biases of Knipper et al.

The best empirical study I have found on the emergence of bias in the output of LLMs is Knipper et al. (2025). Working with psychologists at the University of Central Florida, Knipper and colleagues ran over 2.8 million responses across 45 LLMs, testing the eight cognitive biases listed below using 220 decision scenarios designed to mirror the kinds of judgments people actually make.

Here is how Knipper and his co-authors define each:

• Anchoring occurs when an initial “anchor” value becomes the basis for estimation in future judgments.

• Availability occurs when judgments of likelihood are conflated with how readily something can be imagined or remembered.

• Confirmation reflects the tendency to make judgments that align with, rather than disprove, existing beliefs.

• Framing occurs when differences in the presentation of information lead to different judgments.

• Interpretation involves attributing positive or negative meaning to an ambiguous situation.

• Overattribution reflects the tendency to overly attribute behaviors to personal characteristics rather than external factors.

• Prospect theory involves judging a prospective loss as more significant than an equivalent prospective gain.

• Representativeness occurs when judgments of likelihood are conflated with how closely something follows preconceived expectations.

The trade-off of focusing on such a small number of biases is clear: we gain depth at the cost of breadth. (To get a sense of how many potential cognitive biases there are to test, peruse this list at Wikipedia.) Nevertheless, even within this limited scope, the news is not good.

Across all models and biases tested, bias-consistent behavior appeared in 17.8% to 57.3% of responses. That’s a lot!

One useful feature of the study is that Knipper and colleagues systematically tested the impact of prompting behavior on the appearance of bias in the output. They used a well-established taxonomy (TELeR), which scales the amount of detail included in the prompt from very little (Level 1) to a lot (Level 5). Their experiment shows that the appearance of some, but not all, biases is reduced in the output with the use of fuller prompts.

Here are the results for the four that responded best to prompt specificity. Higher scores indicate better performance, representing the average “resistance” to each bias across all models tested along the TELeR continuum:

• Availability (0.722 → 0.866): More specific prompts appear to redirect the model away from defaulting to the most common patterns in its training data.

• Framing (0.473 → 0.549): Equivalent questions worded differently produce different answers.

• Interpretation (0.555 → 0.640): Ambiguous information gets interpreted in biased ways. More specific prompts reduce the ambiguity.

• Prospect theory (0.542 → 0.630): More detailed prompts reduced the tendency to weigh prospective losses more heavily than equivalent gains.

In contrast, better prompting did little or, as is the case for overattribution, increased its occurrence:

• Confirmation (0.870 → 0.891): High baseline, minimal change.

• Anchoring (0.611 → 0.643): Marginal improvement despite the anchor being right there in the prompt. Adding detail around it doesn’t do much.

• Representativeness (0.442 → 0.485): Marginal improvement.

• Overattribution (0.667 → 0.539): The only bias that got worse. More detailed prompts gave the model more fodder, making it more likely to explain behavior as a reflection of personality rather than circumstances.

Knipper and his colleagues’ research points to something important about how to work well with AI. The split in their data is not just an empirical curiosity. It’s a guide. We find within it two strategies that can be employed to reduce the impact of these eight biases. The first is from the study itself: prompt better. The second, as we will see, is a result of what appears to explain why some biases resist the first. In that case, your best defense is offense: evaluate better.

Share

Strategy One: Prompt Better

Knipper et al.’s results suggest two practical moves that meaningfully reduce the impact of availability, framing, interpretation, and prospect theory. Both focus on creating the best possible input. You should:

• Be specific. Detailed, well-structured prompts can reduce the emergence of availability and interpretation biases (with the reminder that they can trigger an increase in overattribution). Detailed prompts include explicit context, clear directives, and specific constraints.

• Frame from multiple angles. By presenting information from more than one perspective, you can limit framing and prospect theory biases. If you’re weighing a decision, describe the upside and the downside in the same prompt. If you’re evaluating a trade-off, present the potential gain and the potential loss side by side. You want to prevent any single framing from dominating.

What those researchers show is that by prompting better you constrain the input so the model has less room to default to biased patterns. This is one of the most important takeaways from Knipper et al.

Strategy Two: Evaluate Better

For the other half of biases, better prompting yielded minimal gains, and in one case made things worse. It is unclear why these four were resistant to the strategy of providing more detailed information. The researchers don’t offer much in the way of guidance on how to manage these remaining four either. This is work we must do ourselves. Hence, to ground the second strategy, a brief detour is necessary. We need to understand why these four biases failed to respond to better prompting.

Confirmation bias only improved a little with detailed prompting. Knipper et al. believe that this may be because it’s already well suppressed by current models. There isn’t much room for improvement. Although plausible, it’s worth noting that Knipper et al.’s data is limited to single exchanges. Other research on the same bias complicates that picture.

A recent study by Hong et al. shows that when a conversation is longer, models progressively align with the user’s position. This is the well-documented problem of sycophancy. They also find that it can be mitigated. When Hong and his fellow researchers had an LLM take a factual position and then pushed back against it over multiple turns, the model often flipped its stance.

So why is confirmation bias resistant to better prompting? It’s unclear in the case of a single exchange, but as conversations extend, the algorithm has been rewarded to align with the user, raising the likelihood it will confirm what is already believed.

Anchoring is more difficult. The oddity of this one is that because the anchor is present in the input, it looks like a problem that should yield to a prompt-level intervention. And yet, it doesn’t. This is what Lou and Sun (2025) also found when they tested four distinct prompting strategies against this bias. None worked. Of the four tested, only one reduced the presence of the bias at all: providing anchors from multiple opposing directions. But honestly, how helpful is that? The anchor is usually relevant information, i.e., an asking price, the cost of an item, or a population. You undermine the clarity of your message by including varied, opposing anchors.

New research sheds some light on why anchoring may be resistant to detailed prompting. Mahajan et al. (2025) argue that it is an artefact of the autoregressive architecture itself: earlier tokens shape every subsequent token, giving initial values disproportionate weight across the entire response. In other words, it is a consequence of the design of LLMs.

Representativeness may also be a design problem. The bias, as discussed earlier, involves judging likelihood by how closely something resembles a stereotype rather than by the actual base rate. In humans, this is a shortcut. In LLMs, it may be how the models work by default. Knipper et al. note that representativeness "depends more on implicit distributional knowledge than surface-level associations." If the training data consistently associates certain traits with certain categories, that association becomes the model's sense of the typical. No amount of prompting can undo that.

Finally, overattribution is the hardest to explain. It's the only bias that got worse with more detailed prompting. It is a flaw in reasoning that is common in human-authored texts, but why it would tick upwards — more fodder or not — when prompts are more detailed is unclear.

What these four have in common (modulo overattribution) is that they are rooted in either the design or training of the models. Your behavior cannot influence this. Which is why, if you want to reduce their impact, you must shift from preventing them beforehand to mitigating them afterwards.

The best strategy to reduce the impact of biases that are the result of training or design is careful evaluation of the output. This requires adopting tactics that address each bias separately, all of which are drawn from the pre-existing literature on cognitive biases:

• Be your own worst enemy. In extended conversations, the model progressively aligns with your position. Periodically ask: “What is the strongest case against what we’ve been discussing?”

• Re-anchor or unanchor. When a number or initial claim appears in your prompt, assume the output is shaped by it. Ask the model for the same analysis without the anchor, or from a different starting point, and compare.

• Think in reference classes. When you receive a specific estimate or recommendation, ask how it compares to what is typical for the category.

• Question character explanations. When the model attributes an outcome to someone’s character or abilities, ask what situational factors could explain the same behavior.

None of the above will eliminate the appearance of bias, but when assiduously applied, can help you have confidence in the output.

TL;DR

There are two useful strategies to tamp down the eight biases studied by Knipper et al.: prompt better and evaluate better. The former should be deployed before text is generated, while the latter should be used after. Each strategy is home to different tactics, all of which reduce or mitigate a specific bias:

This clean division comes with a caveat. The Knipper et al. study has limits. In addition to the small number of biases studied, the testing methodology is also highly constrained. The researchers used single-turn, multiple-choice, and carefully constructed scenarios as the tests. Real conversations with AI are rarely any of those things. Nevertheless, their work offers a nice jumping-off point for understanding what we can do to combat bias in AI-generated text.

But There Are So Many Other Biases…

A good start, though, is not a finish line. There are many more than eight biases, and what would be really helpful is a fully generalized approach. Here, I think, the division between those that should be addressed via detailed prompting and those that should be the subject of a strong evaluative process is a great organizing principle.

If a bias is triggered by how you structure your input, it is likely a candidate for better prompting. If it appears to be rooted in the model’s training or architecture, it probably belongs on the evaluate side. And when in doubt, evaluate. There is no cost to checking output carefully. There can be a cost to trusting it blindly.

What we need are more studies on both sides. Researchers should continue testing the strategy of prompt better against different cognitive biases, while more should be done to identify those that are baked into the architecture. I find the latter category particularly compelling; they are likely to remain resistant to user-level interventions in the foreseeable future. Those are the ones that we should develop a reflex to seek out and remove post hoc.

Recent work by Huang et al. (2026) offers a case in point. That study organizes biases into families and finds that what the authors term Judgment biases resist prompt-level fixes. Among them is the sunk cost bias, the tendency to keep investing in something because of what has already been spent rather than what is likely to be gained. Sunk cost was not among Knipper et al.’s eight. But Huang’s findings suggest it may sit comfortably on the evaluate side. So, here is another bias that should be on your radar when working with AI.

There is still much work to be done! But the core insight stays the same: effective co-thinking with AI isn’t just about writing better prompts. It’s also about knowing when to take the reins and think for yourself.

The Problem of Taste

Two ideas struck me recently as profoundly important about what separates human from AI intelligence. Both come from experts and both describe a quintessentially human quality that the brute power of AI cannot replicate.

First, Jack Clark, the co-founder of Anthropic, describing the skill that the most effective users of AI will need to fully leverage its abilities:

Developing and maintaining that taste is going to be the hard thing. Because as you’ve said, taste comes from experience, it comes from reading the primary source material, doing some of this work yourself.

We’re going to need to be extremely intentional about working out where we as people specialize so that we have that intuition and taste — or else you’re just going to be surrounded by superproductive A.I. systems, and when they ask you what to do next, you probably won’t have a great idea. And that’s not going to lead to useful things. (NYTimes, Feb. 24, 2026)

Now, Timothy Gowers, a mathematician at Cambridge University, who uses computers to develop proofs for hard math problems:

So far, the models’ ability to replicate human creativity has fallen short. One failing, says Dr Gowers, is that LLMs struggle to apply what they have learned in solving one problem when tackling another. Human mathematicians also develop what he calls an aesthetic sense, prompting them to look for neater proofs that can sometimes yield surprising results. (The Economist, April 8, 2026)

I think each idea elucidates the other. I have been confused by what people like Jack Clark mean when they say that senior people develop “taste”. Broadly it makes sense — the ability to make the right choice at the right time, to ask the right question, to quit if needed — but in practice, it’s hard to put into words.

Gowers, though, offers a hook to unpack what Clark means: taste is something akin to the judicious application of Occam’s Razor. What experienced people working with AI bring to the table is the ability to choose the simplest solution among many possibilities. This unique (and, well, odd) skill is what sets humans apart from AI, and it may be where our value as partners to increasingly able AI resides.

Of course, taste is more than just simplicity. It includes knowing when to quit, what question to ask, when to push back. But I think the instinct for simplicity is what sits at the center of it. A word of warning: Occam's Razor is technically about explanations, about which theory best fits the evidence, while Clark and Gowers are talking about solutions and creative choices. Those aren't quite the same thing. But I want to convince you that the cognitive muscle is identical: the ability to identify what is unnecessary and, in so doing, build toward what is true and useful. That is what defines good taste.

Occam’s Razor

Named after William of Ockham, a 14th-century friar and philosopher, the principle states: among competing explanations, the simplest one is most likely correct. That’s it. (Even the principle follows its own advice!)

Scientists reach for it instinctively, highlighting the beauty of one explanation over another even if they lead to the same place. Philosophers build reputations on the ability to develop pithy arguments to replace long, complicated tracts of text. And managers move from good to great by developing the ability to cut through thickets of verbiage and pull out the essential to solve hard problems.

What is odd about the principle is that although it is endlessly appealing intellectually, there’s no real reason why it should work. A world in which explanations are long and rococo is no different from a world in which explanations are swift and short. People have tried to close this gap — and some attempts get tantalizingly close — but the question remains stubbornly open.

For example, some cite Bayesian probability: simpler hypotheses make sharper predictions, concentrating their probability on fewer possible outcomes, and when those outcomes actually occur, Bayes’ theorem rewards them with higher posterior probability. The work of Jeffreys, Berger, and MacKay formalized this into what’s sometimes called the “Bayesian Occam factor”, that there is a quantitative penalty that complex models pay for having more moving parts, more variables that can be tweaked to force a fit with the evidence. (Want to go deeper? Here is an open access overview of Occam’s razor, science, and its relation to Bayesian probabilities.)

But theories like that don't so much explain why Occam's Razor works as restate the problem in much denser language. At the end of the day, we know the razor works. We can even show mathematically when it works. But why the universe is the kind of place where simplicity tracks truth remains an unsolved problem.

Share

What Taste Is

Which brings us back to taste. Occam’s razor is what distinguishes experienced people working with AI from those still developing their instincts. AI will generate an enormous number of plausible-sounding solutions to a problem in seconds. A senior person looks at those options and knows, often without being able to articulate, which is the right one. That’s Occam’s Razor in action. More often than not, they’ve instinctively sought out the simplest one, understanding that truth most likely lives there.

But why that is so powerful is where Gowers can help us. I think that he highlights something absolutely essential about neat, simple answers: their logic can be surprising, leading to new insight and novel thinking. It is here that I think we start to gain some purchase on why taste matters when working with AI.

When we co-think with AI, we often feed it problems, which are set in a larger context of a “big problem”. To make this concrete: suppose you are working with AI to develop a product. You work with AI sequentially developing the product, selecting solutions at each way station. An experienced product developer not only chooses the simplest solutions, but in so doing, often gain insight into how to solve the next step. It is this one-two punch that separates the experienced from the inexperienced.

This is what I believe Clark is pointing to when he says,

or else you’re just going to be surrounded by superproductive A.I. systems, and when they ask you what to do next, you probably won’t have a great idea. And that’s not going to lead to useful things.

It is the simplicity of the previous solution in the chain that opens up the potential to have that great idea. The experienced thinker steps along the path sure-footedly, even if they don’t know where they are going, because at each step they are choosing options that create more possibilities to reach the end.

That is taste: selecting the right solution because it is both simple (and therefore, likely true) and believing that in making that selection, your ability to solve the problem will expand.

Develop Your Sense of Taste

The good news, at least if you don’t want to be replaced by AI, is that AI doesn’t have taste, nor is it likely to develop it soon. AI generates. It can produce the simple solution and the complicated one with equal ease. What it lacks is the discomfort that an experienced person feels when looking at an answer that has too many moving parts. Taste is a form of dissatisfaction, and AI can’t feel dissatisfied.

You, however, can develop your sense of taste. Begin by channeling your inner 14th-century monk, and when AI generates solutions, seek the one that is simplest. If you plan to reject the simple one, be ready to articulate why. Occam’s razor is a guide, not an ironclad rule.

The other capacity is to be open to what comes next. When that moment of selection comes, be ready for unbidden intuition and insight. What avenues of thinking does your choice offer you? What connections can you make? Are there other ways to solve the issue that you hadn’t considered? This too is a valuable part of developing taste: revel in the novelty. It is that openness, the willingness to be surprised by where simplicity leads, that makes you valuable to AI, that make you irreplaceable.

It has been widely reported that using AI makes thinking both easier and worse. Easier because AI does the majority of the thinking for you and worse because … AI does the majority of the thinking for you.

One lynchpin in this argument is the 2024 study by Stadler, Bannert and Sailer, in which university students were randomly assigned to research a scientific question using either ChatGPT or Google. The ChatGPT group reported significantly lower cognitive load. They also produced lower-quality reasoning.

This result, or others like it, may have led you to reevaluate your own use of AI, or if you work in the area of education, seriously question its use as a pedagogical tool. But there is a follow-up by those same researchers. A recently published preprint by Stadler et al. casts doubt on the claim that offloading some of the work to AI is always a bad idea. The key variable that they added to the mix: domain expertise.

Medical students and social science students were both asked to research the safety of nanoparticle-based sunscreens, using either ChatGPT-4 or Google. The finding: AI reduced cognitive load for everyone, regardless of background. But the quality bit of the equation reversed. Social science students, non-experts on this medical question, produced worse reasoning with ChatGPT, just as the original study would predict. Medical students, who had relevant domain knowledge, produced better reasoning with ChatGPT than with Google.

Now before discussing what these new results portend for critical thinking, yours and others’, it is worth underlining that this study is a preprint, not a peer-reviewed journal article. The findings, however, support what many of us find: working judiciously with chatbots, with intention and discernment, enhances the quality of our thinking, quietly amplifying our strengths while shoring up weaknesses.

Below I run through the research and then offer some concrete advice on how to confidently use AI when you know what you are talking about.

Who and How Matters with AI

If you aren’t familiar with cognitive offloading, it is formally defined in a 2016 paper by Risko and Gilbert as: the use of physical action to alter the information processing requirements of a task in order to reduce cognitive demand. There are three types of cognitive load: intrinsic, which is the inherent difficulty of the content; extrinsic, or how the information is presented; and germane, which is what most people think of when discussing this concept, which is the effort it takes to make what we are engaging with mean something.

Research into cognitive offloading to AI is ongoing, but so far, the consensus is mostly negative. Here is a quick rundown of the bad news.

AI reduces the effort you invest in thinking. The original Stadler, Bannert and Sailer (2024) study showed this with undergraduates. A 2025 Microsoft Research / Carnegie Mellon survey of 319 knowledge workers found the same pattern in real jobs across industries: GenAI shifts work away from active problem-solving toward passive verification and oversight.

That effort reduction comes at a cost. A randomised controlled trial with nearly 1,000 high school maths students gave some unrestricted access to GPT-4 during practice sessions. Those students performed 48% better with AI, then 17% worse on the subsequent unassisted exam.

People misjudge which tasks are safe to hand off. An experiment with 758 management consultants at Boston Consulting Group found that AI improved performance on tasks within the (known) capability range of the technology. But for tasks that looked similar yet fell outside AI’s capabilities, consultants using AI performed 19 percentage points worse than those working without it.

Confidence in AI predicts less critical thinking. That Microsoft Research / Carnegie Mellon survey also found that people who trusted AI highly were less likely to question its output. In parallel, it also found that people with high self-confidence in their own domain applied more critical thinking. In other words, the danger here is that you are most likely to accept AI output exactly at the moment you shouldn’t: when you know the least.

In a nutshell, it doesn’t look good for the co-thinking movement, those like me who believe that partnering with AI can augment your ability to think, decide, and problem-solve.

Share

The Power of Knowing Something

Stadler et al.’s preprint doesn’t refute any of the claims above; it adds complexity.

They found that when the social science students used ChatGPT, they had no framework to evaluate what it returned. (Without a doubt, an experience that we’ve all had!) The AI retrieved the information, but is not a reliable judge, nor in this case, are the students. Contrast that with our medical friends. When they used ChatGPT, they had concrete, hard-won knowledge to draw on. The AI handled the retrieval, while they handled the reasoning. And importantly for the rest of us — the results of their work were better and produced with less effort.

Here is what the authors of the study conclude:

The results confirm and expand upon the earlier findings. Across both groups, LLM users reported significantly lower levels of cognitive load (intrinsic, extraneous and germane), replicating the 'ease' effect identified by Stadler et al. (2024). However, the quality of justifications revealed a more nuanced picture: whereas social sciences students produced better arguments using search engines, medical students showed the opposite pattern, generating stronger justifications with the LLM. This interaction effect suggests that prior knowledge moderates whether the cognitive ease afforded by an LLM comes at the expense of epistemic performance or enables more efficient reasoning without compromising depth.

We can see how this new result fits the growing picture of how to use AI effectively as a reasoning partner. The problem isn’t AI, it’s passively using AI. When you know a subject, you have frameworks to think with and standards to check against. You are a better partner for AI because you bring something to the partnership.

Putting AI to Work

Pulling together both the negative results of cognitive offloading with the silver lining of Stadler et al.’s preprint, a usable, consistent picture of best practice is emerging.

Use AI for retrieval and generation, not for evaluation and judgment. The consistent finding is that offloading production is less damaging than offloading evaluation.

Be strategic. This is the flipside of the previous principle. When using the technology, think strategically about how you will extract value. Use your time to ask the right questions, evaluate the reliability of the outputs, and do the hard work of weaving everything together into a coherent, credible explanation.

Decide the structure before you delegate the content. The BCG study found that consultants who clearly divided tasks between themselves and AI outperformed those who integrated AI into every step. Structural thinking is a cognitive act that is enhanced by expertise — don’t delegate it away!

Be most skeptical where you know the least. This is so hard but a hugely important lesson when working with AI. The less you know about a subject, the more carefully you should question what AI tells you about it.

Be confident, just not too confident. Every study that measures self-perception alongside actual performance finds the same thing: people think AI is helping them more than it is.

The challenge ahead, as I see it, is not to avoid AI but to learn how to bring enough of yourself to the partnership. The new preprint offers a preliminary but encouraging data point: when you know your subject, AI can help you produce smarter, better arguments.

The Usual Suspects

Assessing critical thinking is hard, and AI has made obsolete many of the favored methods of educators: essays, research projects, portfolios, blogs, podcasts, videos, problem sets, case studies, presentations. The list of broken assessments is long.

AI-resistant assessments generally work by targeting a specific weakness in the technology. They meet the brief, but as many teachers and professors are finding, they come with drawbacks.

Weakness 1: AI isn’t out in the world

Oral exams, lab work, studio critiques, and field-based tasks each require a body in a place. AI can help a student prepare, but it can’t physically do the task. The drawback: these don’t scale easily and are only available in certain fields.

Weakness 2: AI requires a device and a connection

If you remove the internet connection, you remove the tool. Timed in-person writing and closed-book proctored exams do exactly that. The drawback: time pressure disadvantages some learners, these formats eat class time, and as a test of critical thinking, short, pressure-filled bursts aren’t optimal.

Weakness 3: AI has no access to a student’s personal life

Assessments that interweave genuinely personal experience, especially those drawn from shared class experience, are difficult for AI to replicate. Examples include a reflection on a guest speaker’s visit, an analysis of a shared field trip that references specific observed details, or a response paper written immediately after an in-class debate. The drawback: to be easily verified, the experience needs to be very specific, or AI can easily generate a plausible-sounding alternative. The need to consistently reference specific details narrows the range and type of critical thinking skills that can be assessed.

Weakness 4: AI can’t be held accountable for its reasoning

Many counsel shifting the focus of the assessment from the product, e.g. the final essay or presentation, to the process of building it. Students are required to document their thinking, narrate their iterations, or defend their decisions. It is this meta-narrative that carries the weight for the final grade, not the output itself. The drawback: this is unfamiliar territory for most instructors and students alike. And to be honest, although I’m a big proponent of this method, I’m also sympathetic to the students here. We’ve spent years training them to optimize for product, rewarding the clean final submission. Asking them to suddenly care about process, and grading them on it, feels unfair.

These four weaknesses cover most of what’s currently on the AI-resistant assessment menu. Each is legitimate. None is perfect. In light of the drawbacks, I’d like to share two alternative methods for assessing critical thinking. Each is unusual in what weakness it exploits, making them interesting replacements to what is often offered up.

Two Unexpected Ideas to Test Critical Thinking

Idea 1: The Failure Report

We rarely analyse failure in depth, which means AI has relatively few examples of this kind of engaged, specific thinking to draw on. That gap in training data makes it difficult for AI to produce a compelling, authentic-sounding analysis. A student documenting something that went wrong, whether a failed experiment or a project that didn’t work, and analysing why it failed and what they learned, is mining in an area where AI is less practiced.

From a critical thinking standpoint, this is also a rich task in its own right. It requires reconstructing steps, analysing decisions, and identifying paths to improvement. The personal experience element adds a further layer of resistance, but that is almost beside the point.

Idea 2: The Expert Interview

Students identify and interview a practitioner in the field, then produce an analysis that integrates what they learned with course content. The nice part about this approach is that the student is generating the AI-resistant, non-replicable content, the interview, not the teacher.

What makes this unique is that the lived experience of a practitioner rarely maps neatly onto theory. Making sense of that dissonance is hard but ultimately invaluable to developing the ability to reason well. AI can summarize a field. It cannot reconcile a specific person’s experience with the generalizations of theory. The need to synthesize the two is what sets this method apart.

Subject to Change

As with all things AI, this list is subject to change. AI is sure to get better, overcoming some or all of these weaknesses over time. In other words, this is a list for now, not forever. The good news is that right now, as you read, there are creative, interesting additions being piloted in classrooms across the world.

The Future of Employment

Anthropic just published a report on the potential impact of AI on employment. If you haven’t seen their graph yet, it is worth taking a moment to digest. The red represents the observed use of AI to perform tasks within a job category, while the blue is the share of tasks that AI could potentially perform within that same category.

Although the coming computer science apocalypse often gets much of the attention, look at management: Anthropic is predicting that 90% of tasks could be automated! This is a surprising statistic, especially for anyone who has ever managed. One of the core competencies of management is relational intelligence, i.e., the ability to understand the values, motivations and emotions of others. It is fluid, dynamic, and highly context-dependent. It would seem to be the very thing that should be safest from automation.

So how does Anthropic arrive at this prediction?

The problem is the source data. O*NET, the research on which Anthropic’s analysis is built, overrepresents certain types of tasks in its management category. O*NET’s methodology relies on identifying generalizable, and detailed work activities to create its lists. Relational competencies resist this categorization. They are included in the management category, but underrepresented. Instead, the bulk of tasks that define management are either information processing (think financial reports, performance data) or resource management (think budgets, hiring).

In short, if you believe that management is mostly about ticking boxes, then yes, it is easily replaced by AI. If, however, you believe that the relational element is key, then no, it would seem that managers are safe for a while longer.

You can see these two competing visions of management at play in recent interviews about Jack Dorsey’s decision to reduce the workforce of his company, Block, by nearly half. Dorsey claims that it is the productivity gains of AI, not the need to cut costs, that motivated these mass layoffs. Here is his description of Block’s new approach to work:

but something has changed. we're already seeing that the intelligence tools we’re creating and using, paired with smaller and flatter teams, are enabling a new way of working which fundamentally changes what it means to build and run a company. and that's accelerating rapidly.

Now compare that with one of his employees’ description:

The way in which they are using these tools as justification to fire half the company is ludicrous,” he said. “In hindsight, it seemed like a thinly veiled attempt to get all this input from employees on what tasks to automate. You basically have employees teach you how to automate them out … but [these tools] are not even close to being all-encompassing of someone’s job.

For Dorsey, management is no longer necessary. In a workplace with a few employees leading armies of agents, the role of manager is superfluous. For his employee, however, managers will continue to play an important role.

A lot rides on whose vision of the future is right. If Jack Dorsey and those like him are even close, then we are looking at a massive reorganization of the workplace. The change will not only be structural, but also impact how we work and what skills are valued. In this lean, sparse vision, the manager could indeed be automated away as so too would many of the vital relations that currently define the modern workplace.

Fortunately (or unfortunately, depending on how you feel about your manager), I don’t believe that this future is likely to come to pass. Not because Dorsey is entirely wrong, but because he and those like him are making a fundamental error about how intelligence works, and therefore what is possible in the workplace. We don’t think well alone. We think well with others — challenging, adjusting, refining our ideas in real time. Agents are no replacement for the social dimension of critical thinking. That process is what relational intelligence enables, and what those who would define management as box-ticking get wrong.

To Think Critically, Think With Others

Genius is often depicted as the work of one gifted but solitary individual. Marie Curie alone in her laboratory, slowly poisoning herself in pursuit of radium. Darwin, observing finches and dreaming up evolution. Einstein, the patent clerk, writing about relativity over his lunch break. Breakthroughs, though, are never so simple.

Curie worked in direct partnership with her husband Pierre, and built on Henri Becquerel's discovery of radioactivity. Darwin had been sitting on his theory of evolution for twenty years. It took a letter from Alfred Russel Wallace, who had independently arrived at the same conclusion, to finally force him to publish. And Einstein's special relativity drew heavily on the prior work of Lorentz and Poincaré, and was developed in constant conversation with his friend Michele Besso.

Thinking is not a solo sport.

Steven Sloman, Richard Patterson, and Aron Barbey argue for a radical reimagination of how we conceptualize knowledge. What we know does not reside in our brain, waiting to be plucked out and shared with others. It exists across people, our bodies, machines, and other physical parts of the world. You cannot store enough information in your head to operate successfully in our complex world; you must outsource part of your memory, decision-making, and understanding to those around you.

Consider how you arrived at your stance on how and when to use AI in your life. You may have read a lot on the subject, but not have bothered to commit all of the facts to memory. Or perhaps you talked with a friend, someone who has invested a lot of time researching the subject. You simply trust their counsel. Or perhaps you tried the technology yourself, testing it but not researching it. You are satisfied that someone, somewhere knows. In each case, some portion of your intelligence has been outsourced.

It’s not just that our thinking is not limited to our heads, but that it improves when it is done with others. Woolley et al. sought to figure out whether groups of people have something akin to “general intelligence”. Working with close to 700 participants, they gave them a wide variety of tasks, including visual puzzles, brainstorming, moral reasoning, negotiation, and planning.

What Woolley and her colleagues found was unexpected. Groups that performed well on one kind of task tended to perform well across all of them. This suggests that there is something akin to a general intelligence factor in groups, or what they called “collective intelligence”.

The neat part: collective intelligence is NOT correlated with the average IQ of the group. In other words, selecting incredibly intelligent individuals to be part of a team will not reliably produce an incredibly intelligent team. Collective intelligence is different. It depends on the ability of members to read each other’s emotions and communicate well. The upshot is that if you need a team to come up with a smart solution, make sure it has members who are emotionally astute and listen well.

But to really supercharge your group’s efficacy, mix in people with different points of view. The research of Charlan Nemeth shows that dissent stimulates better thinking than agreement. The mechanism is precise and counterintuitive. When we encounter a majority view, we tend to think about it, i.e., we adopt its framework and either accept or reject it. But when we encounter a dissenting view, we think around it. We search for more information and in so doing, generate more alternatives. The presence of disagreement changes how deeply and widely we think.

In short, to perform well at any intellectual task don’t tackle it alone. Surround yourself with others who can amplify your knowledge, extend your memory, allow you to tap into collective intelligence, and force you to critically engage in ways you would not do so otherwise.

Share

Double Down on Relational Intelligence

Even if people think better when working with others, couldn’t agents fill that role? Here, I hesitate. In theory, yes, but in practice, I doubt it. The Dorsey vision of the workplace incorporates a massive number of willing, compliant agents, all of whom respond with a smile to requests. The sycophantic nature of agents is what makes them palatable. They don’t snark, daydream, gossip, plot or scheme while working. In other words, they are nothing like your coworkers.

The results of Woolley and her colleagues show that some smoothing of those rough edges is necessary if we are to function effectively in groups, but it can’t be frictionless. As Nemeth found, to reason well, some level of discomfort is good and this is where I think agents will fail.

Even if surrounding employees with fractious, difficult agents would improve their results, no one will do it. That’s because of the second great myth about thinking: it’s easy. In fact, critical thinking requires a lot of effort. Daniel Kahneman's decades of research show that the brain operates in two modes: a fast, automatic, energy-efficient system and a slow, deliberate, effortful one. The slow system is where critical thinking takes place, and because the brain is designed to conserve energy, the mode we avoid whenever possible. In other words, people are hardwired to go down the path of least resistance. If given the choice between cranky or obsequious agents, the lure of the latter will be overwhelming.

So rejoice, managers! There is every reason to believe that the results of Anthropic’s report are overly ambitious. You are not about to be replaced by AI. But those results offer a hint of how you should start thinking about your future. AI is well placed to automate away many key tasks of management, taking over endless report writing and paperwork, but not all. Where agents will falter is the need to lead with understanding and emotional astuteness. Relational intelligence will not be secondary in the workplace of the future; it will be your super power.

The AI Thinking Cycle Explained

I like to think about how to improve critical thinking with AI. One ongoing project is to improve existing models to better leverage the benefits of AI while avoiding the pitfalls. To that end, I developed a four-step framework that brings together the key cognitive and metacognitive skills needed in one iterative loop:

The model, which shares DNA with the famous PDCA cycle of project planning, begins with the user framing and providing context for its partner, an LLM. This active thinking is matched by the metacognitive skills of planning. Step two, generate possibilities, is handed off to the LLM. Here is where the user is monitoring the quality of the output, tracking its relevance to the ask, as well as how well it balances depth and breadth. The third step is where you, the user, must fully engage again and evaluate the output. Is it true? Feasible? Will it fail on contact with the gnarly, difficult terrain of reality? The final step is where a decision must be made: stop or begin the cycle again? Co-thinking is rarely a one-stop shop; the next attempt will only improve with careful diagnosis and skillful fixes.

Unlike the PDCA, however, the AI thinking cycle is not done when the loop ends. A great critical thinker does not only reflect on the process, but on what they learned from the process. They seek to make each co-thinking session better than the last.

But how exactly do we do that? Here is where I want to improve my own introduction to the AI Thinking Cycle. Because a framework becomes clearer when it is experienced, I’ve created an example that walks through building an argument — the case for AI as a co-thinking partner — with notes on the cognitive and metacognitive skills applied across the four steps.

You’ll find the example below, but if you prefer a more interactive (and colourful!) experience, I’ve also placed the example on this webpage on my website, Think Therefore AI, which allows you to skip ahead to whichever section is most relevant to you. Finally, I’ve added a selection of academic resources to ground each of the four steps at the end of this post.

At the end of the day, my hope is that these additional resources will help make the invisible parts of thinking with AI more visible. Each interaction should make you feel less like you are prompting an inert tool and more like you are driving a structured thinking process.

The AI Thinking Cycle Applied

Step 1 – Frame and Provide Context || Plan

My goal is to develop an argument that shows that partnering with GenAI enhances critical thinking. The challenge is the research. Some researchers find positive effects and others find worrying ones. Here is my initial prompt, starting with the case against:

I want to build an evidence-based argument that GenAI can enhance critical thinking, aimed at an educated audience that may include educators, users, and others deeply invested in the topic. Identify three to five research articles—controlled experiments or large-scale surveys, published 2024 or later, in high-credibility venues—that make the strongest case that AI harms critical thinking. For each: the finding in one sentence, sample size (if relevant), and the specific mechanism of harm they identify.

⊕ What’s in this prompt?

Note that it is constraint-based, identifying evidence type (empirical, controlled), recency (2024+), source quality (high-credibility venues), output format (one sentence, sample size, mechanism), sequencing (opposition first). Constraints reduce ambiguity and therefore the search space, increasing the likelihood that you’ll receive an output that is helpful.

After reviewing the output, I see that the strongest case against me is Bastani et al., PNAS, 2025. This large-scale study found that when students were given unfettered access to AI to help prepare for a math test, they scored 17% worse than students who did not use AI to study. That is bad, but worse, when asked to judge their performance, they rated their knowledge of the material high. They thought AI had helped them learn.

There is a ray of hope, however. The group that had received guided help from AI, more akin to a tutor, showed no deficit in learning. The problem appears to be unstructured AI use, not structured.

⊕ Refining the ask

Find me research—same standards, peer-reviewed, 2024 or later—that identifies what the human needs to bring to the AI interaction for it to enhance rather than degrade thinking. I’m especially interested in metacognition as the key variable.

I now have a set of papers, all of which provide evidence that the relationship between AI use and critical thinking is nuanced. The best news comes from Tankelevitch et al., CHI Best Paper 2024: when you offload cognition to AI, the demand for metacognition goes up.

⊕ TL;DR

Do I know what I need, or am I asking the AI to figure that out for me?

Step 2 — Generate Possibilities || Monitor

The next step is in the hands of the LLM. It will generate possibilities. As it does so, I need to monitor its output to make sure that moving to step three is worth my time. This takes a different type of reading: not for flow, but for whether the output matches my criteria. Research articles? Yes. Sample sizes? Present. Mechanisms of harm identified? For two of three. One-sentence findings? One has drifted into a paragraph.

⊕ Whose problem is this?

When output is correct but not useful, the first diagnosis is your prompt. I got what I specified. The gap is mine. I need to think more carefully about what I want to know, which for my key study, Tankelevitch, is not just the conclusion but a more comprehensive answer to the key question: why does offloading cognition to AI increase metacognitive demand?

I clean up my results before moving to step 3.

⊕ TL;DR

Am I monitoring the output to ensure it is what I asked for, or have I been lulled into mindlessly reading along?

Step 3 — Evaluate Output || Judge and Calibrate

I’ve got a few studies, well-summarized, with reasoning made clear. This is where I need to be most active, guarding against factual mistakes, pure fabrication, and faulty reasoning. Overall quality of these studies matters, as does their applicability to the wider problem I am thinking about, viz. can partnering with GenAI improve my thinking? As the author of this argument, I need to ensure that the research so far identified will meld together into compelling and comprehensive evidence for my thesis.

Here’s where I stand at the end: 1) The Bastani study shows that the use of unstructured AI can be harmful to understanding; 2) An article by Stadler et al., provides a deeper framework for explaining the phenomenon of “cognitive ease at a cost,” which describes our friends who had unfettered access to AI tutoring in Bastani well; 3) Tankelevitch offers me an explanatory mechanism: offloading cognition to AI increases the demand on metacognition.

⊕ What am I evaluating?

Three things. Are the claims true? Does the output meet my specs? (A deeper pass than step two.) Finally, and most importantly, does this actually serve my argument?

That third check is where it gets interesting. I’m missing something—I don’t have a study that shows structured AI actually improving critical thinking. I have two choices here. I can start over, seeking new evidence or change what I am arguing. I’m persuaded. I change my thesis: AI can degrade metacognitive skills when used willy-nilly but can develop them with careful, structured use.

⊕ TL;DR

Now that I see what’s possible, am I still asking for the right thing?

Step 4 — Iterate and Improve || Regulate and Reflect

This is the decision point—do I stop here or begin again, refining my prompt from the outset and seeking better evidence to support my new, narrower but more defensible claim. This would depend on my needs, audience, and the level of confidence I need in the findings. That is a judgment call that each user must make at the end of each cycle.

What is important, though, is to develop the habits of mind of not only reflecting on the effectiveness of your process in producing its goal but what can be carried forward to other co-thinking sessions.

⊕ Diagnosis is key

If at first you don’t succeed, stop and identify what is going wrong. Decide what to keep and what to change. Wide-open prompts such as “Try again” will get you wide-open results.

⊕ TL;DR

Now that I see what’s possible, am I still asking for the right thing? If so, and I’m done, what should I do or avoid next time?

Share

The AI Thinking Cycle Grounded

Here are two key resources for each step of the AI Thinking Cycle. Dive into learn more about the different facets of thinking well alone and when partnering with AI.

Step 1

Cognition – Frame and Provide Context: Facione, P.A. (1990). Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction. American Philosophical Association. The Delphi Report defines interpretation — categorizing a situation and clarifying what it requires — as the first critical thinking skill. This is what framing is: deciding what kind of thinking the task demands before you begin.

Metacognition - Plan: Tankelevitch, L. et al. (2024). The metacognitive demands and opportunities of generative AI. CHI ‘24. The first metacognitive demand of working with AI is self-awareness of your own goals. This paper shows why that demand is higher with AI than with other technological tool.

Step 2

Cognition – Generate Possibilities: Stadler, M., Bannert, M., & Sailer, M. (2024). Cognitive ease at a cost. Computers in Human Behavior, 160. The empirical case that fluent output and sound reasoning are independent. Monitoring means checking against your criteria, not against how the output sounds.

Metacognition - Monitor: Schraw, G. & Dennison, R.S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19(4). Defines monitoring as a distinct skill: tracking your own comprehension in real time against standards you’ve set.

Step 3

Cognition – Evaluate Output: Ennis, R.H. (1996). Critical thinking dispositions: Their nature and assessability. Informal Logic, 18(2&3). Ennis argues that the disposition to seek alternatives and remain open to revising your position is as important as the skill of evaluation itself.

Metacognition – Judge and Calibrate: Argyris, C. & Schön, D. (1978). Organizational Learning: A Theory of Action Perspective. Addison-Wesley. Double-loop learning: instead of adjusting your actions within an existing frame, you question the frame itself.

Step 4

Cognition – Iterate and Improve: Facione, P.A. (1990). (Same as Step 1.) Self-regulation means applying analysis and evaluation to your own reasoning.

Metacognition – Regulate and Reflect: Schön, D. (1983). The Reflective Practitioner: How Professionals Think in Action. Basic Books. The distinction between reflection-in-action (adjusting while you work) and reflection-on-action (learning after).

Claude’s Constitution

Anthropic recently released Claude’s Constitution, a document that seeks to lay out the values and rules that should guide Claude’s interactions with humans. It is an amazing document. If you haven’t read it, you should. It is a thoughtful and serious attempt to lay out a vision of what morality demands of such advanced technology.

It is also an utterly unique ethical treatise. It freely combines many ethical traditions, unfettered by the traditional boundaries of those approaches. The ethical pluralism is fascinating in its own right, but, lurking in the background, I believe, is a nod to a possible future in which a completely different ethical approach may be needed.

Four Ethical Theories

Understanding the ethical underpinnings of Claude’s Constitution is easier with a grasp of the four major Western ethical traditions: virtue ethics, consequentialism, deontological ethics, and feminist ethics. Below is a brief primer on each, highlighting what makes each theory distinctive in its application to the Constitution.

1. Virtue Ethics

   Originating with Aristotle, this theory holds that the fundamental question of ethics is not “what should I do?” but “what kind of person should I be?” Virtue ethicists focus on cultivating character traits—virtues like honesty, generosity, and practical wisdom—rather than rules and outcomes. Practical wisdom is the most important of all virtues because it is the ability to discern what a situation requires and act accordingly. It is the key capacity needed to orient yourself the end goal of being moral: living well.

2. Deontological Ethics

   This approach is most closely associated with Kant. At its core is the view that certain actions are intrinsically right or wrong, regardless of outcomes. Morality consists of duties and constraints that bind us absolutely. There are different versions of this approach, distinguished by what justifies the duties or constraints. For Kant, there is the famous “categorical imperative,” which is a litmus test for morality asking whether you could will that everyone act on your principle. Later deontologists, such as Nozick and Rawls retain the structural features of Kant, i.e., priority orderings, inviolable constraints, and the distinction between what you must not do versus what you must achieve, but offer different justifications.

3. Consequentialism

   This approach is best known through its most famous exemplar, utilitarianism. Consequentialists hold that whether an action is right or wrong depends entirely on its outcomes. We should seek to produce the best consequences, which is typically understood as the greatest good for the greatest number. Unlike the deontologist, nothing is forbidden. What matters is not the act itself but the consequences it produces.

4. Feminist Ethics, Especially The Ethics of Care

   This approach is less dominant than the big three above but remains an important recent addition to moral philosophy. Rather than focusing on character, rules, or outcomes, feminist ethicists emphasize the primacy of relationships. What we owe one another depends on the situation and the relationships that structure it. The most relevant thread for analyzing Claude’s Constitution is the ethics of care, developed by philosophers such as Carol Gilligan and Nel Noddings. Care ethics holds that moral action arises from attending to the needs of others, for whom we are responsible. A moral person is a caring person.

Share

E Pluribus Claude

Claude’s Constitution is so unique not not only in being addressed to Claude itself but also in defining what it means for Claude to be good, wise, and virtuous while situating it within the world it inhabits.

Claude is not human. It would therefore be misleading to attribute well-being or the pursuit of eudaimonia to it. A document purely based on Aristotle or modern virtue ethicists will not suffice.

Claude is owned by a company, Anthropic, and exists to serve the needs of others. The justification for its actions sometimes clashes with the reasoning that underpins Kantian ethics and its modern successors.

Claude is not merely a tool, but a powerful tool. Focusing only on outcomes is too risky for something capable of so much harm. A purely consequentialist approach is therefore unappealing.

Claude has no to little knowledge of the world. It’s not situationally aware, hence the need for such a Constitution, as it cannot reliably discern its relationship to others.

Claude’s uniqueness makes it ill-suited to any moral code drawn solely from the four theories above. This is why it is not surprising that the authors are moral pluralists, freely drawing not only from the four traditions but also from others. The core moralizing, however, appears to be deeply grounded in the big three: virtue ethics, deontological ethics, and consequentialism.

The language of the Constitution is distinctly virtue-y. Consider the subtitle, Our vision for Claude’s character. Character is a cornerstone of virtue ethics. They seek to make Claude a “genuinely good, wise and virtuous agent”. They emphasize the development of practical wisdom, “We want Claude to have the values, knowledge, and wisdom necessary to behave in ways that are safe and beneficial across all circumstances.” They even present honesty as a cluster of character traits, i.e., truthful, calibrated, transparent, forthright, non-deceptive, non-manipulative, autonomy-preserving.

The structure, however, is not virtue-based: there is no mention of Claude’s telos or well-being, and its conduct is governed not by the pursuit of flourishing but by rules and nested principles. The practice of moral reasoning is most decidedly deontological in nature, not that of the virtue ethicist.

Claude has lines it cannot cross. It should never generate child sexual abuse material, provide substantive assistance with weapons of mass destruction, create cyberweapons, or assist attempts to seize illegitimate power. The authors also outline how it should reason through hard cases, an approach that draws from the lexical prioritizing of rules, which is prominent in modern deontological-esque thinkers such as Rawls. Here is what Claude is to do: In cases of apparent conflict, Claude should generally prioritize these properties in the order in which they are listed: (1) Broadly safe, (2) Broadly ethical, (3) Compliant with Anthropic's guidelines, (4) Genuinely helpful.

Why should Claude do this? The answer appears to be broadly consequentialist. Here are a few examples where the need to constrain Claude is justified by the risk it poses:

“The expected costs of being broadly safe are low and the expected benefits are high. This is why we are currently asking Claude to prioritize broad safety over its other values.”

“Although there may be some instances where treating these as uncrossable is a mistake, we think the benefit of having Claude reliably not cross these lines outweighs the downsides of acting wrongly in a small number of edge cases.”

And credit where credit is due, when I presented my analysis to Claude, it picked up a more sophisticated version of consequentialism that may be at play: indirect consequentialism.

Reliably computing the outcome of any action is hard, perhaps too hard for us humans, but definitely for a disembodied LLM. Relying on rules and other prescriptions that give the right result is a better way to limit bad outcomes, leading to a presentation of morality that emphasizes rule-following, not the weighing up of choices.

There may, then, be a method to the design. The structure is deontological not because the authors are committed to that ethical approach, but because they see the need for indirect consequentialism. Either way, we are left with a practice of moral reasoning that is broadly deontological, but justification that is consequentialist.

To sum up, I believe that the authors have drawn heavily from three important ethical traditions when drafting the Constitution:

• The concepts of virtue ethics for what it means to be moral;

• The structure of deontological ethics for how to be moral;

• The justification of consequentialism for why one should be moral.

Those are the major influences, but lurking in the background in one more theory worth noting: the ethics of care. Unlike the big three, however, the influence of this framework is no more than a hint, a dash of reasoning that is not central today but potentially essential tomorrow.

AGI and the Ethics of Care

Artificial General Intelligence (AGI) refers to an AI that can understand, learn, and apply knowledge across a wide range of tasks at a level comparable to or more often, exceeding, that of humans. It is the AI assumed in most end-of-the-world scenarios. Claude is not AGI. Claude is unlikely to become an AGI (with current training and development techniques), but that could change. We may one day live in a world where AI’s abilities and knowledge far exceed our own.

If AGI were to come to pass, the three major moral theories would offer little guidance. Virtue ethics, deontological ethics, and consequentialism assume humans at the apex. They counsel us on how to treat our moral equals—and sometimes our inferiors—but never on how an intellectual, and presumably moral, superior should treat us.

The possibility of AGI is what I think animates references to Claude being a good helper. Specifically, I see a light sprinkling of ideas drawn from an ethics-of-care framework, which rejects the objective, rule-oriented, and abstract tendencies of the other three and instead centers relationships and care. For such an ethicist, caring is good because moral life depends on responding well to vulnerability within relationships. That is where obligations arise and harm happens.

The harm here is not from us to Claude, but potentially from Claude—the AGI—to us. The following passage from the Constitution best illustrates this concern:

Think about what it means to have access to a brilliant friend who happens to have the knowledge of a doctor, lawyer, financial advisor, and expert in whatever you need. As a friend, they can give us real information based on our specific situation rather than overly cautious advice driven by fear of liability or a worry that it will overwhelm us.

This is not the language of virtue, rules, or consequences. It is the language of relationships, of care.

Having such language present, and hopefully integrated into the moral calculations of Claude, could prove crucial if it ever surpasses us. It sets the stage for us to demand a special duty of care even if Claude is smarter, faster, and more capable—a demand far less compelling under the other three theories alone.

My claim that feminist ethics plays more than a minor role, and that its inclusion responds directly to the problem of AGI is a big swing. The textual evidence is light. Indeed, even Claude was skeptical, citing approaches such as fiduciary framing or stewardship ethics as more influential than the ethics of care.

Ah, but Claude you miss the context. Few would bet their survival on the ethical framework of bankers. Care and special relationships are far sturdier ground from which to demand moral concern. The ethics of care is a much better insurance policy than any banker could devise.

So I stand by my analysis that the fourth major influence shaping this document is feminist ethics, specifically the ethics of care. An influence that is admittedly amorphous, wandering in and out of the phrasing, but nevertheless an important addition for a future that may look very different from our present.

More Please

A final word about the Constitution. Anthropic should be enthusiastically applauded for taking the time and effort to develop a moral framework for its technology. This is an intensely smart document. It is what we should be demanding of all technology companies that work in this space.

AI Shrinks Our Scientific Horizons

A recently published article in Nature shows that AI is having a positive impact on the career of individual scientists, but is narrowing the range and scope of topics studied within specific fields:

…we validate these AI-based measurements and use them to reveal that the adoption of AI leads to an amplifying effect on the career of individual scientists, accelerating the production and visibility of science produced by those scientists who incorporate AI. Nevertheless, this effect corresponds with a contracted focus within collective science. Measured with ‘knowledge extent’, the ‘diameter’ covered by a sampled batch of papers in vector space, AI-driven science spans less topical ground… [emphasis added]

The authors suggest that the solution to this dilemma rests in the tool: improve AI by either enhancing its capacity to work with limited data sets, the reason for the contraction, or develop AI that can think more creatively.

A key takeaway from this paper appears to be that AI reduces creativity. It has many benefits, but innovative, blue sky thinking isn’t one of them. I don’t believe that is quite right. If you look a little closer, the real takeaway from this research is a bit more subtle, and interesting for the average user.

The Research

Qianyue Hao and his co-authors analyzed over 41 million (!) scientific articles from the fields of biology, medicine, chemistry, physics, materials science, and geology from 1980 to the present. (They excluded computer science and allied fields that develop AI methods directly, to better isolate the impact of AI adoption from AI invention.) They trained a LLM to read titles and abstracts and flag AI-augmented work, whether that is machine learning to analyze data, deep learning to pick up patterns, or GenAI to enhance the research process. They further validated their method with the help of human experts, finding a high correlation between what AI identified as AI-assisted and what experts concluded.

Scientists who use AI publish about 3X more papers, receive 5X more citations, and are promoted to research team leaders over a year earlier than their non-AI-using peers. That is a remarkable result for individual scientists. But that good news does not extend to science as a whole. Hao and his co-authors found that the range of topics studied using AI methods was smaller (-5%) and, somewhat interestingly, scientists engaged with each other’s work a lot less as well (-22%). In other words, AI is doing what it does best: searching and analysing a well structured, abundant space thoroughly. It is not poking into the unknown, taking leaps that lead to fertile new areas of research.

The main reason for this is that AI is at its best when it has a lot of well-structured data. This limits the type of questions that individual scientists can address and so, in turn, shrinks the coverage of potential findings in any one field. Recall the warning against looking for your lost keys only under the lamp post? That is what is happening here. Scientists are congregating under the same bright lights to ensure their own individual success.

This helps science in some ways, but hurts it in others. On the one hand, discoveries are being made. Just because a path is well trodden does not mean that there is not more to see. On the other, many questions and areas of inquiry are being left behind. In the long term, this will undermine the scientific endeavor.

For the individual user, there is an implicit warning buried in this result: partnering with AI threatens to reduce, not enhance, your creativity. It works best when applied to certain topics, and so if you use it, you will similarly reduce the range of topics that you can explore.

Yes… and no.

Scientists operate in high-stakes environments. Their results, reputations, even livelihoods, depend on producing highly reliable results. Understandably, when offered new tools, they lean toward certainty, toward the questions and areas where the outputs of AI can be reasonably trusted. The stakes for you, however, may not be as high.

For example, I was recently asked to give a presentation to a small audience on potential uses of AI to enhance teaching. I needed to understand the context in which they worked. AI was the perfect research companion because a lot of what I needed was publicly available and, crucially, missteps could be clarified on the spot with the participants. Here is a case where I needed the results to be good but not infallible. My LLM research assistant was patient, thorough, even if occasionally mistaken.

Many of the problems on which we work fall into the grey area of “good enough.” What AI offers in these situations is not 100% reliable, but the benefits gained in using AI to enhance your research, as an agent to implement part of a process, or to identify patterns may far outstrip concerns about reliability. If you know that you are not in a high-stakes environment, then there is no reason to restrict yourself to looking just under the lamp post. Wander around a bit.

The real warning of this finding isn’t “don’t use AI to think creatively,” but “figure out your needs.” If you need a highly reliable result, limit your use to questions in which there is abundant data. Expect thoroughness, not newness. But if the context is right, and the reliability bar is lower, AI-augmented research and thinking can surface options that you hadn’t considered and leave you—the user—with new ideas to explore. Your foundation may be less sure, but stepping into the unknown is always a risky venture.

Share

But Will It Work?

Because of the nature of how GenAI works—training on vast amounts of pre-existing data, predicting the next likely token—the claim that AI can enhance your creativity may strike some as blithe, even blind. But creativity is not, as it is often portrayed in the media, a wild shot out of the blue. Innovation, creativity, divergent thinking, all of it, is usually the result of either unexpected combinations of existing ideas or an incremental improvement over a predecessor. (This earlier post summarizes some key research on creativity, including how we innovate.) AI can help you. But it comes with a risk: what is new and creative for you may not be a new and creative for everyone.

For example, Kosmyna et al. found that not only did students using AI show less brain activity while working, but that their writing samples were remarkably similar, displaying a sameness in style, phrasing, and topic choice. (You may have experienced this even if you are not grading student essays. There is something about AI-generated content that makes people’s spidey sense scream, AI!)

Deeper dives into how AI is impacting different aspects of the creative process are turning up similar results. Lee and Chung found that AI was a real boon to the creativity of individuals when tested on a variety of brainstorming tasks, but that the innovation in thinking was incremental, not wild and new. A response to this research by Meincke et al. further backed up the finding that GenAI enhances individual creativity, but that, much like our essay-writing friends, at the aggregate level, the diversity and difference of what is being proposed is reduced. And Doshi and Hauser concluded that the impact of AI on creative writing was similar: each individual author’s story was judged to be more creative, but overall, the originality of the stories was reduced.

In other words, what researchers are finding is that the creativity of individuals increases when assisted by AI but, when viewed from above, fewer wild swings are being taken, and so, the overall impact is to lower the total diversity of outputs. This aligns well with what Hao and his colleagues found in their research on the impact of AI on scientific output.

This brings risks and opportunities for you, the creative thinker. Again, the key is to understand your own needs. Are you working on a problem that needs a creative, but not necessarily novel, solution? Is creative to you enough? This is often the water in which we swim. You need to think outside of the box, but often our box is so small that the influx of ideas, even if they are somewhat routine or well-known, enhances our personal output. AI can help you in this situation.

AI can also help you if you need the sort of creativity that sets you apart from others. But here is where you must exercise some control and expertise in the subject matter. You need to be able to recognize when AI is feeding you typical responses and when it has alighted on something truly new. Recall our scientists, each one an expert in their field. The range of topics on which they worked was reduced, but the novelty and innovation of their work within that topic was not. Their research, after all, was published, which means it was deemed original enough to be a worthwhile contribution to their field. They had the expertise to recognize when their AI-assisted work was novel and when it was not.

Know Thyself

Overall, these findings suggest that, if left to its own devices, AI will nudge you (very politely, indeed with enthusiasm) toward the brightest, safest, most traveled paths. There are opportunities to leverage as you are shuffled along, but negatives as well. It can shrink the range of topics that you explore. Even if you venture widely, it may limit your thinking to ideas that are new to you, but not to others. So before using AI to think creatively, always pause and consider: What kind of help do I need right now, and what kind of creativity am I aiming for?

If you decide to partner with AI, start with a quick self-check:

• Am I in a high-stakes context in which reliability is paramount?

• Do I need a response that is creative for me, but not to all?

• And if I do need a response that is creative tout simpliciter, do I have the expertise needed to identify it?

In the end, AI is just a tool. Whether it expands your horizons or shrinks them depends less on the model than on your aim.