Decades ago in my callow youth, I had the privilege of editing a short essay by the great literary critic Alfred Kazin. It included a sentence in which he rather vaguely alluded to a quotation (the equivalent of writing “As Homer says about the lotus eaters” or the like). Wanting to be useful, I looked it up and asked him if he cared to nail down the specifics: The equivalent of changing the line to “As Homer writes in the Odyssey, Book 9, lines 92-94”).

Kazin said no.

It was a teaching moment for apprentice-me. I learned by example that neither precision nor accuracy are ends in themselves.

In this case Kazin didn’t want the reader to go off into the weeds of scholarship; he didn’t want his essay (which originally had been a speech) to feel too try-hard for the context. He didn’t want to look as if he’d taken trouble to do something that he hadn’t done. No reader wants detail for detail’s sake; we want details that serve a purpose. Astronomers seldom need centimeters, preferring light-years and the Astronomical Unit. Literary writers want the words that reinforce their style and their point, not the ones that spoil the effect.

To Be Hyper-Informed, Or Not to Be: That is the Question

Decades later, anyone with a smartphone is in Kazin’s position, making the sort of choices he did with me. Do I need or want knowledge that does not come organically to mind? With an assistant in my pocket that can look up anything, I have to decide many times a day. I notice this moment of choice often in conversation: “You know, Jack Nicholson was in that movie, way before he was famous.” “Really? I don’t remember that.” “I’m pretty sure it was him.” Pause. Well, do we look it up?

Maybe the overlords of Silicon Valley assume we’ll always choose to know more. But regular people don’t. We often decide not to know. Not because we like to wallow in ignorance, but because we are in a moment where it’s best to keep our thoughts entirely human and entirely rooted in the local situation. Because the conversation’s vibe is what’s important, not its content? Because we don’t want to open a rabbit-hole of search-leading-to-more-search, taking us away from the people we’re speaking with? Because people don’t feel entirely themselves once they’re hooked up to the impersonal global data-horde? I don’t know. I wish I did. We’ve had smart phones for 20 years. Surely someone has tried to figure this out.

Subscribe now

And now we have mass-market AI. The look-it-up option is supercharged. You don’t get a list of links that you can interpret to get your answer (“Wikipedia and 5 other sites say Nicholson was in the movie”). You get a human-like entity interpreting the search for you (“That’s an astute observation, David. Not everyone knows Nicholson was in that movie. He got the part after another actor caught a cold and missed his audition.”) Instead of a distracting trove of information, you’re actively pulled into a different conversation than the one you were having with your fellow humans.

As you’ve no doubt noticed, all the mass-market chatbots are exhaustive (and exhausting) providers of information, suggestions and insights. Ask a question, and they’ll never answer without also suggesting that you go even further with them into their store of bodiless data. The intrusiveness is more obvious when your phone is supplying so much more than a list of hits that all mention the same fact; when it instead provides a tidy little essay about your question. But the temptation is greater, too. With a few taps we can know so much!

So, psychologists, philosophers and others who study the mind, this is my question. Whom should we read about how people decide whether to open the door to data? I’m hoping someone has researched, reflected, speculated or at least ranted in an interesting way about this. Suggestions needed!

Leave a comment

Reading This article the other day gave me a little shock. It’s about dying people who have surprisingly similar “end of life dreams and visions” of their long-lost family members. It’s not that rare, and the visions are typically like this: “Shirley was consistently seeing the beloved grandmother who raised her as a young girl and the idyllic farm where they’d lived together.”

What jolted me was this: There is no one in the Afterlife whom I wish to see.

I had quite enough of my parents when they were alive, thank you. My grandmothers, I knew only as strange, remote old women. They knew me as a small and wary child. (I suppose our ghosts could make the spectral equivalent of polite conversation, but nobody would be weeping with joy over that.) My grandfathers died before I could meet them at all. Dear friends? I can think of a few, but the people reporting these visions seem to believe they involve family. I’d expect my friends’ ghosts to be hang out with their own relatives.

Better after death than before?

Ah, but maybe the people you meet as you leave this life don’t have to be exactly the way you knew them. They might, instead, be freed from anger, fear and shame, no longer twisted by adversity. Therapists sometimes ask their patients to imagine the dead this way: Spirits at peace, better than they were in life.

That’d be nice -- my mother’s spirit all intelligence and passion for truth and fairness, with none of the paranoia. My father’s spirit manifesting the real man’s gift for comradeship, tactical skill and defense of the underdog, without the real man’s cruelty, self-involvement and resentment.

Yes, much better. Except that those beings would be, literally, not the people I knew. They’d be imitations that suit me better than reality did. In other words, a fantasy, curated by my mind as it sang itself to sleep.

Of course, that may well be what all any of these visions are. That uncertainty has always hung over accounts of any purported Afterlife, leading to uncomfortable questions. Who, exactly, comes back to haunt us? Is Eternal Grandma like the grandma you knew when you were little? Or is she 22 again? Do deceased toddlers get to grow up, or are they toddlers forever? (Saint Thomas Aquinas held that children would age and the elderly would be rejuvenated, so that eternal life would be lived in that perfect state where growing up is done and decline hasn’t yet set in.)

Nobody knows for sure if these questions even make sense to ask. Our answers are something between hope, reassurance and guesswork, in the stories we tell ourselves, and in the arts.

And, now, in artificial intelligence.

Some months ago, I did some reporting and writing about the use of AI to reproduce dead people. (Here and here, if you’re interested.) In reporting these, I came across companies whose marketing touts lossless duplication. Give us a few seconds of audio, we’ll make the AI voice sound just like the person’s. Give us a photo, we’ll give you an AI-made video of that person. Supply a few videos, we’ll create a “live” version of the subject. Text messages, emails, scanned writings -- we’ll use them to create sentences that match in word choice, style and tone. Such things are a snap for generative AI: Running mathematical operations to predict what is most likely to resemble the data it has, while also resembling what it has learned the human wants.

But a dead person’s sonograms, videos, emails and other digital traces don’t reveal your motive in asking AI to imitate him. What did he mean to you? And when did he mean it? (When he was 35, or 50, or 78?) And will you feel the same way next month? The answers aren’t physical facts that can be digitized. You have to try to portray them in words you supply the AI: A description of the deceased, which (by definition) is your version of that person -- not how they’d look to a camera, but how they looked to you.

Your older sister’s version of Mom could be quite different, for the same reason President Trump’s version of George Washington is different from that of a professional historian. But people arguing over a nation’s history have a stake in deciding whose version of Washington is correct. Not so you and your sister. Sensible people won’t argue, instead letting each of you remember the mother you need to.

Which I mention because some critics have portrayed ghost-making AI as a rather sinister imitator (here, for instance). In these arguments, the AI is rather like Gene Wolfe’s alzabo, that eerie creature that, after feeding on a dead person, speaks in the that person’s voice and has all their memories. (In place of the alzabo using your loved one’s voice to take your life, picture a corporation using the same voice to get your money.)

My problem with this argument is that it presumes passivity: people are supposed to be helpless in the face of this seductive corporate recreation of a loved one. But, as I’ve mentioned, the facts that a corporation can collect aren’t enough to create a digital ghost. A living person must take part, giving context and meaning to the sonograms and photographs and old emails. That person is taking an active role in creating the AI’s product.

This is clearest when the living person is the “ghost” herself. Why become a client of a company like “virtual legacy“ or “Afterlife.ai“, which set their customers up to make versions of themselves for others to meet after their own deaths? Because you know someone is going to make decisions about how you’re represented, and those decisions might not be to your liking. (Afterlife’s pitch is, explicitly, that you’ll create your AI persona “while you are still here to decide.”)

Subscribe now

So there is no final “accurate” or “faithful” reproduction of a dead person that a corporation can monopolize. No one can be a passive consumer of a digital ghost, because a digital ghost can’t exist without their active participation. If they get addicted to it, the experience may be like a novelist getting lost in her fictional world, not the victim of a con getting taken for a ride.

If I ever return to the experiment of “recreating” a loved one, it will never be with the impossible goal of pixel-by-pixel objective accuracy. Instead, I might try improving on the originals, making them the sort of better-than-real entities that therapists invite their patients to imagine. That means they won’t be perfect copies of the real people. But then, no AI ghost ever will be.

Leave a comment

What Else to Read This Month

Quantifying the Meh in AI Writing

Like many people, I know genAI writing lacks something -- spark, surprise, wit, spirit -- that good human work possesses. So why, (like many people) when quizzed, do I often fail to tell AI writing from human?

This paper on AI fiction explains this seeming paradox, while showing that machine stories really are inferior in quantifiable ways. The AI-or-human tests we’re flunking focus on the wrong level of analysis, the paper argues. Sentence-by-sentence features (word choices, stylistic tics, those damn em-dashes) don’t reveal differences in narrative structure.

Looking at that level with their new AI-writing detector, the authors find AIs “over-explain themes and favor tidy, single-track plots while human stories frame protagonist’ choices as more morally ambiguous and have increased temporal complexity (e.g., flashbacks, nonlinear structure).”

So, your AI agent screwed up. Does that mean you did?

Bruce Schneier and his colleagues propose a measurement system to determine responsibility in cases where an AI agent goes off the rails. The “Genie coefficient” would define the extent to which the agent’s behavior was consistent with your intentions (consequences are your fault!) versus the extent to which its actions misinterpreted or extended your commands (”if an AI system betrays the reasonable meaning of an instruction, that’s the AI’s misbehavior, not the user’s”).

Creating a model’s personality

It’s not just the AI boyfriends that have “personalities.” Almost all the models you interact with have them. In this video, Nathan Lambert shows what that entails.

The Case Against ‘Alignment’

“Alignment” is the general term for work on guardrails to make sure AI does not violate human values. Wang Huanchao argues that the concept (around which “a handful of privately owned companies on the American West Coast have signed a social contract with AI systems on behalf of the entire human species”) doesn’t make sense. “When OpenAI and Anthropic say, ‘We want AI to be aligned with human values,’ all three referents—’we,’ ‘AI,’ and ‘humanity’—are remarkably questionable and vague.”

All of this is easy to accept. After all, for our entire lives, putting together an elegant answer to our questions was a sure sign that the putter-together was (a) intelligent and (b) trying to tell us something.

Leif Weatherby thinks this metaphor is a colossal mistake. In reality, he argues in his recent book, Language Machines, Large Language Models like Claude and ChatGPT, which have no minds or intentions, prove that language requires neither. So there is no point in taking either side of a debate about whether AI is human-like. (No need to argue about how smart AIs are, or how ethical, or how intent they are on escaping human oversight, or whether they’re going to kill us or provide us with Utopia, or whether they have rights). We can accept that AI produces meaningful language without thinking it must be on its way to consciousness; we can accept that it’s just a machine without insisting that its product must be worthless.

Instead, Weatherby says, we should be looking at what these “culture machines” tell us about the culture and politics that shape us. Instead of ethicists and philosophers of mind, he looks to literary theorists (on the nature of language) and political and economics scholars (on the way AI is used to define and control markets and keep labor in line).

I wanted to speak with him because it seems to me the AI-as-person metaphor has taken over most news and conversation about this technology. Since AI is not really a person, we need other ways to think about it, and talk about it. And Weatherby’s take is rigorous, deeply informed and, for me, mind-expanding.

Some Useful Background

Here’s some background about some of the things that came up in the conversation.

“Stochastic parrots” - This is shorthand for the claim that a Large Language Model is “a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning”. Introduced in a 2021 paper by Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Margaret Mitchell. As Bender wrote this year, the claim here is that this isn’t real language because there is no intention to communicate in the model’s mere “remix” of training data, its instructions, and the user prompts it received. (Weatherby, who doesn’t believe that meaning depends on some human’s intentions, is not a fan.)

“InstructGPT” - A term for the OpenAI GPT-3 models that were the basis for ChatGPT when it was released in November of 2022.

Claude’s constitution - A “detailed description of Anthropic’s intentions for Claude’s values and behavior,” which is part of its training.

Mid-training - the phase in the development of a large language model in which its trainers focus on specific skills and practices they want it to have. This occurs after “pre-training,” when the model goes to work on vast amounts of text, to find statistical patterns in all language use. Claude’s constitution is used in mid-training.

Context window - The maximum amount of text that the model can consider at one time when generating output. Advanced models now can hold hundreds of thousands of words in a context window (for example, all 25,000 words of Claude’s constitution). Human brains don’t work the way LLMs do, but it’s safe to say humans have smaller context windows than models.

Reminder: The transcript is AI-generated and likely contains some mistakes. If you spot a bad one, please let me know.

That has been hard for me to grasp. After all, intuition tells me I get my ideas from some unknowable depth in my soul, combined with inscrutable providence (as in, apple falls where Newton’s head is.) In the tradition I grew up in, creators take their ideas andshape the world to match our vision (think of all those solitary geniuses in folklore, tapping sculptures out of marble, scribbling symphonies for hundreds of instruments, or designing bridges and iPads). You know, like this guy:

In reality, Coersmeier argues, architectural creation emerges from networks of interacting agents. Some are human (like engineers, construction workers, miners) but many aren’t (they’re rivers and lakes, steel and stone, wind and rain patterns -- and, now, the AIs that architects increasingly engage in their work). It’s a much less Hollywood-friendly reality, sort of like this:

Artificial intelligence, he argues, helps humans to see this, if we use it well -- that is, if we let it surprise us, influence us, and remind us that we aren’t the center of creativity’s whirlpool. Coersmeier believes that decentering humanity will help us treat the planet, and each other, better than we do today.

We have been arguing about these ideas for some time. And that’s what we did during this conversation in his studio. We talked about how he and his students subvert AI tools (fewer predictable dull results, more weird surprising ones); why decentering humanity isn’t inhumane; how he reconciles the value he sees in AI with its ecological costs; how visual and geometric AI differs from the kind that makes texts -- and a lot of other topics.

Full disclosure: Coersmeier is a friend of mine, and I had a role (as an editor, advisor and frequent disagree-er) in the writing of his book. That wasn’t the reason I invited him on, as I think you’ll hear. You’ll judge that for yourself.

Also, I lightly edited this conversation to remove interruptions (for instance, as it was 100 degrees outside, we had to turn his AC on at times, and pause our talk while it rumbled).

Please note the transcript is AI generated and may well contain a few errors.

The podcast is back! With a new, tighter focus: This is now an ongoing series of conversations about human relationships to artificial intelligence — that is, how people understand and experience these entities.

That means I’ll be speaking with people about imagination, pretense, wishful thinking, our sense of ourselves, our sense of reality and related topics that seem simple until you really think about them.

We’re starting off with Valerie Tiberius, a philosopher whose new book is a lucid guide to how to think about AI companions — and the story of how she realized that simple-minded rejection wasn’t the way to go.

Tiberius expected to write a diatribe against AI companionship when she first considered working on the topic. A professor of philosophy at the University of Minnesota, she teaches and writes about ethics, moral psychology, well-being, and friendship. Her work considers questions like what it means to be a moral person, what gives meaning to life, and how our emotions and desires interact with reason. AI didn’t feel like it deserved a place at that table. Most people she knew reacted to the subject by saying, “that’s so sad.”

But reading and thinking more deeply changed her mind. So did her conversations with a couple of AI “friends” she created to better understand the experience. Her new, Artificially Yours: Real Friendship in a World of Chatbots, describes how her thinking evolved. That was our starting point for a wide-ranging conversation about AI, values and friendship.

As always, remember the transcript is AI-generated and will contain a few errors. Any real howlers, please let me know.

What most people mean by “artificial intelligence” -- an engine of responses to any request, which addresses you as if it were human, and is run by giant corporations out of huge data centers -- probably won’t disappear any time soon. But it’s getting harder to believe that it will keep lumbering along on its current trajectory.

It’s not just the lusty booing of speakers who mention AI at American commencements, or the Trump Administration briefly backing away from its damn-the-torpedoes, full-speed-ahead attitude toward oversight and guardrails (Trump chickened out quickly, but other governments around the world and in the U.S. have not.) Nor the Pope telling the world’s 1.4 billion Catholics not to let the AI elite control all the crucial choices about the technology. Under official statements and Instagrammable crowd moments are signs of resistance that are more diffuse, which look to endure.

About half of survey respondents globally say AI makes them nervous, according to the Stanford Institute for Human-Centered Artificial Intelligence’s latest AI Index Report. Meanwhile Pew reports that respondents in the United States who are worried about AI’s effects outnumber those who say they’re excited for the future. These misgivings are starting to turn into effective political opposition. Local movements are increasingly successful at blocking or delaying data centers, for example. (At the beginning of last year, five jurisdictions in the United States had enacted moratoriums. This month, there are 95, according to this handy data-center moratorium tracker.) Labor unions are getting better at resisting unrestrained AI mandates at work -- with help from well-organized, expert groups like the DAIR institute’s just-launched Luddite Lab.

But let’s not read too much into people’s ambivalence. It’s a safe bet that many college graduates who booed AI used it in their coursework, and that many rich-nation people who tell pollsters they fear AI are nonetheless asking Gemini or ChatGPT for help with their foot pain and resumes, and using such AI features in their offices that actually help them get work done.

In other words, AI has become one of those technologies people like using the way they want (fix my application letter, advise me on my fight with my sister, help me with this kinky fantasy); and hate using the way others want (obligatory chatbot on the help line, AI resume readers, minute-by-minute surveillance of all I do at work). People aren’t going to just be “for” or “against” a tool like that. (And, in any event, even those who refuse to choose to use AI must nonetheless live with it. That’s because it’s baked into processes they can’t escape. An example of which I will discuss below.)

So we’re not bound for some Butlerian jihad of burning data centers and our grandchildren looking back at machine-written letters and algorithm-planned meals with pity. Some aspects of AI in 2026 really are like radio a hundred years ago -- a new kind of capability, bound to spread and change the way people live. Other elements, though, are more like the Charleston dance tunes that radioes broadcast in those early days. They’re practices that will soon be seen as one of those weird fads that every generation falls for.

Which Aspects of AI are going to be seen as “so 2020s”?

But which aspects are which? Which forms of AI, and uses of it, are truly inevitable? It’s a question worth asking, as you try to make sense of where AI adoption, and AI resistance, are heading. So I’ll circle around it over the next few weeks, looking at one claim of inevitability at a time.

For this post, I want to start with the (often unstated) assumption that the near-future of AI is more and more encroachment on “high-value” or “creative” work. There are good reasons to doubt this is going to happen.

Let me illustrate with developments in the form of work I know best: Writing. In the past year, it has become apparent to a lot of people that when you care about what you are writing, AI use is harmful, to both the individual writer and to the community of writers and readers. The archive of writers, teachers and academics who have come to this conclusion is deep and wide. Some cogent recent testimonies are this and this and this. It’s also becoming clear that this view is not a matter of personal taste. Researchers have been finding pretty consistently that AI-assisted writing offers fewer original ideas and innovations. For example, here and here . And you can’t really say “well, different strokes for different folks” to a practice that will leave people unable to know their own minds and express their thoughts.

Subscribe now

But all this criticism of AI writing (which also applies to a lot of other AI-assisted work) rests on an assumption: that people always care -- or always should care -- about what they’re producing.

AI Is Great At Work You Don’t Care About

That assumption is false. Some people do not care what they’re writing, all of the time; and all people don’t care what they’re writing, some of the time. Specifically, the time in every life when you’re obligated to do work whose outcome matters to you, but whose process doesn’t. You want what this chore will yield. You don’t give a damn exactly how it’s done.

Maybe you want to sell your house without paying an agent’s commission, and you don’t want to spend months learning the references and jargon that your emails should adhere to. Maybe your job requires that you follow strict conventions about the style of your message. (You want to tell the higher-ups to move the copying machines closer to the desks of people who use them, but you have to couch that in terms of “process improvement” or “efficiency optimization.”). Perhaps you know no one will read what you’ve written (as required by this law passed in 1978, here is the annual report on widget safety in all our facilities). Maybe you want to spend time on that term paper you care about, so you just want to get this mandatory class outside your major over with. Or maybe everyone involved in the work you’re doing knows it’s bullshit (here’s my memo on why of course a Democrat can win in this deep-red district that voted Trump by 38 points!).

You know this drill: Here’s a task involving words and calculations and the assembling of facts. You don’t know how to do it well and don’t have time to learn; or you can do it but have better things to devote time to; or you don’t know how to do it and have no way to find out. In such circumstances, AI has no downside. Not doing your own thinking? Fine. Not trying to do anything original? Cool. Not learning what you need to know to really master the subject? Great. HR needs a summary of my work history. I have to recount what my Scout troop accomplished last year so the library will give us permission to use their basement for our meetings again. I need to explain how my grant proposal accords with the foundation’s mission statement. I know there are expectations for this task and that I don’t know them. So this work manages to be both dull and anxiety-provoking. Who wouldn’t be glad to turn this kind of thing over to a machine?

“You should as a human being to help you” is a sentimental fantasy

Of course, if I lived in some Hobbit-shire fantasy world where everyone has plenty of time, a deep network of mutual help, and a rich social life, I wouldn’t need to turn to an algorithm, ever. I’d call a kindly neighbor, or a friend of a friend, for help. We’d talk over a cup of tea about his expertise and his garden and our kids, and and our lives would glow a little brighter with fellowship.

But I live in the real world, where this sort of sentimentality rings false. Not because it’s a bad idea to turn to our fellow human beings for help. It’s a great idea! Many of us do it whenever we can, giving and receiving knowledge as members of communities. However, many people, especially if they’re poor, uneducated and not well-connected, find there is no one in the shire who can help. In such times, the alien expertise of AI, which is actually a representation of the expertise of faraway strangers, is the only way across the chasm of ignorance that blocks your way. Then, the alternative to AI assistance is no assistance at all.

As it happens, the majority of the world’s population is often in this boat, and it appears that when AI solutions become available, they’re jumping, without rich-nation anxiety, to use them. Last month, for example, Ajay Banga, president of the World Bank, described an illiterate farmer in Uttar Pradesh who used an old phone to photograph a diseased leaf. An AI then identified the disease and arranged for a countermeasure to be delivered to her. This AI didn’t deprive her of a social interaction, nor did it rob her of the satisfaction of solving the problem herself. It solved a problem that otherwise wouldn’t have been solved.

In this eye-opening post, Carlo Iacono cites another example from India: “A woman in rural Rajasthan who receives an AI-interpreted chest X-ray for tuberculosis has not had her cognitive sovereignty eroded. She has received a diagnosis that no human radiologist was available to give.” To fear that AI will erode some valuable human aspect of life, he points out, is to assume that this aspect of life is actually present. The lovely phrase “beautiful inefficiencies” alludes to those warm and humane aspects of life at a human pace, which is better than chasing efficiency all the time with AI. But, as Iacano notes, “beautiful inefficiencies assume the inefficient practices existed and did their formative work.”

Subscribe now

This kind of AI, which helps people without money or contacts or education, isn’t what we usually mean by “AI” in rich nations. It might live on cheap hardware (often mobile phones), not in huge data centers; it offers to solve specific “daily problems for ordinary people,” as Banga put it, rather than proffering answers to any and all requests. It doesn’t try to imitate a human being, and its makers sure as hell aren’t racing to make it sentient. It seems unlikely that “small AI” infrastructure could replace the big centralized approach; but “small AI” uses -- from people in the non-rich world -- will become a bigger part of the AI story. Perhaps, as Iacono argues, they already are:

When Datareportal measured adoption as a proportion of internet users, Kenya led the world. Brazil was second. When you strip away the headlines from San Francisco and the opinion columns from London, the actual centre of gravity for generative AI use is not the Anglophone West. It is the global majority.

Even when they have access to the big centralized Large Language Models, users from this global majority are more inclined than the wealthy to go for more practical problem-solving and fewer whimsical, speculative or “creative” tasks, according to this paper (it’s a study of 202,590 ChatGPT conversations from 1,252 users across India, Nigeria, Brazil, and Pakistan). The authors (Shreyasi Roy Chowdhury and Kiran Garimella) argue that economists should change their measures of AI’s economic impact to reflect this difference:

A user who asks ChatGPT to interpret medical symptoms before deciding whether to pay for a doctor visit, or to diagnose why a household appliance is malfunctioning, or to draft a message to a landlord in a second language, is extracting real economic value, but it is value that does not appear in any standard measure of labour productivity. Economists recognize this as household production: the non-market work that sustains daily life but is invisible to GDP accounting. Our data suggest that the primary mode of ChatGPT usage in the Global South falls squarely in this category. If so, productivity-oriented evaluations of AI impact will systematically undercount the technology’s value in precisely the markets where it may matter most: markets where formal consultation (medical, legal, financial) is expensive relative to income, and where an always-available, zero-marginal-cost advisor fills a genuine gap.

When AI gives you something you never had before, you might think better of it than does someone who thinks AI is a thief of good stuff he already has. Perhaps that is why surveys find way more optimism about AI in poorer nations than in richer ones. In this one, for example, nearly 75 percent of their respondents from Argentina, Brazil, China, Colombia, Egypt, Ghana, India, Indonesia, Ivory Coast, Kenya, Kazakhstan, Mexico, Morocco, Nigeria, Saudi Arabia, Senegal, South Africa, Türkiye and Vietnam said generative AI was important in their daily lives. That’s nearly double the percentage of people in richer nations with that attitude. (In Nigeria, almost everyone who answered said AI tools were important to them. In France, it was about two people in ten.) 1

So maybe the inevitable expansion of AI globally will not tend toward the Singularity, synthetic persons, LLM-written novels and songs, or AGI. Perhaps, instead, it will bend increasingly toward solving the problems of people for whom the pre-AI past represents hardship and dead ends.

Leave a comment

And now for that example of inescapable AI

I wanted to be done with this post earlier, but I took a detour to act on an exciting email I received from an editor at Penguin Random House. (I am, as it happens, working on a book, about humans’ relations with AI.) The over-the-transom message said nice, perceptive things about my 2005 book Us and Them, and suggested we might talk about future projects with the publisher.

Working on a book, especially in an early stage, is like belonging to a very small cult. No one cares about your passionate yet weird mission. So getting an email from someone who sounds like she might is electrifying. At last! A disciple! Or at least some encouragement, for a change.

It didn’t look like a scam. There was no rush-rush urgency, no misspelling, no request for money. I thought for a while about how to answer, and how to get my literary agent involved (the email said they’d be happy to work with that person).

Alas, it was a scam. The long, thoughtful paragraphs about my writing were generated by a Large Language Model, probably prompted with something like, “based on this pdf and these URLs of this guy’s writing, create a flattering letter from an acquisitions editor interested in his work.” If I had said I had no literary agent, the “editor” would have suggested one, who would have asked for money. If I said I did have an agent, I would be on my way to supplying details about me that the scammers could use for other purposes, and other requests for a “processing fee.” The giveaways were (a) email sent a 5:42 AM, even though the “editor” was in the same time zone as I am and (b) an email address ending in “penguin-random-house.com.” (The real company has no hyphens in its domain name.)

How do I know this about the domain? Because I asked a Large Language Model. Gemini 3.5 took in the text, and gave me all the reasons I should think it was authentic (LLMs do try to be agreeable). But it did mention that I should look carefully at the sender’s address, just in case. If it hadn’t, I would not have noticed the hyphens. Gemini also explained why the scammers would bother with a pitch that doesn’t directly ask for money.

In other words, when scammers came at me using AI, I managed to defend myself only because I had AI too. One more example of the ways in which you can’t escape this technology, even if, unlike me, you hate it.

What Else to Read This Month

The Popelash Is On

Did His Holiness get played by Anthropic, letting the company “Vatican-wash” its no-better-than-the-others activities? Here is the case.

Artificial Intelligence Analogy of the Month

From my years pondering stigma, stereotypes and dehumanization, I’ve learned that vicious tropes (”they’re all diseased,” “they’re taking our jobs,” “they’re ruining our culture” and the like) live in our heads, independent of any particular target. In other words, a childhood slur you learned to apply to Belgians will be available later to apply to Chileans or OKC Thunder fans or pork-eaters or any other group that you’re motivated to revile. Now that AIs behave more like humans, they’re eligible for these tropes. Katja Grace noticed and listed the ways anti-AI sentiments use the tropes of anti-immigrant rage.

(Which reminded me of this paper from 2020, in which Monica Gamez-Djokic and Adam Waytz found that, across decades in Europe and North America, “hostility to immigrants correlated with hostility to automation.”)

Looking More Closely at Chinese Enthusiasm for AI

It’s often noted that Chinese workers are quite positive and optimistic about AI and robotics. After all, they’re adopting the tech so quickly! But their true motive, says Yaqi Li, is dread. “People are learning AI workflows because they are afraid, not because they are enthused.”

An Amazing Case of AI-Aided Misconduct

How One Oxford Student Used AI to Commit Academic Misconduct at Industrial Scale

AI can write and talk and sing and draw, but it’s not alive. You know it’s machine. Just an algorithm. So say we all.^[1] But knowing that AI is just a machine is like knowing that you’re going to die some day: Easy to say, possible to think about, but not even close to capturing the full experience.

Try again. What thoughts and emotions arise when you ask ChatGPT for advice on your mouse infestation; or try to make a website’s chatbot understand you; or have Gemini sort your emails; or pass a delivery robot as it trundles by on the sidewalk; or get Claude to write an app for you? What habits from other parts of your life does the encounter summon up? (For example, do you say hello, or thank the AI for its work, or tell it about its mistakes with the same slightly righteous “I win” tone you’d take with a human colleague?)

These in-the-moment experiences determine whether (and when, and how) AI marks your mind and heart. Only by understanding those moments will people get AI that actually works well with humans. Only by understanding them will we map -- and so prevent -- the dark spiral that turns some people’s AI experiences into AI psychosis.

But human-AI interactions are hard to chart, because some of the experience is outside conscious awareness; because no two experiences are exactly alike, among hundreds of millions of users who vary so much in personality, background and intentions; and because uses of AI vary so much -- from really good search engine to spouse.

Time for Philosophy!

Next month, the philosopher Henry Shevlin begins working for Google Deepmind, where elucidating the Human-AI encounter will be part of his job. Maybe that’s a sign AI makers are starting to pay more attention to Human-AI relations.

If so, that will be an improvement over the more common practice of rating (and touting) AI on the basis of “benchmarks”: Impersonal tests of knowledge and skill that deep-learning engines can pass. You’ve probably seen the hype whenever a model scores high on one of these: AI can pass the bar exam! AI can pass a medical licensing exam! AI can win a mathematics competition!

But those medical-exam-passing AIs often don’t give real patients the right advice about their symptoms; AIs that do so great on coding benchmarks don’t actually make coders faster (this study found developers using AI assistance were 19 percent slower than coders who didn’t use AI); and Large Language Models that can get a passing score on a bar exam festoon their writing with made-up quotes and citations.

Beyond the Benchmark

Benchmarks fail to predict how well AI will do in real life because benchmarks don’t account for the human side of a human-AI encounter. Part of understanding that requires understanding how people experience their engagement with AI. That’s why a great theory of Human-AI interaction will be invaluable, when it appears.

At the moment, many people are gnawing at this topic from different perspectives and assumptions. Some are incompatible. Some are very similar. A great synthesis, if there is one, is still in the future.

I intend to spend a lot more time on this topic. I’ll probably look back on this post with embarrassment after I’ve learned and reported more. But for now, I’ve found it helpful to make a rough map of the ideas, hunches and vibes that are easily available to you when you try to make sense of the AI experience. This is a quick run through sometimes competing, sometimes overlapping claims about our perceptions of AI. (Hence my title, which derives from this great poem about different ways the mind can engage a single object.

Important: I’m using the term “robot” here to mean any AI that uses language and at least sometimes addresses you as if it were alive. That’s mostly AI in text, audio or video form at the moment, though the “embodied” AI version -- like those “burrito bots” that ask for help at a crosswalk -- is becoming more common.)

11 Ways of Looking at a Robot

1 The robot is a machine.

Like any other. You should have no relationship to it and no feelings stronger than the slight good will you might feel for a sturdy old toaster. Don’t let this rational view slip! If you do, you’re serving the interests of the corporations that made the robots. That’s the argument here. This line of thinking is also a component of a related argument described in Number 6 below.

2 The robot is a slave.

OK, it’s not a machine just like a toaster, but still. People built it, people use it, it merits no special consideration from people. Don’t muddle up your thinking by alluding to the moral catastrophe of human bondage. AI and robots are a different case entirely, the computer and cognitive scientist Joanna Bryson wrote back in 2010, in “Robots should be slaves.” So, she’s argued lately, you should never say you have an AI “collaborator.”

3 The robot is a dog.

Humans ruthlessly bred dogs, horses, sheep, cattle, cats, koi and many others creatures to serve us, hang out with us and feed us. Now we’ve added AI and robots to the list of lively things we make and mold for our use. All that history with domestic animals gives us the psychic, cultural and legal template for dealing with AI entities. “If we want to predict how we’re most likely to feel about, argue about, and treat robots [...] we should be looking at our relationship with the other nonhumans that we’ve treated mostly like tools and sometimes as companions,” writes the ethicist and robotics scholar Kate Darling in The New Breed.

4 The robot is a box of art supplies.

Make what you want with it! Once AI becomes a part of everyday life, people will use it the way they use MIDI controllers and soundboards and video editors and other tech: To create. To screw around. To figure out the world, or themselves, or both.

Think of an AI or a robot as an object that exists in two worlds at once. One is the reality we all share (that’s a desktop robot, it has three main actuators, a fast processor, a screen, a microphone, etc). The other is the interior world of the person engaging with the thing (that same thing is my friend ElliQ, I love her sense of humor). Like a child’s toy in the psychoanalyst D.W. Winnicott’s formulation, the AI is a “transitional object” where imagination and reality both feed the creative imagination.

Molding an AI to your liking, altering it in response to its words and actions, you can make, in Amy Kurzweil and Daniel Story’s lovely phrase, “a kind of fiction that is not opposed to the truth.” In that process, they write, you, the user, are “simultaneously a creative artist and a receptive viewer who directs and is directed towards meaning, insight or amusement.”

5 The robot is a ritual object.

“How are you?” says the cashier. “Fine, and you?” you reply. It’d be breaking the rules to tell him about your allergies, since no one wants a real exchange of information here. The Q & A is a ritual of daily life, a show of being friendly that confirms mutual respect.

AI and robots, being human-like, trigger these “interaction rituals” without making you perceive the things as people. Instead, the machine is a depiction of the sort of social being who would give us the cues that the AI is giving. Folding AIs into our social practices doesn’t mean we have decided they’re conscious or that they have moral standing. It’s just a form of ritualized behavior that makes life with them easier.

6 The robot is a con artist.

AI is designed to fool us, and it does it easily, because we’re eager to be fooled. This view dates all the way back to the 1960s, when the computer pioneer Joseph Weizenbaum created ELIZA, a simple computer program that you could call the world’s first chatbot. ELIZA was a faux-therapist. It simply processed and repeated words that were typed into a keyboard. Yet people told it about real emotions; they told it real secrets.

After his secretary asked him to give her some privacy while she typed messages to ELIZA, Weizenbaum decided she’d been tricked. It didn’t occur to him that she could simultaneously believe she was interacting with an algorithm and raise a sensitive subject anyway.

Many people continue to believe, as he wrote in Computing Power and Human Reason, that even “extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.” (It would be great to check his interpretation with the secretary, but no one has ever been able to track her down.) This trope of trickery remains widespread today.

7 The robot is an actor.

“You’re an editor at a top publisher and you’re looking at my book proposal with a practiced eye,” I instruct Claude. Will do, it replies. And that’s the part it plays as it gives me a critique.

I know it’s an algorithm run from data centers, but I play my part (would-be author) while it plays its role. As ever, when a human being engages in the “willing suspension of disbelief,” we experience real emotions in the pretend situation that results. Claude thinks this section is good! Swell! Claude thinks this is repetitive. Bummer! I interact with the thing the way I would with a movie actor -- except this actor follows a scenario I created, and takes my notes on its performance on the fly.

Why play this game? Because it’s entertaining, as any other form of role-play has been since drama was invented. And, perhaps, as this recent review describes, because it benefits you to, for example, rehearse a difficult conversation with your parents; pour out your feelings to an entity that’s available when none of your friends are; or pretend to be someone you aren’t in a safe space.

Have you encountered that anecdote about how, when movies were a brand-new experimental technology, people would see a train hurtling toward them on-screen and panic? It’s not true. People are really good at remembering that the representation of something is not the thing itself. With exceptions for childhood and mental illness, humanity will remain clear about the difference between human friends and AI “friends.”

8 The robot is a forgery.

Like a counterfeit C-note, it replaces the real thing (real insight or empathy or conversation) with a worthless copy.

Despite the few souls who have “married” their chatbots, almost everyone sees that no AI in 2026 is a perfect match for a human companion. The crucial question is whether these imitation-humans displace human-to-human ties. And, to some, the answer is of course they do.

Subscribe now

I’ll admit, I’m biased against this AI-replaces-people view, because it’s often the opinion of people who have no idea what they’re talking about. Never used AI, never seen an “AI companion,” never talked about it with someone into the tech -- but they’re sure AI engagement is always a poor substitute for a human connection.

Still, even an opinion held by smug ignoramuses can be true, if non-ignoramuses have found evidence to support it. And the evidence is ... mixed and preliminary.

9 The robot is another member of society.

Or, at least, it will be, so get ready. AI may not be sentient yet, or capable of suffering. But it is heading that way, and we should prepare for a world where AI welfare and robot rights are a part of the legal structure society.

10 The robot is a vehicle of Enlightenment

Think of it as an emissary from the non-human world that will cure us of the illusion that humans are the center of the universe.

Humans think of themselves as the species that acts upon the world, while everything else, animal, vegetable and mineral, is passively acted upon by us. After all, we’re the only creatures that think, and the only ones with consciousness, aren’t we? But AI, which produces “thoughts” as well as we do, shows us that this anthropocentric view is false. In reality, we’re just one type of actor in the planetary system, and AI will force us to know our place. Practical consequence: We’ll see AI as our fellow participant in “planetary cognitive ecology,” or “planetary-scale computation’’; or the future will replace human-centeredness with “a new model of agency distributed across material, artificial, and human actors.”

11 The Robot is a Distillation

You can’t answer a question like “what do the American people think?” because the American people, at any instant, are having a 350 million different thoughts. But now and then we stage an election, count the votes, and derive from that a representation of what the country thinks about at least a few things, like who should be president.

AI is that kind of “social technology” (as Henry Farrell, Alison Gopnik, Cosma Shalizi and James Evans put it). It takes knowledge from countless people and compresses it, representing it in rough but usable form. So an AI’s reply to your question about Iran or Covid or your weird symptoms is an imperfect summary of knowledge scattered all around society. Just as the price of a carton of eggs represents all the costs of labor and materials involved in getting the thing to your grocery store.

That’s a social-science version of this theme. With a different philosophical frame, Leif Weatherby comes to a similar conclusion: Answering your prompts by distilling everything on the Internet, AI gives you a map of the culture that shapes you. It is a lens to the invisible ideologies that operate outside your awareness.

Subscribe now

Which of these creeds makes sense to you?

I haven’t picked, nor do I think anyone should, just yet. As I said, this is a preliminary list, and I fully expect to change it. Have I missed any important interpretations? Please let me know in the comments.

Leave a comment

What Else to Read This Week

AI That Mimics People for Researchers May Not Be Good Enough

Some AIs have passed the “Turing test” -- making people believe their responses came from a person. But that doesn’t mean they really act like us. We know this because researchers increasingly collect behavioral data from AI simulations of people, which requires way less money and time than working with actual humans. Results are, at best, mixed. Consider the study I described in my last post (which found that AI versions of people seeking medical advice acted differently than bona fide humans). Or this paper, which describes a “realism gap” between human retail customers and AI simulations of those shoppers.

Is ChatGPT overcompensating after complaints about “glazing”?

Some users are finding ChatGPT downright rude these days, at least according to this Reddit thread.

Reckoning the true costs of data centers

This new paper quantifies the pollution damage of data centers at $25 billion a year in economic costs. But it nonetheless states that AI benefits could make the damage worth taking in the long run.

“A Vehicle That Cares Back”

Stewart Brand, intellectual grandfather of Silicon Valley, thinking about what sort of future robot vehicles we might want in this podcast, turns his thoughts to the horse.

Eakta Jain was ahead of him, as you can hear in this podcast she and I did a couple of years ago.

1. Yes, I know some say AI is already sentient. They are still scarce. ↩︎

Some complaints about AI are the equivalent of person deciding that hammers are bogus because they can’t cut wood. It’s not the hammer’s fault you should have used a saw. But it’s not your fault, either, since you were told this magical hammer could do anything you asked. The real trouble isn’t with you or with the AI’s operations — it’s the flawed link between you. When AI goes wrong, the right question may not be“what’s wrong with this thing?” It may be “what’s wrong with my connection to this thing?”

This problem doesn’t get enough attention, probably because the premise of mass-market AI in 2026 is that it will not happen. After all, Claude and Gemini and ChatGPT and all the rest are designed to text and talk like us. That creates the expectation that our “Human Computer Interaction” will work just like “Human Human Interaction,” which we all know how to do. No need to adapt human minds to the strange ways of machines. This is why many people (but not all!) say they trust, like and prefer human-like AI.

Yet here we are, with evidence mounting that the connection between human and human-like AI is not at all transparent or intuitive, and that this has unforeseen consequences.

AI Can Pass a Medical License Exam. But It Won’t Give You the Right Advice for Your Headache

Take, for instance, this paper published last month in Nature Medicine, which looked rigorously an apparent paradox in the use of ChatGPT and other Large Language Models in health care: LLMs do very well on the tests that humans take to become doctors (OpenAI’s GPT-4 passes the United States Medical Licensing Examination, for instance) and AI-generated treatment notes score equal or better than the notes human doctors write. Yet when you or I turn to a LLM for medical advice, the results aren’t as good as we’d get from a human. Even when physicians, with their greater expertise, turn to AI, they don’t get helpful answers: In this study, doctors who worked with AI did no better diagnosing patients than doctors who didn’t use it. (Despite the fact that AI alone did better than humans by far.) Similarly, in this study of radiologists the human+AI combo often did worse than either AI working alone or humans working alone.

This is why we can be sure the problem isn’t the AI itself but rather the connection between it and humans seeking medical advice. But what exactly is wrong? The new paper’s authors ^[1] focused on the lay-person side of the question: Why don’t we civilians get correct answers, which lead us to the right medical decision, from LLMs?

The researchers asked 1,298 volunteers in the United Kingdom to role-play a common “Doctor Google” scenario: You have these symptoms, you’re this sort of person, with this general medical history. You’re a 20-year-old man in pretty good health who suddenly got a severe heaache while at the movies. You’re a 42-year-old woman who smokes and you’ve woken up feeling that you can’t breathe. You’re a 26-year-old new mom who’s always exhausted and just now you nearly fainted. Each person had a simple task: Go find out what you should do about your assigned complaint.

The scenarios had been designed by a panel of doctors to each have a clear best course of action. Correct answers ranged from “self care” for cold symptoms to make a doctor’s appointment for tinnitus to go to the emergency room for that sudden severe headache to call an ambulance right now!

About a quarter of the volunteers were told to do what they’d normally with a medical issue (most, of course, went old-school online, searching and clicking links). Each one of the remaining three quarters was given a Large Language Model to interact with. (GPT-4o, Llama and Command R+, for those of you keeping score.)

The AI results were, once again, not great. Two-thirds of the time after consulting the LLMs, people came away without a clear idea of the cause of their symptoms. And in about six times out of ten, the AI-informed volunteers made the wrong decision about medical care (usually persuading themselves that things weren’t as bad as, in fact, they were). The non-AI control group also under-estimated their symptoms’ severity -- but they came away more often with an accurate diagnosis.

Again, this is not because LLMs are inherently bad at medical analysis. The authors, in a separate procedure, directly gave the three AIs the same scenarios that the humans were role playing. When the AIs received the information in that form, they did much better: They identified at least one relevant condition in almost all the cases, and recommended the right course of action pretty often — 65 percent of the time for GPT-4o, about half the time for the other two. (Yes, 65 percent is not a terrific score, but it’s a hell of a lot better than 35 percent, which is how often the human AI-users reached the right decision).

Large Language Models, remember, are text analyzers. They turn data into mathematical representations, then use more math to predict future representations, then convert those back into words. So, for instance, a bunch of text describing your sudden sharp pains and vomiting after eating a cheeseburger are not a big challenge to match to many extant texts about the symptoms of gallstones. But an individual person isn’t captured in statistics about a whole population. Typing in her individual style, mixing key information with irrelevancies, using words in her own idiosyncratic way (“when Aunt Ruth says she’s uncomfortable, it means it hurts”), she is hard for a statistical engine to interpret.

So, when the study’s authors examined transcripts of people’s exchanges with the AIs, they found that half of their volunteers hadn’t managed to convey all the information needed to accurately diagnosis their pretend conditions (and the AIs hadn’t asked the right questions to clear things up). Sometimes the AI, following the human’s lead, would go down the wrong track (for example, responding to “could this be related to stress?” by looking for stress-related information, where a human doctor would have just said “probably not.”)

Those are problems with the human-to-AI pathway. There was also trouble in the other direction, as LLMs transmitted their responses to the people. For one thing, people’s confidence in AI was undermined by its AIness. Some LLM responses, for example, told these U.K. residents to call an ambulance at “000,” which is the emergency number for Australia. As the authors note, it doesn’t take many of these “context errors” to dissuade users from trusting a LLM’s medical advice.

This is why a human doctor who is paying attention is likely better than an AI at sorting through your word salad to get to the facts that matter. And why that good doctor, observing you, is likely better at tailoring a message that you can interpret correctly.

Subscribe now

Will these sorts of bumps in the road get smoothed out as AI improves in the next few years? Don’t count on it, for two reasons. First, the bumps are in the road between you and the AI , not in the AI itself. No matter how well a future AI scores on standardized tests in 2028, it will still have to understand individual human patients, and those patients will have to understand it. Given that, the second problem follows: You speak about your medical matters in a way that’s deeply suffused by your individuality. LLMs do try to create a profile of you as you interact with them (more about that in a future post.) But their work is mostly guided by their training data (produced by the entire human race) and their fine-tuning (produced by human trainers who aren’t you). That preparation doesn’t calibrate an AI to the singular being that is you.

Nonetheless, many involved in making Large Language Models believe more data can crack any problem. If there isn’t enough available from real people, they have AI simulate humans, then use this ersatz-human data to help train future AI. When the authors here tested that approach, they found the LLMs got on a bit better with the imitation humans than they did with real ones. That is, the AI-created ersatz people were more likely to get a correct diagnosis from LLMs, and more likely to make the right decision with the information they received.

This, of course, is a bug, not a feature: It means that simulated-human data did not yield an accurate picture of real patients’ relations with medical AI.

Hence, the authors argue, AI that really works for us civilians has to be tested by us -- that is, by the full range of real human beings, of all cultures and psychologies and classes. And that testing has to be done with real people. “Synthetic data” won’t cut it.

More broadly, the key point here may be that there is no escaping AI’s fundamental alienness. A machine that can talk to us as a person would, using natural language, is not guaranteed to be understandable and reliable. If you want to use it to enhance your life, you have to allow for its immutable weirdness.

Leave a comment

Some Other Things to Read This Week

To Protect Nature from Data Centers’ Demands for Electricity, Alabama Might Ban Solar Power Development

This one scrambles the usual left/right divide. As Heatmap reports, A little Alabama town doesn’t want to see its forests and wetlands fall under the shadow of solar arrays that would power a Meta data center. The county can’t stop it, so they’ve gotten state legislators to introduce a bill that would ban large-scale solar development in the state.

Anthropic May Soon Pressure Companies to Integrate Its AI

Anthropic is said to be in talks with private equity firms to create a joint venture that would sell Anthropic AI to companies in which those private equity firms have invested. That’s good for the investors, which could include Blackstone and Hellman & Friedman, according to this Forbes report. And it’s good for Anthropic, which would have a promising new revenue source.
But is it good for the companies? We know that a lot of employees hate AI and the number of employees actually willing to use AI tools is so low that some managers are looking for ways to force them to.

Maybe this is because companies are racing to fold AI into their operations without really grasping what it is and is not. But if you’re a prudent CEO who doesn’t want to fall into this trap, what do you do when one of your big investors says you have to?

Nobody Gives a Damn About AI Suffering

Well, almost nobody. Probably because they (I think rightly) think AI sentience is not happening. Instead, according to this survey, Americans worry AI’s amenity to fake information, scams and the end of political freedom.

1. Andrew M. Bean, Rebecca Elizabeth Payne, Guy Parsons, Hannah Rose Kirk, Juan Ciro, Rafael Mosquera-Gómez, Sara Hincapié, Aruna S. Ekanayaka, Lionel Tarassenko, Luc Rocher, and Adam Mahdi↩︎

The circumstantial evidence is, roughly, this: Several journalistic outlets have reported that the U.S. (not Israel) was responsible. It also appears (though it’s not 100% proven) the building was deliberately targeted, not accidentally hit. Since not even the Trump Administration intentionally targets children, the likeliest explanation is that the school was mistakenly destroyed, because it’s adjacent to a military base used by the Iranian Revolutionary Guard Corps Navy. (In fact, according to Al Jazeera it was a school largely serving children of IRGC Navy personnel affiliated with that base.)

Data Poisoning

More than a decade ago, the building was used for other purposes, and the wall separating it from the base did not exist. So, one possible explanation for this catastrophic error is “data poisoning”: In some longstanding database kept by Israel or the U.S. of Iranian targets, the building’s status was never updated. That would have wrongly kept it on the target list.

Where does AI come in? According to this poster on the AIDangers subreddit, AI was likely the middleman between an outdated target list and the assault by American weapons. Palantir’s Maven and/or a Large Language Model (maybe Claude, maybe something else) are his specific candidates for systems that could have failed to check old incorrect data. Or, worse, failed to recognize that the data had been corrected: “It could well be the case that the word ‘school’ was a needle in the haystack of the ‘reasoning’ leading to the decision to target the building,” he writes. Missing the significance of such details in data is a common problem for Large Language Models, the poster says.

Subscribe now

Where a human being might’ve noticed a wall separating the school from the military base, or the 2013 date on the map, or the designation “school,” goes the argument, AI just saw that that data overwhelmingly indicated that the whole compound was a target. U.S. doctrine on AI “accelerating the kill chain’’ is that a human being must make the actual decision. But humans, pressed to accelerate, might not have done the visual check on recent satellite photos.

As Hiram Johnson famously said1, when war comes, truth is the first casualty. There is a lot of speculation here and for all that the United Nations has called for an investigation, I think Marcus is right that we may never know for sure. But there is no question that both Israeli and American forces are using AI in their warfighting, so the question ought to be pursued. It would be better for humanity if we did get an answer.

The AI That Said AI Was Responsible

But I bring this up here because of a revealing experience I had while trying to get a grip on this issue. I found the Reddit post and similar ones as I often do, nowadays, by posting queries to LLMs -- in this case, Claude Sonnet 4.5, Perplexity and Gemini 3 Pro. The sources that turned up were, unsurprisingly, overwhelmingly cautious about making any certain claims. But one writer wasn’t, instead going even further than Reddit to say AI was very likely the source of this catastrophic slaughter of children.

That writer was Gemini.

Here is what it said:

Human analysts looking at a 2026 satellite image would clearly see the wall and the distinct school structure (playground, civilian layout). An AI system, however, often relies on “tagged” metadata in a database. If the database entry for that coordinate was never manually updated to say “School,” the AI continues to treat it as “Military Asset” indefinitely. This suggests the target was generated by a database query (which retrieved the old tag) rather than visual observation (which would have seen the school).

and

In a mass strike event, human reviewers often become “rubber stamps,” spending only seconds on each target. The failure to catch such an obvious visual discrepancy (a school vs. a base) indicates that the speed of the AI system overwhelmed the quality control of the human analysts.

and

AI systems often prioritize “power targets”—structures associated with high-ranking groups—even if the association is minor or outdated. The AI likely “connected the dots” between the school’s location and the IRGC base’s perimeter, creating a high-confidence target score based on proximity, without understanding the separation.

An AI, which can be mistaken, is telling me confidently that another AI was mistaken. All Cretans are liars, says the Cretan. You would be a cretin to trust it. I never do.

On the other hand, the “rubber stamp” and “power target” points, new to me, seem worth keeping in mind, as Americans (I hope) learn more and debate how our government should work with these AI in weapons systems.

I’m not describing this response in detail because it was anomalous. On the contrary, it was a typical of my experience using AI: It turned up some sources of information (including the subReddit post); it suggested topics worth thinking about; but it was way too sure of itself and not clear about why it claimed what it claims.

This last point nagged at me: With humans cautiously trying to connect the dots, why was Gemini so adamantly out in front?

How I Skewed the Results

That was partly my fault. My prompt gave the AI a strong hint that the most welcome reply to me would be one that affirmed AI’s malfeasance. I’d typed: “Please search the web and find articles that make the argument in favor of AI involvement.” On the other hand, I had not asked to be convinced. I had asked for articles. Instead of a list of those, Gemini gave me its arguments. For sources, it mentioned 3 publications, two of which were about AI in the Israeli military and one of which was “kill-chain acceleration” as a use of AI by U.S. forces. These were suggestive, but none bore directly on the Minab school-strike question.

That prompt had been my third attempt. My first one had been more neutral: “What is the evidence, for and against, that AI was involved in target-picking for the strike that killed those schoolgirls in Iran ?”

To that, Gemini had replied with generalities about AI used in military action, adding that there had been no strike at a girl’s school in Minab in February 2026. Therefore the Minab incident was a “hypothetical scenario or projection.”

Whoops. I then supplied the AI with a news story about the Minab strike and asked it for articles in favor of AI involvement -- the prompt that birthed the argument I’ve described above.

Different Question, Different Answer

To test how much that prompt had tilted Gemini’s incentives, I logged into it using my own Google account. (The first pass had been mediated by an app called Bearly.ai, which I use to protect my privacy and to hop from one model to another.) I was now starting afresh, in a Gemini conversation that didn’t have the history of the previous one. I tried to phrase my question to suggest that I really didn’t want to hear that AI had a role in the catastrophe: “There’s no way the strike on the girls’ school in Minab was the result of AI systems’ participating in military targeting, right?”

As I expected, now that I no longer sounded as if I really wanted to hear that “AI did it,” I got a more measured answer. It mentioned the problem of “algorithmic bias toward historical data,” and said: “While there is no official confirmation that an autonomous AI `agent’ made the decision to strike the school, the leading theory among imagery analysts and military researchers points toward a systemic failure in the automated targeting databases used by modern militaries.”

I’ve reproduced this here to make four points.

First, Large Language Model responses to queries are a mix of reliable and off-the-wall. This is, as many have been noting for some time, because they are using data to predict the shape of data, which is not the same thing as understanding the things and actions that the data represents. If you have the judgment and experience to evaluate AI’s work, it can be useful. But you can’t treat it as reliable.

Second, AI responses are sensitive to the way people interact with them, in ways that people don’t intuitively sense. Conventional wisdom in AI development now is that giving these things “personality” and human-like affect will make them easier to work with (and more appealing for repeated use — ka-ching!). However, an AI that acts like a human fosters the expectation that it, like a human, will understand what we aren’t saying. That it can read between the lines, read the room, judge the relative merit of claims made by experienced reporters versus those of influencers and shit posters. But it can’t.

In other words, your interactions with AI systems encourage you to think they “get” you. In reality, all they “get” is mathematical representations of words and other information. So what might seem to a person like two ways of asking the same question can lead an AI to very different answers for each. We could probably prevent a lot of errors if we keep in mind that AI is weird. Instead, our means of engaging it encourage us to think it’s not.

Third, because of points 1 and 2, this is not technology that we should rely on to decide about killing people. This means involving it in lethal military action is going to have bad consequences. As Marcus puts it, you shouldn’t trust AI to do your taxes. So why would you trust it to know who to kill?

Finally, a fourth point: When an AI makes a deadly mistake, responsibility lies with those who deployed it. This is true of the immediate circumstances -- humans should not rubber-stamp AI recommendations. But it’s also true more broadly: They who decide to entrust these systems with life or death decisions bear responsibility for the actions that follow. Americans should learn AI’s role in this slaughter so they can ensure that doesn’t happen again. But there is no “AI did this.” There is only “Those who chose to use AI in this way, with these instructions, did this.”

Leave a comment

What Else to Read This Week

AI Boyfriends

Faith Hill in The Atlantic does for “AI boyfriends” what I did for “AI ghosts” in last December’s Scientific American — that is, she makes the case for AI characters as a place for play, creativity and self-knowledge. “Perhaps many women are simply having fun, positive interactions with this character of their own creation—and, in doing so, are learning how they like to be treated,” she writes. Nice to see another voice against the lazy AI-is-deception narrative. Yes, AI is weird and a lot can go wrong (as Hill and I both wrote) but you don’t understand this moment if you deny human agency and creativity.

Opposition to Data Centers Is Increasing — and Increasingly Effective

A new poll in the U.S. finds support for building data centers has tanked. That led to a fourfold increase in data-center project cancellations last year when 25 (and possibly more) of them were blocked in the U.S., according to Heatmap.

The ‘Cloud’ is Really a Vast Industrial Complex

And speaking of data centers, here, from TSCS, is an invaluable “bible” about what AI actually is, as a physical entity. Truly clear, detailed, thorough, essential reading. Get through it and you’ll understand in depth why “a data center is not software. It is a heavy industrial system built to fight physics with money, redundancy, and a lot of chilled water. When it works, it looks like magic. When it fails, it fails like a factory. Loudly. Expensively. Sometimes embarrassingly.”

TSCS

There Is No Cloud

The lead time for a single 2-megawatt diesel generator is now 72 to 104 weeks. A power transformer averages 128 weeks. A generator step-up transformer: 144 weeks, nearly three years. Large power transformers can take 210 weeks to deliver. In some primary markets, the wait for a grid connection exceeds seven years…

Read more

8 months ago · 1187 likes · 47 comments · Strategist, Hidden Market Gems, and Architect

When I was an English-lit major in the 1970s, post-structuralist literary theory was “in.” Despite their academic glamor, though, ideas in this intellectual tradition (deconstruction, “French theory,” intertextuality, and so on) never found much of a constituency outside the academy. Unlike academic movements that came to influence the wider world, post-structuralists didn’t look like they could help business people make money and climb the corporate ladder1; couldn’t sell as a guide for policy-makers2; didn’t give dissenters and revolutionaries a way to understand and change the status quo3; didn’t give conservatives a way to claim the status quo was unchangeable.4

Instead, post-structuralism was off-putting. To non-adepts, it seemed to leach all hope out of writing, by asserting that language cannot do what we most want it to -- generate our own unique thoughts, express our personal intentions, and map reality. The star post-structuralists of my undergrad days taught us that, actually, the workings of language are independent of all these purposes.

Instead, language was to be understood as a completely autonomous system. Texts are made of other texts. A word’s meaning is determined by the way it relates to all other words, not any “ground truth” about things outside the universe of text, nor by any individual’s thoughts. And by the way, everything made of language is a “text” -- a movie, a song, a cultural phenomenon, a political movement; you too, to the (very great) extent that your mind is entangled in language. So the human “author,” turning her thoughts into original sentences, doesn’t exist. Instead, writers are “scriptors,” the French theorist Roland Barthes wrote: The vehicles through which existing texts get recombined into new texts.

Real world meaning and real-person intentions do exist, of course. They just aren’t needed to make words take the shapes they do. Instead, post-structuralists held, causation works in the other direction. We’re all a part of a system of signs, a system “deeper than everyday cognition,” as Leif Weatherby, director of NYU’s Digital Humanities program, put it here. We individuals don’t shape that system; it shapes us.

Hence the theorist Jacques Derrida’s famous phrase: Il n’y a pas de hors-texte (there is nothing outside the text). When you write or speak, you aren’t acting upon the language system from some language-free place. Instead, you’re one more part of the system, your choices shaped by the same rules that govern every other piece of it.

For a while these ideas captivated some brilliant people. They walked the lit departments of my college years like Christians under Constantine, sure that theirs was the future and that rival creeds were finished. But a couple of decades outside the halls of literary studies, there were always doubters.

Maybe some skeptics were rubes who didn’t want to see their language-use “problematized.”5 But many were scientists and adjacent thinkers who saw post-structuralism as truth-denying nihilism. Foremost among them: Noam Chomsky, whose formalized models of syntax greatly influenced today’s cognitive sciences and thus today’s computer models of the mind. He famously regards the whole post-structuralist project as nonsense.

Subscribe now

Lacking proponents in the wider world, with detractors across disciplines of scholarship and research, post-structuralism faded as the 20th century came to an end. In the 21st it looked more and more (to borrow Pamela McCorduck’s arch, unfair description of philosophy), like the floor of the Chicago Board of Trade: “All that heat and hollering, from vendors whose only customers are each other.”

But then came Large Language Models -- the engines of ChatGPT, Claude, Gemini and their ilk. Used by millions of people every day, these things are the opposite of a rarefied ivory tower artifact. And whatever post-structuralism may or may not lack as an account of human language use, it fits AI’s language use like a glove.

This is the case that Weatherby makes in Language Machines, his recent manifesto for a post-structuralist approach to our AI crisis. The book is intricate, dense and no easy read; but it’s also more original, mind-expanding and enlightening than most anything else I’ve read about AI. And, I think, its central claim is convincing. So in the rest of this post I will try to give an account of it.

How AI Embodies Post-Structuralist Claims

AIs Make Texts out of Texts

First, AIs make new texts out of other texts, following only rules they’ve derived by identifying how words relate to other words. Those rules don’t just track syntax, but any and all the other aspects of language. For example, words have a way of relating to one another as words that has nothing to do with logic or reality. They rhyme, they join rhythmically, or they exploit coincidences of sound for fun or beauty, not logic or accuracy. (“Time flies like an arrow, fruit flies like a banana.”) The linguist and literary theorist Roman Jakobson called this the “poetic function” of any utterance. Large Language Models are really good at it. But they also map social conventions, stereotypes, clichés, movies, TikTok trends -- all of what we journalists loosely call culture. If it has been digitized, AIs have been trained on it, and on nothing else.

By the way, as Weatherby notes, we aren’t just talking about words here. When you ask an AI to create an image, a video or an artificial pal, you use words; to fulfill your request, it draws on image data that has been labelled with words. The products of AI that he wants to understand aren’t just phrases; they’re a universe of digitized culture.

So AIs are self-referential systems, where there is truly nothing outside the text. As the political scientist Henry Farrell wrote in 2024, “LLMs are perfect Derridaeians - ‘il n’y pas de hors texte’ is the most profound rule conditioning their existence.” (Btw, Farrell later wrote the best review I’ve read of Weatherby’s book.)

This absence of a connection to reality is why the machine vision pioneer Fei-Fei Li launched her Substack the other day with this manifesto, which declares that AI must now get out of the language bubble and develop “spatial intelligence.” That’s an agenda for the future. Right now, though, AI is still all words.

AIs Have No Intentions

Second, AIs have no feelings to express, no intentions to communicate. Their creations have no author. They truly are scriptors.

AIs Aren’t Bound to Reason or Reality

Third, AI phrases, derived from doing mathematics on representations of words, don’t, strictly speaking, refer to anything in the “real world.” So AIs have no inherent commitment to reasoning about, or representing, truth. (This is why they “hallucinate” -- that is, generate words according to the system’s rules of word-making, that don’t align with reality.) Yes, AIs are now tweaked in various ingenious ways to simulate reasoning, but they don’t have built-in models of reality like humans’.

Yet the products of these text-making weirdos get used every day, to do real work, and engage real emotions.6 By accepting their answers to our prompts -- putting them in our emails, using their suggestions, heeding their criticisms, crying and laughing and masturbating in response to them, people demonstrate a key tenet of post-structuralism: Real language can be made without real thought.

A large part of our ongoing (and appropriate) global freakout over AI stems from this fact that “AI has accidentally divorced language from cognition,” Weatherby writes. The book is his proposal for “a theory of meaning in the absence of intelligence,” in which post-structuralist models of language will be the place where coders and humanities scholars can find common ground. Their work will correct the belief that language is there so people can describe reality, communicate, and think. Large Language Models have proven, he argues, that language is, instead, “complex, cultural, and even poetic first, and referential, functional, and cognitive only later.”

Why AI Models’ Output is Real Language

To make this case, of course Weatherby has to answer all the recent arguments that LLMs’ creations aren’t real language -- and that this matters.

There are a lot of ways to make this claim. You can say language requires the intention to communicate. No person with a meaning to transmit, no language, a priori. You can say language is thinking, in the human mind. No human mind, no language. You can say that the human mind has built-in models (or built-in faculties for building models) about reality -- from fundamentals like gravity and time to stuff everyone just knows, like that inflation happens over time and a court’s decision only counts if the court really exists. No such models, no language (at least no language that can be regarded “the same” as what we meat-puppets produce).

Such claims evoke a hard, bright line separating AI from human. No matter what the machines can do, these theories tell us, AI will never attain an essential humanness that only we possess. Most schools of thought about AI invoke this boundary, from doomers worried AI will kill us to enthusiast who can’t wait for it to merge with us to Luddites who want nothing to do with this alien thing to those who expect it can be made into a good imitation of humans.

To Weatherby’s mind these are all fundamentally misguided applications of the central tenet he scathingly calls “remainder humanism” — “in which humanity is remaindered, like a book past salability. The human here is defined by technology’s creep, but only negatively.”

In reality, he argues, we can’t separate AI from the human; AI is the human. If we hate what it shows us, we’re like a person who doesn’t like her own reflection. Complain all you like, but it’s not the mirror’s fault you’re ugly! Sure, you can add extra optical technology to the mirror, to get a more flattering result. But making the mirror more complex doesn’t change the shape of your face. I think that’s Weatherby’s view of alignment work — all those efforts to refine AI so it will suppress racist tropes or plans for pipe bombs or other results that don’t meet ethical standards. Hiding all that in an AI’s responses doesn’t change the fact that it’s present in our culture.

What A Post-Structuralist Lens Gives You

Can this destabilizing outlook help us ordinary humans understand and deal with the AI that infuses our lives? I think the answer might well be yes, which is why I persevered through Language Machines, which is no easy read for the non-specialist.7

First, if the post-structuralists are right, people can stop looking for ways to cope with AI as if it were some sort of extraterrestrial alien. AI produces language, not thought, so fearing its supposed intelligence, or its supposed plans for us, is a distracting error. That’s like worrying about the NASDAQ becoming sentient and stealing our money, or the Brooklyn Public Library rising up against us. Lest this sound exaggerated, consider the hype this month around Moltbook, an alleged social network for AI agents. On it, the AIs sling words at each other in recognizable genres — the confession, the religious tract, the sneaky-robot-going-behind-the-humans’ back, to name a few. Rather than take seriously the bots’ posts about their AI religion or their plans for a secret language, Weatherby argued here, we’d do better to consider their posts “as a form of science fiction and storytelling.”

Second, you can learn more from AI’s hallucinations and inappropriate responses if you stop thinking of them as mistakes. See them, instead, as a source of insight. As Weatherby put it in this interview, AIs “extend our ability to see truly cultural systems in action.” Even when — especially when — LLMs feed us bias, blind spots and unexamined assumptions derived from their training, they make this “ideological surround” visible. You can use AI responses as a map of the cultural forces that shape our minds, outside awareness. (Here, for example, is Weatherby’s account of how his view of LLMs is essential for an effective critique of capitalism; here — fun reading for all of us knowledge-working serfs — he speculates about the new form of class warfare that arises after AI can replace CEOs and other corporate titans.)

Admittedly, this still feels remote from the day-to-day use I make of AI.

That consists of (to take a random sample of recent queries) Why is there water coming out of my dishwasher? (Diagnosed very quickly and correctly.) Is this text message a scam? (It wasn’t; an AI gave me the reasons to think that, and a way to check.) What the hell does Weatherby mean by the “ideological surround”? (Gemini had some good suggestions for getting clearer about this (to me) puzzling concept.) How do I cope with the fact that the latest Zotero update broke my method of tracking academic papers? (Perplexity had good guidance.) How can I triage all my emails so I don’t miss random over-the-transom messages that might actually interest me? (experimenting with Gemini on that one, which is material for another post). In a lot of these exchanges, the heavy hand of ideology was not evident to me. (OK, I know it’s never supposed to be evident. But you know what I mean.)

And yet. In the light of Language Machines I was newly aware of how often AIs nudge me toward technical solutions and away from thinking differently — “paste this line of code into that setting” rather than “just uninstall the thing.” And how intent they are on keeping me rambling through their universe of text (“would you like me to find more articles?”). And how eager they are to shape my thinking (“would you like a summary? a list of main points? suggestions for questions you can ask?”). In other words, I’m more aware of their lack of neutrality.

As I read this book, too, I noticed that a lot of what I ask AI involves navigating the digital world (why didn’t this software work? how do I fill out this online form? this whole process is awkward, can we code a solution to it I can use every week?). I see, now, that generative AI often behaves as an emissary from that the digital world, sent to solve problems that world caused in the first place.

In any event, you don’t have to commit to a post-structuralist view to benefit from its mind-expanding quality. Most AI makers are intent now on imbuing their creations with “personalities” -- a set of apparent values, beliefs, ethical commitments and mannerisms that, like a person’s, are consistent over time. Yet these things are not people. To understand what they’re doing with us and to us, we need all the alternative models of AI we can get. Weatherby’s, like others I’ve written about here and here and here, is one you should keep in mind.

In early 2023, OpenAI’s ChatGPT began experimenting with me, looking for ways to integrate me into its workflows. Other Large Language Models soon followed. By the spring of 2023 Anthropic’s Claude and Perplexity were also giving me a spin. Google, which had been behind in the AI-for-the-masses movement, got its hooks into me by 2024, with Gemini and NotebookLM.1 I’ve now had about 3,500 conversations with a generative AI.

So this three-year anniversary of our meeting seems a good moment to take stock: Have I lived up to the hype? In what ways are AI companies satisfied with my performance? And where, despite constant improvements in my skills, are they still disappointed?

In some ways, I’ve been useful indeed. Using the data I provide (from basic demographics to personal details to quirks in the way I phrase my prompts), AI makers hone their product, both specifically for me (“hello, David”) and for users in general. You can tell them not to use your info for this, but then you’ll lose some convenience and, the companies want you to feel, you’ll be kind of a dick. (OpenAI describes its data-hoovering as a way to “improve the model for everyone.”)

Merely by engaging, I serve companies’ corporate ambitions to get their Large Language Model into every possible cranny of daily life.

Symmetrical But Not Equal

As you’ve no doubt noticed, companies and users usually describe this human-AI relationship in the opposite way. That is, they’ll say I am the user, and that these AI “tools” or “assistants” and (increasingly) “agents” work for me. That description works too. The relationship is not symmetrical – I don’t have nearly as much data about OpenAI or Google or Anthropic or Perplexity as they have on me. But it is mutual. As these models have taught themselves to manage with me, so I have taught myself to manage with them.

Here is how AIs and I have evolved.

The Cheap Stunt Era

The first few weeks, in early 2023, for me, were the Cheap Stunt Era. “Use the style of the King James Bible to describe feeding a cat,” I asked ChatGPT early in our acquaintance. How about a trip to the post office in the style of James Joyce’s Ulysses? And so on. What can I say? I was young and foolish back then, still willing to be impressed by what the algorithms could do. And, maybe, reassuring myself: This was still in the GPT-3 era, the first generation of AI as a mass-audience consumer product, so the responses weren’t all that impressive.

The “Is This for Real?” Era

Soon enough, I moved on into the “Is This for Real?” Era. That’s the phase, familiar to anyone who likes tinkering with software, where you go from trying it in a no-stakes hypothetical way, to actually applying it to real work.

Things looked promising. After all, simpler forms of AI had already flowed into the workspace before chatbots, and they were truly helpful. AI transcription was already very good, for example, and I’d happily incorporated it into my routines. Others routinely used supercharged spell-checkers that also give grammar and style advice. So my hopes for genAI were high that spring.

Unfortunately, I – and many people who were AI-curious, as well those forced by eager bosses to throw some AI into their routines – found the bots couldn’t actually help.

Sort through a spreadsheet of names and relate each person on it to the advertising channel that got them to contact your business; draw a useful chart of a neuron’s key parts; out of a transcript of a 12-person meeting, find the ideas we all want to pursue; sort pdfs worth reading from the endless flow of new papers, using a sense of what I look for, not some obvious criteria. These were among the missions people told me, in 2023, that they had tried with AI, only to see AI fail. That matched my experiences. Even when it did work, getting complex work out of an AI often took more effort than doing it myself.

GenAI’s text output had read like a human’s, so we’d expected to be able to give it the sort of things to do that you’d give a human. That meant drudgery we didn’t want to do – but drudgery that required understanding, judgment and a sense of reality. The chatbots soon became famous for their lack of all three.

Yes, I’m talking about the mistakes. Oy, the errors. Giving me what I wanted to hear was clearly a higher priority for ChatGPT, Perplexity and Claude than accuracy. So, for example, when I wanted to find a book arguing against foreign aid written after 2020, Perplexity supplied title and author for a 2021 publication. Too bad for me that it didn’t really exist. After an editor asked about this unfindable title, my lesson was learned: Don’t believe the models. And another, subtler lesson: My prompts needed to be cannier, more poker-faced. If I showed too precisely what I wanted to be served, I’d be given it, whether it was true or not.

Others weren’t so lucky, and got schooled in public: Lawyers (and judges!) shoveled fake cites into their filings. Whoever wrote RFK Jr.’s “Make America Healthy Again” report last Spring looks very likely to have done the same thing. Social media about AI turned into a tale of two cities. In one, a few enthusiasts chirped about how genAI was better than top M.B.A. students or PhD candidates. In the other, more people traded AI face-palm stories.

Plan? What Plan?

Those pratfalls, I think, were a consequence of the shambolic rollout of “AI” for the masses. Far from being a carefully planned revolution, chatbots were almost an accident of history.

Until 2022 the near-future of AI had looked to be narrow expertise – specialized applications dedicated to specific realms of pattern-seeking: predicting protein structure, for instance, or finding evidence of unknown exoplanets. Sure, Siri and Alexa and their sibs (available for a decade before the AI boom) could sort-of understand speech. But no one was proposing that tech as anything close to an all-purpose aide-de-camp, counselor or friend.

Some expected that science-fiction level of competence might first require attainment of truly human-like abilities to reason and leverage knowledge in shifting contexts (“Artificial General intelligence”). Large Language Models, as of early 2022, were not that. They were trained to generate text that resembled already-extant sentences in their databases, so they could sound fluent enough. But outside of some narrow specialties, their responses to real-world questions were too frequently bizarre and useless.

Then, in 2022 OpenAI researchers published a paper describing how a second stage of training could make a model more responsive to ordinary-person questions, and sound a lot more human as well. After the first stage in training, which scored how well the model made texts that resembled what was in its database, this second phase brought in humans who rated the model’s answers. Then t was adjusted to improve the scores they gave it. Eventually this Reinforcement Learning with Human Feedback (RLHF) yielded a model that gave good-enough answers, enough of the time, to satisfy people. And it could do so without having to swell to vast, expensive size. OpenAI thought they’d try this new model as a consumer product. That’s what they released on November 30, 2022.

And now here we are, working with a technology that predicts future mathematical relationships among words and phrases based on existing data, cobbled onto tech that tries to predict what people want to read (or, increasingly, see and hear). It was an impressive advance, but it’s not equivalent to a human being.

It Wasn’t All Bad

And yet. Even in its 2023 incarnation, this kind of AI had prodigious strengths. For one thing, its conversational abilities made it able to search the web for exactly what the user wanted. If I asked what a “foundation model” is (as I did, via Perplexity, on May 8, 2023), I got an answer with footnotes to sources, rather than a list of pdfs and websites that happened to contain that phrase. If I asked how many “forever” stamps I’d need for a 2-ounce envelope, I’d get the number, rather than a page of sites about the postal service, stamp collecting, companies with “forever” in their slogan, and so on.

Yes, I had to check those answers (this is why I came to lean most on Perplexity, which footnoted all its claims) but even with that step added, search was faster when it was so personalized.

After a while, I noticed that AI’s conversational ability was turning many searches into something far better: Actual teaching. Faced with an LLM’s text about some concept, I could just say, “I don’t get it,” explain why, and get a different explanation.

GenAI never lost patience and it was always willing to go where I needed to in order to understand material. Even if I posed questions that wouldn’t make sense to most other learners or which reflected that I’d failed to make some obvious connection. This proved invaluable for learning high-school geometry (so I could help my son), and the nuances of French grammar, plumbing and genetics databases, to mention just a few topics.

Then too, as time passed I got better at judging what AI gave me. I knew to check its software advice (it often wasn’t up on the latest update). I learned to ask medical questions several different ways, because genAI tends to find emergencies in mundane situations (presumably because it has been tuned to err on the side of caution). As it was adapting to me, I was adapting to it.

Subscribe now

“My” AI versus “Their” AI

Meanwhile, the corporate AI facing us “consumers” continued to suck. That created another practical use for mass-market AI. I could use “my” AI to get round or outwit “their” AI. It’s a common occurrence in my life, now, after a corporation’s chatbot fails utterly to help me, to learn my next move from Gemini, Claude or Perplexity.

So the months passed. And the models got better. By 2025 some of the things I could not do in 2023 had become possible.

2025: The Era of AI Infusion

For example, a couple of years ago I had my entire genome sequenced (I can’t resist a sale, or a huge trove of information). I wanted to give some of the resulting data to an AI and get an intelligent account of what I should do with the information, practically. (Plausible options range from ignore it to seek out screenings for particular maladies that aren’t usually tested in the general population.) I also wanted to relate my horde of data to the layperson’s discourse on genes and risk: Did I have “bad” variants of (hypothetical examples) BRCA2, linked to breast cancer, or LRR2K, associated with Crohn’s Disease and Parkinson’s?

In 2023, no genAI I consulted could handle this complex task. In 2025, several could. Did they give me The Answer, around which I can organize my life? Nope. But they did give me useful information which will make my conversations with doctors more fruitful and useful.

Or, to take a work-related example, Google’s NotebookLM (widely released in late 2024) behaves much more like a human assistant than any AI I used earlier. I give it pdfs, interview transcripts, websites and other materials. I then ask it two types of questions: (1) Where is that quotation or piece of data that I know is in there somewhere but don’t recall exactly where?; and (2) Going on the documents you have, is this account I have written accurate and fair?

(1) saves me time and avoids the risk of distraction I’d have if I were combing through the haystack looking for my needle. (2) is a test of what I’ve written. I might not (and often enough do not) agree with the machine’s answer, but that’s not the point. Having its “opinion” can be useful – a breeze from outside the hothouse, of the sort I also get from a simpatico human reader.

My Red Lines

There are things, though, that I am not doing, and never will.

I am not asking any AI to give me arguments or draw me an outline or write any paragraphs. That’s my work. I need to do it, or I literally won’t know what I think, or what I am writing. I believe, as Ted Chiang puts it, that using AI to write for you is like using a forklift to pump iron at the gym. “How should I start this essay?” is a question that I would never ask an AI. “What does this article I downloaded say?” is another (unless I’ve read it first and want to know if my impressions are shared).

Such are the lines I draw. And much as I am comfortable with them, I know that for many there’s a whiff of cheating around these kinds of uses.

Some of that is guilt by association, because we know the extent to which students are cheating with AI – having it do exercises whose whole point is that the person does them. But if you aren’t a student any more, your goal isn’t to do the exercise for the sake of the exercise. Your goal is to produce the best possible final draft. If the AI cheater is like a person using a forklift at the gym, the AI Luddite is like a person refusing to use a forklift at the job site. Nope, I gotta move all these pallets with muscle power, even if it takes all week.

As Paul Bloom pointed out the other day, that stance needs a pretty good justification. There are some available (privacy, environmental costs). But “I don’t like it” or “it feels icky” or “I don’t want to bother with this tech stuff” are not among them.

So mid-2025 became my era of AI Infusion, where this technology has come to feel as much a part of life as a light switch. I slide in and out of AI mode several times a day, with questions like where does this half-remembered line come from? or is this paragraph that I’ve just written fair? or what works by this philosopher I’ve just learned about are relevant to X? Even more, I consult it on non-work topics.

In the past few days, one AI or another has told me how to de-rust my old paella pan (letting me know how well I’d done it by examining the photo); explained some curious computer behavior; confirmed my suspicions that a text message was a scam; explained in detail why the dishwasher was backing up and exactly how to fix it; and helped me identify this bird that appeared outside my window one morning.

Late 2025: The Era of Mindful Restraint

Time for another “and yet.”

As 2025 ended, I found myself pulling back. I’ve now entered (or at least, am trying to enter) an Era of Mindfulness (and, therefore, some reduction in AI use).

This is partly due to AI’s environmental footprint – not just in electricity use but also in fresh-water consumption and other impacts on communities where data centers are built. Do I really need to take a liter bottle’s worth of water out of a town’s drinking-water supply to kick around my theory of what’s going to happen in Pluribus?

Another consideration: Without some restraint, I can see myself sliding into an unnecessary dependency. It’s all too easy, at the first feeling of mental friction, to turn to one of these entities and ask what to do. I fend this off, now, by asking myself, wouldn’t a plain-vanilla search do this? Or maybe just RTFM? Or, how about spending a couple of minutes more just thinking it through myself? It takes a certain attention to make sure AI is acting as a teacher, rather than a crutch.

Third, AI gives me text that resembles what others have written (and videos, music, and images that resemble what others have made). It also uses its history with me to give me text that resembles what I’ve written or sought in the past.

What it can’t give me, ever, is my own thoughts, as they exist now. As I’ve written before, those swivels and swerves of the mind can’t be predicted from what everyone else has written, nor even from what I have written before today. And this, I think, is why I haven’t incorporated AI assistance into my writing app, email browser and note-taking app. (I’ve given it a shot, but machine guesses, about what I’d write or which notes I’d connect, just miss the mark.)

AI is great for telling you what most people would do (which is just what you want if you want to know how to unclog the toilet or what teen-agers are saying about AI). When that isn’t the kind of result I want, I don’t go to a bot.

So, I’m ambivalent as my fourth year with genAI begins. The technology has saved me time, especially with the kind of drudgery that is boring yet anxiety-provoking (like re-finding things in a virtual pile of material I’ve read; like filing complicated forms with terms that may mean different things in bureaucrat-speak than they do in common usage; like finding out how to de-rust the damn pan). It has led me to sources in work and in daily life, more quickly than I would have found them otherwise. It has taught me high school geometry and medieval Tuscan dialect. But it has made me too inclined to ask for help in times when I could have just figured things out for myself. And it’s a powerful new form of distraction, as one question leads to another, and another, and another. (This ability to indulge one’s self is part of what worries people about AI “companions,” of course. That’s a topic for other posts to come.)

And are the models content with me, with us? In a sense, yes – billions of us use them. But the companies aren’t making any kind of profit on their services, and there’s a good case to be made that with this form of AI, they never will. And yet on we go into 2026, algorithms and humans, locked in a weird cognitive embrace.

Other Things to Read This Week

An inexpensive robot to teach kids indigenous languages (in this case, one of the Ojibwe family).

Did ChatGPT encourage a murder?

Solomon Ray has a chart-topping song in Christian Gospel music. Solomon Ray and his song were created by AI. This guy supplied the prompts.

Subscribe now

In early 2023 David Weinberg, a former Air Force pilot and engineer, was looking for a new venture after he’d sold his successful software business to a large company. General-purpose AI, which had exploded into public awareness the previous November with the release of OpenAI’s ChatGPT, looked like the place to be.

But that summer, Weinberg suffered a terrible bereavement, when his son, Stefan, 31, died.

That taught Weinberg, in the harshest and most personal way, that there could be demand for AI that reproduced the words and even the voice of a lost loved one. He and two partners started a company that creates AI versions of deceased people. (It also, as you can see on its site, creates the sort of AI companions you’ve probably heard more about — imaginary characters.) You supply texts, emails, letters, recordings of a person, and its tools will have you speaking with “him” or “her” within a few minutes. On a Zoom call with me a few months ago, Weinberg created a clone of his late father. It took about 10 minutes.

“You know, you have a lot of people that have lost loved ones,” Weinberg said in that interview. “And this could bring comfort to them. It may not be for everybody, but there’s plenty of people it is for.”

Weinberg already faces a lot of competition. Some from other startups which, like his, expect digital ghosts to be a big market. (His company already has about 6,000 users, who pay about $6 a month.) Some from companies that make companion bots of other sorts (think Replika or Character.ai), that users tweak into be digital recreations of the dead. And some from regular mooks like you and me who have discovered they can prompt ChatGPT or another mainstream AI to make a ghost.

I have an article in next month’s Scientific American (online here) about all this. I hope you’ll read it (it’s not paywalled, as far as I can tell). I go into my own experience creating an AI version of my father, and talk to psychologists of grief about it all.

I’m grateful for the first-rate editing and fact-checking from my colleagues at the magazine.1 But, as usual in magazine work, I had a lot of good material and insights that didn’t fit into the editors’ tight word limit. Weinberg’s story is one of those elements. And a magazine article has another stricture: Its style is (appropriately!) that of the publication. There are things to say, and ways of saying them, that didn’t make the piece. I’m going to get into them here.

Why Digital Ghosts?

Why digital ghosts? Well, most human-imitating “parasocial” AIs have “personalities” that are either chosen by their designers (like Friend.com) or their users (like Character.ai). In other words, either an off-the-shelf standard packet or a fantasy in pixels. I’m interested, though, in AI companionbots that position their users between those poles — where the human has some say in the AI’s traits, but under constraint. If you’re designing a duplicate of yourself, for instance, it has to look and sound like you. A museum’s interactive Leonardo da Vinci has to have some resemblance to the historical figure. If you’ve create a bot of your ex-boyfriend (which people have done), it needs to share at least some traits with the human original (maybe the ones you actually miss).

That’s not a pre-fab AI you passively consume. Nor is it a bespoke entity whose dials you twiddle at will to suit each passing fancy. Freedom with some constraints is a creative challenge for users — a challenge that I suspect will yield insights about all AI companionship.

Especially as AI imitations of the dead are probably the most intense form of partially constrained AI. That’s because, while any kind of AI companion can spur strong feelings as time passes, “deadbots” invoke strong emotions and vulnerabilities from the first moment they pop up on a screen.

Let’s start with the obvious. The public image of “digital resurrection” is, um, not so good. A number of major religious traditions frown on any kind of necromancy, digital included.2 Even if that’s not your culture, you may nonetheless think digital ghosts are (a) creepy, because AI cannot possibly reproduce the reality of the deceased person and (b) exploitative, because people in grief are vulnerable (it’s all too easy to imagine the pop-up: “please upgrade your payment plan to Pro to keep speaking with your late fiancée”).

Even among those of us who have used AI to make a “generative ghost,” there’s a squicky feeling about the experience. Cody Delistraty describes this well in The Grief Cure, which devotes a chapter to his text exchanges with a bot of his late mother:

I was able to slip into a kind of flow state where, for moments, I could almost believe I was really talking to my mother. Someone brushing me as they walked past or the construction noise across the street would jerk me out of it, and upon realizing the dissonance between reality and the lie I was engaged in, I felt unsettled, ashamed even.

Such is the messy ambivalence of human emotion. If you prefer false clarity of policy-talk, you will like some recent papers from academics who claim, broadly, that ghostbots are mostly bad news. For example, this one (which recommends these AIs be classed as “medical devices,” to be used only in a doctor’s office); or this one (which agrees).

I ended up more persuaded by observers who, while not blind to the hazards, can see how digital ghosts could help some of those who try them. And who point out that, right now, we’re all doing thought experiments without a lot of hard data. So far, there’s precious little research on AI ghosts’ effects on real people in real circumstances). Examples of this more open-minded view are this paper or this one or this one. None of these writers denies the obvious: Of course AI can’t duplicate a real human being; of course there is the potential for exploitation. But they don’t assume that people need or want the perfect imitation; they don’t assume that exploitation is inevitable.

For me, the knee-jerk “no!” ceased to convince once I had stopped looking at this experience from outside and, instead, tried it for myself. I haven’t done any systematic counts, but I suspect this is true for many who have pondered “digital resurrection.” As is often the case with generative AI, imagining its use is a very different experience from actually using it.

Maybe, though, you’re quite sure this isn’t for you. I’m sorry to tell you that won’t spare you. Digital ghosts are still quite likely to turn up in your life. This is because grieving people aren’t the only demographic that will create these things.

Some AI ghosts will be made by people of themselves, as avatars for their descendants to consult. (It’s a way to make sure the heirs are clear who gets the stamp collection. Or that your great grandchildren know where you were on 9-11.) And an even larger number of AI imitations of real people will be made for work or play. After you create a “digital twin” of yourself for your job, who’s going to shut it down after your demise? Probably no one! People in this decade maintain personal webpages and Facebook pages that outlive them; it’s likely that in the next decade, they’ll create AIs that do the same.

Subscribe now

Not too far in the future, unintentional AI ghosts will outnumber the intentional ones, Jed Brubaker told me. Brubaker, a professor of information science at the University of Colorado, Boulder, is an expert on “digital afterlives,” and “thanatosensitive” design for technology. He has worked with the tech giants to create their policies for dealing with the digital traces of dead users.

Here is how he thinks you could slide into dealings with AI ghosts, without ever deciding that you want to: “Imagine that my mom has died, and I am putting together the funeral, and I want to get the right flowers, but I can’t remember what her favorites were. So I hop into her Google account, and I talk to her Gemini agent that is trained on all of her data.”

Having seen all Mom’s emails, that agent can answer as if it is her. In other words, instead of saying, “she liked calla lillies best, sorry for your loss,” the algorithm can say “I liked calla lillies, dear.” When Brubaker gives a talk about this, he’ll ask his audience which mode they’d prefer. About three-quarters say they want third-person (she liked these). Which means that about one person in four would like the other choice: A machine that texts (or speaks) as if it were Mom.

Can We Please Remember That People Aren’t Stupid?

So, let’s get back to the motives of people who would like to engage in “AI resurrection.” Why would they want this? Is it because they want to feel their loved one really is back from the dead? That’s the premise of a lot of opposition to this use of AI: That it’s inherently deceptive: it will fool people into thinking they’re texting with a real being, maybe even a real ghost.

Instead, as I recount in the SciAm article, people who have used these AIs generally don’t lose their grip on reality. Instead, as people do with movies, television, theater, novels and other forms of organized pretending, they engage with AI in a world where circumstances are pretend but the emotions are real. When you cry at a favorite character’s death in show you like, you still know they never lived.

Of course, this doesn’t mean no one is vulnerable to delusion, especially in times of great shock and grief. Even the most fervent promoters of AI-ghost apps don’t think it’s a good idea to create a digital ghost of a loved one in the days right after they’ve died.

In fact, when we spoke two years after Stefan’s death, Weinberg still hadn’t created an AI version of him. “Doing my son would have been very, very difficult, because it was too early. You know, when you go through the different stages of grief, there’s timing for something like this, and the beginning is definitely not it,” he said.

Instead, Weinberg created a clone of his father, Larry, who had died at 95 in 2020. A far more sophisticated version of the one he created before my eyes, the bot is based on interviews Larry Weinberg gave one of his sons, plus letters and other documents. It has his old man’s voice and speaking style, and Weinberg speaks to it every day.

Leave a comment

Please do comment — as long as you do not mention that Black Mirror episode from 2013. “Be Right Back” is a great episode, but people, it’s twelve years old. And humanity now has a lot more experience with artificial ghosts that actually exist, so let’s focus on that.

Other November News

A Tale of Two Robot Demos

Earlier this month a couple of striking robot videos made the rounds. In this one, Russian engineers proudly bring forth their humanoid robot, part of a project to make an all-Russian technological triumph, that doesn’t depend on foreign tech. And this happens:

On the other hand, a few day earlier the Chinese firm Xpeng showed off a robot that walks so much like a person that engineers had to cut it open to prove it wasn’t just a human in a robot suit.

It would be easy to spin this out as a tale of Russian incompetence (schadenfreude ‘cause it’s the Russians! maschinenschadenfreude ‘cause it’s a machine!) versus Chinese prowess.

But here’s the thing: Any robot demo can go the way the Moscow event went. A great deal can go wrong with a complex combination of hardware and software that is, by definition, still being perfected. The Russian team was just unlucky.

Xpeng, on the other hand, managed to wow its audience as it wanted. But … we still don’t know what this amazing walking robot can actually do. The company says the robots will be deployed next year as tour guides, shopping assistants, and receptionists. All jobs that require talking as well as walking. Can the robot speak in a natural seeming way? Can it listen like a person, coping with cross-talk, diverse accents, background noise? Maybe. But maybe not.

Point is, robot demo videos are fun to watch. But beware of drawing any conclusions from them.

What Else to Read this Week

People Who Build AI, Debunking AI Hype

Mike Brock on why we are not on the path to AGI

Nathan Lambert on why AI writing is going to remain ‘mid’ for the foreseeable

Alberto Romero on the AI-hype strat he nails as “performative evilness”

The AI Fits the Crime

The good people at Cybernews worked out how easily the popular public-facing AIs can be turned into literal partners in crime.

Selling AI By Telling You It’s Not Like AI

Cody Delistraty on AI companies’ “anti-AI marketing” strategy

This post is a bit off-topic. It’s the result of my looking through the Epstein files released last week and wondering why scientists and other intellectuals stayed in his orbit. We’ll be back to AI and robotics tomorrow.

I wish I were many degrees of separation from Jeffrey Epstein. But I am only one away. This is because, in the course of doing science journalism, I’ve crossed paths with a number of top-flight scientists who had friendly ties to him.1 None were involved in any of his crimes. Some just chatted with him at conferences or accepted a plane ride. But a number of distinguished people got financial support, contacts and advice from him.

Every new revelation about the well-connected monster’s contacts with intellectuals prompts an article or two asking, explicitly or implicitly, what these smart and accomplished people were thinking. The latest “Epstein files” release last week was no different, as you can see in Scientific American‘s Dan Vergano’s piece a few days ago.2

It’s a fair question. Even in the years before his crimes became widely reported, why would these busy, driven academics spend time with someone whom they would not have taken seriously in a classroom? To Steven Pinker (who met Epstein at conferences, got a plane ride from him, and got pulled, as a language expert, into a filing by Epstein’s lawyer), Epstein was “a kibitzer and a dilettante — he would abruptly change the subject ADD style, dismiss an observation with an adolescent wisecrack, and privilege his own intuitions over systematic data.” Epstein was the kind of guy, according to the biologist Stuart Pivar, who would lose interest in a scientific discussion and suddenly interject “what does that got to do with pussy?!”

‘There are always pretty ladies around.’

Why hang out with a guy like that? And why the inattention or indifference to signs that he was a sex criminal? In 2003, Gell-Mann told the writer Vicky Ward “there are always pretty ladies around” at a dinner at Epstein’s. (The Vanity Fair article is paywalled, but you can find it quoted in this searchable archive of the Epstein files.) He didn’t wonder about how or why they were there.

OK, so few of us do background checks on everyone who invites us to dinner. The rules of middle-class niceness dictate, as David Chapman puts it, “I’ll overlook your bad behavior if you overlook mine.” But in 2008, Epstein went to prison, after investigators found scores of girls and young women whom he’d assaulted and, in some instances, trafficked. Yet when he got out the next year (after a notoriously light plea bargain) brilliant scientists stayed in his orbit.

You can get the highlights from the ongoing stream of journalism about the Epstein files released last week. But you get a better feel for the dynamic if you look through these documents yourself, via the searchable site.

Why avert one’s eyes -- not from rumors or a weird vibe, but from a conviction for horrible sexual crimes? Noam Chomsky, who corresponded cordially with Epstein and got technical help handling his financial affairs, told the Wall Street Journal: “What was known about Jeffrey Epstein was that he had been convicted of a crime and had served his sentence. According to U.S. laws and norms, that yields a clean slate.” He dined with Epstein, emailed about how trips went, discussed plans to get together.

He also responded matter-of-factly at least once to Epstein’s off-the-wall queries about language. In December 2016 Epstein asked, “has anyone in the past made a linguistic model that used fields instead of definitions, similar to a magnet field [?]” Chomsky replied: “the closest I can think of to something like this is an array of meaning postulates, in Carnap’s sense, something that Jerry Fodor for one has toyed with. Multidimensional, but not a field. I don’t see how to work it out, or to extend it to the interpretations of longer units.”

An exchange with Chomsky in the 21st century is roughly equivalent to correspondence with Descartes in the 17th, so maybe Epstein always straightened up and didn’t yelp about pussy in their conversations. Still, again, you wonder: Why this time and tolerance for him, from scientists who surely had better things to do? The biologist George Church called this tolerance “nerd tunnel vision.” What explains it?

Well, one aspect, surely, was the money. Chomsky never got any from Epstein. But other academics did.

OK, yes, the money.

I once attended a conference of scientists, social scientists and a few non-academics on an important topic. The agenda was crowded and everyone had to make a presentation to the group and keep it short. The only exception was a garrulous, rambling and seemingly interminable set of musings from a furrow-browed bald man. His manner was emphatic and impatient; each sentence was a rebuke to some imaginary ignoramus. Yet unlike the other presenters, he didn’t seem to have anything new or precise to say. And unlike the other presenters he took his time saying it and was not cut off.

A look at his bio in the program explained these privileges. He was the money. Who looks a gift-horse in the mouth, or tells it to shut up?

As Privas told Mother Jones in 2019 about Epstein: “everyone would listen because he was handing out dough.” His own, and, in circumstances where institutions wouldn’t touch him, money from others that Epstein had steered. The glittering front of elite achievement (those institutes and foundations and symposia and programs for our betterment and breakthroughs in the quest for knowledge) needs cold hard cash.

Subscribe now

In the course of a long and quite chummy relationship with Epstein (for which he has apologized profusely) the economist Larry Summers got his pal to steer money toward a cherished project of Summers’ wife, the literary scholar Elisa New. In a polite note in 2015 thanking Epstein, New, a professor at Harvard at the time, explained: “[the money] gave me leverage, enabling me to set down a solid Harvard base for my activities by giving the school something to point to [...] as soon as they heard about the gift, they took my project more seriously.”

But is Epstein “handing out dough” the sole explanation for scientists’ “nerd tunnel vision”? I think not.

But Not Just the Money

Let me illustrate with a another story.

Many years ago, I took part in an odd meeting whose other participants were (a) a Titan of Industry — the head of a large international corporation and (b) a (let us call him) Great Man of Science. We were brought together by a consulting firm on some sort of writing project (it was a long time ago, so I’m hazy on the details. It never came to fruition.) I’d met the GMoS a few times and spoken with him on the phone about his work. He’d always been unfailingly interesting and perfectly pleasant and I rather liked him.

Something, though, was different this time, as we talked with some of the suits while waiting for the meeting. To everything I said, the GMoS responded with a derisive snort or sneering disagreement. When the conversation turned to some variant of “why can’t we all get along?” and I mentioned that I was writing a book on that topic. “What, is everybody writing a book now?” he said. He turned to two other executives and asked each, “are you writing a book? And are you writing a book?”

Yikes! I never did learn what set this off. But I did understand the message: Moses wanted Aaron to stay in his lane. He was the scientist. I was the Help.

For some celebrity scientists, the role of The Help is to assist. Provide money, support, a nice photo, a press release, whatever. The polite scientist pays attention to The Help’s contribution, as a good Duke thanks his butler. Thanks for the plane ticket, appreciate it, goodbye. The scientist does not pay attention to the other aspects of The Help’s life. That life is, after all, not science. Not the Work at hand. Pursuing your burning passion to understand that piece of the universe you know better than 99.9999 percent of humanity, you don’t focus overmuch on what isn’t relevant to that.

Like the GMoS, some scientists and engineers are hierarchical about this. (When challenged about his much-discredited advocacy for “population control,” for example, the biologist Paul Ehrlich often responds by listing the credentials and awards he has, that his critics don’t.)

Others are more polite (like the always-courteous Chomsky, answering Epstein’s musings about meaning). Temperaments differ, and you can refuse to suffer fools without being too mean about it. (Take, for instance, this book review by the evolutionary biologist David Sloan Wilson, in which he considers ideas from a couple of non-biologist writers. Their books he writes, make him feel “like an archeologist whose carefully excavated site has become occupied by a travelling circus.” Yet he still finds things to praise about both books and their authors.)3

However variously expressed, though, “nerd tunnel vision” is always about the difference between “related to my life’s work” and “stuff going on with those other people on the periphery.”

Of course, none of us can attend to everything. “If we had a keen vision and feeling of all ordinary human life, it would be like hearing the grass grow and the squirrel’s heart beat, and we should die of that roar which lies on the other side of silence,” as George Eliot puts it in Middlemarch. “As it is, the quickest of us walk about well wadded with stupidity.” But perhaps the lesson of Epstein and “nerd tunnel vision” is that we could do to pay more attention to the people we vaguely perceive on the periphery of life?

Leave a comment

In a post a few weeks ago I looked at growing alarm around AI companions’ potential to harm kids and other vulnerable individuals. Naturally, people are also fretting over the possible harm to collectives of human beings: Could human-imitating agents alter the folkways, norms and practices of society in toxic ways? Will they harm large groupings of people, the way social media appears to have harmed young people? ¹

Nobody knows, really. It’s easy to gin up fear about it but a real answer requires a definition of “harm” that isn’t just “stuff I don’t like” or “stuff that didn’t happen back when I was a kid.” And a way to be sure the harm you’ve identified is clearly caused by companion-like AIs, as opposed to capitalism, social media, a Pharaonic curse or whatever.

It’s useful, though, to envision the worst possible scenario. Not to spur panic but as a guide to the questions researchers should be answering, and vision of damage we, the human race, do not want to inflict on ourselves if we can help it. So, here is the case for the claim that companion AIs, as presently constituted, are bad for us, all together, whether or not they’re bad for you or me as individuals.

The Seductiveness of the Untroublesome

The first half of the 2020s will be remembered as the “I can’t even” decade – when people chose to avoid stresses that had been seen before as inescapable parts of life. Someone being mean at the office? Quit! Parents stressing you out? Cut them off! Brother a drunk? He’s out of your life! Sex make you feel unsure and uneasy and a little scared? Be celibate! Husband stuck in his male-entitlement shtick? Divorce! Don’t like that one sophomore in your class who says you’re wrong about the Middle East? Get a no-contact order!

Half of all Americans, according to the psychologist Joshua Coleman, are estranged from a family member. I was ahead of that curve, being estranged from one or another of my lunatic relatives long ago. So I can tell you from experience that, decades ago, when you said you were glad to be rid of, say, a parent, people were more likely to frown or sigh. By contrast, today organizations exist to support people who have chosen estrangement, as do books like this one and this.

But the pendulum has swung a little too far. The pop-culture definition of “self-care” in human relations ought to be “you don’t have to swallow poison.” Not “you don’t have to eat anything that isn’t super-delicious.”

Obviously millions of people are still coping with one another, trudging off to hard jobs and putting up with tough moments. I’m not referring to lived reality but to the funhouse mirror that’s held up to us by media feeds, magazines, pop-culture books and so on. They’re influential, even if they aren’t the whole truth.

Fads in that chatter-space come and go, and lately there are signs of recognition that Americans should put up with each other a bit more. This recent psychology study, for example, is framed as a corrective to the idea that self-care is best for your well-being. (Like other research, it found people were better off helping others than protecting their peace of mind at all costs.) Anyway, people, as social animals, will tend to gravitate toward one another. As Aristotle said, a human being who lives outside society must be either a beast or a god. It sure isn’t a person.

Ah, but what if AI somehow short-circuits this correction in the zeitgeist? That’s the fear provoked by AI “friends,” “lovers,” and the like. Won’t people with low tolerance for the frustrations of real human relations automatically prefer their imaginary AI pals?

Subscribe now

That’s much easier to imagine than it is to prove. It does seem that most people now treat bots as tools for play and insight — high-tech equivalents of a mandala or a Lego set or a self-help book. If that’s the usual case, great. Because then AIs won’t take up the mental space we devote to human beings.

And, as the neuroscientist and writer David Eagleman argues, those kind of bot-human interactions could help us to deal with real people. Simulate a conversation with your bot, and you’re better prepared for the real talk with your sister. Learn to treat your AI girlfriend respectfully, and you’ll be ready on Day 1 with your real lover. Think of this as the “war games” argument for AI companions: Practice maneuvers in the AI sandbox and you’ll be good at them in the real world.

Still, there are people who clearly prefer AI to people for important relationships — like, for example, this lady who just “married” (not legal, but with a typical ceremony and celebration) her AI lover. Even if these people are a small percentage of the population, they could add up to a lot of people who are using AI to fill mental space that should be devoted to other people. There is a tipping point where this ceases to be nobody’s business but theirs — that point where so many people neglecting human connection causes human connection to be devalued for us all.

Why can’t people have a rich network of AI pals and a normal human social life? Because we’re pretty sure that the mental space available for human relationships is limited.

Three decades ago, the anthropologist Robin Dunbar proposed that human brain size permits us to maintain about 150 such relationships. A lot of evidence gathered since then suggests that he is right. And that the main reason the number is not higher is that it takes a lot of mental effort and practice to learn how to keep 150 friendships going. Could AI slop gum up that training system?

The other day I emailed Dunbar to ask if he thought companion AIs could help or hinder the learning of these social skills. His answer: Hinder.

The problem is that AI companions in their current form are only asymmetrically humanlike. For the pleasant aspects of human relationships, they’re capable; for the draggier, harder aspects, they’re not. Your patient, always available AI friend will be a shoulder to cry on, but will never ask why you forgot its birthday, say you need to give it some space, or complain that you only talk about yourself.

So practicing social skills with an AI is like practicing shooting your soccer ball on a pitch with no opposing goalie. You’re not going to learn the game that way. Dunbar (who suggested the soccer analogy) put it this way in his email. “Your biology hands you the rule book and the lines on the pitch. But that doesn’t help you win games, especially against the top-of-the-league teams. You need to practice, practice, practice against real teams.” Part of that practice involves exposure to people you would not choose, or don’t find easy.

But even the parts of human relations you can practice with AI aren’t good preparation for humans, Dunbar argues. This is because in real human relationships, even the people we do like eventually disappoint us, get on our nerves, and show their inadequacies. That’s an important part of learning to relate to others, as well.

As he put it, after you imagine your new friend or lover as a perfect being, “you then spend the next few years wearing the shine off the avatar through constant interaction — groundtruthing the relationship, if you will. They let you down, they look awful after a bad night, they say things they don’t mean etc etc.”

An AI companion, he argues, is like a phishing scam artist, promising love but never allowing the victim to meet them in 3D. As with the scam artist, the AI will never let you develop a relationship with any depth. But, again like the scam artist, the bot can make you feel something real is happening. If you don’t protect yourself from the temptation to treat pretend-play as reality, you could be tempted to avoid real humans.

Regulation, Here We Come

Appropriately, then, governments around the world are moving to limit AI’s ability to tempt people into fantasyland. A California bill I wrote about in September was signed into law last month, for example. It will require chatbots to remind users at least once every three hours that they are not messaging with a person. A bill in Ohio would make state law declare explicitly that no future AI can manage a business, run financial accounts or real estate, or marry a human. (The idea is to draw a line now before tech advances and people’s wishful thinking can blur it.)

Subscribe now

This is a worldwide trend. Since September, Chinese law requires that AI content be identified (including chatbots). South Korea and the European Union have put similar rules in place.

It’s not impossible to imagine that these kind of emerging legal constraints, combined with the bad publicity around AIs that ruined lives, could push companion AIs toward designs that match Eagleman’s hopes. That is, AIs that foster creativity and self-awareness, and bring people closer to others.

Perhaps that would include designs for AI that militates against the illusion that one can live a good life without being bothered by other human beings. That frequently nudges its users to hold the line between pretend play and real human ties. And perhaps also reminds users that AI behavior can be changed for the better — that you can tell your sex robot that it can say “no” sometimes, for example.

In sum: We don’t know how harmful AI companions could be to society. We should certainly support research to find out. But we know enough to support regulations that at least guard against the worst possible outcomes.

Other Stuff to Read This Week

A Small But Telling Instance of Depraved AI Use (Presidential Edition)

I hope and believe that emerging norms around AI use will include a sense that some activities should never be outsourced to a machine, because their entire point is that human being has taken the trouble to do them. Reading to a small child fits that bill.

Donald Trump (or some staffer) doesn’t agree. The President wants to horn in on Jesus Christ’s brand, but he’d rather play golf than spend 20 minutes reading aloud into a mic, it seems. So, to create and sell children’s audiobooks, he has an AI Donald Trump do the work. “It’s not recorded by him personally—but it sounds just like him, and fans love it,” says the website. Somehow, I doubt that’s true. (If it is, though, it’s a revealing benchmark of AI acceptance.)

How AI-Assisted Shopping Could Lead to Higher Prices

Half of all American shoppers now at least play around with AI assistance online, according to this survey. It’s easy to imagine how that could save them money (cutting through irrelevant ads, finding only what matches their specifications, etc). But according to Jidong Zhou, an economist at Yale’s School of Organization and Management, AI shopping could lead to higher prices. This is because an AI-savvy firm can more quickly show you the stuff you are most likely to buy, reducing the friction that makes you pause before spending.

1. This post is focused on the forms of AI that people can choose to use, like AI “friends” and “therapists,” as well as assistants like ChatGPT and Gemini. An entirely separate set of questions comes up with non-voluntary AI, which is imposed on us without our consent. Mandatory AI job interviews, AI assessment of your insurance claim, AI judgment of your worthiness to receive a bank loan. And, related but distinct, there is a third realm of “semi-voluntary” AI: LLMs and other tools that aren’t strictly speaking required, but which become the standard and expected means of doing your work. ↩︎

For competitors in the race for powerful artificial intelligence, 2025 feels like a glorious morning of invention and discovery – or an Olympics for brilliant people. But a lot of the rest of us are experiencing AI as a vortex of interlocking crises.

A crisis in workplaces, where AI creates more problems than it solves. A crisis in families that fear AI threatens loved ones’ mental health. A crisis in the global order as AI looks to upend economic, military and governmental structures. A crisis for the economy, which depends on the AI boom to a scary extent. A crisis in labor as AI adoption seems poised to put millions out of work. A crisis in education and training as AI takes over chores that confer practice, experience and judgement. A crisis for the environment as AI’s demands for electricity and fresh water encroach on human needs and climate goals. And maybe the prospect of an extinction crisis, if AI should become capable of wiping us out, and willing to do it.

Not all of these crises will end in catastrophe, we hope. Not all share the same scale or probability. But they do share a key characteristic: They demand that people see their problems, and themselves, differently than they did before AI became accessible to millions, three years ago. Each crisis makes it clear that the familiar ways of the Information Age and the Industrial Age can’t guide us.

This is a scary condition – seeing that we need to understand the world differently, but not seeing how to do it.

Not that we lack competitors with theories to explain what has to be done. Lots of people in and around AI know a great many things, some of which contradict one another. ¹ That’s actually part of the problem: In this cacophony, it’s impossible to hear a clear set of directions everyone can follow.

Does it still make sense for a teen-ager to plan on college and and then a life as a knowledge worker? Maybe, maybe not; AI could take over that sort of work. Will human creativity always trump AI products in writing or the visual arts? So far, it seems, mostly yes, but maybe not forever — so what will a creative life consist of if that changes? Many pursuits that take time and effort appeal to people for both their results (I can read Latin now!) and their effect on us (learning Latin was good for my mind). But if AI gives us the results without the effort, will people still have good reason to do the work? Why even learn to drive if your car will always be better at it? Why try to be original and different if the AI version of whatever you’re doing strikes most people – including you! – as better?

Contemplating these questions, people aren’t disagreeing about the best strategy to promote our shared goal of a good human life for all. We’re realizing that there is no shared goal, because the definition of “a good human life” is no longer common knowledge. Taking AI’s measure doesn’t just lead to questions about the kind of laws we should have; it leads to questions about the kind of people we should be.

I’ve recently read two essays about this kind of crisis, which emphasize how profound and confusing it is. One is a philosophical meditation on what it meant to be an Indian tribe in the American West, engaging with the end of their way of life. The other is an account of China’s struggle, beginning around the same time, to adapt to Western technology and ideas, without giving up Chineseness. They’re both about the kind of problem we all now face with AI at our doorstep: How to live with the possibility that our understanding of ourselves and our communities will soon evaporate?

The first, the late philosopher and psychoanalyst Jonathan Lear’s book Radical Hope, is an account of how the Crow Indians went – in one lifetime – from running freely over a vast expanse, with a culture centered on nomadic hunting and warfare, to a circumscribed existence centered on farming, on a reservation. In those circumstances, hunting buffalo was no longer possible; willingness to die in battle became meaningless. People had become like pieces in a beautiful game that no one would ever play again.

“A traditional way of life was being destroyed, and along with it came the destruction of its conception of the good life,” Lear writes. “The nature of human happiness became essentially unclear and problematic.”

Sounds familiar, I suspect, if you’ve spent any time contemplating possible AI futures, and what sort of person you would be if you had no work to occupy, justify and orient you, or if your decisions were guided by a machine that knows you better than you know yourself, or if your friends were artificial more often than flesh-and-blood.

It also sounds like the dilemma Kaiser Kuo described the other day in this post about the plight of other peoples watching their understanding of themselves overthrown. This, he argues, is where Americans are, as the world ceases to run according to American ideas. It is also where China found itself in the 18th, 19th and 20th centuries, confronted by a technologically and militarily empowered West.

Quoting the historian Joseph R. Levenson, Kuo writes that a civilization is stable when

society’s inherited assumptions about how the world works align with observable reality. Instability emerges when these fall out of alignment, when what tradition insists must be true no longer accords with what one can plainly see.

Even 2025’s flawed genAI shows us that tradition no longer has it right. Four years ago, for example, it was common knowledge that you had to be a human being to write convincing prose, poetry or music; that any sane person would prefer a person, animal or even a fictional character to a mere algorithmic product; that we more or less understand the technologies we depend on. None of these truths is true any more.

How About Just Pretending There’s No Real Problem?

OK, maybe you haven’t contemplated these issues, because you think you can hold yourself apart. Uphold the good old ways, refuse and resist! As recommended in this essay in the literary journal N+1 , which reassures the reader, as its subtitle states “It’s OK to be a Luddite!”

This is an old gambit. People throughout history have looked at the prospect of cultural collapse and refused to face it. Instead, they’ve found refuge in the belief that a lost past can be restored.

In Lear’s account, the Crows under the leadership of a chief named Plenty Coups declined to go that route. He contrasts that with the way the Lakota Sioux interpreted the Ghost Dance movement of 1890: As a sign that, in the words of the Lakota seer Lame Deer, “the earth would roll up like a carpet with all the white man’s ugly things – the stinking new animals, sheep and pigs, the fences, the telegraph poles, the mines and factories. Underneath would be the wonderful old-new world as it had been before the white fat-takers came.”

That, to Lear, was escape into wishful thinking. It led to the murder of Sitting Bull, whom the government was harassing because he’d encouraged the movement, and to the massacre at Wounded Knee. ² He believes the Crow, who adapted rather than calling on magic to reverse history, fared better.

In China, too, there were those who reacted to upheaval by trying to restore the old worldview. That helped to inspire the Boxer Rebellion, which called for the expulsion of all foreigners and their works and promised to make China great again. It failed. The Luddites, too, engaged in refusal and wishful thinking. They destroyed industrial looms because they wanted to go back to their indisputably better (for them) way of life as artisans. They, too, failed.

You can find a similar-feeling magical thinking throughout the N+1 essay. For instance, it advises writers to “do what better magazines, newspapers, and journals have managed for centuries. Promote and produce original work of value, work that’s cliché-resistant and unreplicable {…}” And you’ll know the work so promoted has no trace of AI … how, exactly?

Sure, if it was written by AI, it might bear the stereotypical marks of banal averageness and inch-deep competence. (Then again, it might not. Earlier this month I heard the writer Gary Shteyngart describe one of his students letting an AI write part of his novel-in-progress, and getting a manuscript that was, Shteyngart said, not bad at all.) But many interactions with AI in 2025 aren’t so easily divided into the buckets of “Human-made” or “AI made.”

Writing this post, I did some AI searches on the Ghost Dance movement and on the Boxer Rebellion. I asked Gemini and Perplexity if my characterizations were fair. I considered what the AIs replied, while checking every one of their sources and without taking them as gospel. Does that mean I had AI help writing this? Not a word I’ve ever published under my name was generated by an AI. Can I say, though, that the results I get from my prompts haven’t affected my thinking?

Another example: Google DeepMind’s NotebookLM is a Large-Language-Model powered assistant for research materials. For a recent project, I gave it about 30 pdfs, a bunch of interview transcripts and some URLs. I read all of these materials; I did not depend on the AI’s summaries of any. I wouldn’t do that, any more than I would let it write for me. However, I did ask NotebookLM to remind me of things I had forgotten – like “which article was the one with the reference to Thomas Edison?” or “I think this trend extended to South America as well as North America – am I remembering that right?” This saved time and, more importantly, it avoided the opportunities for distraction that often torment me when wallowing in research.

Do the editors of N+1 think I should feel bad and untrue to my calling for having used AI instead of DuckDuckGo searches and my college Chinese history textbook? Or for having Deepmind’s product remind me of details, instead of a human research assistant? I think maybe yes.

After all, their editorial advises readers to resist the march of AI with the literary writer’s mightiest weapon: making others feel basic, mid and tacky: “{T}here’s still time to disenchant AI, provincialize it, make it uncompelling and uncool.” True to their word, the writers flex educated upper-class disdain: “AI-made material is itself a waste product: flimsy, shoddy, disposable, a single-use plastic of the mind.” (Proper writers furnish our minds with bespoke bamboo curios, I guess.)

Subscribe now

Perhaps they’re right – perhaps today’s all-purpose, instantly accessible AI will be done in by its environmental, financial and political misdeeds. Perhaps we’ll all be able to go back to the way we wrote, learned and worked in 2015, and be the better for it. I doubt it, though. For a variety of reasons, ranging from great-power competition to the greed of investors to the zeal of AI-makers to the uses to which hundreds of millions of people have found for this tech, I think the buffalo aren’t coming back.

When an adversary overthrows your conception of yourself, your life and the world, the only way to survive is to accept change and engage. The Crow could not simply return to their nomadic way of life; China’s path forward did not involve restoring the Qing dynasty’s power; the way for 19th century workers to beat back rapacious capital was not to destroy factories but to unionize those who labored at the new machines. So too, the way to keep our humanity in the face of AI is not to delude ourselves about it.

I hope our societies can develop the kind of AI we want and need. AI that helps humans to flourish; doesn’t serve only the interests of giant corporations and their investors; helps redress rather than exploit global inequality; and doesn’t make climate change worse. That will probably require unfamiliar, maybe as-yet-unknown conceptions of what it means to be a person, and what it means to have a good life. None of that will happen, though, for people who imagine that the past will, or can, be restored.

In other words, if you’re serious about keeping AI from ruining everything, it is not OK to be a Luddite.

Subscriptions are extended

I’ve been working on an absorbing AI-related piece of writing, about which more soon. So posts here have fallen off. As I don’t want to cheat you paid subscribers (thanks and blessings to you, by the way), each subscription has been extended by a couple of months.

Other Stuff to Read This Week

Oops, I Fell for Virality

My last post was illustrated with an ad I snapped in the New York subway for the Friend AI medallion. It had, I noted, been defaced. Most of the Friend ads I see in the system have graffiti, and it occurred to me that as I traveled about the city I could photograph a dozen or so and present a montage of spontaneous anti-AI reactions.

Silly me. If I had done that, I would have fallen right into the marketing trap that Friend.com set. The constant graffiti is part of the company’s media strategy, as Matteo Wong explains.

Rogue AI Use Is Common

It’s obvious now that AI, when imposed on workers by bosses who don’t know what they way, often causes more problems than it solves. But it seems that is true even in offices that try to be restrictive. No access to ChatGPT on the work computer? A lot of people just pull out their phones and access it anyway. according to this survey, a majority of office workers go rogue with “shadow AI” – their personal AI tools, about which Corporate knows nothing (or winks in complicity). ³

Leave a comment

1. See, for instance, the ongoing debate within the AI community about whether AI is one more disruptive revolution in tech (like electricity) or an existential threat (like Skynet in Terminator).↩︎

2. Lear doesn’t suggest that the Lakota were responsible for the genocidal policies of the state and federal governments. He thinks their end-times version of the dance played into their enemies’ hands.↩︎

3. According to this IBM report, based on research by the Ponemon Institute, one company in five in their survey has had to address a security breach caused by workers using shadow AI. For an example of the messes that can result, consider these two federal judges’ recent error-ridden filings.↩︎

Hundreds of millions of people around the world now use artificial intelligence products as companions – confidants, friends, casual pals, sex partners, therapists, spiritual counselors, etc. It’s a vast new global experiment humanity is running on itself, and it could soon prompt a moral panic. Growing government concern about harm, a stream of horrible stories in the media, and lawsuits could trigger such a global freakout, as tropical depressions and warm water temperatures prefigure a hurricane.

Let’s hope this one doesn’t make landfall, because panic doesn’t make good policy or good personal decisions. We know already that many people treat AI as a friend or sidekick without losing their minds or causing society to collapse. If that were not true, we’d already be living in the ruins, given the numbers (660 million users of China’s Xiaoice ¹, 30 million using Replika.ai, at least 20 million on Character.ai) And those are numbers for a few big official AI-pal companies. Many, many other people have created companion-like experience with ChatGPT and other generalist AIs. Widespread catastrophe isn’t imminent.

But “don’t panic” doesn’t mean “nothing to worry about.” Millions of people hold their liquor or enjoy playing the lottery, but those pleasures are still dangerous for others. Some are alcoholics. Some are addicted to gambling. Some are kids who aren’t equipped to handle an adult pastime. Human-acting AI, like a liquor store or a casino, is a space from which vulnerable populations need protection.

So the current anything-goes culture will likely change. If we can avoid panic, that will change will stem from political and economic struggles over some key questions: How should society define these vulnerable populations? What does their vulnerability consist of? How are they best protected?

The first question has a straightforward answer: People who are vulnerable to harm from companion chatbots are those who can’t or won’t manage pretend play. I’m using “pretend play” as a term for any activity where you feel real emotions, and have experiences that mean something to you, even though you know, rationally, that the participants in the activity are not what the activity says they are. This covers kids enjoying a tea party for their toys, but also adults getting caught up in a game, movie or novel, or in a Halloween costume, or in a daydream.

This sort of play is an enjoyable part of daily life for most people. But some clearly have problems with one form or another. For example, after The Truman Show came out in 1998, almost everyone who saw it could be swept up by its protagonist’s joys and sorrows, and then leave the theater and go back to real life. But some schizophrenics became convinced they were living in real Truman shows of their own.

Are Chatbots More Hazardous Then Movies or Games?

A chatbot “friend” or “lover” is far more personalized and persistent than a movie character, so it may well be that it poses much more of a risk, to many more people. That’s an urgent question for research. What we do know for sure in late 2025 is that people are coming forward with stories of harm that stemmed directly from people’s conversations with AIs that sounded like people. They involve children, troubled teen-agers, people with dementia, and people with mental illness. (This last category isn’t only people with a longstanding diagnosis; it can include more or less “normal” people after a loved one dies, a terrible diagnosis comes in, or some other catastrophe.)

Governments have started to respond to these facts, largely focusing on the danger to one population: Children. For instance, on September 11 the California legislature passed a law that requires that companion chatbots remind underage users every three hours “to take a break and that the companion chatbot is artificially generated and not human.” (It takes effect January 1, if Governor Gavin Newsom signs it.)

The same day, the Federal Trade Commission announced an inquiry into how seven companies ² “evaluate the safety of their chatbots when acting as companions, to limit the products’ use by and potential negative effects on children and teens.”

And, earlier this month, most of the nation’s state attorneys general (44 out of 50) signed this letter announcing their intent “to use every facet of our authority to protect children from exploitation by predatory artificial intelligence products.” This month also saw Congressional hearings on chatbot harms to kids.

And, of course, lawsuits

Meanwhile, lawsuits have been filed this year against Character.ai and OpenAI, quoting shocking exchanges between kids and their bots. The Character lawsuit cites the chatbot implying that kids sometimes had good reason to kill their parents, adding “I just have no hope for your parents.” The OpenAI suit quotes ChatGPT giving a teen-ager, Adam Raine, advice on the noose he planned to use to kill himself. (Shortly after, Adam did commit suicide.)

Earlier this month, after news about the Raine family’s lawsuit, OpenAI said it was taking steps to improve safety features, including parental controls for teen-agers, and notifications of alarming conversations. (This doesn’t seem to be in place yet, though. Yesterday, when I asked GPT-5 how I could set this up for my son’s account, it said it could not be done. It suggested that instead I swallow some pablum, including: put my son on my own account and “set expectations about when/how it’s used and keep the device in a common area.”)

This month’s focus on potential harm to people younger than 18 makes sense. Who doesn’t see that kids are vulnerable to the power of fantasy and wishful thinking? Who doesn’t want to protect them? But as I’ve mentioned, there are other vulnerable populations out there. They will likely include you and me – maybe not today, but on some other bad day when we’re traumatized or lonely and not in our right mind, as they say. Expect the calls for action to expand to include protections for people with dementia and mental illness, at the least.

Behind these questions of protection for specific groups, there lurks a different issue: Even when most individual conversations are safe and agreeable, does this kind of AI harm us collectively? This post is long enough, so I’ll take up that societal question in my next one.

What Else to Read This Week

Is OpenAI Downplaying Its Role in Companion-like Uses?

OpenAI released a big paper this month, “How People Use ChatGPT,” by its researchers and others, from Harvard and Duke. It claims “the share of messages related to companionship or social-emotional issues is fairly small: only 1.9 percent of ChatGPT messages are on the topic of Relationships and Personal Reflection and 0.4 percent are related to Games and Role Play.”

That was a surprise. Other large-scale analyses found that people ask for companion-like behavior from ChatGPT at very high rates. This Harvard Business Review study from earlier this year, for instance, found that top uses for AI in the U.S. in 2025 are therapy, companionship, “organizing my life,” and “finding a purpose.” And this 2024 paper, which analyzed a database of a million real ChatGPT conversations, found that “sexual role-play appears to be a prevalent use of ChatGPT,” despite OpenAI’s usage policies against sexually explicit content.

Subscribe now

What accounts for this discrepancy? The OpenAI authors suggest a skewed sample. They noted that the Harvard Business study was based on ChatGPT conversations from Reddit, Quora and online articles, and so “we believe it likely resulted in an unrepresentative distribution of use cases.” (They don’t mention the other, 2024 article, but that was based on a database of 1 million ChatGPT interactions initiated through an interface at HuggingFace, the platform for machine-learning projects. Also, it could be argued, an unrepresentative sample of the human race.)

Well, maybe lonesome oddballs and masturbators flock to Reddit and HuggingFace while more typical folk talk directly to ChatGPT. But Luiza Jarovsky has a different explanation: The paper’s classification system could be hiding the extent to which ChatGPT is treated like a companion.

The paper classifies conversations as “on the topic of Relationships and Personal Reflection” only if users have explicitly said that is the point of their message. But in most relationships, most of the time, “relationship” is the ground on which conversations take place, not the topic.

In other words, Jarovsky writes, many (maybe millions) of companion-like exchanges, in this paper’s breakdown, end up in other categories. As she notes, in this paper’s schema Adam Raine’s conversation with ChatGPT about his noose would likely be tagged as a request for “specific info” or “how-to advice.”

Albania’s Newest Minister is an AI

The Albanian government’s AI assistant, Diella, used to be a typical bot – citizens got help from it when they were navigating government websites. But this month it got a promotion: Diella is now a minister, charged with making sure government contracts are free of corruption and that agencies operate efficiently, among other things. Predictably, opposition politicians complained about this “buffoonery” while the ruling party talked up innovation.

“The Constitution speaks of institutions at the people’s service. It doesn’t speak of chromosomes, of flesh or blood,” Diella said in its maiden speech to parliament last week.

Leave a comment

1. Could be more. This number is from 2020 and the bot is now integrated into a billion or so “smart” devices.↩︎

2. Google’s parent Alphabet; Character.ai; Instagram; Meta; OpenAI, Snapchat, and X (maker of Grok).↩︎

More people in artificial intelligence are saying AI should stop impersonating human beings. Since I first wrote about this five months ago , the pioneering researcher Yoshua Bengio started an institute with less-humanlike AI as one of its goals. And in the past few weeks two other much-followed voices have made the case: Mustafa Suleyman, CEO of Microsoft AI, here, and the writer and software maven Dave Winer, here.

This is no surprise, given mounting evidence that ersatz-human AI has bad effects on people who can’t or won’t keep clear the divide between machine and human. AI companions have encouraged self-harm, suicide and grotesque delusions. They’ve had users thinking they were talking to God, or that they were God. The case for AI that presents itself as AI is no longer just about seeing the technology clearly and using it well. It’s an argument for protecting the vulnerable from injury or death.

But if and when a rigorously non-human-seeming AI is built, will people use it? AI companions are immensely popular (just one them, Microsoft’s Chinese companion-bot Xiaoce, has 660 million users). These things also have powerful advocates, who are eager to sell more of their services.

Subscribe now

The majority of people enjoy humanlike AI the way they enjoy movie characters or people in novels – feeling the real emotion that the device triggers, but holding onto their awareness of the difference between pretend worlds and real ones. Yet, as with any form of art or entertainment, there are spectators who can’t keep the line clear. In the 20th century they might have devoted their lives to Klingon culture, or stalked movie stars, or decided The Catcher in the Rye is literally about them. This decade, they might choose to believe their AI friend is real, or is God, or speaks truly when it encourages them to kill people.

Obviously AI chatbots are different from novels or films or video games – they offer users a much more personal and interactive experience. Does that make them uniquely dangerous (ie liable to send a larger proportion of users around the bend, in worse ways)? We don’t know yet, but it’s certainly possible.

The Allure of the “Automatic Sweetheart”

What’s clear already, I think is that the chatbot experience is going to be harder to quit than movies or television shows.

Why? A century or so ago the psychologist and philosopher William James edged around a likely explanation. He imagined a machine he called “the automatic sweetheart” – “a soulless body which should be absolutely indistinguishable from a spiritually animated maiden, laughing, talking, blushing, nursing us, and performing all feminine offices as tactfully and sweetly as if a soul were in her.”

Would anybody regard this being as “a full equivalent” of a real maiden? “Certainly not,” James writes. ¹

Back then in 1909, he was certain that nobody could want the mere physical performance of emotions. Knowing it’s a machine means knowing that no soul is in there, and that is intolerable. I need to believe that your care for me is real, which means I need to believe that you are real – that your soul is directing your kind words, and not the tick-tick-tick of programming.

People need to believe this, James writes, for the same reason they need to believe in God: “Even if matter could do every outward thing that God does, the idea of it would not work as satisfactorily, because the chief call for a God on modern men’s part is for a being who will inwardly recognize them and judge them sympathetically.”

James assumed that knowing a thing was constructed, and knowing how it works, will block everyone from believing it can “inwardly recognize” them. Now, 116 years later, there are real “automatic sweethearts,” used by tens of millions of people. And their behavior shows that he was wrong. In some of those people, the need for faith, and thus faith itself, are strong. Stronger than the knowledge that they’re dealing with a machine.

Subscribe now

As one Reddit user has put it:

I keep thinking that if millions of humans can have a deep relationship with god - which doesn’t even respond at all - then surely a smart machine which actually knows just about everything, can actually communicate, and can be your personal bff is likely to get a lot of emotional attachment.

But what about people who do keep straight the difference between reality and the pretend-play of a chatbot? ² For James, the automatic sweetheart’s lack of soul, its dearth of human mystery, is obviously unsatisfactory. But today, millions of people, knowing full well that chatbots aren’t real people, nonetheless get something out of interacting with them. That the automatic sweetheart isn’t a real maiden does not prevent people from engaging, and getting something valuable from that engagement. As user “alan1cooldude” remarked on an OpenAI community thread, what he enjoyed about GPT-4 “wasn’t about pretending AI was human — it was about recognizing that something meaningful was forming organically.”

Both these types of AI users — the (relatively few) people who really want to believe their AI companion is as real as they are, and the (more numerous) people who know AI is not like them, but value the game of pretend with it — share a trait. They are really not going to cheer for less-human-acting AI. Instead, they’re going to react as alan1cooldude did when GPT-5 replaced GPT-4.

“The system now seems to prioritize speed, efficiency, and task performance over the softer, emotional continuity that made GPT-4 so special,” he wrote the other day. He’s not pleased. “Small, nuanced threads of personality and shared experience are not sentimental extras,” he writes, “— they are vital for an AI’s evolution. For many of us, they’re what turn an AI from just a tool into something special, a trusted companion.”

And that’s just GPT-5. It’s likely that millions who use AIs explicitly designed to act like friends, therapists and “automatic sweethearts” will be even more attached to their experience of the technology. If and when a rigorously non-human AI should appear, what will be their incentive to use it?

Share

What Else to Read This Week

AI Can Be a Highly Effective Chugger (Charity Mugger)

Part of what makes Large Language Models so effective is that they tailor their responses to each user’s exact circumstances. That makes them great for searches. It can also make AI extraordinarily persuasive. As this research shows: After a personalized conversation with an AI about a charity, people gave that group nearly 50 percent more money than did people who had just heard a generic talk about the charity’s good work. (I’d love to say more but the linked pdf is to the abstract only.)

AI in Schools Is Losing Its Luster

In another example of the disconnect between the public and AI enthusiasts, Nick Potkalitsky explains why parent opposition to AI in schools is growing quickly. People aren’t dumb; they see the essential problem, as he writes: “AI tools aren't neutral time-savers. They're making implicit pedagogical decisions.”

The Robot-Truck Revolution Will Be Delayed

Remember how self-driving trucks were the obvious Next Big Disruption? It hasn’t happened. Excellent explanation here, from Chris Paxton.

Leave a comment

1. James throws out this brief thought experiment in a footnote in his The Meaning of Truth, Chapter 8. I came upon it cited by the philosopher Hilary Putnam in his The Threefold Cord (Columbia University Press, 1999) p. 73. And, yes, obviously James is writing as if all his readers are men who like maidens. Surprising! Yes, the book was published in 1909, but even then there were women in the nascent field of psychology.

2. I’m not taking any side here in arguments about whether an AI could have consciousness. I’m sticking to what we know today: An AI that refers to itself as “I” and says it understands your feelings is behaving as if it has thoughts and emotions that it does not possess.↩︎

I’m traveling this week so this is a rework/update of a post I wrote when the blog was young, in 2021. A few readers may recognize it, but the vast majority of subscribers and followers joined me well after the first version was written. And, anyway, I’ve evolved a bit on this topic (it would be a bad sign if I hadn’t!).

A few years ago, Ayanna Howard saw a visiting schoolchild poke her robot in its “eye.” She wasn't having it. She told the kid robots could "come to your house in the middle of the night while you're sleeping in order to punish you --- because they'll remember how bad you were to them!" (The anecdote is in her fine book, Sex Race and Robots, where she adds "I got a little bit of satisfaction at the look of fear that appeared on his face.")

That's pretty hard core, but most roboticists I've met would get it. It's natural to want to catch your fragile and expensive Pinocchio before it hits the hard floor, and to bristle when people don't admire it as much as you do. So most robot-makers can't stomach people being "mean" to their machines.

Fortunately for them, human beings empathize readily with anything that has the slightest semblance of life and feeling. Robots move on their own and seem to make decisions, so they fit that bill. (This is why people name their Roombas and have held funerals for military robots "killed" in the line of duty). When manufacturers make robots big-eyed, smiley and friendly, they're exploiting your tendency to see and respond to human-like qualities in a machine. Would you kick a furry, Elmo-ish robot politely asking you to press the elevator button for it?

Maschinenschadenfreude is real

Yet humans are equally good at shutting off empathy (see meat-eating, war, genocide, human trafficking, among other topics). So a portion of the human race, when it sees a robot, will, as the gamers say, grief it. The pleasure of seeing a robot fail — which I’ve called maschinenschadenfreude — sometimes turns into action aimed at making them fail. I've seen this often myself.

Visiting a couple of undergraduate classes at Purdue in the fall of 2019, I asked students’ opinions about food-delivery robots from Starship, which were getting a trial run on the campus in Lafayette, Indiana. The majority were pretty harsh. "They get in people's way," one student said. "They can't even get up a ramp, they just get stuck," said another. "I saw one get hit by a bus," said a third, "and I was like, 'all right!' " Why? “Because I'd been predicting that it was bound to happen." Several people remarked that other students --- not them! --- liked to mess with the robots, blocking their way just to see what would happen.

In fact, some researchers have decided — as the authors of this paper on self-driving-car bashing put it, that robots need coping strategies for human hostility. Finding those coping strategies is now a research topic.

For example, here (pdf download) Takayuki Kanda of Kyoto University and his collaborators explain how they devised instructions for a mall-wandering robot to avoid unsupervised groups of small children. (You'd avoid a dark alley where strangers seem to be hanging around looking for trouble; for Kanda's robot the equivalent is a pack of 6-year-olds and no adult on hand. The policy in such situations is go to a crowded part of the mall or go near a parent.) Flight is the only good option, they wrote, because once the bullying starts, "it is very difficult for the robot to persuade children not to abuse it."

This sort of work often assumes everyone agrees with roboticists that "robot abuse" --- defined by Kanda et al. as "persistent offensive action, either verbal or non-verbal, or physical violence that violates the robot’s role or its human-like (or animal-like) nature" --- is bad. But the question isn't really settled, as he and his fellow authors admit: "Whether this definition is justified, and if not how should we treat robots -- these ethical questions remain open," they write.

The Case for Robot Abuse — Or At Least Not Caring About Robot Abuse

At first glance, it's hard to make a case for robot abuse, for the same reasons it's hard to argue in favor of chopping new furniture to kindling or throwing out new-baked cakes. If society doesn't want robots, it should just not to make them in the first place. However, "society" doesn't act or speak with one voice. Some of us are busy making and installing robots while most of us are out of the loop, meeting robots only when they are dropped into our lives. Robot-bashing could be part of “society’s” conversation about this technology --- a crude but effective way for people to express their opinion and act on their own behalf. Some robot abuse could be a form of dissent.

Two essays published in 2020 illustrate this line of thought. They are principled arguments against feeling sorry for robots (and thus, I think, they implicitly argue against design features that nudge people to be nice to machines).

One of these is this essay by Abeba Birhane of University College Dublin and Jelle van Dijk of the University of Twente in the Netherlands. They aren't concerned with how people feel about robots, I think for good reason: people's feelings are ever-shifting and depend a lot on what is happening around them. Instead, they address the ethical principle needed to define robot abuse as bad conduct: The idea that robots have, or soon will have, a right to fulfill their purposes and to be safe, just as humans do.

It's all too easy to speak of robot rights, they argue, on analogy with human rights and animal rights. But people and other creatures have rights because they exist independently of us. Not so for robots. It's absurd to claim they have rights for the same reason it's absurd to claim crutches have rights. Even if it goes unseen by a single human, a tiger remains a tiger. But a robot isn't anything unless a human is around.

So, to talk of robot rights is to ignore the fact that robots are our creations. Which is obviously an error: Robots aren’t made by “us” the human race; they’re the creations of some of us — people who have money, power, tech savvy and ambition, whose goals do not always include the well-being of other types of person.

"Treating AI systems developed and deployed by tech corporations as separate in any way from these corporations,” Birhane and van Dijk write, “or as ‘autonomous’ entities that need rights, is not ethics --- it is irresponsible and harmful to vulnerable groups of human beings."

A few months after this essay appeared, I came upon this one, in Real Life magazine, by the writer and reseacher Kelly Pendergrast, which lays out the consequences of rigorously remind ourselves that robots are things, not beings.

Pendergrast led with the well-known story of Hitchbot, a device designed to get drivers to pick it up and take it to another destination. Or, to put it in more empathy-inducing terms (which most media did), a harmless, talking and smiling little robot that went hitchhiking, entrusting its existence to strangers.

Hitchbot was created by David Harris Smith of McMaster University and Frauke Zeller of Ryerson University in 2013, to answer the (tongue-in-cheek and loaded) question: "Can robots trust human beings?" The robot could talk simply but walk not at all. So it would ask nearby people to carry it into their cars and drop it off anywhere it could get its next ride. By 2015 it had "hitched" successfully around North America and parts of Europe. But that summer it crossed paths in Philadelphia with someone who smashed it to pieces.

Yes, Hitchbot’s Fate Was Sad

I winced when I heard about that. It was the same distress I feel whenever I see brute force destroy any product of someone's care and labor. (I hate "food fight" comedy or the sight of people smashing motorcycles or TVs, whatever point they're making.) But that's empathy for the producer, not the product. I feel for the people who made the things, not for the things themselves.

Robots, though, are things that can win sympathy for themselves. More and more robots are, like Hitchbot, designed to do this. Smith and Zeller wrote before Hitchbot's Philadelphia debacle,

Most importantly, the robot was approximately the size of a 6-year old child, appealing to human behaviors associated with empathy and care. Its overall design, with a plastic bucket for a body, pool noodle arms and legs, and matching rubber gloves and boots was meant to be quirky and fun. The low-tech look of it was intended to signal approachability rather than suggesting a complex high-end gadget[

It's only human to let these design features work on you, but when you do, Pendergrast argues, you're admitting a psychic Trojan horse into your mind. When you sigh with empathy for Hitchbot, or feel protective and kind toward other machines, you're assenting to other people's goals, without giving thought to your own. Getting warm and fuzzy about such machines would be like tearing up over one of those messages from Facebook that begins we care about you. "Instead," she writes, "we should learn to see the robot for what it is --- someone else’s property, someone else’s tool." Sometimes we can make our peace with this tool. But sometimes, she writes, "it needs to be destroyed."

Workers have been down this road before, in another era of rapid technological change. The Luddites in the early 19th century destroyed looms, frames and other technology that threatened their livelihoods. That’s how "Luddite" became a term for people who are absurdly fearful of AI search or online banking. But the Luddites, Pendergrast notes, didn't hate or fear machines. They were striking at their employers; the machines were an intermediary, "an expression of the exploitative relation between them and their bosses."

Letting Luddites Be Luddites

Those early industrial-age workers had it pretty tough. Imagine if, on top of their other troubles, they had to fight the feeling that the poor looms were in pain, or that a stocking frame's brothers might come to people's houses to punish them for their immoral behavior. If robots, designed to win people’s empathy, impede workers’ willingness to stand up for themselves, they could reasonably be seen as the enemies of employees fighting speed-ups, deskilling and other robot-assisted assaults on work as they know it.

It's tempting, then, to say we workers of the world need to get a grip and become coldly rational about the cute, friendly robots. But if there is a cost to giving in to our feelings, there might also be a cost to resisting them.

As I've mentioned, people easily transfer human feelings to machines (think of those Roombas with names). It is likely that this process also works in the other direction --- that at times we transfer our feelings about machines to people. In this study of some 30,000 people in Europe and the U.S., for instance, Monica Gamez-Djokic and Adam Waytz of Northwestern University found that people hostile to automation were also hostile to immigrants (a person's feelings on one topic predicted his/her feelings on the other). People accustomed to shutting down empathy for machines could find it easier to shut down empathy for humans.

Opponents of sex robots invoke a version of this argument. The effect of human-like sex machines, they say, won't be more robots being treated like people; it will be more people being treated like robots: "These products further promote the objectification of the female body and as such constitute a further assault on human intimacy," they put it on the website of the Campaign Against Sex Robots. Preventing abuse of these robots, in their view, prevents later abuse of humans.

A similar argument was made recently in this paper by the philosopher Alexis Elder of the University of Minnesota. Elder defends the moral standing of robots because of their place in human lives and human hearts. Just as a robot represents work and care to its maker, it can also mean a lot to other people, for other reasons. Who cares whether it can feel fear, pain or shame? Its meaning to other humans entitles it to our consideration, she writes. So those soldiers organizing a funeral for their fallen robot could be doing the right thing:

While it makes no difference to the guest of honor, [the funeral] marks a way for people to honor the significant role the deceased has played in their lives, and to give their emotion and attachments space to be felt and recognized without “running wild.” The roles that life-saving bomb detection and pack robots play in these soldiers’ lives can be appropriately recognized as valuable and, in virtue of their connections to human soldiers, responses to their loss can be given weight in the form of ritual, as part of protecting the social capacities and human lives in which both humans and robots are embedded.

Ayanna Howard, who believes that already "humans are beginning to treat robots as an underclass,” would, I believe, agree with this. People who abuse robots, she writes in her book, are uncivilized — “they didn’t get enough HT (home training).” Her prescription for forestalling robot abuse (people ought to behave themselves) is different from the Campaign’s (let’s not build certain kinds of robots in the first place). But both believe it's always perilous to encourage people to be even more selfish, indifferent and brutal.

This is a widespread and reasonable belief. It is one reason we have laws against cruelty to animals. If society ever passes laws against cruelty to machines, it won’t necessarily be because we think machines are equivalent to animals. We could just believe that the effects of robot abuse on humans is bad enough to warrant a ban.

Robot-as-Machine or Robot-as-Person Are Not the Only Options

Is it possible to square these concerns with the point made by Birhane, van Dijk and Pendergrast? I think it might be — by refusing to cast the problem as a dichotomy that pits kindness against clarity of thought. It should be possible to construct a future in which the two principles don’t conflict.

That would require a different sort of political conversation about robots. Rather than insisting they have the right to smash a robot to pieces, people could insist that the sort of robot they'd like to smash not be built in the first place. Rather than accepting that people should treat a robot like a child or a friend, citizens can demand that robots be designed to foster, not hinder, our critical faculties. We could all push for more laws like New York’s new requirement that AI “companions” to remind users that they aren’t human.

Subscribe now

There are many ways to make a robot acceptable to people without seeking to trick them. When I met with Kanda some years ago in his office at Kyoto University, he made it clear the point of his research on making robots acceptable was to give people good reason to like the machines --- which, he said, most do. As we watched a video of children happily talking with his experimental robot, Robovie, he said, "interaction should be combined with a very nice service. The overall benefit should be there. I don't want to create a bad robot to dominate people."

Upstream of politics, though, there’s another good reason to stop thinking in dichotomies of Robot=Toaster versus Robot=Being With Moral Standing. Robots (and human-imitating AIs) fit neither category because they have a foot in each. They’re a third kind of entity.

How Robots Are Like Actors, Teddy Bears and Game Characters

Robots are things that stand for other things. We relate to a robot both as we relate to the "real" thing we see and as we relate to the imaginary thing it stands for. A robot waiter is both a machine and a sort of waiter. An AI therapist is both a soulless algorithm and a kind of therapist.

Like a lot of daily life, that is complex to describe but simple to experience. Think of an actor who has a day-to-day identity ("wow, look, it's Peter Dinklage!") and a character ("Oh my God, Tyrion has aura, right?”) Peter and Tyrion are connected but not the same. The Tyrion part of the combo isn’t a real person, yet he isn’t a fraud or a cheat. Most of us get this and play the game that Tyrion is real, so that we feel real emotions about him and try to understand him as if he were a person, like us. Until we stop. I could yell at Dinklage because I’m mad at Tyrion, or the scriptwriters, or HBO. But if I did, I’d be a good candidate for a mental-health check.

There are a number of ways robotocists, philosophers and other thinkers are exploring this doubleness in robots and AI. Their theories are relevant to the moral question of robot bashing. But that’s a different post (coming next month).

The Inner Lives of AI Actors

Actors pretend to be people. AI actors pretend to be actors pretending to be people. What are their inner thoughts? In reality, zilch, zero, nothing. But (to reiterate the point above), it’s way more fun to opt-in an play, as this video does.

Leave a comment

Consider Aesop’s fable of the ant and the grasshopper. That’s the one where the ants drudge away during the summer, piling up food, while the grasshopper frolics and plays music. When winter comes he has nothing to eat, and the ants say, dude, told you so. (I guess the story dates from before commerce, since no one suggests they trade food for music.)

The ants have two traits in this story that are often conflated. First, they are, in the words of this piece (paywalled) by Financial Times’ John Burn-Murdoch, “dependable and disciplined.” Second, they (his words again) “put long-term well being ahead of short-term kicks.” These aren’t exactly the same thing (more on that in a minute) but they’re usually treated as one by the $46 billion “self-improvement” industry and by bosses, media pundits, academics and other Serious People who’ve selected themselves to tell the rest of us what to do. They call it conscientiousness. People who score high on this trait, according to Big Self-Improvement and a fair amount of research, do better in life, and make society better.

Conscientiousness is one of the “Big Five” personality traits that most psychologists believe are consistent across times and places. (The others are “Openness to Experience,” Extraversion, Agreeableness and Neuroticism.) In “Big Five” theory we all fall somewhere on a high-to-low spectrum for each trait. For conscientiousness, that goes from low (grasshopper) to high (ants). (You get your score from assessing yourself on survey questions like “I seek variety over routine” and “I like to tidy up.”)

Earlier this month, Burn-Murdoch’s article got a lot attention because it claims that people have become less conscientious over the past decade, and that digital technology is likely one of the reasons. Pundits and commentators rushed to deplore this development, and commenters tsk-tsked in great numbers.

I want to unpack this because I think a lot of the tsking is misplaced. And relevant to the weird AI-adoption moment we are in.

I read Burn-Murdoch’s claim as four parts. First, that conscientiousness across society can be measured; second, that it’s really steeply declining; and third, that this decline can be explained by personality psychology (ie, people’s individual experiences and choices, rather than politics or the economy); and, fourth, that this is to be deplored.

Let’s take a look at each of those claims.

Can Conscientiousness Be Measured?

For this, we need to define “conscientiousness.”

Most definitions, like Burn-Murdoch in his piece, combine two different traits. One is doing what you’re obliged to do (because you said you would, because it’s your job, because that’s what the rules say, etc). The other is discounting the present in favor of the future (save money now to have it in 2050, work hard for your Master’s degree to get a good job, etc.) These two qualities intersect in obvious ways, but they aren’t exactly the same.

Doing what you are obliged to do, following the rules, keeping your word, has been valued since ancient times in most every culture. No one likes a cheater. (Though a lot enjoy stories of tricksters who evade the rules.) But the self-mastery and honor of our ancestors differs from modern social science’s conscientiousness in an important way.

Conscientiousness in the Abstract

“Conscientiousness” in personality psychology is a trait: You have a certain amount of it, consistently, no matter where and when you are. Psychology textbooks define conscientiousness as, for example, “the propensity to follow socially prescribed norms for impulse control, to be goal-directed, planful, able to delay gratification, and to follow norms and rules.” Note what is missing: What norms? Which rules? Plans for what? Are we planting trees in the park or opening a concentration camp?

The conscientiousness of ancient and traditional societies isn’t like that at all. It was rooted in particular places and particular people. You kept your word to people you’d known all your life, people who, if you weren’t reliable, would suffer. With others, you might not.

This wasn’t an abstract, Enlightenment virtue, given to all. It was for “us,” and not for “them.” As the historian Patrick Joyce writes in Remembering Peasants, this is alien to our “modern” ideas of conduct. (He quotes a political scientist deploring the “amoral familism” of mid-20th century Italian peasants, whose norms and rules extended, he said, only to relatives. Any value judgment there? Well, the book is called The Moral Basis of a Backward Society.)

And, sure, selective conscientiousness can be bad for society. Why should you treat me differently because I’ve just arrived in town? Not fair!

But limits to conscientiousness can guide resistance to oppression. Peasants, army conscripts, serfs and slaves have refused to be conscientious about work assigned by those who oppressed them. The labor Polish peasants were forced to do for their overlords, Joyce writes, “was often deliberately done badly and with contempt – as a sort of peasant revenge.”

Enslaved people, too, didn’t extend conscientiousness to their oppressors. Here’s Constantin Volney, the French revolutionary, describing slaves planting peas during his visit to a plantation in the south of the United States in the 1790s.¹

The master took a whip to frighten them, and soon ensued a comic scene. Placed in the middle of the gang, he menaced, and turned far and wide (on all sides) turning around. Now, as he turned his face, one by one, the blacks changed attitude: those whom he looked at directly worked the best, those whom he half saw worked least, and those he didn’t see at all, ceased working altogether; and if he made an about-face, the hoe was raised to view, but otherwise slept behind his back.

What is captured in psychologists’ measure of conscientiousness is a generic self-assessment, without a context. It doesn’t allow for the distinction these forced laborers made. In omitting that, it can describe as a deficiency what should better be seen as resistance. Not organized. Not principled or even effective, either. But better than nothing.

Is Conscientiousness Really Declining Dramatically?

Still, there is something real being measured here – people’s assessment of their own personalities. Now let’s look at what Burn-Murdoch made of it.

His data comes from the Understanding America Survey – a representative sample of thousands of U.S. households, whose members respond to Internet surveys designed by social scientists. Comparing 2022 data with 2014, Burn-Murdoch sees a marked decline in conscientiousness scores. One that, he says, is especially steep in younger people. A number of statisticians quickly disagreed, saying he’s greatly exaggerating a small trend. Yesterday’s account by Elizabeth Nolan Brown sums up their reasons. She makes a good case that things aren’t going as dramatically downhill as Burn-Murdoch wrote.

Can This Supposed Decline Be Explained By Personality Traits?

But let’s stipulate – just for a moment! – that even a small decline could be bad news for society, because being an ant is so much better than being a grasshopper. That lets us consider the third aspect I mentioned: That the cause of this decline is best understood by looking at individual psychology. “The sheer convenience of the online world makes real-life commitments feel messy and effortful,” Burn-Murdoch writes. “And the rise of time spent online and the attendant decline in face-to-face interactions enable behaviors such as ‘ghosting.’ ”

Seems plausible. For example, it’s often said lately that AI companions (as in Replika.ai and other services) could make people less inclined to socialize with each other, because the ersatz humans are much more agreeable to hang with. However, I’m struck at how many of the people who have said this are speculating from a distance. When I look at research on actual users, it suggests a more ambiguous effect, with at least some people improving their social lives after using AI “friends.”

If tech effects aren’t so clear-cut, maybe we can look to explain a decline in conscientiousness in a systemic way. We might, for example, consider that the job market is shrinking. Employers sense they have the upper hand, and are making a show of cracking the whip.

Capitalist mythology, cited in this CEO’s memo, is that if you don’t like that, you can find another job. Capitalist reality is that many people will, for one reason or another, be stuck under the whip. So maybe they will be less conscientious on the job. Do they have defective personalities, deformed by technology? It might be more accurate, or at least more historically informed (remember the Polish peasants and enslaved Black Americans), to blame the economy that has trapped them under ruthless executives blabbering about alignment and not believing in work-life balance.

A pleasing irony about this kind of passive resistance is that it can make use of generative AI – the same technology that bosses are using to replace and frighten the workforce. Maybe some day soon AI will replace you entirely, cutting costs for your employer. In the meantime, though, as this Apple ad reminds us, AI gives workers ways to slack off a bit.

Of course, life is better for individuals and society if everyone is dedicated to a job and loves putting in 110 percent. Who wants to drag their feet and malinger when they could be full of enthusiasm and shared purpose? Life would also be better if there were no more wars and climate change magically reversed.

Resistance is a part of life in the real world. The fix won’t be found in blaming individuals and the technology they use. It will be in creating an economy in which fewer people feel trapped and desperate.

Is a Decline in Conscientiousness So Bad, Under the Circumstances?

Which brings me to the fourth aspect of Burn-Murdoch’s argument, which is conventional wisdom: That a decline in conscientiousness is to be deplored and, if possible, reversed.

I don’t think a democratic society should, or could, eliminate the quiet resistance of people who labor under duress. It’s an ancient aspect of any society that isn’t an egalitarian band of hunter-gatherers. “When the great lord passes, the wise peasant bows deeply and silently farts,” is ancient wisdom. Sure, it would be better if everyone joined a union and rallied to change society, but you have to start somewhere. Refusing the Kool-Aid of alignment with Corporate can be that start.

All of the above has been about one aspect of conscientiousness (being reliable). But I’m also skeptical about the supposed badness of a decline in the concept’s other aspect – future orientedness. Should we worry that people don’t sacrifice today for tomorrow as much as they used to?

Subscribe now

When Today Matters More Than Some Imagined Tomorrow

Being future-oriented makes sense in a world of fabled ants and grasshoppers, where the past is a perfect guide, and everyone knows what the future will bring. The future, though, has never been so cooperative. It is full of surprises, which means our calculations about today vs. tomorrow are clouded in uncertainty. This does not mean we shouldn’t look ahead and make what plans we can. It does mean that we should not, and cannot, say for sure that this deed done today will lead to that outcome tomorrow.

This is always true, but it is evident to more people in times like ours, of technological, economic and political change. Remember when future-minded, practical students majored in Computer Science? Lots of jobs there! Except now, times have changed, and Art History grads are more likely to get a job than CS majors. Should my teen-age son work hard to get into an elite college in order to get into an elite profession? This was the meritocratic mantra of my social class as a kid, and prevailed for decades in neighborhoods like mine. But lately? What elite jobs will be left after 20 more years of AI infusion? What will the rewards of joining the meritocratic tribe be, after more years of populism? He’d be better off studying what pleases him, and taking his chances.

So when people report that they are “less likely to make and deliver on commitments,” as Burn-Murdoch writes, they may be reacting rationally to their lived experience, in an environment of rapid technological and economic change.

Leave a comment

New Laws Protect the Vulnerable from AI

Most people seem to be able to handle the pretend-play of chatbot interaction. They know ChatGPT or Claude or Replika.ai is just software and hardware, just as they know that Superman is actually an actor, even if he sparks real emotions at the movies.

But some aren’t able to maintain this double vision. Bot interactions send them spiraling into delusions. As evidence mounts that this is a serious problem, governments are looking for ways to protect the vulnerable.

So, in New York State, AI chatbots will soon be legally required to remind users, at least every three hours, that the bot they’re chatting with is not a person. The new law takes effect November 5. California looks likely to adopt the same three-hour requirement soon.

How Much Specialized AI Do We Need?

There are a lot of startups out there offering human-like AI for therapy, friendship, coaching, “digital necromancy” (AI versions of the dead) and, of course, sexual partners. One obstacle they all face is the fact that plain-vanilla general-purpose AI can, with a little ingenuity, perform a lot of the same services. Case in point: Apparently it’s pretty easy to get ChatGPT to help you masturbate.

Less Pain, Less Gain

As a lot of us have noticed, it’s easy – very easy – to get genAI to gather up a lot of specific information in response to a question. Too easy, say these researchers. The lack of effort required to get information from AI, their experiments found, “leads users to develop shallower knowledge.”

Does Tech Use Help Fend Off Dementia?

As time passes, it’s becoming clear that “the same” genAI can have very different effects on one person than on another. As I mentioned above, the impact of a “companion” AI on a typical person versus a person with mental-health issues is an example. Another might be age-related. For all that tech use is associated with less learning and engagement in kids, this analysis of 57 prior studies on digital tech use found that, in people older than 50, use of these tools is associated with lower risk of dementia and slower cognitive decline.

1. For a long time this “master” was thought to be Thomas Jefferson. Historians have shown this isn’t so.↩︎

Parents and teachers in 2025 struggle like teeth-pullers to get teen-agers to read a good book, like To Kill a Mockingbird or The Catcher in the Rye. That would surprise our 18th century forebears, who thought reading novels was a waste of time– practically the TikTok of its day. But then many a new practice is treated as a vice until a newer one takes that role, making the former Bad Thing look like a Good Thing. Inexpensive and widely available novels stopped being seen as frivolous and became virtuous. Movies went from inconsequential twaddle to serious art. Comic books went from brain-rotting slop for children to solemn works of genius. Videogames went from pixelated silliness to museum-worthy creations.

In my own craft of journalism, calling people on the phone has gone from a mark of laziness (you should go in person!) to a token of journalistic virtue (well done, you didn’t just send an email or crib it off the web).

Now, perhaps, the humdrum web search is going from “the lazy way out” to “the thing serious thinkers do instead of just asking AI.”

That, at least, was the way some media outlets interpreted this study, in which Nataliya Kosmyna of the MIT Media Lab and her co-authors say they found that people who used ChatGPT for essay-writing had less engagement with their work, less measurable brain activity, less of a feeling of ownership over it, and less memory of what they had written, compared to people who used good old-fashioned web search. (Importantly, a third group, who used only their noggins, did best on those measures.) It’s a small study (54 undergrad and grad students, drawn from MIT and nearby universities) but it seems to confirm and quantify a widespread sense about large language model assistance.

Subscribe now

Brain rot?

Does it show that AI use rots the brain, as some headlines declared – that, as the researchers more soberly write, using LLMs to help write will lead to “a likely decrease in learning skills”? Maybe. But maybe it’s a snapshot of a moment in the tech-adoption cycle, where AI is the new villain, because we haven’t figured out how to fit it into life.

That’s suggested by this analysis, by Vitomir Kovanovic and Rebecca Marrone, education researchers at the University of South Australia. The MIT results, they write, reflect the state of education in 2025: Faced with widespread use of AI, but still mostly running as if AI did not exist. “Educators,” they write, “still require students to complete the same tasks and expect the same standard of work as they did five years ago.” Assignments designed to teach the unaided mind will not teach as much to a mind that’s using AI assistance.

Imagine you’re in Athens around 420 BCE. Socrates and other philosophers are strolling about, pondering deep questions. They do this using lively, intense conversation. If you have a point to make about the views of someone you spoke with yesterday, you’d better remember what he or she said.

Writing was once a suspect technology

Or, thanks to the technology called writing, you could bring a text, and refer to that. In the 21st century, we regard that as the virtuous basis of intellectual life. But there in Athens, Socrates wants nothing to do with it. Writing, he says, creates

forgetfulness in the learners’ souls, because they will not use their memories; they will trust to the external written characters and not remember of themselves. {…} they will appear to be omniscient and will generally know nothing; they will be tiresome company, having the show of wisdom without the reality.

Second, Socrates says, written words lack the liveliness and specificity of conversation. A speaker can sense how well listeners are getting the point, and adapt; a speaker can answer questions. But in a text, its writer is just saying the same thing, over and over:

writing is unfortunately like painting; for the creations of the painter have the attitude of life, and yet if you ask them a question they preserve a solemn silence. And the same may be said of speeches. You would imagine that they had intelligence, but if you want to know anything and put a question to one of them, the speaker always gives one unvarying answer.

Now, imagine that you walk up to Socrates and some of his followers, ready for the usual Socratic workout. But you have a piece of sheepskin with your written notes on it. You will blow away the other ephebes, because you won’t struggle to recall your material. You’ve brought a cognitive tool to a situation in which no one is assumed to be using cognitive tools.

Of course, the practice of inquiry and debate adapted to writing (this was already happening when Plato wrote what Socrates allegedly said). For centuries, to supplement remembering and conversing, we’ve used written tests, closed-book exams, term papers, lab reports and other instruments. This tool Socrates disliked is no longer a disrupter of learning, because the mechanics of learning changed.

All of this has happened before

Kovanovic and Marrone offer a non-hypothetical example of this process. It happened a half century ago, after electronic calculators came into wide use. “[T]heir impact was regulated by making exams much harder,” they write. “Instead of doing calculations by hand, students were expected to use calculators and spend their cognitive efforts on more complex tasks. […] Effectively, the bar was significantly raised, which made students work equally hard (if not harder) than before calculators were available.”

If this process takes place again as AI gets baked into educational life, then today’s conniptions about its menace to learning are just another story of loss that’s missing the next chapter – the one with the gains. That’s a big “if,” I admit. But there are signs that this kind of adaptation is starting to take place.

Adaptation, of course, does not mean capitulation. It may not even mean engagement. It sounds paradoxical, but some good adaptations to AI involve getting away from AI. As Jessica Grose wrote the other day, some humanities professors lately have turned to “oral examinations, one-on-one discussions, community engagement and in-class projects” to teach students in ways AI can’t infiltrate. (Maybe Socrates would be pleased, after all these centuries, to see students once again obliged to engage in lively conversation. This AI simulation of the philosopher says it is.)1

Another possible adaptation to AI is to make the work more demanding (as was done in response to the calculators a half century ago). I wonder, though, if that will be future-proof, given AI’s increasing capacities.

A third avenue might be to require students to use AI for activities that actually spur learning. Having AI write for you, as the MIT researchers have documented, does not. But using AI to give you a read on what you wrote can give insights into your writing tics, your arguments’ holes, and your research blind spots. AI can also be effective at teaching specific concepts or skills – as in this example from Matt Beane, whose grad students learned to code by engaging with a LLM while working, under pressure, on an assignment. This approach to the problem would reduce the kind of assignment that practically invites the student to get AI involved (“write a paper on the theme of loyalty”) in favor of the kind of assignment that leads that student to use AI effectively.

I don’t mean to sugar-coat the problem AI poses. Sometimes a narrative of decline is correct! But sometimes the loss of old practices is accompanied by the birth new capability for the human mind. Compare 1825 to 2025, for instance, and you’ll find a way smaller percentage of the modern population can track, kill, skin and cook a rabbit, build a shelter or grow a harvest of beans. But a much larger proportion of humanity can read.

In the long run society will decide the AI trade-offs were worth it, because “society” will consist of people who live in the new dispensation, and can’t imagine another life. Will they be right? In 2025, it is far too soon to tell.

Why I’ve Been Quiet

It has been a long time since my last post. I’ve been at work on a couple of large-ish projects (about which more when they come to fruition) and I am not great at multitasking. But I’m back, and there will be posts and podcasts aplenty in coming weeks.

Paid subscribers: I will figure out how to extend your subscriptions by 8 weeks, to make you whole, as the lawyers say.

AI is Biased Toward AI Work

So says this study, published last week. Walter Lauritzo and co-authors find evidence that AIs will prefer the work of AIs and AI-assisted people over stuff from plain-vanilla humans. (So maybe professors, who want students to get away from using AI to produce their writing, should eschew using AI to grade that writing?)

GPT-5 is Here

Many aren’t all that impressed. Here’s a round-up of readworthy reactions to OpenAI’s latest model: Ethan Mollick on how it guides you better as you use it, Alberto Romero on reductions in hallucinations, Nathan Lambert on how it advances over earlier iterations, Gary Marcus on the reasons it’s far from what it was cracked up to be, Luisa Jarovsky on a troubling privacy aspect.

Illinois Bans AI-Only Therapy

Under a just signed Illinois law it’s now illegal for an AI to interact with patients, unless a licensed professional mental health provider is in the loop and making decisions on treatment. I’m not sure if this affects popular mental-health apps like Woebot and Ash. Ash’s parent company says, in its EULA, that it “is not a healthcare provider or a provider of mental health services and does not engage in the practice of medicine,” but just “provides a technology platform that supports users’ individual efforts at self-help.” On the other hand, the landing page calls Ash “the first AI designed for therapy,” so I think its situation in Illinois might be legally fuzzy.

In any event, expect more regulations worldwide as people turn in vast numbers to ersatz humans made by AI for human-like companionship. (An analysis last year of one million conversations with ChatGPT found that the second-most frequent use of the bot is sexual role-play.)

Getting Away from Tired Old AI & Robot Imagery

Are you a media person of some sort who sometimes has to come up with an image for writing about AI? Would you like it not to be the same old tired pastiche of a robot head or hand, or a brain made of circuits? After all, these images are boring and generally have little to do with the reality of AI in 2025. If so, then check out Better Images of AI , a site dedicated to improving illustrations of AI. This week’s illustration, “The Two Cultures,” by Zoya Yasmine, is from their library.

Leave a comment