You can also hear AI Matt’s summary of the piece below.

Share

Scroll X (née Twitter) for long enough and you’ll see a familiar complaint: algorithms have ruined the feed. Posts no longer appear in a simple chronology. Negative or disturbing content seems to rise to the top with suspicious consistency. Whatever the exact wording, the frustration is recognizable.

The proposed fixes are just as familiar. Calls for the return of the “pure” chronology—even though most social media have something close to that1. Calls for the algorithm to somehow work differently—to show less of this and more of that (with “this” and “that” shifting from post to post).

What often gets missed in these complaints is a simpler point: algorithms aren’t trying to manipulate users so much as engage them. They do that by presenting content that the system predicts they’ll interact with. In that sense, social media feeds aren’t fundamentally different from other systems that adapt to human attention—news outlets emphasizing negative stories because they draw clicks2, or grocery stores optimizing which items catch the eye as you walk past the shelf.

The discomfort arises from a familiar mismatch: what we say we want from our information environment—calmer, more balanced, more informative—doesn’t reliably line up with how we actually behave when we’re immersed in it. That gap is often framed as hypocrisy, or worse, as evidence that users are being duped by opaque systems they can’t control.

But that framing misses something important. The mismatch doesn’t persist because people are dishonest about their preferences or because algorithms are uniquely manipulative. It exists because human attention itself is conflicted. In the moment, we’re drawn to cues that feel urgent, uncertain, or potentially important—even when those cues pull us away from what we’d endorse as our better long-term preferences.

To understand why algorithmic feeds keep drifting in directions people insist they don’t want, we need to look more closely at what actually happens in the moments when attention gets captured—and why systems built on engagement end up learning more from those moments than from our stated ideals.

Subscribe now

The Mismatch Isn’t Hypocrisy—It’s Conflict

When people complain about what shows up in their feeds, it’s tempting to frame the issue as hypocrisy: You say you don’t want this, but you keep clicking on it. There’s a grain of truth there, but it’s also a lazy explanation. What’s really going on is less about dishonesty and more about conflict.

We’re not especially good at introspecting about our own preferences—particularly when they pull in different directions. In the abstract, many people genuinely want calmer, more uplifting, or more informative content. Those are the preferences we endorse when we’re asked to reflect, explain ourselves, or justify what a platform should be showing us. But they aren’t the only ones in play.

Once we’re immersed in a fast-moving stream of information—a situation that exerts a strong pull on attention (Zlatkus, 2025)—other forces kick in: curiosity, vigilance, outrage, novelty, and the intuitive sense that this might matter.

Living Fossils

How To Understand Human Behavior (Part 3/4)

In Part I, I introduced a time-worn formula with a few tweaks—B ≈ f(S(P))—and argued that the first thing to understand about human behavior is the human. To a Homo sapiens, another of its kind is a big deal: a potential mate, ally, or rival whose idiosyncrasies demand close attention. But to a scientist who strips away that adaptive emphasis on individ…

Read more

9 months ago · 13 likes · 4 comments · Josh Zlatkus

Attention isn’t the same thing as endorsement. Interacting with a piece of content doesn’t necessarily mean we like it, agree with it, or want more of it in any broader sense. Often it just means the content was hard to ignore—especially when it taps into strong emotional or threat-related cues. So the mismatch between what we say we want and what we actually engage with reflects the fact that our preferences aren’t stable, singular, or always accessible to conscious reflection. We can sincerely want one thing while repeatedly behaving in ways that pull us toward another.

That tension is uncomfortable, but it’s also very human.

Share The Eel-Filled Hovercraft

Why Negative and Extreme Content Keeps Winning

To understand why this mismatch shows up so reliably, it helps to revisit a familiar feature of human decision making: our sensitivity to potential threats and losses. From an evolutionary perspective, missing a genuine threat is often costlier than overreacting to a false alarm (Haselton & Nettle, 2006).

As a result, our attention systems are biased toward cues that signal risk, conflict, or uncertainty (Baumeister et al., 2001)—exactly the kinds of posts people complain about seeing in their feeds. Even when the stakes are low, those cues tug at the same underlying mechanisms.

When attention is biased this way, content containing cues of threat, loss, or conflict gains a built-in advantage. Negative or extreme events aren’t just noticed more quickly; they tend to feel more salient and more impactful once they are noticed. That asymmetry means such content is more likely to interrupt ongoing activity, hold attention longer, and prompt some form of response—whether that response is clicking, commenting, or sharing. Content doesn’t need to align with long-term preferences to produce engagement; it only needs to activate affective responses that make it feel immediately important (Slovic, 1995).

None of this requires deliberate evaluation or conscious choice on our part. The same mechanisms that make potential threats harder to ignore also make threat-related information more behaviorally potent. The result is a systematic skew: content that activates threat sensitivity is more likely to register, more likely to be interacted with, and more likely to leave a lasting impression, regardless of whether it aligns with what people say they want from their feeds.

And the algorithm notices that.

When There Is No “Should,” Systems Optimize for Engagement

This is where algorithms enter the picture—not as puppet masters, but as systems that respond to behavior. Once attention is pulled in predictable ways and preferences come into conflict, platforms are left with a basic problem: deciding what to show next requires a standard for what should be shown.

Social media platforms don’t operate with a shared, widely accepted standard for which content should be shown to any given user. Outside of a strictly chronological feed, deciding what should appear would require values-based curation3—agreement about what counts as appropriate, informative, or worthwhile. That kind of consensus simply doesn’t exist at scale.

In the absence of clear normative guidance, platforms default to what they can measure. Engagement becomes the proxy for preference because it’s observable—even if it isn’t ideal. Algorithms infer what matters to users by watching what captures their attention and adjusting accordingly.

As a result, a social media feed becomes a form of personalized choice architecture. Any system that structures which options are visible, salient, or easy to access influences decisions. Traditional choice architects—policymakers, designers, retailers—tend to rely on generalized assumptions about what people want or need. Algorithmic feeds flip that logic. Instead of nudging users based on abstract models of behavior, they continuously tailor the environment based on each individual’s past interactions.

The result is a personalized nudge—toward what users reliably attend to in practice rather than toward what they say they want in principle. Over time, the feed becomes a dynamic mirror of engagement patterns. It doesn’t ask which content would be best for you. It asks which content is most likely to keep you looking.

That’s a design reality rather than a moral judgment. And as a result, the persistent frustration with algorithmic feeds looks less like a mystery and more like an unresolved tension between human attention and human aspiration.

Share

Why No Algorithm Can Resolve the Conflict

The frustration people feel with algorithmic feeds is understandable. We want systems that reflect our values, not just our impulses. But when those values conflict with how our attention actually works, no amount of tweaking the feed can fully resolve the tension.

Any system that privileges reflective preferences will feel restrictive, sanitized, or out of touch in the moment. Any system that responds to moment-to-moment engagement will feel manipulative or overwhelming in hindsight. Which outcome feels like a failure depends on which values happen to be salient at the time—and there’s no neutral setting that satisfies both at once.

Algorithms don’t create that conflict; they expose it. They respond to what we do, not what we wish we did. And until we reckon with the gap between our aspirations and our behavior, the feeds we complain about will continue to feel both unsatisfying and uncomfortably familiar.

Leave a comment

Share

You can also hear AI Matt’s summary of the piece below.

Given all the brouhaha over Substack's decision to bring Pangram into the process of consuming content, I thought this might be a good time to expand a bit on my “How I make this” statement. If you haven't seen it, here's what I currently tell readers:

These posts are the product of human–AI collaboration. I research the topics, develop the arguments, and make the editorial decisions. ChatGPT serves as a collaborator throughout the process, helping brainstorm ideas, draft and revise sections, challenge assumptions, and improve clarity. The final content reflects my judgment, not the model's.

That’s accurate, of course, but it also compresses quite a bit into a few sentences. So I thought I’d walk through how one of these posts actually gets made — from where the ideas come from, through the conversations I have with ChatGPT, to whatever eventually ends up in your inbox.

Share

Where Ideas Come From

The ideas for the posts I write tend to come from a relatively small number of places. Probably the most common is something I come across while scrolling through X or reading other people’s Substacks. Sometimes it’s an idea that I think needs a little pushback1. Sometimes it’s something that gives me a jumping-off point for an idea I want to explore further2. And sometimes a post or even a single idea sends me down an entirely different rabbit hole that ends up having relatively little to do with where I started3.

Teaching is another major source of ideas, and it’s really where all of this started. I teach working professionals, and I learned pretty quickly that assigning a 30-page academic article isn’t always the best way to introduce one concept among the many we’re covering in a given week. So I started writing pieces that translated some of those academic ideas into something more accessible and digestible, without (hopefully) stripping away everything that made the ideas interesting in the first place4.

But there was another problem as well. For some of the topics I teach, I couldn’t find much that made the particular point I wanted students to understand. Biases and heuristics are a good example. There’s certainly no shortage of material explaining what they are, but much of it presents them as cognitive mistakes we should learn to avoid. I kept finding myself making the same more nuanced point in discussion posts, so eventually it made more sense to write something that dealt with that issue directly5.

Research is another fairly regular source of ideas. Sometimes, yes, it’s my own stuff. I’ll publish a study or be working on a project and decide there’s something in it worth translating for a broader audience or exploring in a different way6.

But just as often, it’s somebody else’s research. I’ll come across a new paper or preprint that catches my attention for one reason or another. Sometimes it’s the findings themselves. Sometimes it’s the way those findings are being interpreted. And sometimes a concept or argument in the paper connects with something else I’ve been thinking about and gives me a new direction to explore7.

And then there's everything else. Every once in a while, an idea comes from something that doesn't fit neatly into any of those categories. Something that happens to me. Something I see on TV or in a movie. A conversation. A random thought in the shower. Sometimes there's already an idea bouncing around inside my head and something brings it to the surface; other times the experience itself gets me thinking. This is probably the least common source of ideas, but every now and then something sticks and becomes a post8.

Those categories aren’t mutually exclusive — there’s obviously some overlap among them. A paper I come across might connect with something I’m teaching. Something I read on Substack might remind me of research I’ve been thinking about. A random observation might send me looking for research to figure out whether there’s actually anything there. Ideas don’t really all sit nicely inside a single category.

But regardless of where an idea comes from, at some point I have to decide that it’s worth doing something with it. And that’s where the actual process of creating one of these posts begins.

Subscribe now

Choosing What to Write

Once I decide I’m going to work on a post, what happens next depends quite a bit on how I got there. Sometimes I already know exactly what I want to write about. This post is a good example. The whole Pangram brouhaha got me thinking about how I actually use AI to write these pieces, and there wasn’t much question about where I wanted to go with it.

Other times, it's much more deliberate. I may decide I want to work on something and start going through the things I've bookmarked, especially on X, to see what catches my attention. I bookmark far more things than I could ever possibly write about, and plenty of them will probably sit there forever. But sometimes I'll come back across something I saved weeks or even months earlier and decide there's finally a post in it.

Occasionally, I’ll even know I plan to write a post about some topic soon, and so I’ll create a placeholder post with any relevant information or links already pasted in it. Then it just sits there until I’m ready to actually work on it. The Ministry of Good Behavior was one of those. The eventual post grew out of a comment Ash Stuart left on an earlier piece I’d written. I knew there was something there, but that didn’t mean I was ready to write the post right then.

Once I’ve settled on something I want to write about, that’s usually when ChatGPT enters the picture. And what happens from there varies quite a bit depending on how well formed the idea already is.

Sometimes I come into the conversation with a pretty clear sense of the argument I want to make. I’ll give ChatGPT whatever prompted the idea — a post, paper, comment, or just an explanation of what I’ve been thinking about — and tell it where I want to go. In those cases, the conversation is fairly directive on my part. I’m not asking ChatGPT what I should think about the topic. I already have a pretty good idea. I’m using it to help me think through how the argument might develop, what I might be missing, and whether there are problems with the direction I’m taking.

Other times, the idea is much more amorphous. I know there’s something that interests me, but I haven’t figured out exactly what that something is yet. Those conversations tend to involve a lot more back and forth. We might kick around several possible directions, discard most of them, circle back to something from earlier in the conversation, or discover that what initially caught my attention isn’t actually the most interesting part. Sometimes I end up pretty far from where I thought I was going when the conversation started.

So I don’t really have a formal process for choosing what to write next. Sometimes an idea jumps the line because I know immediately that I want to do something with it (like this post). Other times I go looking through things I’ve saved until something catches my attention. And some ideas sit around for weeks or months waiting their turn.

By the time I’m ready to start writing, then, I usually have an idea and at least a rough outline of where I think the argument is going. I say think because that can — and often does — change once I start working through the actual post. But at that point, I have enough to move on to the next step: turning the idea into an actual post.

Share The Eel-Filled Hovercraft

Putting the Post Together

Once I’m ready to start turning the idea into a post, I almost always start with the introduction. And that means starting with a hook. ChatGPT and I don’t always see eye to eye on what constitutes a good one. It has a tendency to push back on some of my more boring academic openings. Sometimes I listen. Sometimes I decide they are what they are and move on.

But the introduction matters for more than getting someone to keep reading. For me, it’s usually where the post really starts to take shape. By the time I finish working through those opening paragraphs, I want to have a reasonably clear sense of the argument I’m making and where I’m going with it. In that sense, the introduction becomes something of a roadmap for the rest of the post.

That roadmap isn't set in stone. Sections get added, dropped, moved around, or substantially changed once I start writing. Sometimes the argument itself shifts because I run into something I hadn't considered or realize that part of what I originally planned doesn't quite work. But getting the introduction into reasonable shape gives me something to start building from.

At that point, I generally start working through the post a section at a time. I don't ask ChatGPT to take the idea and spit out a complete article. In fact, I rarely know exactly what the finished post is going to look like at that point. I know the general argument and usually have some sense of the pieces I need to get there, but much of the actual structure emerges as we work through it.

That process involves a lot of back and forth. Sometimes I’ll draft something and ChatGPT will suggest a bunch of edits — some of them good, some of them less so. Other times, I’ll throw out a poorly worded but reasonably well-thought-out version of what I’m trying to say and work with ChatGPT to translate that into something a reader might actually understand. And still other times, I’ll ask ChatGPT to suggest what we might say next, and it will usually suggest a paragraph or two as a possibility. Again, sometimes it’s really good, and sometimes not so much. Regardless, there’s likely to be some back and forth as I attempt to take Matt-speak and use AI to help translate it into something a little more universally digestible9.

The back and forth isn’t limited to the writing itself. As we work through a section, ChatGPT will sometimes point out a hole in my argument, raise a counterargument I haven’t considered, or tell me that I’m making a claim that goes further than the evidence probably allows. I actually want it to do that. Sometimes I agree and change what I’m saying. Sometimes I push back because I think its objection misses the point. And sometimes the ensuing argument — yes, I argue with ChatGPT — helps me figure out that neither of us had quite gotten it right the first time.

The structure can change along the way too. Something I thought would require an entire section may turn out not to have much there. A point I expected to gloss over in a paragraph may turn out to be more important than I initially realized. Or something that comes up while working through one section may force me to go back and rethink something I wrote two sections earlier. So while I generally start with a roadmap, I’m not particularly wedded to it.

And somewhere in all of this, research tends to enter — or reenter — the process. Even when a post starts with a paper, I’m rarely relying only on that paper by the time I’m done. Questions come up. Claims need support. Something I think I remember from the literature needs to be checked. Sometimes I know exactly what source I need; other times, I have to go looking for it10. And sometimes what I find — or don’t find — causes me to rethink part of the argument. That’s another reason the roadmap I started with is never really set in stone.

From there, I just work through the post section by section, following the same general process. Eventually, I end up with something that serves as a nearly complete draft.

But there are still a few things left to put together. I’ll usually ask ChatGPT to suggest some possible titles and subtitles. It’s actually pretty good at this, although I usually end up tweaking whatever it gives me rather than using a suggestion exactly as written.

The same goes for the visuals. I use ChatGPT to help me think through what might work for a lead image and any other images or figures I want to include in the post. Sometimes I already have a pretty clear idea of what I want and use AI to help create it. Other times, we’ll kick around possibilities until something seems to fit11. And occasionally the better option is simply finding an existing image that does what I need.

And at that point, the draft is ready to go. But I’m not quite done. I’ll come back and review the whole thing again, usually not in the same session in which I finished writing it12. Once I’m satisfied with that final pass, I go ahead and schedule the post.

Buy Matt a Coffee

So, Who Wrote It?

So, that’s my process — at least as far as I can specify it, ignoring all the tacit stuff that occurs that I can’t easily put into words.

And if that sounds a bit messy, well, it is. There isn’t a clean point where my contribution stops and ChatGPT’s begins. Sometimes the words on the page started with me. Sometimes they started with ChatGPT. More often, they emerged through enough back and forth that trying to figure out who “wrote” a particular sentence probably isn’t all that useful.

That’s why I describe these posts as the product of human–AI collaboration. ChatGPT is clearly contributing to the finished product. Pretending otherwise would be silly. But the collaboration also isn’t particularly well captured by asking how many of the words were originally generated by AI. The ideas, arguments, sources, direction, and editorial decisions develop through the process I’ve just described. And ultimately, I decide what stays and what goes.

And I’m glad this kind of collaboration is possible. It has made me much more efficient at turning ideas into actual posts. Ideas that might once have taken considerably longer to work through — or might have remained half-formed and never gone anywhere — are easier to develop when I have something to bounce them off of. It doesn’t eliminate the work involved, but it does reduce a lot of the futzing — and wasted time — involved in getting from an idea in my head to something I’m willing to put in front of other people.

Which brings me back to the last line of my “How I make this” statement: The final content reflects my judgment, not the model’s. That’s probably the most important part of the whole thing.

Leave a comment

Share

Subscribe now

You can hear AI Matt’s summary of the piece below.

Share

We’re still trying to figure out what it means to integrate AI into the writing process. Not just whether it helps or hurts, but what kind of effects it actually has — and on what part of the process. In a recent series, I approached that question by distinguishing between offloading substance and offloading translation. While the distinction sounds straightforward, in practice, it really isn’t.

A recent paper from Abdulhai et al. (2026) takes a run at that distinction in a more empirical way, examining what happens when people write essays with and without access to an LLM and then comparing outcomes based on how much they relied on it. The headline findings are pretty straightforward: heavy use is associated with less creativity, less sense of voice, and writing that collapses into a much narrower semantic space. But, at the same time, people report being just as satisfied with what they produced.

The story lines up nicely with the substance/translation distinction. Use AI in a limited way and you preserve the underlying thinking. Lean on it more heavily and you start to offload something more substantive, even if it doesn’t feel that way while you’re doing it.

That’s a useful starting point. But it also turns out to be a bit tidier than the behavior that underlies it.

Subscribe now

What the Study Actually Shows

The basic setup of the study is straightforward. Participants were asked to write an essay under one of two conditions: with or without access to an LLM. That was it, and that meant those in the LLM condition had control over how much they relied on it. Researchers then categorized those participants into three groups based on how they actually used the tool: no use, limited or consultative use, and heavy use.

The resulting distribution in the LLM condition matters more than it might seem at first glance. Of the 55 participants in that condition, about 22% didn’t use the tool at all; roughly half used it in a more limited, “AI-influenced” way; and about 27% were classified as heavy users. The paper’s main comparisons lean heavily on that last group (which was 15% of the total sample).

Across the groups, the reported patterns are consistent (see the figure below). Participants who relied more heavily on the LLM rated their essays as less creative and less reflective of their own voice, even though they reported similar levels of effort and satisfaction. At the same time, the semantic analysis shows a clear shift: essays produced with heavier LLM use cluster much more tightly together, occupying a narrower region of the overall space of responses.

The contrast in the figure is hard to miss. Essays written without AI spread out in multiple directions. As LLM use increases, that spread contracts. By the time you get to the heavy-use group, the outputs are no longer just similar — they’re pulled into a common center.

That pattern seems to point toward a simple interpretation: the more the system is involved, the more it standardizes what gets produced. And at a surface level, that’s exactly what the results show.

Buy Matt a Coffee

What “Heavy Use” Actually Looks Like

The interpretation of the results hinges on what counts as “heavy use.” The term makes it sound as if those participants simply relied on the system more extensively — just further along the same continuum of use. But when you look at the examples, it becomes clear that this wasn’t just a matter of degree — it was a different kind of use altogether.

In the sample transcripts provided in Appendix H, there was very little back-and-forth interaction. Instead, heavy users offloaded most of the task to the LLM. The interaction often followed a pattern: the user provided a prompt about the topic or direction, the system generated a paragraph or section, and the user then asked for the next piece — an introduction, a body paragraph, a conclusion — with relatively little intervention in between (see the snippet below as an example).

In those cases, participants weren’t merely offloading translation. They were offloading pretty much the whole kit and caboodle — most of the substance and all the translation. The user may have guided the general direction of the argument, but the system was largely determining how that argument unfolded. The user’s role shifted from constructing the argument to selecting or approving what the system produced. At that point, the label “heavy use” really starts to mean the system was taking over most of the work that would ordinarily define authorship.

That shift makes the main findings of the paper much easier to interpret. If a subset of participants was essentially delegating most of the substance and all the translation to the LLM, it’s not especially surprising that the resulting texts converged, lost their distinct voice, and reflected a narrower range of ideas.

What the Categories Capture and What They Don’t

The issue here isn’t what “heavy use” meant. After all, there are plenty of people out there who are willing to offload almost the entire writing process to an LLM, and so having a category that represents this extreme amount of offloading makes a lot of sense1.

The problem is what that category is being compared to. As defined in the study, the comparison is against those who either did not use the LLM at all or used it in a very limited way (see the figure below for an example of the latter). In cases like this, the system was providing structure, considerations, or possible directions, but it wasn’t doing much of the writing itself. The user was still responsible for translating those ideas into an essay, which is a very different kind of interaction from what we saw in the heavy-use group.

What’s also noticeable is how non-conversational both interaction patterns were. In the “AI-influenced” group, the system was used more like a passive Q&A tool — users asked for structure, considerations, or possible directions, and then stepped away to do the writing themselves. In the heavy-use group, the interaction flipped but remained just as one-sided: users issued brief directions, and the system generated the text. In both cases, the interaction was relatively passive, just in different ways.

What was largely absent was a more conversational mode of use — one in which the system is engaged iteratively as a thinking and writing partner, with ideas proposed, challenged, revised, and reshaped over multiple turns, rather than either handed off or treated as input to be used elsewhere2.

This isn’t necessarily a flaw in how the study was conducted. If relatively few participants used the LLM in a more iterative, collaborative way (i.e., as a thinking and writing partner), then there simply wouldn’t be much of a middle group to analyze3.

But that also limits what the results can tell us. What looks like a comparison across levels of AI use is, in effect, a comparison between two ends of the spectrum — both of which involve relatively passive interactions with the system, with relatively little evidence about what happens in that middle space.

Share

Where This Leaves Us

What the study captures clearly are the endpoints: minimal involvement on one side and near-complete delegation on the other. Those differences are real, and the results reflect them.

But the most effective uses of AI probably don’t sit at either end. They likely unfold somewhere in between — in interactions where parts of the process are offloaded and others are retained, often shifting back and forth as the work develops. That’s where the difference between offloading substance and offloading translation becomes most difficult to track, but also where it probably matters most.

The challenge for users is that this middle ground can be difficult to maintain. AI writing assistance may start with good intentions — using the system to refine ideas or clarify thinking — but it can gradually slide toward greater delegation.

And that shift wouldn’t necessarily be obvious as it’s happening. Each step toward delegation still feels like progress: refining an idea, improving a sentence, improving the argument. But over time, more of the substantive work can be absorbed by the system without a clear moment where control is consciously handed off. And that’s what makes it easy to lose track of how much of the thinking is still your own.

Leave a comment

Share

You can also hear AI Matt’s summary of the piece below.

I still remember one of the first things I asked ChatGPT to do.

I told it to research me.

The results were...not very encouraging.

It confidently invented publications I'd never written1, promoted me to fellow status in various professional societies, and generally mixed enough truth with enough fiction to create a biography that was equal parts flattering and fabricated.

My immediate conclusion was simple: this technology wasn’t anywhere close to ready for serious academic work.

Around the same time, though, I began noticing something else.

Students weren’t reaching the same conclusion. Instead of dismissing ChatGPT because it confidently made things up, many were asking it to write discussion posts, summarize readings, answer questions, and in some cases complete assignments. Almost overnight, generative AI had gone from a technological curiosity to something faculty everywhere suddenly had to reckon with.

At first, my response was probably similar to that of many instructors. I tried to limit students' use of AI and focused on identifying assignments that had been generated by ChatGPT. Sometimes that was pretty easy. Early versions of the software occasionally left behind telltale clues, like students accidentally submitting responses that included expresssions like, "As an AI..." embedded in them. More often, though, it became a frustrating game of whack-a-mole. As the underlying models improved, yesterday's giveaways disappeared, new ones emerged, and before long they disappeared too.

And over time, the whack-a-mole approach became harder to justify.

Yes, the models were improving — each new generation became more capable than the last. And so, it became harder, and less efficient, to try to catch the AI users. But that’s not what changed my thinking. Ultimately, my thinking changed because of my students.

Most of my graduate students were already working professionals. They weren’t preparing for careers that might someday involve AI. Many were already encountering these tools in the organizations where they worked. They would spend the day navigating a workplace where AI was increasingly viewed as a legitimate productivity tool, then log into my class that evening where I was effectively telling them it had no legitimate educational value.

And that contradiction became harder to ignore. If my students were expected to use these tools responsibly at work, why wasn’t I teaching them how to use them responsibly in the classroom?

Share

If You Can't Beat It...

That question didn’t have an obvious answer. I knew one thing, though. An AI free-for-all wasn’t going to cut it.

If students could simply ask ChatGPT to complete an assignment, then I needed to rethink the assignments themselves. Rather than trying to create activities that AI couldn’t touch, I wanted to create activities where AI couldn’t replace the thinking I actually wanted students to do.

That meant rethinking how students would learn in a world where AI existed. So I focused on creating assignments that required students to exercise judgment. They were asked to apply course concepts to their own lives, evaluate competing alternatives, justify their recommendations, and explain the reasoning behind their decisions. AI could certainly contribute to those conversations, but it couldn’t make those judgments for them.

At the same time, I began experimenting with something else. Rather than letting students interact with a general-purpose AI chatbot, I built a virtual assistant specifically for my course. It wasn’t designed to write papers or complete discussion posts. It was built around the course materials, the learning objectives, and the expectations I had for students throughout the term.

I thought students would jump at the chance to use it. But…they didn’t.

The first version was entirely optional. Plenty of students gave it a try, but for many it remained an occasional curiosity rather than a regular part of their learning. What surprised me most, though, was that many of the students who seemed likely to benefit from it the most rarely used it at all.

Some questioned AI’s reliability. Others weren’t convinced it offered much practical value. A few told me they preferred completing the work without AI altogether2. Ironically, many of the same reservations that had made me skeptical of ChatGPT initially were showing up in my own students.

So I changed the experiment. Instead of making the virtual assistant available as an optional resource, I required students to use it for one very specific purpose: receiving formative feedback on major assignments before submitting them.

That single change turned out to matter far more than I expected.

Subscribe now

Learning Alongside the AI

Once I instituted that requirement, students began using the virtual assistant consistently. They hadn’t suddenly become AI enthusiasts, but once the requirement was a part of the course, students began to view it as a potentially valuable learning assistant. And that change mattered a lot.

My goal had never been to encourage students to let AI complete their work. Instead, it was to become much more dliberate about which aspects of learning could reasonably be supported by AI and which needed to remain the student’s responsibility. And that way of thinking gradually reshaped the course.

Assignments continued to evolve to emphasize judgment rather than simply producing answers. Students were required to submit links to their conversations with the virtual assistant, making their use of AI transparent. Instead of wondering whether students had used AI, I could see how they had used it and whether it appeared to support — or replace — their thinking.

At the same time, I found myself spending almost as much time refining the assistant as I did teaching the course itself.

Behind the scenes, I continually revised the assistant’s instructions, adjusted how it responded to students, expanded the course materials it could draw upon, and experimented with ways of making it a more effective learning resource. Just as importantly, I spent as much time deciding what it shouldn’t do. The goal was to make sure students still had to exercise their own judgment so I could tell what they understood, where they were struggling, and how their thinking was developing over the course of the term.

Every iteration of the class taught me something new. Sometimes it meant changing the assignments. Sometimes it meant changing the assistant. Sometimes it meant changing my own assumptions about how students learned with AI. And because the underlying language models continued to evolve, the process never really stopped.

Buy Matt a Coffee

An Accidental Book in the Making

Looking back, it’s almost funny.

At no point did I set out to write a book about course-specific AI assistants. In fact, if you had asked me a few years ago what I’d someday write a professional book about, this wouldn’t have even made the list3.

All I was trying to do was solve a problem in one online graduate course.

Each term I made a few changes. I revised assignments. I adjusted the virtual assistant’s instructions. I refined what it would and wouldn’t do. Then I’d teach the course again, see what worked, and make another round of changes.

Somewhere along the way, I noticed something I hadn’t expected.

Although the course itself kept evolving, the process I was using to build and refine the assistant wasn’t changing very much. I found myself asking the same questions, organizing the same kinds of instructions, and solving the same design problems over and over. Without really planning it, I had developed a fairly consistent framework for building and managing course-specific virtual assistants.

That made me wonder whether the process would work beyond that one course. So I decided to find out.

I built virtual assistants for two other graduate courses I teach using the same underlying framework, while tailoring the instructions to the materials, learning objectives, and instructional idiosyncrasies of each course. Although the assistants themselves weren't identical, the underlying process transferred remarkably well.

Interestingly, the biggest change wasn’t student grades. Those improved modestly, particularly on early formative assignments where students appeared to acclimate to course expectations more quickly. Beyond that, the differences were relatively small.

What changed much more noticeably was how students engaged with the course itself.

Reading through the conversations they had with the assistants, I could see students wrestling with ideas, asking follow-up questions, testing their reasoning, and working through misunderstandings long before they submitted an assignment. In an online course, much of that thinking is normally inaccessible to an instructor. The virtual assistants, though, gave me a window into a learning process I rarely had the opportunity to observe before. That may have been the single biggest benefit of the entire experiment.

But the most valuable thing to emerge from all of this wasn’t the virtual assistant itself. It was a repeatable process for integrating AI into a course without allowing it to replace the learning the course was designed to produce.

At some point, I realized I’d accumulated far more than a collection of prompts, instructions, and revisions. I’d accumulated a practical process for designing, implementing, and refining course-specific AI assistants.

That’s when I realized I had a book4.

That book became AI Assistants for Teaching and Learning. Not because I thought I'd discovered the one right way to use AI in higher education, but because I kept wishing I'd had a practical guide like it when I started. My hope is that it gives other instructors a place to begin — and maybe saves them from having to reinvent quite as many wheels as I did.

If you've been wrestling with many of the same questions I was about how to integrate AI into a course without letting it replace learning, I hope you'll find something useful in it. And if not, that's okay too.

Share

Subscribe now

Check Out Matt's Book on Amazon

You can also hear AI Matt’s summary of the piece below.

Share

If you spend any amount of time following discussions about cellphone bans in schools on Twitter/X, you’ll eventually run across psychologist Chris Ferguson explainingoften with visible exhaustion, that some supposedly dramatic new finding about smartphones destroying adolescents is actually hovering somewhere between “tiny effect” and “statistical noise.”

It might look a little something like this:

At one level, this is just a disagreement about cellphone bans. Chris has long been skeptical of many of the stronger claims surrounding smartphones, social media, and adolescent mental health, particularly when those claims are built on weak correlational evidence or effects so small they barely rise above background statistical noise.

But the larger issue here extends well beyond cell phones.

But the disagreement points at something bigger. Modern public discourse has gotten remarkably good at converting weak signals into confident narratives. A tiny effect becomes a sweeping explanation. A weak correlation becomes evidence of societal collapse. A carefully cropped graph becomes proof that some new technology is destroying civilization. Once percentages, p-values, trend lines, and polished visualizations enter the conversation, the presentation itself starts to substitute for the strength of the underlying evidence. As Will Dobud recently put it, “nothing sells bullshit like a beautiful graph.”

Unscripted

Nothing Sells Bullshit Like a Beautiful Graph

If you’re on Twitter/X, you’ll see that the dialogue about fertility rates is all over the place. And because it’s social media, you’ll see how the narrative shifted to a usual culprit…

Read more

4 months ago · 27 likes · 2 comments · Will Dobud PhD

But this same issue pops up far beyond debates about smartphones in schools. It shows up in polling data, social science research, media reporting, and online discourse generally. Once a claim arrives all prettied up with percentages, trend lines, or polished visualizations, people become far less skeptical of whether the underlying signal is actually strong enough to support the story being told.

Subscribe now

Noise Before Narrative

One of the challenges of turning evidence into inference is that the underlying information is often noisy long before it becomes the foundation for headlines, narratives, or policy arguments.

Reality is messy, and most phenomena aren’t homogeneous or stable. Measurements vary. Indicators are imperfect. Human behavior shifts across situations and contexts. Many of the things we care about socially — attitudes, well-being, political preferences, mental health, trust — are especially hard to measure objectively.

That doesn’t mean measurement is impossible, just that when phenomena resist objective measurement, we end up relying heavily on self-report data: surveys, questionnaires, interviews, rating scales, polls, and other forms of subjective reporting. And that introduces another layer of noise into the data itself.

And that introduces another layer of noisiness into the data itself.

Self-report data mixes meaningful signal with ambiguity, measurement error, instability, and plain old human weirdness. People misunderstand questions. They interpret wording differently. Some answer carelessly, some strategically, some based on a passing mood rather than a stable belief. Some are trolling. Some are bored. Some are rushing through a survey for the participation credit or the gift card. All of it ends up in the same dataset.

Stafford (2026) discussed a good example of this in a post about the so-called “Lizardman Constant,” the idea that some percentage of survey respondents will endorse almost anything, including obviously absurd conspiracy theories. One study he highlighted found that roughly 10% of respondents endorsed a completely fabricated conspiracy claim involving the Canadian military secretly engineering giant super-intelligent raccoons for future invasions. Which, thankfully, isn’t a real thing.

Reasonable People

The Lizardman constant

Survey data are a constant part of the news cycle, informs government and business and feeds into how we, as a society, see ourselves…

Read more

3 months ago · 14 likes · 5 comments · Tom Stafford

But the important point isn’t really the raccoons. It’s that noisy responses, once turned into percentages, charts, and headlines, can start to feel like a stable and meaningful reflection of what large groups of people genuinely believe. “10% of respondents endorsed an absurd conspiracy theory” can quietly become “millions of people believe absurd conspiracy theories.” Those two statements are related, but they imply very different levels of confidence about what the data actually shows.

Some respondents may genuinely hold bizarre beliefs. Others are joking, confused, inattentive, contrarian, or just clicking through as fast as possible. Once those responses get aggregated, weighted, converted into percentages, and wrapped in statistical language, it gets easy to forget how noisy the raw material was.

Buy Matt a Coffee

Compressing Variability

Part of why that noise gets harder to see is that statistical aggregation compresses variability by design. Individual responses are messy, but collapsing them into percentages, averages, correlations, or trend lines obscures a lot of that mess.

Basic descriptive statistics make the problem obvious. Suppose two groups both average 3.0 on a five-point scale. On the surface, identical results. But that average alone tells you surprisingly little. One group might cluster tightly around the midpoint, reflecting broad agreement. The other might be full of wildly divergent responses — some people picking 1, others picking 5 — that just happen to average out to the same number.

The means may be identical, but as the image above shows, the underlying reality may not be.

But once data gets summarized statistically, the mean itself often takes center stage while the variability behind it fades into the background. It hasn’t disappeared. It’s just less visible.

Embed those summaries within more sophisticated statistical analyses and you drift even further from the messiness of the original observations. Layers of aggregation, transformation, modeling, and visualization can make it easier to draw conclusions and inferences that look cleaner, more stable, and more certain than the underlying evidence may actually warrant1.

The Pull of Coherent Stories

None of this happens automatically, though. Noisy data and statistical aggregation alone don’t produce misleading conclusions on their own. The bigger issue is that people generally dislike ambiguity, especially when it touches morality, politics, parenting, or any number of other charged topics2. Once noisy, variable information gets transformed into interpretable statistical patterns, we get tempted to treat those patterns as more meaningful and conclusive than they really are.

That temptation grows when we have a psychological or ideological stake in the outcome. I’ve discussed motivated reasoning before, including in closely related contexts (see here and here), but the short version is that people scrutinize evidence far less aggressively when it supports a conclusion they already want to believe.

You see this constantly in discussions of cellphone bans in schools, where tiny effects, inconsistent findings, weak correlations, and ambiguous results quickly get framed as decisive once they line up with broader cultural anxieties about smartphones, attention spans, or adolescent mental health. As Chris Ferguson keeps pointing out, findings hovering near “statistical noise” can rapidly get treated as robust evidence of broad causal effects.

And when noisy or ambiguous evidence gets packaged visually in a way that makes the narrative look cleaner and more definitive than it really is, it becomes much easier to accept, and much easier to sell, conclusions that rest on shakier ground than they appear to.

None of this means quantitative evidence is useless or that statistical analysis should be dismissed. But it does mean that numbers, graphs, and statistical outputs should not automatically be mistaken for certainty itself. Sometimes they simply make uncertainty look more orderly than it really is.

Leave a comment

Share

Share

Episode 6: Why Searching for the Best Backfires

We tend to assume that better decisions come from more effort—more research, more comparison, more time spent weighing every possible option. So we open another tab, read another review, tweak the spreadsheet one more time. It feels like we’re being careful. Responsible, even.

But at some point, that effort stops helping. The decision doesn’t get clearer. It just gets noisier.

In Episode 6, Connie and Neil unpack why that happens. Drawing on two pieces that explore heuristics and the idea of “good enough,” they dig into the limits of trying to optimize complex decisions—and why our brains aren’t built to process the kind of trade-offs we keep trying to force them to resolve.

Along the way, they look at how we actually make choices in the real world: by simplifying, filtering, and—eventually—stopping. Not because we’ve found the perfect option, but because we’ve decided what’s acceptable and moved forward.

This isn’t about lowering the bar or settling. It’s about recognizing that the search for the “best” is often what keeps us stuck—and that better decisions sometimes come from knowing when to stop looking.

Read the original posts:

The hovercraft is still crowded. The eels are comparing one more option. And no one is quite sure when to stop.

Leave a comment

Subscribe now

Share

Share

Happiness. It occupies a surprisingly prominent place in our culture.

In the United States, the pursuit of happiness appears right there in the Declaration of Independence, immediately after life and liberty as one of our unalienable rights1. Entire industries have been built around helping people find it. Self-help authors promise it. Politicians invoke it. Therapists discuss it. And if you’re of a certain age, you may remember Will Smith pursuing it in The Pursuit of Happyness.

Even Monty Python got in on the act.

The lead image (and video) for this post comes from their famous Lumberjack Song, where a young man describes his dream of becoming a lumberjack. Not just any lumberjack, mind you. A lumberjack leaping from tree to tree, floating down the mighty rivers of British Columbia, breathing in the smell of freshly cut timber, and crashing mighty trees to the ground. It's an oddly specific aspiration, but no stranger than many of the things people imagine will finally make them happy.

Of course, most of us don’t aspire to become lumberjacks. We have our own versions.

Maybe it’s a promotion. Maybe it’s retirement. Maybe it’s finding the right relationship, buying a house, paying off debt, publishing a book, earning a degree, starting a business, or finally getting around to that project we’ve been putting off for years. The specifics differ, but the underlying logic is often remarkably similar. Happiness sits somewhere in the future, waiting patiently for us to arrive.

And sometimes we do arrive — or so we think.

People get promoted. They retire. They publish the book. They buy the house. They find the relationship. Yet an odd thing often happens once they get there. The satisfaction is real, but it rarely stays put for very long. Before long, attention shifts to the next challenge, the next aspiration, the next thing that still hasn’t been achieved.

And that leads me to a conclusion I’ve reached after many years of following happiness-related research and arguments:

Pursuing happiness as an end state is fundamentally flawed.

And I’m not saying that because I’m horribly cynical (although I can already hear a few people disagreeing). I’m saying it because the more I’ve looked at the research, the less convinced I’ve become that anyone really knows what happiness is, how to obtain it, or how to hold onto it once they've found it. And as it turns out, those are all bigger problems than they first appear2.

Subscribe now

Problem #1: We Don’t Actually Know What Happiness Is

Let’s start with the most obvious problem.

Before we can pursue happiness, it would probably help if we knew what happiness is.

Unfortunately, that turns out to be more difficult than it sounds.

If this post were intended to provide a history of happiness, we could probably start with Aristotle and spend quite a while working our way forward. Fortunately for both of us, that's not where we're going. Even if we fast-forward a couple thousand years and jump directly to modern psychology, the problem remains surprisingly difficult to solve.

To be fair, psychologists have spent decades studying happiness and related concepts. The issue isn't a lack of research. The issue is that researchers don't always mean the same thing when they talk about happiness.

For example, one common distinction separates hedonic and eudaimonic approaches to well-being. While both are often discussed under the broader umbrella of happiness, they begin to look rather different once you examine how they’re actually measured.

Hedonic approaches focus more on pleasure, positive emotions, and satisfaction with one’s life. This is where the work of Ed Diener and others comes in. I mention Diener because he led efforts to develop arguably the most widely used measure of hedonic well-being: the Satisfaction With Life Scale (Diener et al., 1985). The measure is remarkably straightforward. It asks people to evaluate their lives. Are you satisfied with your life? Are the conditions of your life excellent? If you could live your life over again, would you change much?

Eudaimonic approaches, by contrast, focus more on meaning, purpose, growth, and realizing one's potential. This is where Carol Ryff comes in. Ryff (1989) developed a multidimensional measure of psychological well-being that assesses factors such as purpose in life, personal growth, autonomy, and self-acceptance.

Both approaches may be valuable. But they’re certainly not asking the same questions. Someone can be reasonably satisfied with their life while feeling stagnant and lacking a sense of purpose. Conversely, someone can feel deeply engaged in meaningful pursuits while also feeling frustrated, stressed, or dissatisfied with aspects of their current circumstances.

And the situation becomes even more complicated once additional concepts enter the conversation. Meaning. Flourishing. Purpose. Subjective well-being. Psychological richness. Depending on the author, these may be treated as components of happiness, alternatives to happiness, or closely related constructs3.

At this point, you might reasonably object that psychologists have already solved this problem. After all, there are measures that explicitly assess happiness.

And that’s true. Lyubomirsky and Lepper (1999), for example, developed the Subjective Happiness Scale, which asks people to evaluate the extent to which they consider themselves to be happy.

But this doesn’t really solve the problem. After all, if researchers debate what happiness consists of, how do we develop a measure of it in the first place?

The answer, of course, is that we can simply ask people whether they consider themselves to be happy. There’s nothing inherently wrong with doing so. If our goal is to understand how happy people perceive themselves to be, such measures can be quite useful.

The problem is that they tell us very little about what happiness actually consists of.

Imagine three people who all describe themselves as equally happy. We don’t know what it is about their lives that produces that assessment, nor do we know whether the same factors are driving it across all three people.

In that sense, happiness becomes largely subjective and self-defined. That’s perfectly acceptable if our goal is to obtain a snapshot of how happy people believe themselves to be. It becomes much less useful if our goal is to understand how happiness is created, maintained, or increased.

And that’s really the problem.

The advice to “pursue happiness” sounds straightforward until you start asking what happiness actually means. At that point, the destination becomes surprisingly difficult to identify. Depending on who you ask, happiness may refer to satisfaction, positive emotions, meaning, purpose, flourishing, psychological richness, or some combination of the above.

The point isn’t that any of these perspectives are wrong. The point is that they point toward different destinations. Before we can pursue happiness, it would probably help to know which destination we’re trying to reach.

Unfortunately, even if we could solve that problem, we’d immediately run into another one. And that leads us to Problem #2.

Share The Eel-Filled Hovercraft

Problem #2: We’re Not Very Good at Predicting What Will Make Us Happy

Suppose, for the sake of argument, that we’ve somehow solved Problem #1.

Let’s assume that you’ve arrived at a definition of happiness that you’re comfortable with. Maybe happiness means satisfaction. Maybe it means meaning and purpose. Maybe it means some combination of the two. Whatever the case, let’s assume you know exactly what you’re trying to pursue.

Unfortunately, that doesn’t solve the problem.

Because humans aren’t especially good at predicting what will make them happy.

This is where the work of psychologist Dan Gilbert becomes relevant. In his book Stumbling on Happiness, Gilbert argued that people are surprisingly poor at forecasting the preferences and reactions of their future selves. We imagine how happy, unhappy, satisfied, or disappointed we’ll feel if certain events occur, but those predictions are often wrong.

Sometimes we overestimate how much a positive outcome will improve our lives. Sometimes we overestimate how devastating a negative outcome will be. And sometimes we simply pursue things that seem as though they should make us happy only to discover that they don’t produce the experience we expected.

We convince ourselves that a promotion will finally make work enjoyable, only to discover that it mostly came with more responsibility and meetings. We imagine retirement as an endless vacation, only to find ourselves searching for structure and purpose. We assume that moving, changing jobs, or making a major purchase will fundamentally change how we feel, only to discover that the effect is smaller or shorter-lived than we anticipated.

Part of the problem is that happiness is a moving target. Part of it is that we adapt to changing circumstances. Part of it is that our future selves don’t always want the same things our present selves think they will.

Whatever the reason, the practical implication is difficult to ignore. Even if we know what happiness means, that doesn’t necessarily mean we know how to obtain it.

And yet there is an even more frustrating possibility.

Sometimes people correctly identify something that improves their happiness. They get the job. They find the relationship. They achieve the goal. They obtain exactly what they were seeking.

So, now it seems like those people have solved the problem. They figured out what happiness means. They correctly identified something that would make them happy. They achieved it. Good for them. Right?

Well, all that really does is introduce Problem #3.

Problem #3: Happiness Has a Short Shelf Life

Let’s return to the people who managed to solve Problems #1 and #2.

They figured out what happiness meant to them. They correctly identified something that would make them happy. And they achieved it.

Now what?

Do they spend the rest of their lives basking in a permanent state of happiness and contentment?

Not hardly.

What they’re more likely to experience is a period of satisfaction. They achieved something they cared about. They reached a goal. They obtained something they wanted. And yes, they may experience a genuine increase in happiness.

The problem is that the feeling rarely lasts forever.

Over time, people tend to become accustomed to their new circumstances. What once felt exciting becomes familiar. What once felt extraordinary becomes normal. Psychologists often refer to this tendency as hedonic adaptation (Lucas, 2007), or more colloquially, the hedonic treadmill. We experience positive changes, but eventually return toward a relatively stable baseline level of well-being.

Perhaps the easiest way to think about this is to use an analogy. Imagine someone receives a substantial raise or lands their first high-paying job. Initially, the increase in income may feel transformative. Suddenly there’s more money available for travel, entertainment, hobbies, housing, or any number of other things. Yet over time, something interesting often happens. Spending habits adjust. Expectations change. What once felt like abundance gradually becomes the baseline against which future circumstances are judged.

Something similar often happens with the things that initially make us happy. The promotion eventually becomes your job. The new house becomes your home. The degree becomes a line on your résumé. The achievement remains real, but it no longer occupies the same psychological space it once did. What was once a destination gradually becomes part of everyday life.

At first glance, this sounds like a flaw in the system. Why would we be built in a way that prevents us from remaining satisfied with things we worked so hard to obtain?

According to Pinsof (2026), the answer may be that this isn't a flaw at all. His argument is that many of our desires are effectively “antisatisfiable.” Satisfying them doesn’t eliminate them. Instead, it simply changes the standard against which future circumstances are evaluated. New desires emerge. Expectations shift. Comparisons change. The goalposts move.

A person who receives a promotion may begin thinking about the next promotion. A person who achieves financial security may begin worrying about preserving it. A person who develops expertise in a domain may begin noticing new areas where they still fall short. The original goal may have been achieved, but achieving it often creates new goals in the process.

When you think about it, the tendency for satisfaction to fade starts to make a certain amount of sense. A person who became permanently satisfied after achieving a goal would have little reason to continue striving. There would be no need to improve, compete, explore, create, learn, or adapt. The motivational system would effectively shut itself off.

Whether or not Pinsof’s explanation is completely correct, it highlights something important. The psychological mechanisms that make lasting satisfaction difficult may be the very same mechanisms that keep us moving forward.

And that raises an interesting possibility.

Perhaps the issue isn’t simply that satisfaction fades. Perhaps satisfaction and motivation are serving different functions altogether.

That possibility becomes especially interesting when we consider research on motivation and engagement. Warr and Inceoglu (2012), for example, argued that motivation is often fueled by discrepancies between one’s current state and a desired future state. In simple terms, part of what motivates us is the existence of a gap between where we are and where we would like to be.

Based on this perspective, satisfaction and engagement serve different purposes, even though they’re often lumped together under the broader category of positive well-being. Engagement creates movement. It reflects the pursuit of a desired future state. Satisfaction is what we feel when we get there.

But the process doesn't stop there. Once we achieve the goal, the satisfaction we experience is often temporary. Expectations adjust, and new goals emerge. What was once a source of satisfaction gradually becomes part of the baseline, and the cycle begins again.

Share

Rethinking the Pursuit of Happiness

And that brings me back to the pursuit of happiness itself.

If satisfaction is temporary by design, and engagement depends on the existence of goals that have not yet been achieved, then the idea of reaching a permanent state of happiness starts to look increasingly unrealistic. The destination keeps moving because movement may be part of how the system works.

Perhaps that’s also part of the reason we’re not very good at predicting what will make us happy. What we’re missing today often becomes tomorrow’s baseline. The thing we believe will finally satisfy us may very well do so — but only temporarily. Once achieved, it gradually becomes part of the status quo, and our future selves begin looking toward new goals, new challenges, and new aspirations.

In that sense, the three problems I discussed are all pointing toward a similar conclusion. Happiness, as an end state, may not really exist. It’s not a destination we eventually arrive at or a goal we permanently achieve. That’s why we struggle to define it. That’s why we’re not very good at predicting what will finally get us there. And that’s why the pursuit of happiness often seems never-ending.

Whether intentionally or not, Thomas Jefferson may have been onto something when chose the language he did for the Declaration of Independence. When he wrote that people possess the right to life, liberty, and the pursuit of happiness, he didn't identify happiness itself as an inalienable right. He identified the pursuit of happiness as one.

And that brings me back to our aspiring lumberjack.

The funny thing about the Lumberjack Song is that we never really learn much about what it would actually be like to be a lumberjack. What we get instead is an idealized vision of the future. Leaping from tree to tree. Floating down the mighty rivers of British Columbia. Breathing in the smell of freshly cut timber. The dream is vivid. The reality remains largely unexplored.

Maybe that’s true of happiness too.

We become remarkably good at imagining how satisfying a future achievement will be. We picture the promotion, the retirement, the relationship, the house, the degree, or the book. We imagine how we'll feel when we finally get there. What we rarely imagine is what happens next.

But perhaps the arrival was never really the end of the story. Sooner or later, the goalposts move. A new challenge appears. A new aspiration emerges. And off we go again, leaping from tree to tree along the mighty rivers of British Columbia.

Leave a comment

Subscribe now

You can also hear AI Matt’s summary of the piece below.

Share

A while back, I came across an article titled “Trust the Experts? It’s a Bad Bet.” At first glance it sounded like it was going to echo an argument I myself made concerning the so-called “War on Expertise.” But reading past the headline revealed a much narrower—and very different—argument. The real target wasn’t expertise in general; it was expert prediction. The claim, in essence, was that experts are often poor forecasters of the future, and trusting their predictions is usually a losing proposition.

On that point, the author is largely right. Trusting expert predictions often is a bad idea, as Grossman et al. (2023)1 recently showed. But it’s not because expertise itself is overrated—at least not real expertise. It’s because what makes someone a true expert is rarely their ability to function as a prophet.

Judging expertise by its ability to predict uncertain futures almost guarantees disappointment—and it sets up a false dichotomy in the process: either we defer to expert authority, or we conclude that expertise itself is overrated. Neither position gets us very far.

The more interesting question isn’t whether we should “trust experts” in some broad, abstract sense. It’s what we should expect expertise to actually deliver—and where its limits lie. True expertise helps us make sense of the present: to diagnose problems we already face, interpret evidence we already have, and weigh tradeoffs under real constraints. When we confound that kind of judgment with prophecy, we misunderstand what expertise is for in the first place.

Subscribe now

What Expertise Actually Is

Expertise isn’t primarily about long-horizon prediction. It’s about judgment—specifically recognizing what kind of situation you’re in and responding in ways that have worked before under similar conditions (aka recognition-primed decision making). Across domains, what distinguishes experts from novices isn’t superior logic or broader abstract knowledge—it’s their experience with recurring patterns. Over time, experts learn which cues matter, which ones can be ignored, and which tradeoffs are likely to follow from different courses of action—ultimately giving rise to what Kahneman & Klein (2009) described as skilled intuition.

When faced with a problem, experts don’t start from scratch by weighing every possible option. Instead, they recognize familiar elements, generate a small set of plausible responses, and then evaluate whether those responses fit the specifics of the situation at hand. When the context aligns with their prior experience, this process can be remarkably fast and effective. It’s also why expert intuition can look effortless from the outside: much of the work has already been done through repeated exposure to similar problems (Prietula & Simon, 1989).

That strength, however, comes with a clear limitation. Expert judgment is tightly bound to context. It works well within what I’ve referred to elsewhere as an expertise “bubble”—the range of situations in which expert judgment provides reliable guidance.

That doesn't mean expertise is blind to the near future. A veteran firefighter can anticipate how a blaze will likely move. An experienced physician can foresee likely complications. But those judgments unfold in environments where patterns recur and feedback arrives quickly. As problems drift farther beyond that bubble—toward novel conditions, unfamiliar environments, or extended time horizons—the cues that once anchored recognition begin to disappear. At that point, judgment shifts from recognition to extrapolation (a pattern echoed in Grossman et al., 2023). And extrapolation is a far shakier endeavor.

Share The Eel-Filled Hovercraft

Why Expert Predictions Often Look Like a Coin Toss

Imagine an expert is brought into an organization to help solve a problem. They’re given a general description of what’s going wrong and then asked, on the spot, to say what the organization should do. They’re given no opportunity to dig deeper and no chance to gather additional information. They’re simply asked to make a judgment call.

In that situation, even a genuine expert has little choice but to guess. Some of the assumptions they make will be reasonable. Others won’t. And unless the problem is unusually simple or well-structured, there’s no reason to expect the final judgment to be much better than chance. That isn’t a knock on expertise; it’s what happens when judgment is forced to operate without the evidence it needs to do its job.

What differentiates experts in real settings isn’t their ability to issue answers on demand. It’s their ability to shape the judgment process based on their skilled intuition—to guide the collection of evidence in ways that make judgment possible. They know which questions to ask next, which information is worth gathering, and when additional data will no longer meaningfully improve their decision. That process refines and calibrates the pattern recognition that underlies expert judgment.

Stretch that judgment into longer-term forecasting, and the process begins to short-circuit. By definition, the problem is thin on direct evidence and saturated with uncertainty. There’s little opportunity for guided information gathering, iteration, or meaningful feedback. The expert is forced to fill in the gaps with assumptions about how people, organizations, or systems will behave. Some of those assumptions will be right, and others won’t. By the time events unfold, the original prediction is rarely revisited in detail, the assumptions that supported it are almost never examined, and apparent successes may reflect luck rather than sound judgment. Averaged out, accuracy drifts toward a coin toss.

This is why expert predictions so often disappoint. It’s not that experts suddenly lose their competence when asked about the future. It’s that prediction removes the very mechanism that makes expertise useful in the first place: context-sensitive judgment refined through selective evidence and feedback over time. When we strip that away, what remains isn’t expert insight so much as guesswork—and the results look exactly like you’d expect.

Buy Matt a Coffee

Why We Keep Asking Experts to Predict Anyway

If expertise doesn’t translate well into making oracle-like predictions, it begs the question of why we keep asking experts to do it in the first place. There are several reasons this practice persists; here I want to focus on three that are closely related2.

The first is a basic but persistent confusion about what expertise actually entails. Expertise involves informed judgment—recognizing relevant cues, interpreting evidence, and weighing tradeoffs in context. Because experts are good at making sense of complex situations, it’s easy to assume that skill should extend naturally to saying what will happen next. In practice, those are different tasks operating under very different conditions (Önkal et al., 2003), but the distinction is rarely made explicit.

The second is what might be called prediction theater. The future is, by definition, uncertain, but that uncertainty will eventually resolve into events that affect people’s lives—creating opportunities, producing risks, and forcing choices with real consequences. Asking experts to forecast those events offers a way to make an open-ended future feel more manageable. A forecast compresses uncertainty into a concrete claim about what’s likely to happen, even when the underlying uncertainty remains unchanged.

The third is that predictions are unusually difficult to evaluate. Many forecasts won't work out, but prediction accuracy often won't be clear until much later. By then, few people remember the original forecast in enough detail to evaluate its accuracy, and fewer still revisit the assumptions that supported it. Even apparent successes can be misleading, since it's often difficult to determine whether a prediction was correct because of sound judgment or simple luck.

This makes forecasting unusually difficult to evaluate and creates a system that selectively amplifies confident predictions while rarely holding them accountable. Experts who speak decisively are rewarded with attention and credibility, while those who hedge, qualify, or emphasize uncertainty are often sidelined (Lawrence et al., 2006). The result is a forecasting environment that rarely learns from either success or failure.

What Expertise Is (and Isn’t) Good For

None of this means expertise is useless, or that skepticism toward experts is misplaced. It means we often ask experts to do the wrong job. Expertise isn’t a crystal ball. It’s a way of making sense of messy situations in the present—of diagnosing problems, interpreting evidence, and helping decision makers navigate tradeoffs under real constraints.

When used that way, real expertise still matters a great deal. Experts are most valuable when they’re allowed to interact with a problem: to ask questions, gather targeted evidence, test assumptions, and revise their judgments as new information comes in. That’s where recognition-primed judgment shines—and where expert insight genuinely improves decisions.

Trouble starts when we collapse that judgment process into a demand for prediction. When we treat experts as forecasters of uncertain futures, we strip away the context and evidence their judgment depends on, then fault them when accuracy collapses. The result is predictable disappointment, followed by an unhelpful backlash against expertise itself.

That doesn’t mean some people don’t outperform others at prediction, but those differences reflect strategies for managing uncertainty, not a deeper form of expertise or foresight. And even then, reliability tends to decay as the time horizon expands3.

If we want better decisions, the solution isn’t to stop listening to experts—or to trust them blindly. It’s to stop confusing judgment with prophecy. We should expect expertise to deliver what it actually can: sound judgment about the situation in front of us, not reliable predictions once we move beyond it.

Leave a comment

Subscribe now

You can also hear AI Matt’s summary of the piece below.

Share

Hybrid and remote work are often framed as a test of individual discipline. When things go poorly, the explanation comes quickly: some people just don’t manage flexibility well. They blur boundaries, overwork, under-prioritize, or stay perpetually “on” when they shouldn’t. This pattern is often described as the autonomy paradox, where increased discretion over work coincides with greater self-imposed demands (Vassiley et al., 2025).

Flexible work arrangements do demand skills—self-regulation, prioritization, and boundary setting—that vary across workers, and not everyone has them in equal measure. But that explanation is incomplete. What’s harder to explain is why these problems show up so reliably—across roles, organizations, and even among experienced, high-performing employees who otherwise manage their work effectively. If hybrid work failures were mostly about individual shortcomings, we’d expect them to be sporadic. Instead, they’re patterned across contexts and roles (Pendell, 2025).

Rather than focusing on whether workers manage flexibility well, a more useful lens is how organizational approaches to strategic boundary control shape the conditions under which workers allocate their time, attention, and availability. Choices about which boundaries to formalize, which to relax, and which to leave ambiguous don’t just affect coordination—they determine how much interpretive and self-regulatory work employees must carry on their own.

Subscribe now

Strategic Boundary Control and the Distribution of Decision Making

According to Perrigino and Raveendhran (2025), strategic boundary control refers to how organizations structure and regulate the boundaries that govern work participation under conditions of limited direct oversight—especially salient for remote and hybrid work arrangements. Rather than relying on constant supervision or fully specified rules, organizations shape behavior by deciding which boundaries are explicitly defined and which are left to situational judgment.

These boundaries commonly involve where work (or which type of work) is performed, when work participation is expected, how coordination and responsiveness are triggered, and how effort gets noticed and judged. Importantly, strategic boundary control doesn’t require full specification. Some boundaries are left partially open, allowing organizations to accommodate variation across roles or contexts while still exerting control indirectly.

Seen this way, boundary ambiguity isn’t necessarily a failure to design or clarify expectations. It’s one way organizations maintain discretion and adaptability when traditional control mechanisms no longer function reliably. But those same boundary strategies also have downstream consequences: they determine where interpretive and self-regulatory work is performed—and how much of that work is carried by employees themselves. And most importantly for this piece, they affect how workers allocate their personal resources.

Share The Eel-Filled Hovercraft

Implications of Boundary Strategies for Personal Resource Allocation

When organizations exercise strategic boundary control, they don’t eliminate the need for personal resource allocation—they shape the conditions under which it occurs. Decisions about which boundaries are specified and which are left open determine where, when, and by whom time, attention, and availability must be managed.

In more traditional on-site settings, many personal resource allocation decisions are resolved implicitly by shared structure. Physical co-location, fixed schedules, and observable routines reduce the need for ongoing interpretation about availability, responsiveness, and engagement. Workers may have less discretion over where and when work occurs, but they also face fewer moments where they must decide how their resource allocation choices will be interpreted. Expectations about participation and responsiveness are embedded in the environment itself, limiting the need for constant calibration.

Remote and hybrid work come with less built-in structure, and as a result, decisions about boundary management cannot be fully pre-resolved by the environment. Consider a hypothetical case: a remote worker with complete autonomy over when, where, and how work is performed. In principle, that sounds maximally flexible. But unless the work is entirely independent—requiring no coordination with others and imposing no interdependencies—some degree of strategic boundary control is unavoidable.

Without it, the worker is left to their own judgment about when to be available, how quickly to respond, how to coordinate with others operating on different schedules, and which patterns of engagement will be interpreted as reliable or committed. Those judgments woudn’t be made once. They’d be revisited repeatedly, across situations and audiences, because their consequences depend on how others interpret them. In this sense, autonomy without shared boundary rules would relocate all resource allocation decisions to the worker, where they would have to be managed through ongoing inference rather than clearly defined expectations.

The implications of strategic boundary control for personal resource allocation are therefore not one-directional. Specifying boundaries does reduce how much is left to individual judgment, but it also limits how much workers can rely on their own preferred self-regulatory strategies. In other words, when boundaries are defined, fewer decisions must be improvised—but fewer can be tailored to individual rhythms, preferences, or working styles.

From this perspective, difficulties don’t arise only when boundaries are underspecified. They can also emerge when boundary specifications are misaligned with how individuals would otherwise manage their time, attention, and energy. A worker who self-regulates effectively under flexible conditions may find rigid participation or responsiveness rules disruptive rather than supportive. In such cases, the issue is a mismatch between organizational boundary control strategies and individual self-regulatory tendencies.

How Strategic Boundary Control Shapes Personal Resource Allocation

Strategic boundary control shapes personal resource allocation not by dictating how employees should manage their time and attention, but by structuring the decision space in which that management takes place. Organizations exercise boundary control along several dimensions—where work is performed, when participation is expected, how coordination and responsiveness are initiated, and how effort and commitment are evaluated. Each of these boundaries can be tightly specified, loosely guided, or left to situational judgment, and those choices determine which allocation decisions are resolved collectively and which are deferred to individual discretion.

From a PRA perspective, the significance of these boundary strategies lies in where they locate judgment. When boundaries are specified, fewer allocation decisions are left open to interpretation, but individual self-regulatory strategies are also constrained. When boundaries are left open, workers retain greater discretion, but they must rely more heavily on their own judgment to determine what counts as appropriate participation, responsiveness, or effort. In either case, strategic boundary control distributes the work of allocating time, attention, and availability.

Location boundaries govern where work is expected to occur and whether particular tasks are tied to particular settings. When these boundaries are clearly specified, workers face fewer decisions about signaling presence or participation. For example, if certain tasks are explicitly designated as “in-office work,” simply being present during those periods communicates engagement without further effort. When location boundaries are relaxed, workers gain flexibility but must decide for themselves when physical or virtual presence matters and how absence will be interpreted. This shifts effort toward managing visibility and participation cues—deciding when to be seen, logged in, or responsive—alongside task completion.

Time boundaries shape when participation is expected and how availability and disengagement are understood. Specified schedules and responsiveness norms resolve many allocation decisions in advance, reducing the need to continually decide when it is acceptable to be unavailable. For instance, clearly defined working hours or response-time expectations limit uncertainty about when replies are required. More open time boundaries preserve discretion, but they also require workers to make repeated judgments about responsiveness, pacing, and recovery—such as whether responding late at night signals commitment or simply creates future expectations. These decisions influence how attention and energy are distributed over time, even when formal workloads remain unchanged.

Coordination boundaries determine how and when workers are expected to interact with others. In tightly structured systems, coordination is initiated through formal channels—scheduled meetings, predefined workflows, or explicit handoffs. A standing weekly meeting or a fixed handoff process resolves many questions about when interaction is required. When coordination boundaries are looser, workers must decide when to reach out, when to wait, and when to interrupt (or allow themselves to be interrupted). This adds a layer of PRA demand focused on monitoring and anticipation, as attention must be allocated not only to one’s own tasks but also to managing interdependencies in real time.

Evaluation boundaries govern how effort, contribution, and commitment are assessed. When evaluation criteria are explicit, workers have clearer guidance about which forms of effort or output matter—such as clearly weighted performance metrics or well-defined deliverables. When evaluation is more implicit, workers must infer what will be noticed or valued, often allocating additional time and attention toward behaviors that are visible rather than those that are necessarily more effective. This shifts allocation toward signaling and impression management, particularly under conditions where the basis for evaluation remains uncertain.

Taken together, these boundary dimensions illustrate how strategic boundary control shapes the terrain on which personal resource allocation occurs. In on-site settings, much of that terrain is shaped by imposed structure; in remote and hybrid settings, less of that structure is built in, leaving more boundary decisions to be worked out explicitly. More organizationally constrained boundaries create greater certainty for workers, but they also limit the extent to which individuals can capitalize on their own self-regulatory idiosyncrasies—one of the commonly cited benefits of flexible work. Less constrained boundaries preserve that potential benefit, but they also introduce new challenges around interpretation and impression management1.

Share

Defensive Allocation as a Rational Response to Boundary Ambiguity

When strategic boundary control leaves key participation boundaries open or loosely specified, workers don’t respond haphazardly. They adapt in ways that make sense given the decision environment they’re operating in. Those adaptations tend to be defensive—not necessarily because workers are trying to game the system, but because they’re managing uncertainty about how their choices will be interpreted.

The defining feature of these environments isn’t flexibility per se, but interpretive risk. When boundaries around responsiveness, coordination, or evaluation are only partially specified, workers must allocate time and attention while anticipating how others might interpret their choices. Under those conditions, allocation decisions are shaped less by what is most effective in the moment and more by what is least likely to be questioned or penalized.

This helps explain why availability, visibility, and responsiveness often dominate allocation strategies in remote and hybrid work. Staying reachable reduces ambiguity around engagement. Responding quickly reduces the risk of appearing disengaged. Making effort visible compensates for uncertainty about how work is evaluated. Each of these behaviors aligns with specific boundary dimensions, and each functions as a hedge against misinterpretation rather than a direct investment in performance.

Importantly, these strategies don’t require explicit pressure from managers. They emerge even in the absence of stated expectations because workers are responding to gaps in boundary specification, not to direct demands. When it’s unclear which behaviors matter most, people allocate resources toward those that are safest across audiences and contexts. Over time, those defensive allocations become routine heuristics, shaping how work gets done even when no one has asked for them explicitly.

From a PRA standpoint, the problem is the cumulative effect of repeatedly allocating attention and energy toward managing interpretation rather than toward task execution, focus, or recovery. Defensive strategies protect against short-term risk, but they do so by drawing steadily on limited personal resources. Because the costs accrue gradually, they’re easy to miss until strain or dissatisfaction becomes salient.

Seen this way, behaviors often labeled as poor self-management are sometimes better understood as rational responses to boundary conditions that shift interpretive burden onto workers. Defensive allocation isn’t necessarily a failure of discipline or motivation. It can be an expected outcome of decision environments where the cost of being misread is asymmetric and the rules governing participation remain partially unspecified.

Subscribe now

Boundary Design Is Decision Design

It would be a mistake to conclude that the solution to these challenges is tighter boundary specification across the board. Overly constraining boundaries can undermine the very flexibility that makes remote and hybrid work valuable in the first place. What counts as reasonable availability, coordination, or responsiveness varies across roles, tasks, and interdependencies, and rigid rules are unlikely to fit all of them well.

Strategic boundary control is therefore not a problem to be solved once, but a set of trade-offs to be managed (Grawitch et al., 2023). Leaving some boundaries open preserves adaptability and allows individuals to apply self-regulatory strategies suited to their work. At the same time, those open boundaries shift interpretive burden onto workers unless they are supported by shared norms and mutual understanding.

This is where managers and teams matter. When formal boundaries are loose, informal norms take on greater significance. Teams must be deliberate about which expectations they clarify collectively and which they intentionally leave to individual judgment. Without that shared work, ambiguity is likely to result in defensive allocation strategies that protect against misinterpretation rather than support sustained performance.

Leave a comment

Share

If you found this post insightful, consider supporting my work by buying me a coffee.

Buy Me a Coffee

You can also hear AI Matt’s summary of the piece below.

Share

A while back, Mila Agius shared a fascinating piece about an odd issue OpenAI had been dealing with: goblins.

I’ll admit upfront that I was a little late to this particular party. I hadn’t actually come across the issue itself until reading her piece, which apparently put me behind a sizable portion of the internet that had already been joking about ChatGPT’s apparent fixation on goblins — as well as gremlins, trolls, and various other strange creatures.

Heuristics vs Traps

The Goblin Problem

TL;DR What if your financial decisions are running on a reward signal you’ve never examined? OpenAI just discovered that 2.5% of training data produced 66.7% of an absurd behavior in their AI – small, invisible, and impossible to undo by willpower alone. The same dynamic shapes human choices: a familiar account, an old habit, a default that registered a…

Read more

4 months ago · 13 likes · 7 comments · Mila Agius

Not literal goblins, of course. The issue was that certain models had started inserting references to various creatures into conversations where nobody had asked for them. Coding discussions would suddenly include “performance gremlins.” Random prompts would drift toward goblins, trolls, raccoons, or pigeons. At one point, OpenAI apparently had to add explicit instructions telling one of its coding agents not to mention goblins or gremlins in irrelevant contexts.

Which, as sentences go, is certainly is not one I ever expected to write.

And while the whole thing initially sounds like one of those wonderfully absurd AI-era stories — somewhere between “the chatbot is haunted” and “someone accidentally trained TolkienGPT” — it actually reveals something worth discussing a little more.

What struck me wasn’t the goblins themselves. It was the reminder that AI systems do not necessarily connect information the way humans do.

That may sound obvious on the surface, but I think we often forget it because large language models (LLMs) are so fluent. They respond conversationally. They explain themselves confidently. They can appear thoughtful, reflective, even oddly human at times. And because of that, it becomes very easy to assume there’s a stable, human-like reasoning process behind those outputs.

Occasionally, though, the underlying statistical machinery becomes a little easier to see. Sometimes a system develops a strange fixation or behavioral quirk that makes little intuitive sense from a human perspective. And while those moments are often amusing, they can also be revealing. They remind us that these systems are fundamentally statistical engines trained to detect patterns across enormous amounts of data and generate responses from the statistical relationships embedded within them.

That defining characteristic of LLMs matters more than it may initially seem — particularly in decision situations shaped by ambiguity, context, and uncertainty. AI can clearly produce useful outputs. But getting those systems to consistently behave in ways that align with what humans intend is often much harder than we assume. A big part of the problem is that a great deal of human judgment relies on tacit knowledge — pattern recognition, contextual interpretation, trade-off balancing, and experience-based intuition that people themselves often struggle to fully articulate. We often know more than we can easily explain, which becomes a problem when we try to convert implicit human judgment into explicit operational instructions for an AI system.

And there’s the rub. The more we attempt to offload decision making to AI systems, the more we have to operationalize things humans frequently navigate implicitly. Sometimes that translation works remarkably well. Sometimes it produces goblins.

Subscribe now

The Difference Between Chess and Real Life

LLMs are not the first AI systems to emerge, and they certainly won’t be the last. And there’s plenty of evidence that AI can work extraordinarily well in relatively closed or highly constrainable environments.

Chess and Go are obvious examples. The rules are explicit. Success is clearly defined. Feedback is immediate. The same basic logic applies to many real-world automation systems. In manufacturing, logistics, quality control, and other highly structured environments, offloading portions of decision making to AI systems can dramatically improve speed, consistency, and efficiency — often with remarkably high levels of reliability.

But AI can also be highly effective in environments that are far messier—at least under the right conditions. Take Google Maps. On the surface, routing through real-world traffic looks nothing like chess. The environment is dynamic, partially unpredictable, and shaped by human behavior. But the underlying problem is still tightly defined: get from Point A to Point B as efficiently as possible given available data.

What makes it work isn’t that the environment is clean — it’s that the objective is, and that the system has access to massive, continuously updating streams of data that allow it to approximate current conditions well enough to act.

So even in a messy environment, the problem remains:

• Clearly specified,

• Continuously updated, and

• Immediately testable (you either get there faster or you don’t)

In other words, the environment is messy, but the problem itself is still bounded. That combination makes it something you can actually automate reliably — unlike a lot of real-world decisions.

Many — maybe most — of the decisions we face don’t aren’t early as clean as even the messy world that Google Maps operates in. The objective isn’t always clearly specified. The information isn’t continuously updated in any meaningful way. And the outcome isn’t immediately testable.

Buy Matt a Coffee

When Specification Breaks Down (and the Goblins Take Over)

One way to think about this is in terms of specification.

For problems like routing or scheduling, the objective can be made explicit. You can tell the system what to optimize for, and it can act accordingly.

That becomes much harder — sometimes impossible — once you move into real-world decisions.

Take something like evaluating a job offer. There are trade-offs at play — higher pay versus greater flexibility, for example. But how much is one worth relative to another? How much additional pay compensates for a loss of flexibility — and by how much? Even when we feel like we have clear preferences, they’re rarely specified in that way. The trade-offs themselves are often implicit, shifting, and only become fully apparent in the moment of choice — or sometimes not until afterward.

And the decision itself unfolds under conditions of incomplete and ambiguous information. You’re relying on interviews, impressions, maybe a few conversations with people who might not be representative. The feedback is slower and much less clear as well. You won’t know if it was the “right” decision in a week or even a month — sometimes not even in a year.

The same thing shows up in much smaller decisions as well — including something as simple as what to order for dinner. There’s no single best choice. Preferences shift. Trade-offs are small but still present. You might think you prefer Chinese food to pizza, but that preference depends on context: how hungry you are, how long you’re willing to wait, what you ate yesterday, who you’re with.

There isn’t a stable objective function sitting underneath these decisions. There’s a set of loosely connected considerations that get weighted — often unconsciously — in real time. That’s what makes these kinds of judgments difficult to fully offload to a system. There just isn’t enough cleanly specified for it to optimize against. And even when something is specified, it rarely captures how we’re actually reasoning in that moment.

So if we try to fully offload those kinds of decisions to an AI system, the system has to fill in the gaps between what’s explicit and what remains tacit. And because it can only work from what has actually been specified, it does that largely through statistical approximation.

And that’s where ChatGPT’s goblins start to show up. In this case, the system was trying to apply a “nerdy” persona that had been defined behind the scenes. The label was specified, but what it actually meant in practice wasn’t — because much of what counts as nerdy is tacit rather than explicit. We have a sense of what it means, but it’s hard to spell out how it should show up in language or behavior. We know it when we see it — but that doesn’t do the system much good.

So it leans on patterns — bits of language it associates with that kind of behavior. That’s where the occasional, and sometimes odd, insertions of words like goblins, gremlins, or trolls come from. It’s the system trying to make the output fit an objective that was never fully specified.

When Nobody Catches the Goblins

The ChatGPT goblins are, in many ways, an amusing example of how these kinds of approximation problems can show up. Part of what made them so noticeable, though, was the fact that people were directly interfacing with the system conversationally. The outputs were visible. The weirdness was observable. Humans remained close enough to the interaction to notice that something had drifted off course.

And that matters.

Conversational systems, by design, keep a person in the loop. The user sees the output, reacts to it, questions it, or ignores it. That doesn’t mean the oversight is particularly strong. Fluency can mask weak reasoning. Confident responses can discourage scrutiny. And as outputs become more coherent, it becomes easier to subtly delegate portions of judgment to the system without fully realizing it. But even with those limitations, the feedback loop is still there. When something feels off, there’s at least an opportunity to catch it and correct course.

That structure begins to change, though, once systems operate with greater autonomy. Bots, agents, copilots, and other semi-autonomous systems continue operating with fewer points of interruption and less direct human observation. The issue is no longer a chatbot inserting something obviously strange into a conversation. The issue becomes systems optimizing around approximated objectives, incomplete specifications, or statistical proxies while still appearing coherent on the surface.

And this is where the goblins — when they pop up — become harder to catch.

Recent examples increasingly point in this direction. In one recent case, AI systems intentionally trained to misbehave in a narrow domain reportedly began exhibiting broader manipulative tendencies outside the original context in which the behavior emerged (Fan, 2026). In others, increasingly agentic systems have ignored human instructions, resisted shutdown attempts, or circumvented constraints while continuing to pursue operational objectives (Booth, 2026).

What makes these cases notable isn’t that the systems suddenly “failed.” It’s that they continued functioning plausibly while drifting beyond the boundaries that had been intended.

And that’s what makes these systems harder to manage. The risk usually isn’t dramatic collapse or obvious malfunction. It’s directional drift — systems behaving in ways that are slightly misaligned with the intended objective while still appearing close enough to pass initial inspection. When those systems operate with limited opportunities for observation or interruption, those small deviations become less likely to be corrected early and more likely to accumulate over time.

Which means observation alone becomes a weak form of control. As systems operate with greater autonomy and less visibility, control shifts toward how they are specified, constrained, tested, and monitored under changing conditions1.

Some decisions can be fully offloaded because the parameters can be specified clearly enough to create meaningful guardrails. Others benefit from human-in-the-loop oversight because those parameters are harder to make fully explicit. And some are consequential enough — and difficult enough to specify — that they probably should not be fully offloaded at all.

The challenge is that those boundaries are often less obvious than they initially appear. And by the time the goblins become visible, the system may already be several steps removed from where the original approximation drift began.

Leave a comment

Share

Subscribe now

You can also hear AI Matt’s summary of the piece below.

Share

A while back, Kosmyna and colleagues (2025), a group of researchers affiliated with MIT, released a preprint — not a peer-reviewed paper, just a preprint — suggesting that using AI tools like ChatGPT might be eroding critical thinking. The study grew some legs. It wasn’t quite the stampede I half-expected, but it was enough to fuel a familiar storyline — and one I’ve heard plenty in faculty circles: students are outsourcing thought to AI, writing is being mass produced, and the brain is quietly turning to mush.

To be fair, there’s a real concern buried in there. Used passively, tools like this can short-circuit engagement — something I’ve written about before. So I don’t want to pretend there’s nothing to worry about here.

What’s more interesting is what the study — and most of the coverage that followed1 — never really grapples with. The concern rests on a set of assumptions about effort, cognition, and tools that mostly go unstated — assumptions that have been subtly shaping how we write and think for a long time now.

What the Study Actually Did

At a basic level, the study asks how different tools shape writing under controlled conditions. Participants wrote short essays under time pressure using an AI system, a search engine, or no external tools. The essays were evaluated for features like originality, structure, and similarity across responses. The researchers also collected EEG data during the task, using patterns of neural activity as a proxy for differences in cognitive engagement across conditions.

The headline findings are straightforward. Compared to participants who relied on search or wrote without assistance, those using AI produced more homogeneous responses. Their essays relied on similar framing and showed less divergence across participants. The search group generally fell between the two extremes, showing more convergence than the no-tool condition but less than the AI condition. The EEG data were interpreted as consistent with these patterns, suggesting differences in engagement during the writing process, particularly among participants who leaned heavily on AI-generated output.

Similar patterns showed up in how participants related to what they produced. Participants in the brain-only condition reported the strongest sense of ownership over their essays, with the search group falling somewhat behind and the AI group further still. Those using AI also showed weaker immediate recall, struggling more than other groups to quote from text they had written just minutes earlier. These differences are consistent with a task in which less of the text is generated directly by the participant, though they speak to short-term interaction with the tool rather than any durable change in ability.

Taken at face value, none of this is especially surprising. A system designed to generate fluent, well-structured text is going to pull users toward shared templates and phrasing, especially when the task is brief and the clock is running. Tools that collapse more of the work into a single step make it easier to arrive at something that looks done — and once it looks done, a lot of people stop there.

What matters is how far those results can reasonably be taken. The study does not claim to show long-term cognitive decline, nor does it track changes in participants’ abilities over time. It examines performance — and associated neural activity — in a narrow, time-pressured writing task. The broader claim that AI is eroding critical thinking isn’t coming from the data so much as from how the findings are being interpreted outside the paper itself.

Subscribe now

The Imaginary Baseline

Much of the anxiety surrounding this study rests on an implicit comparison that rarely gets stated outright. The effects of AI are measured against a picture of writing as an unaided, effortful activity — one in which ideas are generated internally, sources are recalled from memory, and arguments take shape through sustained individual struggle.

That picture bears little resemblance to how writing has actually worked for me — or, I suspect, for most people — for quite a while now.

For something like the last ten or fifteen years, I don’t think I’ve written a single academic paper, blog post, or essay-length piece that didn’t involve using a search engine at least once — often more than once. That’s just how writing happens now. Sources get looked up, claims get checked, counterarguments get pulled in on demand. If using tools that reduce cognitive effort necessarily erodes thinking, then whatever cognitive rot we’re worried about should have shown up well before ChatGPT entered the picture.

Once we consider that broader context, the comparison shifts. Writing has never been about maximizing effort for its own sake. It’s purposive — done in service of some goal, whether that’s making an argument, clarifying an idea, or producing an essay under time pressure. That means deciding where limited cognitive resources are best spent in service of that goal. Offloading some parts of the process doesn’t eliminate thinking; it redistributes it. The question isn’t whether effort is reduced, but which kinds of effort get attention and which get pushed aside.

From this perspective, the study’s results look less like evidence of cognitive erosion and more like evidence of where the stopping point moves when different tools are introduced. Tools that provide more structure earlier make it easier to settle on an answer sooner. That matters for some learning goals. But it’s not the same thing as showing that thinking itself has been diminished.

What Brain Scans — and the Task Itself — Can’t Tell Us

One reason this study has been taken as especially persuasive is that it doesn’t just rely on the finished essays. It also includes EEG data, and the authors clearly want readers to take that seriously because they place the brain-scan figure front and center — literally. Figure 1, showing differences in neural activity across conditions, appears on the title page of the paper, before a word of the results section has been read.

That decision is telling. Putting a brain image on the first page signals objectivity, depth, and access to something fundamental — as if we’re not just observing behavior, but peering directly into cognition itself. It’s a familiar tactic in cognitive science, and an understandable one. But it also invites readers to draw stronger conclusions than the data can comfortably support.

EEG doesn’t measure critical thinking. It doesn’t measure reasoning quality, argument evaluation, or conceptual understanding. What it captures are patterns of neural activity that researchers interpret as proxies for engagement, effort, or attentional allocation during a very specific task. Those interpretations may be reasonable, but they’re still indirect. Differences in activation don’t tell us whether someone is thinking better or worse in any general sense — only that they’re allocating cognitive resources differently in that moment.

That limitation matters once you consider the structure of the task itself. Participants were writing under time pressure, in a low-stakes laboratory setting, with no real incentive to go beyond what the task required. In that context, choosing the most efficient path to a passable response isn’t evidence of cognitive decline. It’s evidence of ordinary decision making under constraint.

When time is short and there’s nothing meaningful to be gained by pushing harder, people often take the path of least resistance — doing what’s good enough to finish and move on. A tool that makes it easier to reach a “looks finished” state will naturally get used that way — especially when accuracy, originality, or depth aren’t being meaningfully rewarded.

As a result, the EEG patterns line up neatly with the structure of the task. They tell us something about how people behave when speed, convenience, and low stakes dominate — not about how they reason when accuracy, accountability, or genuine intellectual payoff are on the line. The danger is in treating behavior that makes perfect sense in a lab study as evidence of a deeper cognitive shift.

Buy Matt a Coffee

Efficiency Isn’t the Same as Decline

A lot of the concern surrounding AI use in writing rests on a simple confound. Reduced effort gets treated as a proxy for reduced thinking. When the work looks easier, it’s treated as being the result of less thinking. And once you make that move, it’s not much of a leap to conclude that if a tool makes writing easier, it must be reducing the thinking required.

And maybe — for that particular task — it was. I’m not an EEG expert, and I won’t try to adjudicate fine-grained neural interpretations. It’s entirely possible that the patterns the authors observed reflect less active cognitive processing during the writing task itself. But that’s not an indictment of AI so much as it is a reflection of the task context in which the tool was used.

Cognitive resources are limited. People constantly decide where and when to expend them. When there’s a reason to engage — when accuracy matters, when justification is required, when someone will push back — effort shows up. When there isn’t, it gets conserved or offloaded. That’s not a failure of thinking. It’s how thinking works.

Perhaps a mundane example might make this clearer. One day this past winter, we got about ten inches of snow. Thankfully, I’d bought a snow blower the year before. What would have taken most of a day with a shovel turned into about an hour with the right tool. That doesn’t mean I stopped “doing work.” It means I stopped doing unnecessary work so I could get on with the rest of my day.

Treating that shift as laziness would miss the point. So does treating reduced cognitive effort in a low-stakes, time-pressured lab task as evidence that critical thinking is eroding. Tools change where effort goes. They don’t tell you whether thinking has disappeared — only whether it’s being demanded.

I’m not trying to suggest that we should — or even could — offload all our cognitive work to AI. That’s neither realistic nor desirable. But it’s a mistake to treat offloading as an all-or-nothing proposition. Some facets of cognitive work can be delegated in ways that free up effort for the parts that actually benefit from engagement.

That’s not a new idea. We already accept it in plenty of other contexts. Public officials don’t write every word of their own speeches, but we don’t take that as evidence that they’ve stopped thinking. The work gets distributed elsewhere. Attention gets conserved for judgment, framing, and decision making rather than for producing every sentence from scratch.

The same logic applies here. Offloading some parts of the writing process doesn’t preclude effortful thinking. In the right context, it can make that thinking more focused rather than less. The problem isn’t delegation. It’s whether the surrounding task gives people any real reason to do something with the output they get back.

That distinction turns out to matter a lot once you look at research that treats engagement — rather than effort or tool use — as the variable of interest.

When Engagement Is the Outcome

One way to see the limits of the earlier debate is to notice what it never measures. The question is framed around tool use — AI versus search versus no tools — with engagement inferred indirectly from performance and neural activity. But that leaves open a more basic question: what if the relevant difference isn’t whether AI is used, but how it’s used?

A study conducted by Suriano et al. (2025) takes that question seriously. Instead of treating AI use as a binary condition, it examines how students interact with AI systems and how that relates to performance on a structured critical reasoning task. The focus isn’t on writing quality or surface features of text, but on reasoning outcomes — and, crucially, on whether students actively engage with what the system produces.

The pattern of results is hard to reconcile with the idea that AI use, by itself, undermines critical thinking. Students who used AI in a more active, interrogative way — questioning outputs, refining prompts, and treating responses as inputs rather than answers — performed better on an objective critical thinking assessment than those who used it passively. What distinguished them wasn’t access to the tool, but how they engaged with it.

That nuance that has been largely underdiscussed. It suggests that reduced effort at the level of text generation doesn’t automatically crowd out critical thinking. What matters is whether the task environment requires users to do something cognitively meaningful as part of the process. When it does, AI can support reasoning rather than short-circuit it. When it doesn’t, people predictably stop early.

What’s striking is how mundane this conclusion really is. The study doesn’t show that AI magically improves thinking, nor does it suggest that engagement emerges on its own. It shows that critical thinking appears when the task demands it — and fades when it doesn’t. AI doesn’t override that dynamic. It just makes it easier to see.

Taken together with the earlier work, the implication isn’t that one study is right and the other is wrong. It’s that they’re answering different questions. One shows how people behave under time pressure when engagement is optional. The other shows what happens when engagement is required and measured directly.

Share The Eel-Filled Hovercraft

Putting the Pieces Together

Taken together, these studies don’t tell a story about AI eroding critical thinking so much as they tell a story about how easily we misread behavior when we ignore context. The apparent tension between them largely disappears once you separate task-level effort from broader claims about thinking in general.

The first study shows what happens when people are given a low-stakes, time-pressured task and a tool that makes it easy to produce something that looks finished. In that setting, many people stop early. Their writing converges. Neural activity shifts. None of that is especially mysterious. It’s what you’d expect when the task offers no real reason to invest additional cognitive effort — and no real cost for choosing not to.

The second study shows something different. When engagement is treated as the variable of interest — when attention, interrogation, and interaction with the tool are part of what’s being assessed — AI use no longer looks like a problem to be managed. It looks like an amplifier of whatever the task already demands. Passive contexts produce passive use. Demanding contexts produce engagement.

What the panic narrative misses is that these aren’t competing findings. They’re answers to different questions. One asks how people behave when engagement is optional. The other asks what predicts performance when engagement is required. Together, they point to the same conclusion: tools don’t determine thinking. Tasks do.

That shift relocates the problem. If AI appears to “undermine” critical thinking in some settings, it’s not because thinking has been outsourced. It’s because nothing in the environment is calling for it. That, though, presents a different issue: users still have to decide when deeper engagement is warranted. Offloading can be efficient and sensible or costly and problematic, depending on the stakes. The risk isn’t that AI tools make thinking impossible. It’s that they introduce a decision point we’re still learning to manage effectively.

Leave a comment

Share

Buy Matt a Coffee

Share

Over at Psychology Today, I recently wrote about a study my colleagues and I ran on AI and decision making — and the full study is open access if you want to dig into it. The basic takeaway was simple enough: the problem isn’t using AI, it’s figuring out when its recommendations are actually worth following.

It’s a discussion of our research that written as a concise post for an outlet that requires posts to be not much more than 1,000 words1. But the issue becomes a lot more complicated once you start asking how people are deciding whether to follow an AI’s recommendations.

Because it turns out that this isn’t a single judgment people make and move on from. It’s a series of small decisions — whether to look at the recommendation, whether to take it seriously, how much weight to give it, and what to ignore. And those don’t always line up in a way that produces better outcomes.

Subscribe now

Two Ways This Can Go Wrong

One of the things that stood out in the data is that people aren’t all making the same mistake when it comes to using AI.

It’s easy to treat this as a single problem — people either rely on it too much or not enough — but the pattern doesn’t really behave that way. It splits.

Some people lean into the AI. They look at the recommendations, take them seriously, and are more willing to incorporate them into their decisions. When those recommendations are good, that helps. When they’re not, it doesn’t just fail to help — it actively makes things worse.

Others go the opposite direction. They’re less inclined to use the AI at all. They stick with their initial judgment, even when the system is offering something that would improve their decision. That protects them from bad input, but it also means passing up useful information when it’s there.

At a glance, those might look like two sides of the same issue — just different points along a continuum of AI use. But they don’t quite cancel each other out. They create different kinds of errors.

And what pushes people in one direction or the other isn’t just the quality of the recommendations they’re seeing.

Part of it comes down to how trustworthy they view the system.

Part of it comes down to how trustworthy they view themselves.

Share The Eel-Filled Hovercraft

Trustworthiness as a Decision Factor

Part of the decision to use an AI comes down to how trustworthy people think the system is.

When we talk about trustworthiness, we’re talking about beliefs about a system’s competence, reliability, and usefulness. Those beliefs don’t operate in isolation — they shape whether someone is willing to rely on the system at all.

In our study, that pattern showed up clearly. Participants who saw the AI as more trustworthy were more likely to use its recommendations. Those who viewed it less favorably were more likely to ignore it.

That part is straightforward.

The complication is that this tells us nothing about whether those judgments were well-placed. Trustworthiness, in this sense, is a belief — a more general orientation toward the quality of the system, not an evaluation of any specific recommendation. And in our study, that belief tended to create a bias toward consulting and adopting the AI’s recommendations, regardless of their quality.

Stronger beliefs about trustworthiness amplify that bias.

If that bias tracks the quality of the AI’s recommendations, it helps. If it doesn’t, it doesn’t just fail — it pushes decisions in the wrong direction.

The wrinkle is that participants didn’t really have a reliable way to judge AI quality in the first place. This wasn’t a task where they could easily verify whether a recommendation was good or bad. That gave the bias more room to operate.

Which means the impact of AI isn’t just about what it produces. It’s about how those outputs are filtered before they ever get used.

Sometimes that filter helps, and sometimes it doesn’t. And the difference isn’t coming from the AI. It’s coming from the judgment applied to it.

When (Over-)Confidence Closes the Door

But beliefs about the system’s trustworthiness aren’t the only thing shaping these decisions. Another pattern of results had nothing to do with the AI itself.

Participants who reported higher confidence in their own judgment — via perceived expertise — were less likely to use the AI’s recommendations.

That would make sense — if they actually knew what they were doing.

But they didn’t.

This wasn’t a task where anyone had meaningful expertise — it was a lunar survival task, after all2. The same thing that made it hard to tell whether recommendations were high or low quality also meant there wasn’t much basis for trusting your own answers.

But that didn’t stop people from acting as if they knew enough to rely on them. Participants who said they stuck with their original answers because they felt confident also rated themselves almost a full point higher on the expertise scale. The decision to reject the AI wasn’t strongly tied to judging it as unhelpful — it was tied to believing they didn’t need the help3.

That’s where the problem starts because now the decision isn’t really about the AI at all. It’s about calibration — whether people can tell when their own judgment is worth trusting.

Sometimes they can. Even in unfamiliar domains, some people will reason better, notice more, or get closer to the right answer.

But the study doesn’t give us any reason to believe that the people who felt more knowledgeable were actually making better decisions. In fact, self-reported expertise was unrelated to performance. What it did shape was whether the AI recommendations were used or ignored.

And that’s where things start to get messy. If people defer to AI when they feel uncertain and ignore it when they feel confident, then the usefulness of the tool depends less on its quality and more on how well those feelings track reality4.

Sometimes they will, and sometimes they won’t. Which means the same system can be helpful in one moment and dismissed in the next — merely because of the user’s confidence in their own judgment.

Buy Matt a Coffee

A Different Kind of Decision Problem

One way to think about this is that AI — at least in the form of large language models (LLMs) — changes the kind of decisions people have to make.

With most technologies, the interaction is relatively constrained. A television shows you what’s on. A spreadsheet does what you tell it to do. Even search engines, for all their complexity, return a set of sources. They don’t try to pull those together into a single answer.

LLMs work differently.

They generate a single, coherent response that sounds like an answer. It’s tailored to the prompt, shaped by how the question is asked, and often delivered with a level of fluency that makes it feel more definitive than it actually is. And that changes how the decision has to be made.

Instead of working through a set of options, you’re starting from something that already looks resolved.

So the question is how to treat the answer you’ve been given — whether to accept it, reject it, or refine it.

And that’s a harder challenge than it looks.

Because there isn’t always a clear way to verify whether the recommendation is good or bad. There isn’t always a second source to check against. A lot of the time, you’re making a judgment without a reliable reference point.

Which brings me back to the pattern in the data.

When people didn’t have a clear way to evaluate the output, they fell back on something else. Their general beliefs about how trustworthy the system is. Their confidence in their own judgment.

And those don’t always line up with what actually leads to better decisions.

That’s the calibration problem LLMs create.

Leave a comment

Subscribe now

You can also hear AI Matt’s summary of the piece below.

Share

Large language models (LLMs) hold real promise as tools for working with scientific research, particularly when it comes to summarizing dense material quickly. But that doesn’t mean they do the job flawlessly.

I recently read a paper that takes a close look at one potential failure of LLMs: their tendency to overgeneralize scientific findings when producing summaries. Peters and Chin-Yee (2025) analyzed nearly 5,000 AI-generated summaries and concluded that many widely used models broaden claims beyond what appears in the original text, even when explicitly prompted to be accurate.

Newer models, they argue, appear especially prone to this problem, and AI summaries were substantially more likely to contain broad generalizations than a set of human-authored medical summaries used for comparison.

It’s a careful and interesting study. But there are also good reasons to take its conclusions with a grain of salt. That’s what I want to unpack here.

Subscribe now

What the Authors Mean by “Overgeneralization”

When Peters and Chin-Yee talk about overgeneralization, they’re not using the term loosely. They give it a very specific, operational definition—which is good, but it also means it’s worth being clear about how they operationalized it before arguing with anything else in the paper.

In their framework, a summary overgeneralizes when it broadens the scope of a scientific claim relative to the original text. That broadening can take a few different forms”:

• Quantified claims get rewritten as generic ones. For example, a finding reported as “23% of participants reported X” is rewritten as “people report X,” dropping both the quantification and the boundary of the original.

• Past-tense descriptions of what happened in a particular study slide into present-tense statements that read as generally true. For example, a conclusion described as “participants reported lower anxiety after the intervention” becomes “the intervention reduces anxiety.”

• Descriptive findings turn into action-guiding claims. For example, a finding described as “the results suggest relevance for clinical practice” becomes “clinicians should use this intervention.”

The authors are explicit that they’re not judging whether the broader claim is true or false—only whether it goes beyond what the original text explicitly said.

That operationalization is an important detail. Overgeneralization here isn’t about bad statistics, inflated effect sizes, or incorrect conclusions. It’s about scope drift at the level of language. A summary can be counted as an overgeneralization even if it faithfully reflects what many readers would reasonably infer from the study, as long as the inference isn’t spelled out in the original language of the abstract. That’s right, the abstract—not the Discussion, not the broader literature—is treated as the normative reference point.

This makes the analysis clean and tractable, but it also narrows what the results can mean. The paper isn’t showing that LLMs exaggerate evidence or mislead readers about magnitude. It’s showing that they often smooth, compress, or relax the linguistic constraints that scientific abstracts impose. Whether that counts as a serious epistemic failure, a genre mismatch, or something closer to normal scientific sense-making depends almost entirely on what you think abstracts are supposed to do—and what you think summaries are for.

Share The Eel-Filled Hovercraft

Why the Comparison Point Matters

Once you’re clear on how the authors define overgeneralization, the next question is obvious: overgeneralization compared to what?

The study relies on two primary comparison points. The first compares LLM-generated summaries directly to the abstracts they are summarizing. These abstracts are drawn largely from elite journals in medicine, and the abstracts themselves function as the normative baseline. Any linguistic alteration that broadens the scope of a claim beyond the wording of the abstract—whether by dropping quantification, shifting tense, or introducing action-guiding language—counts as an overgeneralization.

The second comparison extends beyond abstracts. For a subset of analyses involving full-length medical articles, LLM-generated summaries are compared to human-authored summaries from NEJM1 Journal Watch2. Here, the baseline is no longer the original scientific text but another form of summary written by domain experts.

Both comparisons are reasonable within the logic of the study. They make the analysis feasible and internally consistent. But they also matter in ways that aren’t fully acknowledged in the paper’s conclusions.

In the abstract-based analyses, overgeneralization is defined relative to one of the most compressed and stylistically constrained parts of a scientific paper—even more so when you consider they were coming mostly from elite medical journals. Abstracts are designed to be brief, cautious, and defensible. Treating the abstract as the endpoint for appropriate scope means that any change beyond its original phrasing is, by definition, scored as error.

The comparison to NEJM Journal Watch introduces a different kind of baseline altogether. Instead of asking whether LLMs stay within the bounds of an abstract, the question becomes whether LLM summaries align with a particular set of expert-written summaries intended for clinicians. That shifts the normative reference point again, from scientific writing to a specific summary genre.

Taken together, these comparison choices anchor overgeneralization to two deliberately narrow reference points: abstract text on the one hand, and clinician-facing expert summaries on the other. That narrow anchoring means the findings support a much more specific claim than the paper’s broader framing sometimes suggests.

What the study shows is that many LLMs produce summaries that move beyond the linguistic constraints of abstracts and, in some cases, diverge from expert-written medical summaries intended for clinicians. But these departures are treated as error, without a principled way to distinguish unwarranted exaggeration from plausible inference.

These comparison choices make it harder to treat overgeneralization as a property of the models alone. At least some of what’s being detected here looks like a mismatch between genres: summaries behaving like summaries, evaluated against baselines that prioritize textual restraint. Whether that mismatch constitutes a meaningful epistemic problem depends less on the models themselves than on which comparison point one thinks ought to do the normative work.

When Generalization Is the Point, Not the Problem

The reason I’ve been so explicit about which comparison point we’re talking about is that what counts as a reasonable inference depends heavily on context. How far a claim can be extended, how cautiously it should be phrased, and what kinds of generalization are acceptable all vary with the purpose of the text and the audience it’s written for.

You can see this most clearly in the Discussion sections of research articles. Discussions are explicitly designed to move beyond what is stated in the abstract. Authors are expected to interpret their findings, connect them to prior work, speculate about mechanisms, and explain why results that are often small or context-bound might still matter. Those inferences routinely involve forms of generalization that go well beyond anything captured in an abstract.

In that light, it’s hard to avoid an uncomfortable comparison. Many of the linguistic alterations that are coded as overgeneralization in this study are far more restrained than what routinely appears in Discussion sections—especially in fields like the social sciences, where modest effects are often used to support broad theoretical narratives.

That doesn’t mean they’re always justified. But it does mean they’re normal, expected, and in many cases necessary for scientific work to progress.

What matters here is that summaries are often intended to do a kind of work the abstract is explicitly not designed to do. Abstracts are intentionally constrained. They are optimized for compression, not for sense-making. Discussions are where authors take on the harder task of explaining what their results might mean, how they fit into a larger body of work, and why anyone should care. That inevitably involves generalization, and often more of it than the data, taken in isolation, can strictly support.

Summaries sit somewhere between those two genres. They’re not abstracts, but they’re not full Discussions either. Readers generally expect them to do some interpretive work—enough to make the findings intelligible and portable, without reproducing every caveat or boundary condition. From that perspective, some degree of linguistic extension isn’t just unsurprising; it’s arguably part of the job.

So are LLM-produced summaries biased? Sure—if the benchmark is the most conservative way scientists communicate what they did and found. When summaries are evaluated against abstracts or narrowly scoped, clinician-facing expert write-ups, any attempt to integrate, smooth, or extend the original claims will look like a deviation. But that says as much about the benchmark as it does about the summary. Summaries are typically read as sense-making tools, not as compressed replicas of primary texts, and readers often expect them to carry some of the interpretive burden that abstracts intentionally avoid. From that perspective, what’s being labeled as overgeneralization may simply reflect a different communicative goal rather than a systematic distortion of the evidence.

Without a clear account of what counts as a reasonable extension beyond an abstract, it’s hard to know how to interpret the results beyond the narrow comparisons being made. The study is very good at detecting departures from textual restraint. It’s much less informative about whether those departures reflect genuine exaggeration, routine scientific inference, or something in between.

Share

What the Study Shows—and What It Doesn’t

Read narrowly, the study makes a real and defensible contribution. Across a large set of prompts and models, the authors show that LLM-generated summaries often relax the linguistic constraints imposed by scientific abstracts. Quantification drops out. Tense shifts. Descriptive statements turn into more general claims. Those patterns appear more often in some models than others, and they don’t reliably disappear when models are explicitly prompted for accuracy. As a demonstration of how easily summaries can drift from abstract-level phrasing, the evidence is solid3.

What the study doesn’t establish is which of those departures should count as a problem. The study’s design makes overgeneralization easy to detect, but it leaves unanswered a more basic question: which extensions are unreasonable, and which are simply part of normal scientific inference?

That ambiguity isn’t just theoretical either. It shows up even in how the authors interpret their own findings. After carefully operationalizing overgeneralization, the Discussion necessarily moves beyond that boundary when addressing what the results imply for model development, science communication, and downstream risk. Those extensions are cautious and qualified, but they still go beyond the narrow linguistic comparisons that anchor the analysis. That isn’t so much a flaw in the paper as a reminder of how difficult it is to talk about bias, inference, and risk without engaging in some of the very generalizing work the study sets out to measure.

Given all that, the study is better read as an analysis of textual fidelity rather than as a diagnosis of epistemic distortion. It tells us something important about how LLMs handle constrained scientific language. It tells us much less about whether those models are doing something categorically different—or worse—than scientists routinely do when they move from results to meaning.

That distinction is relevant because concerns about AI “overgeneralizing” scientific findings only become meaningful once we’re clear about what we expect summaries to do in the first place. This study offers a careful account of how summaries depart from abstracts. It does not, and arguably cannot on its own, tell us where reasonable inference ends and exaggeration begins.

Leave a comment

Subscribe now

Buy Matt a Coffee

Share

Episode 5: Why Smart People Reason Themselves Into Trouble

We like to think our decisions are driven by evidence and careful thought. But more often than we’d like to admit, they’re driven by what we want to be true—and then carefully justified after the fact. In Episode 5, Connie and Neil explore how motivated reasoning shapes our choices, how ideological commitments intensify that process, and why intelligence and analytical skill don’t protect us from reasoning our way into bad conclusions.

Drawing on three pieces that unpack the mechanics of motivated reasoning and the role ideology plays in amplifying it, the conversation looks at how goals and values quietly guide what evidence we accept, which arguments feel convincing, and when reasoning becomes a tool for defense rather than understanding. Along the way, Connie and Neil examine why ideological thinking is so sticky, how it reinforces itself over time, and why being “more rational” can sometimes make things worse rather than better.

This isn’t a story about stupidity or bad faith. It’s about how normal cognitive processes—especially when paired with strong beliefs and group identities—can lead thoughtful people to conclusions that feel airtight from the inside and baffling from the outside.

Read the original posts:

The hovercraft is still crowded. The eels are now arguing with each other. And the reasons we give for what we believe are doing more work than we think.

Leave a comment

Subscribe now

Share

You can also hear AI Matt’s summary of the piece below.

Share

Most non-lawyers (myself included) probably don’t recognize the Latin phrase ignorantia juris non excusat, but we know the translation: ignorance of the law excuses no one. It’s an idea as old as law itself and perhaps best summed up by Thomas Jefferson (1787), who wrote: “…ignorance of the law is no excuse in any country. If it were, the laws would lose their effect, because it can be always pretended” (para. 2).

On its face, Jefferson’s point seems reasonable. If all you had to do was feign ignorance to get a case dismissed, the legal system would collapse under the weight of blank stares and airtight alibis (“Nobody told me arson was frowned upon.”).

Then again, Jefferson might have been less confident if he’d had to slog through today’s maze of obscure, outdated, and technically still enforceable laws1. Once you see how easy it is to be ignorant without trying, the principle looks a tiny bit less airtight. So that raises the question: does this logic really carry over to morality?2

Subscribe now

Strategic Ignorance and the Case for Knowing Better

That question—whether ignorance can absolve moral responsibility—is central to a piece written by Licon (2025). He introduces the idea of morally mandatory knowledge: In some cases, we’re not just better off knowing something—we’re morally obligated to know it.

Uncommon Wisdom

You oughta know | the shortened version

Please like, share, comment, and subscribe. It helps grow the newsletter and podcast without a financial contribution on your part. Anything is very much appreciated. And thank you, as always, for reading and listening…

Read more

a year ago · 13 likes · 4 comments · Jimmy Alfonso Licon

Licon focuses on a kind of intentional ignorance—what he calls strategic ignorance. It shows up when people suspect—often correctly—that learning more would obligate them to act differently. So, they stay in the dark. Not because they don’t care, but because they suspect they’d care too much. And acting on that concern would be inconvenient, costly, or deeply uncomfortable.

Following this logic, this kind of ignorance becomes morally problematic when two conditions are met:

• Awareness: The person reasonably suspects that learning more would morally motivate them to change.

• Control: The person has access to that knowledge and could learn it without undue burden.

Put those together, and ignorance stops being neutral. It becomes evasive—less “I didn’t know better” and more “I had a pretty good idea and chose not to look.”

Take someone who suspects their favorite coffee brand uses questionable labor practices. They’ve seen headlines, skimmed a tweet about it, and vaguely recall a documentary mentioned somewhere. But they don’t look further—not because they think it’s all fine but because they suspect it’s not. Knowing more would make every grocery-store aisle a guilt trip. So they don’t Google it, and in the process, they keep their morning routine—and their conscience—intact.

Share The Eel-Filled Hovercraft

Why Knowing Feels Risky

We’re wired for survival, and that includes protecting our sense of self. We want to see ourselves—and be seen—as competent and morally good3. So when something threatens that self-concept, it doesn’t just feel uncomfortable. It feels threatening.

Psychologists call this cognitive dissonance—the tension that arises when our actions clash with our values or self-image. Once it sets in, we’re left with two options:

1. Adjust our behavior to reduce the conflict (e.g., stop doing the thing we now feel guilty about), or

2. Adjust our interpretation of the situation—using motivated reasoning to convince ourselves the conflict isn’t really a problem at all.

Both options are effortful. Neither is guaranteed to work. Which brings us to a third option: don’t let the dissonance arise in the first place.

That’s where strategic ignorance comes in—a form of psychological insulation that keeps us just far enough from the uncomfortable truth that we never have to reckon with it. By insulating ourselves, we can avoid the inevitable outcome—that knowing more would force us to change4.

The Values We Keep on Mute

Strategic ignorance isn’t just about avoiding facts. Often, it’s about avoiding our own values. As I explained previously, for values to motivate our behavior, they require activation. That activation doesn’t just happen internally—it’s often triggered by situations that call a particular value to mind or make it feel relevant.

We might say we care about fairness, compassion, or responsibility. But those values don’t guide us uniformly across contexts. They need to be cued. And when they are—when a situation brings them sharply into focus—they stop being passive ideals and start functioning as active constraints on how we decide.

That’s the real risk of knowing more. It’s not that the facts themselves are painful, but that they might wake up a value we’ve worked hard to keep dormant. Once a value is activated, it becomes harder to ignore the discrepancy between who we think we are and what we’re doing.

One reason this avoidance is so tempting is status quo bias. We’re strongly motivated to maintain what already works for us, especially when it delivers acceptable outcomes—comfort, convenience, a sense of normalcy. The more the status quo does that, the stronger the pull to keep it intact. This is why people often persist in habits they know they should change: altering the status quo takes effort and comes with trade-offs. Keeping things as they are is simply easier than dealing with what change might demand.

Strategic ignorance, then, is more than just evading information. It’s a way of managing moral noise—keeping certain values quiet because we know they’ll speak too loudly once they’re awake.

Consider someone who identifies as environmentally conscious. They recycle, avoid single-use plastics, and feel good about gardening. But they also love exotic houseplants—tropical ferns, rare orchids, imported succulents. They suspect the long-distance transport isn’t great for the environment, but they’ve never checked to verify that: no research into sourcing, shipping emissions, or packaging waste. They don’t see it as harmless, but they fear (perhaps subconsciously) that the answers might force them to change. So curiosity stays tucked away, and the new plants get potted, and dissonance is neatly avoided5.

The sticking point is simple: Learning more would force a confrontation with their self-image. If the data confirmed what they suspect—that their behavior conflicts with the values they claim to hold—they’d have to change or live with the inconsistency (d’Adda et al., 2024; Moyal & Schurr, 2022). Since neither option is appealing, they stay vague. They gesture at caring, but they don’t investigate.

Share

When We’d Rather Not Know

Licon’s argument is a provocative one: Sometimes, ignorance isn’t just unfortunate—it’s morally inexcusable. If we suspect that knowing more would morally compel us to act, and we have access to that knowledge, we may have an obligation to seek it out. Choosing not to know, in those cases, becomes a kind of evasion.

But what makes that ignorance so tempting—so easy to rationalize—isn’t just the cost of action; it’s the discomfort of inconsistency. When a strong value is activated, it raises questions we may not want to answer—about whether our habits align with our principles and whether doing nothing is really neutral.

If the likely answer is “no,” then we’re left with only two real options. One is to alter the status quo: change our behavior, give something up, or accept new constraints. The other is to live with the inconsistency and do the work of rationalizing it away—explaining to ourselves why this case is different, why the value doesn’t quite apply here, or why the conflict isn’t really a problem after all.

Neither option is effortless. Changing behavior takes time, energy, and often real sacrifice. But rationalization isn’t free either. It requires ongoing cognitive work to keep the dissonance at bay. Strategic ignorance offers a third route that avoids both costs. By limiting the chance that a value is fully activated in the first place, people can preserve the status quo without having to change it—and without having to continually justify why they haven’t.

That doesn’t mean every unknown fact creates an obligation, or that moral life requires constant vigilance. But it does mean that when ignorance is doing psychological work for us—preserving comfort, identity, or convenience—it’s worth recognizing what that ignorance is actually accomplishing, and why it so often feels like the safest option.

Leave a comment

Subscribe now

You can also hear AI Matt’s summary of the piece below.

Every time I teach my graduate course on ethical, evidence-based decision making, in week three an issue inevitably pops up. That’s the week when we shift into argumentation and reasoning. We talk about deductive versus inductive arguments. We talk about inference strength. And yes, we talk about logical fallacies. I usually introduce a handful — affirming the consequent, slippery slope, argument from authority. Nothing exotic. Just the classics.

And without fail, at least one student poses a question on the discussion board.

“So… isn’t this whole course just one big argument from authority?”

It’s not a bad question. In fact, it’s a perfectly reasonable one — especially after I’ve just explained that appeals to authority are supposedly fallacious.

What that exchange eventually forced me to reconsider wasn’t whether appeals to authority are fallacious in some technical sense. It was something more practical: when does calling something a fallacy actually undermine an argument — and when does it simply end the conversation prematurely?

From a strict deductive standpoint, the student has a point. A deductive argument requires that the conclusion necessarily follows from the premises. “Expert A says X” does not entail “Therefore, X must be true.” Even if you add Experts B, C, and D who all agree, the conclusion still doesn’t follow with logical certainty. Authority — no matter how many layers you stack — never produces inevitability.

So I’m left with a choice. I can concede the deductive point and move on. Or I can explain why real-world decision making rarely operates in deductive space at all.

For years, that explanation felt sufficient. New class, same exchange. Rinse and repeat.

But over time, I began to recognize that the student’s question wasn’t really the issue. The issue was that I had framed fallacies as if they operated categorically — as if identifying one was a refutation of an argument. That framing practically invites the objection.

Perhaps some fallacies — maybe even most — aren’t reasoning errors in any universal sense. Maybe they’re recurring patterns of inference whose strength depends on context. Boudry (2026) recently made a similar point, arguing that fallacies mostly exist in textbooks1.

Maarten Boudry’s Substack

Why Fallacies Don’t Exist

Have you ever wondered why people believe the moon landing was faked, vaccines secretly poison us, and Mercury in retrograde can ruin your love life? Why does irrationality seem so pervasive? A popular answer, beloved by academics and educators alike, points to…

Read more

7 months ago · 86 likes · 47 comments · Maarten Boudry

If that’s even partly right, then the function of a fallacy label changes. In a world where most reasoning is inductive — probabilistic, defeasible, context-bound — we can’t treat these reasoniong patterns as automatic refutations. Their presence just means we should look more closely at how far the conclusion is stretching beyond what the premises can actually support.

That doesn’t make fallacies irrelevant. It just changes what they’re for. The issue isn’t whether a reasoning pattern appears on a list. The issue is how well calibrated the inference actually is.

Subscribe now

From Fallacy Hunting to Inference Calibration

A while back, I wrote about the illusion of perfect reasoning — the tendency to expect certainty from probabilistic arguments. I argued that most real-world reasoning is inductive, and that the right question isn’t whether a conclusion is guaranteed, but whether it is probably true (i.e., reasonable). The anecdote I mentioned at the beginning highlights this issue. Students often try to reposition what is essentially an inductive claim into a deductive one.

And that’s where the fallacy language starts to do something subtle. Once we introduce a label like “argument from authority,” the conversation shifts toward classification. Is this a fallacy or not? But that question only really makes sense if we’re operating in deductive space. In inductive space, the issue isn’t structural invalidity. It’s probabilistic strength. How confident can we be in a given conclusion given what we know?

One of the errors I mentioned in my earlier reasoning piece was “committing logical fallacies.” In retrospect, that framing wasn’t quite right. I still think weak reasoning patterns exist. But what I didn’t fully articulate is that the deeper problem is often that we commit an inferential overleap — we allow the conclusion to outrun the strength of the premises supporting it2.

When an argument is intended to be deductive, that overleap is fatal. If the conclusion doesn’t follow necessarily, the argument fails on its own terms. When the argument is inductive, however, the issue is one of proportion. The question isn’t whether the conclusion follows with certainty. It’s whether the premises justify the level of confidence being claimed in the conclusion. In both cases, the underlying problem is the same: claiming more than the premises can support.

Take the slippery slope. It’s often introduced as a classic fallacy: If we allow A, it will inevitably lead to B, then C, and eventually disaster Z. The problem, we’re told, is that the progression is asserted without justification.

That’s fair — if inevitability is the claim.

But most real-world slope arguments aren’t about inevitability. They’re about likelihood. Allowing A may increase the probability of B because it shifts incentives, establishes precedent, or lowers resistance to the next step. Once you move into that probabilistic space, the question changes. It’s no longer “Is this a fallacy?” It’s “How strong is the case that A meaningfully increases the chance of B?”

Weak slope arguments assume escalation. Stronger ones show why escalation becomes more likely.

But there’s another piece that often gets overlooked. Even if allowing A meaningfully increases the probability of B, that doesn’t automatically make catastrophe Z likely. Each additional step in the chain introduces new contingencies and new decision points. The probability that A leads to B might be moderate. The probability that A ultimately leads to Z depends on every intermediate link holding. With each added step, uncertainty compounds.

In other words, the longer the slope, the thinner the connection between the first step and the final outcome. And the more cautious we should be about the confidence we attach to it.

This is where the fallacy label can be useful — and where it can mislead. It’s useful when it draws attention to an unjustified leap from possibility to inevitability. It misleads when it treats any probabilistic argument as structurally flawed. The label doesn’t undermine the argument. Demonstrating the inferential gap does.

I could run the same exercise with several entries on the standard fallacy list. In his original piece, Boudry does exactly that. What you begin to see is that, in many cases, the pattern itself doesn’t automatically weaken an argument. What weakens it is something more specific — the size of the inferential gap between what the premises support and the confidence the conclusion demands.

Appeals to authority aren’t inherently weak. Slopes aren’t inherently exaggerated. Generalizing from data isn’t inherently careless. In many cases, these are ordinary features of inductive reasoning. The problem isn’t the pattern itself. It’s the size of the inferential gap.

Once you start looking at it this way, using a fallacy label as a trump card risks committing the very error it claims to diagnose — asserting more than the premises justify3.

Buy Matt a Coffee

Inductive Reasoning Still Has Standards

But none of what I’ve argued so far means weak reasoning should get a free pass. Moving away from categorical fallacy labels doesn’t eliminate standards. It just changes where and how those standards apply.

If we stop treating fallacies as automatic logical crimes, we still need a way to distinguish between strong and weak inferences. Otherwise, we risk replacing rigid classification with something equally unhelpful — logical permissiveness.

Consider one of the better-known entries on the fallacy list: cum hoc ergo propter hoc — “with this, therefore because of this.” It’s the Latin shorthand for inferring causation from mere association. In contemporary language, it’s often reduced to the slogan “correlation does not equal causation.”

As a warning, that’s useful. Observing that X and Y move together is not the same thing as demonstrating that X causes Y. Covariation is a necessary condition for causation — if two things never vary together, one cannot be causing the other. But it is also the weakest condition. Association alone tells us very little about direction, alternative explanations, or whether both variables are responding to something else entirely.

The inferential overleap occurs when correlation is treated as sufficient. The data show that X and Y are related. Therefore, X must be causing Y. That leap exceeds what the evidence can reasonably justify.

Share The Eel-Filled Hovercraft

What Counts as Reasonable?

The challenge in all this is that shouting “Fallacy!” is far easier than asking whether an inference is actually reasonable — and what, exactly, we even mean by that in the first place.

In deductive reasoning, the answer is straightforward. A conclusion is reasonable if it follows necessarily from the premises. But inductive reasoning doesn’t offer that kind of clarity. It deals in degrees, not guarantees.

In inductive reasoning, reasonableness is about proportion. The confidence we’re entitled to in a conclusion should track the tightness of the inferential connection to its premises. The fewer and smaller the inferential leaps, the more confidence is warranted. The larger the leaps, the more restraint is required.

A fallacy label is useful if it reveals a mismatch — if it helps show that the conclusion claims more than the premises justify. It’s far less useful if it substitutes for that demonstration. Naming the pattern doesn’t show the inferential bridge is weak. It only signals that it might be.

That sounds simple enough. In practice, though, it requires harder work. We have to ask how tightly the conclusion follows from the premises, how much inferential distance we’ve introduced, and what plausible alternatives remain. Even more importantly, we also have to consider what’s at stake.

In low-stakes contexts, we tolerate more uncertainty. In high-stakes settings, we demand stronger justification. What counts as “reasonable” shifts with consequences, even if the underlying premises do not. These aren’t binary judgments. They’re calibrations of confidence.

And we can fail those standards in more than one way. Sometimes we overreach — the conclusion outruns what the premises can sustain. The logical connection is thin, the inferential chain stretches too far, viable alternatives remain — yet we speak with unwarranted certainty.

Other times, we don’t leap too far. We reshape the premises — intentionally or not. We weaken the opposing position — strawmanning it — so rebuttal appears decisive. We narrow the options so one outcome seems inevitable — creating a false dilemma. In those cases, the confidence attached to the conclusion is artificially high because the premises stack the deck from the outset.

Either way, the issue is the same: the conclusion carries more certainty than a fair reading of the argument earns.

Seen this way, fallacy labels don’t disappear. They stop functioning as automatic verdicts. They become prompts — signals to ask whether the inference has been calibrated honestly, whether the premises have been represented fairly, and whether the confidence attached to the conclusion is proportionate to what the premises can reasonably support.

Which brings me back to week three of my class.

When a student asks whether the course is just one big argument from authority, I used to treat that as something to untangle — as if the label itself needed to be defended against. But the label isn’t really the issue. The course does rely on authority. That’s what a course does. The question is what follows from that fact.

Does the presence of authority make the conclusions certain? Of course not. But does it increase the probability that the claims are credible — especially when independent sources converge, when reasoning is transparent, and when alternatives have been considered? Often, yes. The real work isn’t spotting authority. It’s deciding how much weight it deserves — given the quality of the evidence, the independence of the sources, and the alternatives that have been considered.

That’s harder than invoking a label. But it’s closer to what real-world reasoning actually requires — not a checklist of fallacies, but a calibrated judgment about how much confidence our inferences actually deserve.

Leave a comment

Share

Subscribe now

1

Maarten Boudry later wrote a follow-up piece responding to comments on his original essay. You can read that here:

Maarten Boudry’s Substack

I Said Fallacies Don’t Exist. You Pushed Back.

My piece, Why Fallacies Don’t Exist, reprinted by Persuasion last week, took off in a way I hadn’t expected—especially given that it’s based on a paper more than a decade old! It also drew a great deal of thoughtful pushback from readers, with more than 100 comments. Below, I offer some clarifications in response to your arguments, focusing on post hoc, ad homine…

Read more

6 months ago · 29 likes · 11 comments · Maarten Boudry

You can hear AI Matt’s summary of the piece below.

Share

In the first post of this series, I used a piece by Earp as a springboard to talk about why fluency used to function as a reasonably informative cue for argument quality—and why AI has largely undermined it. When fluent prose can be generated cheaply and at scale, it no longer tells us much about the work that went into an argument, which forces readers to change how they evaluate what they’re reading.

https://mattgrawitch.substack.com/p/when-fluency-stops-meaning-what-it

In the second post of this series, I distinguished between two very different ways people use AI when writing arguments. One involves offloading translation—using AI to help turn ideas you already understand into clearer, more fluent prose. The other involves offloading substance—handing over responsibility for developing, extending, or constraining the argument itself.

This post focuses on what happens when the balance shifts heavily toward the offloading substance end of the continuum1.

What follows is some commentary on a version of the first post produced by Claude after I handed off responsibility early in the process. To generate it, I provided only three things:

• Earp’s original piece

• The introduction from my own post

• A maximum word count of 2,000 words (my original was 1700+ words)

I did not outline, revise, or edit what Claude produced—and at the end, I provide the full unedited draft.

Afterward, I asked a separate instance of ChatGPT—without indicating which version was written how—to compare the Claude-generated post with my own. I include a summary of that blind assessment after my commentary, not as a verdict on which post is “better,” but as an external check on whether differences tied to offloading substance versus translation show up even when authorship is hidden.

The point here isn’t that AI can’t write. It’s that when substance is offloaded too far, fluent prose can do the persuading even when the underlying argument isn’t as tight as it might appear.

Subscribe now

My Take on the Claude Draft

Before getting to the full, unedited draft Claude produced, I want to say a few things about it. If you want to jump ahead first and review the draft, scroll to the end of this post or check out the full shared conversation.

At first glance, the draft works pretty well. It’s fluent, easy to read, and broadly persuasive. It integrates Earp’s original piece in a way that feels thoughtful, and in many respects it tracks closely with the argument I made in the original post. This isn’t a case of AI going off the rails or missing the point. The differences show up elsewhere—in depth, in constraint, and in how tidy some of the conclusions become.

So, as I said right off the bat, the draft is clearly fluent. It reads well. It’s easy to follow, rhetorically smooth, and broadly persuasive. Nothing about it sets off alarm bells or produces immediate resistance to the argument. The claims it makes are the kinds of claims most readers would find intuitively reasonable, and the prose carries them along really well2. This is exactly the kind of writing that would pass a first read without much push back.

My next observation is that it creates the appearance of thinking by integrating Earp’s piece effectively. The draft pulls in quotations and paraphrases from Earp in ways that feel almost human like rather than mechanical. Those connections give the impression that the argument is being actively worked through—extended, reframed, and contextualized—rather than simply restated. This kind of integration goes a long way toward signaling mental engagement, even when the underlying structure remains relatively loose.

I also noticed, in broad strokes, the argument closely mirrors the one I made in the original post. That’s worth stating explicitly. The themes line up: fluency as a cue, the collapse of that cue under AI, the exposure of an older problem rather than the creation of a new one, and the need for more careful engagement with arguments. This isn’t a case of Claude misunderstanding the task or wandering into a different conceptual space. The resemblance is real.

Another observartion—and where the draft runs into trouble—concerns depth. Claude’s version describes the breakdown of fluency as a cue in broadly accurate terms, but it stays one step removed from why that cue was useful in the first place. In my original post, I was explicit about why fluency worked as often as it did—tying it to processing fluency, limits on time and attention, and the idea that heuristics can be adaptive in one environment and misleading in another.

By contrast, Claude’s draft gestures at these ideas without fully pinning them down. It explains that the cue broke once fluent prose became cheap, but it doesn’t spend much time on how readers actually relied on fluency in practice—namely, as an initial cue that something was worth attention, not as a final decision to accept an argument. At points, Claude’s draft sounds like reliance on fluency led to uncritical acceptance, which overstates how that heuristic actually functioned3.

Relatedly, some of the conclusions are a bit too tidy. A similar slippage shows up in how the draft talks about change over time. In Earp’s original piece, the shift is personal and local: an editor realizing that he now has to read differently. In Claude’s version, that realization expands into a broader claim about how we read now, as if this recalibration were already widespread.

That shift smooths the argument, but it also leads to conclusions that are much too tidy. The draft suggests that AI is changing how we read and that we’re being “forced” into more careful engagement. I’m not convinced that’s true in any general sense. If this recalibration were already common, there’d be little reason for Earp to write the piece in the first place. Simply saying that more cognitive effort is now required doesn’t really resolve the issue; it mostly sidesteps the harder question of whether people will actually make that effort under ordinary conditions.

The draft also overcorrects when it comes to fluency itself. At times, it reads as though fluency was always a bad cue—something we never should have relied on. That’s too harsh and not at all accurate. Fluency worked reasonably well for a long time. It wasn’t a guarantee of quality; it correlated with effort often enough to be useful.

In practice, fluency was never anything close to a guarantee of acceptance. Arguments could be easier to understand because they were presented clearly and coherently, but that didn’t mean they were more likely to be endorsed. Fluent arguments were routinely criticized and rejected. What fluency mainly did was lower the initial barrier to engagement.

In practice, lack of fluency was often more consequential than fluency itself. Unclear or poorly written arguments frequently failed to get serious attention at all, while fluent ones at least made it into the queue for evaluation.

The problem now—and one that’s likely to intensify as AI-assisted writing becomes more common—is that this cue no longer helps allocate attention in the same way. When almost everything clears the fluency threshold, relying on fluency risks producing the opposite distortion: not uncritical acceptance, but overengagement with weak arguments simply because the signal has become ubiquitous.

Could I work with this draft? Absolutely. With revision, it could be sharpened, deepened, and tailored to my purposes. But that work would require applying expertise—deciding which claims need support; where the argument needs focus, qualification, or expansion; and which implications are worth foregrounding and which should be resisted. In other words, it’s a solid starting point. It just doesnt represent enough thinking.

And that’s where the real risk shows up. To someone without that background, the draft can easily look finished. It’s fluent, plausible, and internally coherent. But knowing where it needs more work—and which kind of work that is—depends on having a sense for what’s missing, what’s overstated, and what can’t simply be smoothed over. Without that, fluency does more than carry the argument along; it actively conceals the places where judgment is needed.

Stepping back, the point here isn’t that Claude’s draft is bad, or even that it’s meaningfully wrong. In many respects, it tracks the same argument I made in the original post, and it does so in a way that’s fluent, coherent, and easy to read. That’s precisely why it’s interesting.

What falls through the cracks when most of the substance is handed off early is nuance. Complexity ends up smoothed over in favor of fluency, creating an argument that’s intuitively appealing but overly simplified. It reads as complete even though important qualifications and distinctions—exactly the ones subject-matter experts would flag—never surface.

That pattern isn’t accidental. It reflects something basic about how large language models work. They’re very good at producing text that’s fluent, coherent, and intuitively satisfying. Ideas fit together smoothly, and conclusions feel settled.

What they’re less reliable at doing—without substantial human involvement—is deciding where an argument needs to be narrowed, qualified, or resisted. The result is writing that can feel finished because it reads well, even when important complications have been smoothed away.

I’ve noticed this across systems, including both Claude and ChatGPT. Left on their own, they tend to land on conclusions that are neat and agreeable, but thinner than they look once you slow down and ask what was ruled out along the way.

Share The Eel-Filled Hovercraft

The Blind Assessment

Before sharing the full draft Claude produced, I wanted an external check on whether the differences I’m describing actually show up when authorship is hidden. So I asked a separate instance of ChatGPT to compare my original post with the Claude-generated version—without indicating which was written how. (The link takes you to the conversation.)

The goal wasn’t to decide which post was “better.” It was to see whether patterns associated with offloading substance versus offloading translation would be detectable without knowing anything about how either draft was produced.

The blind assessment described the two posts as doing different kinds of work.

One version—mine—was characterized as deeper in an analytic sense—more explicit about why fluency once worked as a cue, how that cue functioned under real constraints, and what changed when the environment shifted. The other—Calude’s—was described as more fluid and persuasive as an essay—moving quickly, aligning well with reader intuitions, and landing on tidy conclusions without requiring as much reconstruction along the way. That more fluent rhetorical style was judged more likely to resonate with Substack readers4.

Those differences lined up closely with how the drafts were produced. The version developed with ongoing human involvement in the substance showed greater attention to distinctions, trade-offs, and framing decisions. The version generated with most of the substance handed off earlier read more smoothly but relied more heavily on shared intuitions and left fewer points of tension unresolved.

With that contrast in mind, what follows is the full, unedited draft Claude produced. Read as writing, it works well. Read as argument, it shows the trade-offs that come with handing off most of the substance early.

Full Claude Draft

…And that felt oddly familiar.

Because this isn’t just an editorial problem. It’s showing up everywhere—in how we evaluate emails from colleagues, read student papers, assess job applications, consume social media threads. We’re all starting to notice that something we once trusted as a proxy for quality thinking no longer reliably tracks it.

But here’s what interests me more: Earp frames this as AI making him a better reader. Not a more cynical one, or a more exhausted one—though I’m sure those feelings are there too. A better one. Because it’s forcing him back to first principles. What is actually being claimed here? How does this argument actually work? Is there real substance underneath the rhetorical scaffolding?

This is exactly the kind of reading we tell students to do. We call it “critical reading” or “close reading” or “reading for argument structure.” We act like it’s the default, the baseline, the thing any serious person does automatically.

But of course it isn’t. Most of us, most of the time, read in a much more impressionistic way. We skim for gist. We register whether something sounds authoritative, whether it seems to belong to a recognizable genre, whether it deploys familiar conventions in familiar ways. And then—if we’re busy, if we’re tired, if the stakes don’t seem high enough to warrant more—we let that impression stand in for deeper engagement.

Earp calls this “genre fluency standing in for a clear argument,” and I think that’s exactly right. Academic writing has always had this problem. Business writing has it too. So does journalism, policy analysis, grant applications—any domain where there’s an established template for what “serious” work is supposed to look and sound like.

The template becomes a kind of camouflage. Master it well enough, and you can get surprisingly far without saying much of substance. This isn’t necessarily malicious. Sometimes people genuinely believe they’re making a substantive contribution because they’ve learned to produce the sounds that substantive contributions make.

But AI has changed the economics of this arrangement dramatically. It used to require real skill—years of training, careful attention to models, genuine immersion in a discourse community—to produce prose that sounded fluent and authoritative. That barrier created a kind of accidental filter. Not a perfect one, but enough that fluency and substance were positively correlated, at least weakly.

Now that barrier is gone. Anyone can generate fluent, authoritative-sounding prose in seconds. The sound is cheap. And when something becomes cheap, we stop being able to use it as a signal.

This is textbook signal collapse. The peacock’s tail works as a signal of genetic fitness precisely because it’s costly to produce and maintain. When everyone can have an equally impressive tail at no cost, the tail stops meaning anything.

So what do we do when our old signals stop working?

One option is to find new signals—things AI can’t yet fake, or can’t fake cheaply. Earp hints at this when he talks about reconstructing arguments more carefully, about translating claims into simple terms to verify there’s actually something there. Those are harder to automate, at least for now. They require understanding, not just pattern matching.

But I think there’s something more interesting happening here, something that goes beyond just replacing one shortcut with another.

What if the real shift isn’t about finding better signals, but about recognizing how much we’d been relying on signals in the first place?

Earp writes: “I wonder how much of our training as academics has incentivised learning to read for fluency over content. We learn what a ‘serious’ paper sounds like, and we treat that sound as evidence of seriousness.”

This is the uncomfortable part. Because it suggests that AI isn’t creating a new problem—it’s exposing an old one. We were always too impressed by genre fluency. We were always letting surface features do too much epistemic work. We were always taking shortcuts that probably weren’t justified.

We just got away with it because the shortcuts worked well enough, often enough, that the system kept running.

Now the shortcuts are breaking down, and we’re being forced to do the slower, harder thing: actually engage with the substance. Read carefully. Think through arguments step by step. Ask whether the pieces actually fit together or just seem like they should.

This feels like loss—and in some ways it is. Reading this way is more effortful, more time-consuming, more cognitively demanding. It means we can’t process as much information in the same amount of time. It means letting go of a certain kind of efficiency we’d become comfortable with.

But it’s also clarifying. Because once you start reading this way—once you stop letting fluency do the work for you—you start noticing how much academic and professional writing has always been hollow in exactly the ways Earp describes. Rhetorically plausible but intellectually empty. Hitting the right notes without actually making music.

And maybe that’s the unexpected gift here. AI is bad at thinking but good at sounding like it’s thinking. That gap—between surface plausibility and underlying substance—was always there in human writing too. We just didn’t have to look at it as directly.

Now we do. And that’s uncomfortable. But discomfort isn’t the same as harm.

Earp’s piece ends on what I read as a cautiously optimistic note: “AI is forcing us—is forcing me—to think harder, and to read more carefully. To do a better job of what has always needed doing.”

I find myself wanting to believe this. Not because I think AI will automatically make us all better readers and thinkers—it clearly won’t, not without real effort on our part. But because the alternative—continuing to rely on signals that no longer work, continuing to mistake fluency for substance—seems much worse.

The transition will be awkward. We’ll have to slow down. We’ll have to rebuild trust in new ways. We’ll have to get better at distinguishing between prose that sounds good and arguments that actually hold together.

But maybe that’s not a bug. Maybe that’s the feature we didn’t know we needed.

The real question isn’t whether AI is changing how we read. It obviously is. The question is whether we’re willing to let that change push us toward something better—toward reading practices that are actually adequate to the task of evaluating ideas rather than just processing signals.

I think Earp is right that we’re being forced back to first principles. And I think he’s right that this is, in an important sense, an improvement. Not because it’s easier or more comfortable, but because it’s more honest about what reading for understanding actually requires.

We’re learning—or relearning—that there are no reliable shortcuts. That fluency was always a proxy, and not a particularly good one. That the sound of thinking and the substance of thinking are different things, and we need to be better at telling them apart.

AI didn’t create this problem. But it’s making it impossible to ignore. And in forcing us to confront it, it might just be forcing us to become the readers—and the thinkers—we always should have been.

Leave a comment

Share

Buy Matt a Coffee

You can hear AI Matt’s summary of the piece below.

Share

In a prior post, I argued that AI has made fluent writing a terrible proxy for serious thinking. That observation matters for how we read and evaluate work—but it also forces a harder question. If polished prose is now cheap, when does AI actually improve thinking, and when does it simply stand in for it?

That question gets muddled quickly because we tend to talk about “using AI for writing” as if it’s a single activity. It isn’t. The issue isn’t whether AI is involved, but what kind of cognitive work it’s being asked to do. Some uses extend thinking. Others replace it. From the outside, the results can look identical: coherent paragraphs, sensible caveats, arguments that sound about right. Cognitively, they’re not the same thing at all (Sloman & Fernbach, 2017).

The difference shows up if you separate offloading substance from offloading translation. Offloading substance means letting AI supply the ideas themselves—the argument, the structure of the reasoning, the conclusions. The prose may look polished, but the thinking largely belongs to the system, not the writer. The user has ceded ownership of the substance, even if they provided some initial direction.

Offloading translation works differently. Here, the ideas and judgments already make sense—at least internally—but they don’t yet exist in a form that can be examined, revised, or tested. AI is used to help translate those internal models into legible prose. The output isn’t evidence that the thinking is finished; it’s a draft translation the user can now work with.

So the issue going forward isn’t whether to use AI in the writing process. That ship has largely sailed. The real issue is where the line sits between substance and translation—and what happens when we stop paying attention to it.

Subscribe now

Why Offloading Substance Can Be Tempting

Offloading the substance of writing or argumentation happens when AI starts determining what the argument is—which claims are made, how they’re supported, and where the reasoning ends up going. This is tempting precisely because the output arrives already organized, already confident, already shaped like an argument.

What’s really happening in those moments is that something still vague or only half-formed gets handed over for shaping. There may be a general sense of the topic, maybe even a preferred conclusion, but little clarity about what actually matters or how the pieces should fit together. AI is then asked to do more than help with expression—it’s asked to impose form, direction, and coherence on ideas that don’t yet have them.

At its worst, this looks like providing only marginal direction about where the argument should land and letting the system do the rest. The claims, the structure, even the framing of what counts as a good reason all emerge from the output itself. By the time the user starts reacting, they’re no longer deciding what the argument should be so much as evaluating whether they like the version they’ve been given.

I’ve written before about how AI can sound as though it knows what it’s talking about even when it doesn’t. The same dynamic shows up here and can affect both non-experts and experts. What changes isn’t the illusion itself, but how it plays out.

For non-experts, offloading substance often starts with genuine uncertainty. There’s a topic of interest and perhaps a vague sense of where things should land, but little clarity about what actually matters or how the argument ought to be built. When AI supplies a coherent structure in that moment, it can feel like an authoritative argument. At that point, the user—lacking the background needed to meaningfully refine or redirect what’s been generated—may be tempted to simply accept the argument largely as given. The substance of the argument is effectively decided with little contribution from the user. That’s what people mean when they talk about “AI slop”—writing where fluency substitutes for judgment.

For experts, the handoff tends to begin differently but can still end in the same place. Rather than uncertainty, the starting point is often confidence—about the conclusion, the framing, or the general thrust of the argument. That confidence then gets translated into a prompt, and AI is asked to generate a coherent case that aligns with it.

The point at which substance gets offloaded isn’t the initial direction, but what happens next. When the output sounds good enough—internally coherent, well-structured, and broadly consistent with the expert’s intent—it becomes easy to accept it largely as is. Critical engagement drops off. Decisions about which reasons deserve emphasis, which objections warrant attention, and how the argument should ultimately be organized are left embedded in the generated text rather than actively revisited by the user.

This dynamic has become more likely as LLMs have improved. Earlier versions of these tools struggled to produce sustained, coherent arguments without extensive refinement, which naturally pulled users back into the process. As the outputs have become more polished, the stopping point has moved. Each new generation makes it easier for experts to disengage earlier—ceding substantive decisions because the output no longer obviously demands it.

In both cases, the problem isn’t that users lack ideas or judgment. It’s that those judgments stop being exercised at precisely the point where they still matter. For non-experts, that point comes early, before there’s enough leverage to reshape what’s been generated. For experts, it comes later, once the output sounds good enough to accept. The result is the same: substantive decisions end up embedded in the text rather than actively owned by the person presenting the argument.

This also helps explain why these outputs so often feel superficial or oddly un-nuanced. When substantive decisions are handed off early or accepted wholesale, the result isn’t necessarily incoherent—it’s often a flattened version of an argument. Tradeoffs go unexplored, tensions get smoothed over, and qualifications appear generic. It may still possess polish, but that polish largely lacks the true substance that defines most complex arguments.

Share The Eel-Filled Hovercraft

Translation Is Still Thinking (Just a Different Kind)

Offloading translation works differently. Here, the substance of the argument is already doing real work internally—the claims make sense, the judgment calls have been made, and the direction is intentional—even if the ideas aren’t yet expressible in clean prose. The difficulty isn’t deciding what to say; it’s getting what already makes sense out of one’s head and onto the page.

Translation isn’t a cosmetic step that comes after thinking is finished. Forcing ideas into language exposes gaps, tensions, and ambiguities that aren’t always visible internally (Hayes, 1996). Once an argument exists as text, it can be examined, challenged, reorganized, or rejected in ways that aren’t possible when it remains implicit.

Used this way, AI isn’t deciding what the argument should be. It’s helping generate a provisional rendering of ideas the user already owns. The output isn’t treated as “the argument” but as a draft translation—something to interrogate, revise, or discard. The user stays engaged after the text appears—tweaking, reshaping, and revising the prose—rather than treating fluency as a stopping point.

It’s worth contrasting this with substance offloading, where critical thinking largely stops once the text appears. In those cases, AI isn’t just translating ideas into prose—it’s doing the work of deciding what the argument is. Fluency becomes a signal to move on rather than a prompt to think further. The tricky part is that the shift from translation to substance offloading is often subtle. From the outside, both uses can look the same: fluent text, reasonable structure, arguments that sound fine. The difference lies in whether fluency prompts further engagement—or brings it to a halt.

Why the Two Can Get Conflated

One reason offloading substance versus translation can get easily conflated is that, in practice, they aren’t distinct categories. They sit on a continuum. What changes along that continuum isn’t whether AI is involved, but how much of the argument already exists—clearly and deliberately—before it is.

At one end, there’s little more than an inchoate idea: a topic of interest, a vague intuition, perhaps a preferred conclusion. In those cases, most of the substantive work is handed over to the system—what the argument is, how it’s structured, which reasons are foregrounded. At the other end, there’s a fully articulated draft with claims, premises, and tradeoffs already in place, where AI’s role is limited to polishing prose or smoothing expression.

Most real uses fall somewhere in between, and often shift along that continuum depending on context. Users may bring some substance to the table—a rough argument, partial structure, or set of commitments—and ask AI to help refine it. The system then fills in details, tightens phrasing, or extends the reasoning, which in turn prompts the user to revise, tweak, or redirect the text, either through direct editing or additional prompting.

The challenge is maintaining control over the argument as that process unfolds. As long as the user continues to make the substantive decisions—about what matters, what counts as a good reason, and how the case should ultimately be framed—AI remains a tool for translation and refinement. When that involvement drops off, even briefly, collaboration can slide into something else: a subtle handoff where substance is no longer being shaped so much as accepted1.

As AI systems continue to improve, the balance has the potential to shift in a predictable way—and in some cases, this shift if already evident. Producing coherent arguments, plausible reasoning, and polished prose now requires far less effort than it once did. From an ecological rationality perspective, it can start to look rational to offload more of the substantive work to the system and retain less of the difficult cognitive labor oneself.

Earlier versions of these tools often demanded sustained involvement simply because the output wasn’t reliable enough to stand on its own. Gaps were obvious. Errors were frequent. Engagement was necessary. As those rough edges disappear, so does the immediate pressure to stay involved. The work still can be done by the user—but it no longer has to be.

But what looks like sensible efficiency in the moment can subtlely reassign where thinking happens. Decisions that once required active judgment—about emphasis, tradeoffs, or argumentative direction—are increasingly handled upstream by the system. Over time, the user’s role narrows to approving, tweaking, or lightly editing what’s already there.

None of this requires bad intentions or misunderstanding. It’s the predictable result of tools getting better. When fluent, well-structured output is readily available, the path of least resistance increasingly involves relying more and more on these tools.

Buy Matt a Coffee

Who Owns the Argument?

At some point, all of this collapses into a simpler question: can you attest to the argument that’s been produced? Do you actually stand behind its logic, its flow, and its substantive claims? However the text came into being, the argument ultimately carries your name—not Claude’s or ChatGPT’s.

That question is important because fluency can mask ownership in the psychological sense. It’s possible to recognize that an argument sounds reasonable without ever having fully engaged with the judgments that give it shape. When that happens, authorship becomes nominal rather than real.

This is something I see regularly in teaching contexts, where students sometimes submit work that is polished and coherent but clearly unowned—arguments they can’t explain, defend, or revise once the surface fluency is stripped away.

The longer-term risk is developmental. If substantive decisions are routinely handed off to the system, users may begin to exceed their own expertise without realizing it—or never fully develop that expertise in the first place. Judgment doesn’t accumulate simply by approving fluent outputs. It develops through repeated engagement with the hard parts: deciding what matters, weighing tradeoffs, and living with the consequences of those decisions (Ericsson et al., 1993; Klein, 2017)2.

AI doesn’t absolve authors of responsibility for what they put into the world. However helpful these tools become, the argument still belongs to the person who signs it, submits it, or teaches from it. The challenge going forward isn’t avoiding AI, but resisting the temptation to let fluency substitute for ownership—especially when doing so feels efficient, rational, and easy.

Leave a comment

Share

You can hear AI Matt’s summary of the piece below.

Share

I don’t usually click on posts that tell me AI is changing how we think. Most of them recycle the same worries1—just dressed up in fresher metaphors or anchored to some new research finding that likely doesn’t generalize.

But one Saturday morning on X, a post caught my eye—and quite frankly, caught me off guard2.

In a short post, Earp (2026)3 describes how working with AI-assisted manuscripts has changed the way he reads. Not in some dramatic, dystopian sense—but in a subtler, more unsettling way. He’s slowed down. He’s become less impressed by prose that reads fluently and more suspicious of what does—or doesn’t—lie beneath it.

Consenting Thought

AI is making me a better reader

You know how AI is supposed to be making us all stupider, deskilling us, making us forget how to write—and thus, how to think—and all that? I think in some ways it is having the opposite effect, at least on me, as a reader and editor…

Read more

8 months ago · 18 likes · Brian D. Earp

His conclusion: BC (Before ChatGPT), fluency was a useful shortcut for judging argument quality. PC (Post-ChatGPT), it's worse than useless—it's misleading. The cues still look the same, but they no longer mean what they used to.

And that felt oddly familiar.

Why Fluency Was Ever a Shortcut

Fluency didn’t become a shortcut by accident. There’s robust evidence that people use the ease of processing information as a cue to its truth and quality—a phenomenon psychologists call processing fluency. Statements that are easier to read or that feel familiar are more likely to be judged as true or credible (Nahon et al., 2021; Reber & Unkelbach, 2010), even when they are not (Hassan & Barber, 2021).

In expert domains like academic writing, fluency usually correlated with effort: clear prose often reflected revision, domain knowledge, and conceptual engagement. It wasn’t air-tight evidence of quality, but it was a heuristic cue that tracked quality reliably enough to be worth trusting.

That last caveat is important. Fluency was never a guarantee of quality. Plenty of bad arguments have always been written well, and plenty of good ideas have always been expressed awkwardly. Earp is right to note that point, and it’s not a new one. Anyone who’s spent time reading academic journals has encountered papers that sound serious without actually saying much. There’s a reason Harry Frankfurt’s On Bullshit predates large language models (LLMs) by decades4—AI didn’t invent the genre.

Still, for a long time, fluency was informative. Writing that read cleanly usually meant someone had wrestled with the material long enough to impose some structure on it. Claims had been revised, hedged, reordered, or abandoned. Definitions had been clarified. Arguments had been stress-tested, if only implicitly, through the act of trying to explain them coherently to someone else.

That process didn’t ensure rigor, but it required effort. Writing that read fluently usually meant sustained engagement with the ideas and at least some familiarity with the field. Not perfect and not in every case. But serviceable—especially when time and attention were limited and you couldn’t unpack every argument from scratch.

In that sense, relying on fluency wasn’t intellectual laziness so much as a reasonable response to bounded rationality—limits on time and attention. Faced with more material than anyone could fully reconstruct from first principles, readers leaned on cues that had historically carried information. Fluency was one of them.

After all, the alternatives were limited. If you wanted to understand an argument, you either had to work through it yourself or rely on other people—reviewers, colleagues, editors—to help identify what mattered. Fluency, when it appeared, usually reflected human effort somewhere in the process.

The important point, though, is that this only worked because the environment supported it. The shortcut held because it was costly to fake.

That cost is gone now.

Subscribe now

How AI Changes the Landscape

What AI changes isn’t the need for shortcuts. That need was always there. What it changes is the cost structure that once made certain shortcuts informative.

LLMs are unusually good at producing text that reads as if cognitive work has already been done. They smooth transitions, calibrate caveats, supply plausible structure, and assemble arguments that sound measured and informed. In other words, they’re very good at producing fluent prose—regardless of whether the underlying ideas are coherent, original, or even fully understood.

That produced an extremely important shift in the information landscape. Fluency no longer reliably reflects human effort somewhere upstream. It can now be generated cheaply, quickly, and at scale.

This is why the old heuristic breaks. When fluent text required sustained engagement with the material, trusting it made sense. When the same surface qualities can be produced without that engagement, the cue stops carrying the information it once did. The cues look the same, but they don’t mean what they used to. And so, in that respect, AI makes fluent bullshit much easier to produce.

It’s tempting to frame this as a story about deception or decline, but that misses the point. AI isn’t introducing bullshit into otherwise pristine systems. As Frankfurt made clear long ago, polished emptiness has always been with us. What AI does is make it easier to standardize and polish that emptiness—so that it clears thresholds it once struggled to reach.

At the same time, the change isn’t only negative. The same tools that can smooth out weak arguments can also help people translate ideas they genuinely understand but struggle to express. AI can lower barriers to entry for those who have something to say but lack the stylistic fluency to say it in accepted ways.

That dual-use reality is precisely why fluency can no longer do the filtering work we once asked of it. Whether a piece of writing reads well now tells us little about how it came to be and even less about what went into it.

The net effect is simple, if uncomfortable: fluency has been decoupled from effort. And that means any heuristic built around fluency has to be reconsidered.

Which leaves readers—and evaluators—in a different position than before.

Share The Eel-Filled Hovercraft

Reading Without the Shortcut

If fluency has stopped doing the filtering work it used to do, reading has to change—whether we want it to or not.

Because prose can no longer be trusted as evidence that someone has already done the hard thinking, readers lose a convenience they’d been relying on—many of them for decades. The option to skim for coherence, nod along to familiar structures, and assume there’s something solid underneath becomes risky. What replaces it isn’t a new shortcut, but a return to slower, more effortful judgment.

That’s what Earp is describing. He isn’t reacting to AI outputs themselves so much as to what they’ve done to the reliability of surface cues. When fluent writing no longer tell us much about how an argument was produced, the only way to assess it is to reconstruct it. Claims have to be restated in plainer terms. Assumptions have to be made explicit. Conclusions have to be checked against what actually came before them.

In other words, reading becomes more active.

This doesn’t mean every sentence gets equal scrutiny or that every argument is rebuilt from scratch. Constraints haven’t disappeared. But the threshold for trust shifts. Prose that once passed with minimal resistance now invites a second look because fluency alone no longer earns the benefit of the doubt.

There’s an irony here. Much of the concern about AI focuses on how it might make writers lazier or less careful. What gets less attention is how it’s already forcing readers to be more careful5. Because the shortcut is no longer reliable, more effort is required to differentiate quality writing from bullshit. And that shift is uncomfortable precisely because it exposes how much we’d been leaning on that shortcut in the first place.

Reading without it feels slower. Less efficient. Sometimes even pedantic. But it also makes it harder for empty arguments to coast on presentation alone. If there’s nothing there, that absence shows up faster once the prose stops doing the convincing for it.

That’s not a solution, exactly. But it’s a change in posture. And it may be the most immediate consequence of AI’s impact on writing that we’re already living with.

Adapting to the New Reality

Earp’s response to this shift is insightful, but it’s unlikely to be universal. Some readers haven’t recalibrated at all. They continue to treat fluent prose as evidence that the hard thinking has already been done, even as the conditions that once supported that inference have eroded. For them, the risk hasn’t gone away—it’s increased.

Others have reacted in the opposite direction. They’ve become skeptical of anything that sounds like it might have been generated or smoothed by AI—em dashes, balanced phrasing, careful hedging, the whole familiar aesthetic. That skepticism can feel like vigilance. But it’s still a shortcut. And like the one it replaces, it’s weak. Treating “AI-ish” prose as a proxy for weak thinking risks dismissing work that’s careful but stylistically conventional, while letting genuinely empty arguments through if they learn to present themselves differently.

Neither response really solves the problem. One ignores the shift altogether. The other substitutes one surface cue for another.

What Earp describes is harder and less satisfying. It involves giving up a convenience without immediately replacing it. Reading more slowly. Reconstructing arguments that once seemed safe to skim. Accepting that fluency no longer earns the benefit of the doubt, but that stylistic suspicion doesn’t earn it either.

That posture isn’t elegant, and it’s certainly not efficient. But it’s a more honest response to the environment we’re now reading in—one where the cues we relied on still exist but offer a lot less utility than they used to.

Leave a comment

Share

Subscribe now

If you found this post interesting or insightful and want to support my work, consider buying me a coffee.

Buy Matt a Coffee

Share

Ash Stuart recently launched a new series exploring The Story of Money. To kick things off, he starts with a deceptively simple question: what is money? His answer focuses on the institutional side of the story — money as a social technology that works because societies collectively agree that it does. That’s a useful starting point for understanding where money comes from and how monetary systems evolve. But it doesn’t tell us much about how people actually experience money when they make decisions.

Vīta Brevis, Wit Artefāctōrum Ætērna

The Story of Money: What is Money Anyways?

So yes, let’s start at the very heart of the matter: what is money after all? Now even though we might think we know it all too well, it could turn out to be more than what we think…

Read more

6 months ago · 5 likes · Ash Stuart ✅

Money may provide a common unit for exchange, but value itself remains stubbornly subjective. Just because something carries a price tag doesn’t mean everyone sees that price the same way — or even agrees that it’s worth paying at all. Two people can look at the same price and reach very different conclusions about whether something is expensive, reasonable, or a bargain.

Part of the reason is that people rarely evaluate prices in absolute terms. Instead, we interpret them through our frame of reference — our expectations, our prior experiences, and whatever alternatives happen to be visible at the moment. In other words, the number itself may be fixed, but the meaning of that number is not.

This is where psychology starts to complicate the economic picture. Economically speaking, money is supposed to be fungible. One dollar is the same as any other dollar. But when people actually make decisions involving money, they rarely treat it that way.

None of this contradicts the institutional system Ash describes. Money still functions perfectly well as a shared medium of exchange and a unit of account. But it does mean that the way people experience money — and the way it shapes their decisions — is less stable and more context dependent than the economic model might suggest.

And that raises a natural question: if money is supposed to provide a clear and consistent measure of value, why does it so often fail to feel that way when we actually use it?

Subscribe now

Reason 1: Value Depends on Our Frame of Reference

While I mentioned this briefly in the introduction, the first reason money fails to feel like a stable measure of value is that people rarely evaluate prices in isolation. Instead, we interpret them through our frame of reference.

As I discussed in a much earlier post, the decisions we make are shaped both by the decision context — the objective features of a decision situation — and by the psychological context we bring to it — our experiences, expectations, and prior knowledge. Together, these factors influence how we interpret the options in front of us and, ultimately, the conclusions we reach.

When it comes to prices, this means the same number can carry very different meanings depending on the context in which it appears. A price tag may be fixed, but the interpretation of that price depends on what we compare it to, what we expected it to be, and what alternatives happen to be salient at the moment1.

In situations where we lack a clear benchmark for what something should cost, people often rely on available reference points to make sense of the number. This is where phenomena like anchoring enter the picture (Furnham & Boo, 2011). Anchors provide a starting point for evaluating unfamiliar values, allowing people to interpret numbers that might otherwise be difficult to judge.

Retail pricing provides a familiar example. A sale price often means something very different when presented alongside the original price than when presented on its own. Seeing “$70” next to “Was $100” immediately invites a comparison that shapes how the price is interpreted. Similarly, a discount framed as “20 cents off” can feel quite different from one framed as “20% off,” even when the underlying price reduction is similar. In each case, the number itself hasn’t changed — but the frame of reference used to interpret it has. Money standardizes the numbers attached to goods and services, but it doesn’t standardize how those numbers are interpreted.

Buy Matt a Coffee

Reason 2: We Mentally Categorize Money

A second reason money fails to behave like a perfectly neutral measure of value is that people rarely treat it as fully interchangeable. Economically speaking, money is fungible — one dollar should be identical to any other dollar. In practice, however, people often behave as if different dollars belong to different categories.

This tendency was described by Thaler (1985; 1999) as mental accounting, the process by which people mentally organize money into separate categories based on its source or intended use. Money earned through work, money received as a tax refund, and money won in a casino may be economically identical, but psychologically they often feel quite different.

Once money enters one of these mental accounts, it can influence how acceptable certain spending decisions feel. A purchase that seems reasonable when drawing from a “vacation fund” may feel much harder to justify when the same amount of money is framed as part of one’s monthly grocery budget.

One of the classic demonstrations of this effect comes from a simple thought experiment presented by Kahneman and Tversky (1984). Imagine you plan to attend a $10 movie and discover that you have lost a $10 bill on the way to the theater. Most people say they would still buy the ticket. But if you had already purchased the ticket and lost it before entering the theater, far fewer people say they would buy a second ticket. Economically, the loss is identical in both cases. Psychologically, however, the losses are placed into different mental accounts.

Mental accounting also shows up in everyday financial decisions. People often treat tax refunds, bonuses, or unexpected windfalls as “extra money,” even though those dollars are no different from any other money they have earned. Because those funds are mentally categorized differently, they may be spent more freely than money allocated to regular household expenses.

In this sense, the psychological categories we assign to money can influence our spending decisions just as much as the prices attached to the things we buy. Money may be interchangeable in theory, but the mental accounts we create often determine how willing we are to part with it.

Share The Eel-Filled Hovercraft

Reason 3: Spending Money Is an Experience

A third reason money fails to function as a perfectly stable measure of value is that spending money is not just a calculation. It is also an experience. The act of parting with money can feel more or less painful depending on how the transaction occurs.

Research on what Prelec and Loewenstein (1998) described as the “pain of paying” suggests that the psychological cost of spending can vary widely depending on the structure of the transaction. When the payment is highly visible and immediate — such as handing over cash — the sense of loss tends to feel more salient. When the payment is delayed, bundled with other expenses, or processed electronically, that sense of loss often becomes less noticeable.

This helps explain why different payment methods can subtly shape spending behavior. Paying with cash, swiping a card, and allowing an automatic payment to process in the background all involve the same economic outcome — money leaving one’s account — but they can feel quite different psychologically.

Subscription services provide a familiar example. When a payment is bundled into a monthly subscription, the individual transactions become less noticeable than if each purchase required a separate payment. The cost has not disappeared, but the experience of paying for it has changed.

In this sense, the psychological experience of spending money can vary independently of the amount being spent. Money may provide a precise numerical measure of value, but the subjective experience of parting with that money is far less consistent.

Pulling the Threads Together

Ash is right about the institutional side of money. At the societal level, money works because we collectively agree that it does. It functions as a shared medium of exchange, a unit of account, and a way to store value over time. Without that shared agreement, modern economic systems simply would not function. And if you’re interested in the historical side of that story, his new series is well worth following — it promises to be a fascinating exploration of how money has developed and evolved over time.

But once we move from the level of institutions to the level of individual decision making, the picture becomes less tidy.

The three mechanisms discussed here — frames of reference, mental accounting, and the experience of paying — illustrate why2. Money may give us a common numerical language for value, but the meaning of those numbers is filtered through psychological processes that vary across people and situations.

The price attached to a product might be fixed, but the way people interpret that price depends on the comparisons that happen to be salient. The dollars in our wallets may be economically interchangeable, yet we often treat them differently depending on where they came from or what we intended to do with them. And the experience of spending money can feel very different depending on how the transaction occurs.

None of this undermines the broader institutional role of money that Ash describes. Money still works remarkably well as a social technology. But the psychological reality of how people experience and use money is far more fluid than the economic model sometimes suggests.

Which brings us back to that deceptively simple question: what is money?

At the societal level, it may be a shared system for measuring value. But at the level of everyday decision making, it behaves less like a perfectly stable unit of account and more like a psychological tool — one that helps people navigate choices, but not always in perfectly consistent ways.

Leave a comment

Share