It’s one of the final days to join my next Claude Code for Customer Insights cohort on Maven. It’s a Maven Top 100 course, rated ✦ 4.8/5 (ratings keep going up!) — and I just made some pretty good updates for this run, if I do say so myself.

Join the course

The first agent I created that finally did things right took me a week. The third one: three hours.

You read all this advice that says you need agents for everything, and here’s how to set them up and they make it sound so easy, but they forget something: you need to improve them, and that’s hard.

I have yet to meet anyone who has created an agent from scratch that was truly the perfect hire from day one. The non-human employee that never needed feedback about how they communicate or a fully revised SOP after one month. It doesn’t exist!

My first agent took me a week because I painstakingly hand-wrote half of the instructions. I copied my own reliable SOPs for analyzing customer calls from different teammates with different levels of interview skills, and added rules about how to handle them.

Then I ran it.

I’d written prescriptions for cases and failure modes that hadn’t even come up after feeding it 45 transcripts. The failure modes that did come up hadn’t been solved. The instructions ended up being about half as long with a better coverage of the right failure modes. I couldn’t have figured that out from day 1.

I kept giving my agent feedback and fixing things by hand, transcript after transcript. We spent a lot of time together.

Finally the agent was a master at handling all kinds of interviews, from all kinds of people playing interviewer.

The second time I improved an agent…I made all the same mistakes! The third time, I learned about self-improving feedback loops. I did everything differently, and the results were just as good (but faster).

After many more agents, I finally accepted that iterating to a high performing agent does not depend on me sitting at my laptop and constantly correcting it myself. With the right set of information, an agent can do that without me.

“But Caitlin, this topic is way over my head!”

The idea of setting up agents that have their own ways to self-improve probably sounds like a lot to some of you - beyond where you are with AI today, what you feel technically capable of. Far beyond what you have the bonus headspace for on Friday evenings. I want to get you running your first simple self-improvement loop today, possibly within 10 minutes (yes, really).

In this edition:

• 🎒 Set it up — four files, already written. Copy them, run one command, and watch your agent catch itself. ≈ Ten minutes

• 🗺️ What’s actually in a loop, and who decides it got better? — the three parts, and the essential thing an agent can’t do for itself

• ⚠️ Troubleshooting ahead — the ten-second test that would have saved me a day

What you’ll need, before you get your loop going:

• An AI tool that works with files on your machine — Claude Code, Codex, or similar. Not the chat window in a browser. This whole thing runs on files sitting in folders, and a chat tab can’t reach them the same way.

• Mac, ideally. One of the four files is a shell script. Windows needs WSL, and I haven’t tested it there.

• 3+ interview transcripts you don’t mind experimenting on. Wherever they already live is fine. You’ll copy the file path.

• A consistent way of labeling who’s speaking. The setup assumes P01, P02 and so on, in the file and in the filename. If yours say Interviewer: or use names, that’s one line to change in one file — and the first thing you run will tell you exactly which line, before you waste a run on it (see below in THE ROUTE)

If you’re using something other than Claude, everything here still works, you’ll just swap a couple of folder names in the instructions for whatever your tool calls them.

🗺️ THE ROUTE

Your agent + a starter self-improvement loop

You’re setting up an agent that checks its own work before it hands it to you, and keeps a record of what it got wrong. That’s how it can improve over time.

I will start by acknowledging that creating self-improvement loops is complicated. There’s a lot to get right.

So what I’ve set up for you here today is an attempt to get you using and playing around with one self-improvement loop and understanding what the pieces are that are built in. I want to get you past the fear of this type of tool.

Your job is to run this simple process as I’ve built it for you, look at what it catches, and open the files up if you want to see how the pieces work.

It takes three steps and ≈10-15 minutes.

Step 1 — Download the four files

Put them wherever you like. A folder on your Desktop is fine.

—

Step 2 — Have Claude set them up

You don’t have to install any of it yourself. Download the four files, then paste this into Claude Code — or Codex, or whatever similar tool you use — filling in the two paths (ex: “PATH-TO-MY-INTERVIEW-TRANSCRIPTS” should not remain like that).

I've downloaded four files into FOLDER-YOU-PUT-THEM-IN.
Set them up for me. Don't rewrite any of them.

1. Copy interview-analyst.md into ~/.claude/agents/ so Claude can find it.
   Leave the other three where they are.

2. Run this and show me exactly what it prints, line for line:
      ./verify.sh "PATH-TO-MY-INTERVIEW-TRANSCRIPTS"

3. If anything comes back NO, tell me what needs changing and wait.
   Don't change it yourself without asking me first.

4. If it all passes, say so and stop there.

That second path is just the folder you already have some interviews in. Nothing needs moving or renaming for this.

What’s this verify.sh file? It breaks things on purpose in a temporary folder, checks that the loop notices, cleans up after itself, and tells you either “your loop is working” or exactly which line to change. It never touches your real files.

—

Step 3 — Ask for an analysis

You’ll just write the agent’s name, like interview-analyst in the prompt, then tell it your request. ”Look for why users cancelled in interviews P02, P03 and P05,” or whatever your test request is. My long request below is just me copying direct file paths for my three interviews (you can also just name them), then I asked:

“What caused cancellations, and where was value fragile?”

(There was no fancy prompting here, just a simple ask).

Two more files will be created on their own in the folder for this. They will be created the first time the agent runs, and then reused in every run:

interview-analyst.md   ← the four you downloaded
check.py
loop-hook.sh
verify.sh

loop-log.md            ← these two get written for you
loop-trail.log

loop-log.md is what went wrong, over time. loop-trail.log is what the hook decided, run by run (the hook being the bit that won’t let the agent finish without checking its work). They’re the memory. If your folder still has only four files after a run, that’s your first sign something didn’t work the way it should.

Watch the agent run. The interview-analyst agent writes the analysis. And before it can hand that to you, its work gets checked. Anything that failed comes back as a list, it fixes its own output, and tries again. You might see “running check.py” come up at the bottom of the terminal window where status is displayed.

When I first ran this on three real interviews, eleven of its seventy-eight quotes weren’t word-for-word. It caught all eleven, fixed them in about a minute, and handed me the corrected version. I’d never have known — and a reworded quote is exactly the thing that ends up in a client deck.

The four checks are “true or false” questions

The four checks it runs through are four questions with true-or-false answers (that’s important for simplicity here):

• Does every quote actually appear, word for word, in a transcript?

• Is every quote attributed to a participant?

• Did every transcript it said it read actually get used?

• Does every theme rest on more than one person?

None of those checks need your “opinion”, or anyone else’s. They are things that can be checked, even without a strong reasoning model. The quotes match letter for letter, or they don’t.

On clean runs (no errors) and checking that the loop works

One thing to expect: your first run very well could pass clean! The agent I’ve handed you has already gone through some corrections. But you can re-apply the same loop to an interview agent you built and see if it has more to tell you. Just ask Claude to set it up for you with another interview agent’s file path.

And if you want to watch it catch something right now, you can force it. Open analysis.md, find any quote, and change one word. Take out a “just”. Swap “really” for “very”. Then ask for the analysis again.

It should find that altered quote, tell you which one it was, and put the word back into the analysis and log the correction.

Keep in mind…

It’s keeping score over time. Every failure goes into loop-log.md, including the ones it fixed itself — a mistake it caught is still one it tends to make. When the same check fails on three separate analyses, that stops being a bad day, and it comes to you and asks for a rule. Roughly never in week one. It can become a very useful part by week four, let’s say.

But when the log stays empty, that means one of four things. Only one of them is good:

• The work was solid, no issues

• Or the check never ran as planned

• Or the checks ran fine and don’t cover what actually went wrong (it wasn’t measured, couldn’t be checked by the checks set up in the loop)

• Or the checker itself broke and is reporting success when it shouldn’t (which happened to me many many times while building self-improving loops)

loop-trail.log settles the first two issues: one line per run saying what it decided, so an empty log next to a fresh timestamp means ran, found nothing.

The third issue listed above is something nothing can settle for you automatically. Your checks catch what you thought to check. Everything you didn’t think of sails through looking perfect, forever, and no amount of green tells you otherwise. So read the output yourself sometimes. Not every run, but often enough that you’d notice something nobody ever asked it about.

“But the agent I’d want to set up to self-improve doesn’t do interviews.” This is just one example. You can swap it. Open check.py. The four checks are about sixty lines of ordinary Python near the top. Swap them for four questions about your own work that a script could answer. Everything else stays exactly as it is.

🏕️ BASE CAMP

What’s actually in a loop?

You need three parts at a minimum. If any one is missing, it’s not a complete loop. No exceptions.

Capture. Writing it down the moment it happens.

Your agent hands you something not quite right and you fix it. Rewrite the sentence, delete the paragraph, move on, because the fix took forty seconds.

I have done this hundreds of times. And every one of those forty seconds was the most useful thing that happened all day. But I threw it away. (Because I was oh so busy, too many other things to do to stop and think about how valuable documenting that might be!).

That’s all capture is. It’s keeping the receipt, putting it where it can be found and used. It has to happen on its own, because agents are terrible at last steps, and so are we. That’s the whole reason there’s a hook.

Something to measure against.

I gave a stripped-down version of my analysis agent a few thousand forum posts and asked what we should build. It came back confident and well-sourced, every quote real, with a product roadmap for a business I don’t own. Every instruction followed. It was no use to me at all.

But no simple “format check” catches something like “is this relevant and useful?”. So a really good self-improving loop needs two levels:

• Check against the answer key — does this quote appear word-for-word? Catches wrong.

• Check against the question that was asked — catches useless.

You can get full marks against the answer key and still have answered something else entirely. Ask me how I know. 😑

And neither one can be written by the agent about work it already did. Ask it to write a rubric for something it just produced and it’ll write one that passes. Of course it will. You would too.

Approval and a record (loop-log.md).

Nothing changes unless you say so.

I got this wrong first few times in the most embarrassing way: I let the agent decide when something deserved to be a permanent rule. It set both its own standards, told me I’d agreed, and wrote “decided with Caitlin” into a file. I’d said nothing! I hadn’t even been asked! Yikes.

So now I know, and I decide when it doesn’t get a vote, and how hard the stop is before updating anything. Typically, for agents with medium- to high-risk tasks, my self-improvement loops watch, writes things down, and that’s where the authority ends.

Everything ends up in one append-only file. It includes every failure, which ones it fixed itself in the outputs of the task (e.g. “I got that wrong, went back and fixed it”), and every rule you agreed to add in to newly improved instructions. Nothing gets edited without me in those cases.

A bit of troubleshooting for your future self-improving loops

I wrote a check that was wrong, and you might too.

The check said: every transcript in the folder appears in the output = correct. So as long as the analysis results included all transcripts that exist in the folder, the coverage was considered good enough.

This was reasonable-sounding, but the wrong metric. Teams analyze a subset of what’s in their data folders all the time. So if I told my agent to run on a subset of the interviews in my folder, the loop would go, “that was wrong! Incomplete, must be fixed!”

My loop was about to start failing good analyses, then proposing rules to make the agent better (that would actually be worse), citing failures that were not failures.

But the loop was technically working the whole time. It was functional. My definition of good was the broken part, and a loop can’t see or judge that.

How to get this right the first time (or at least really close).

Two questions, ten seconds each. Every time I create a new improvement loop with checks the agent’s improvements will be based on, I asked this:

What’s a good piece of work that this check would reject?

What’s a bad one it might let through?

It won’t catch every single thing, but it will get you a lot closer to writing checks to measure what will actually improve your agent.

I still think you can’t hand over that kind of task. You have to figure out that “what good looks like and passes” part yourself. Self-improving agents, both simple and complex, have to start there and it starts with you.

—

Keep moving.

— Caitlin

If you’ve ever typed “I already told you that!” to an AI tool, it almost certainly wasn’t ignoring you.

Last week I watched workflow results tell a product team to run a study they’d already run and delivered. Despite that Claude’s memory files told it about the experiments. This shouldn’t have happened.

The problem was reliance on Claude’s automatic memory. A few details, rules and requirements they’d mentioned in sessions like “don’t forget this!” had been successfully saved to the tool’s automatic memory.

Claude wasn’t intentionally ignoring anyone (well, as far as I know). It just hadn’t accessed the information in the right moment.

This edition covers where memory and context live, why the file you write is a better foundation than the one your AI tool writes for itself, and what to put where so that anything you’d be annoyed to repeat only gets said once.

There’s also a 📹 demo that shows the first problem happening live, and the fix that handles multiple memory problems.

Let’s dig in.

In this edition:

• 🏕️ Base camp — where memory actually lives, when memory doesn’t follow with you, problems worth knowing and designing for - plus a demo video

• 🗺️ The route — fixes, a what-goes-where memory table, and a five-second audit

• 🌰 Trail mix — the benchmark that grows an agent’s context until it breaks, and more on memory from Anthropic

🏕️ BASE CAMP

Problems 1 and 2: Memory that goes missing

Two of the three big issues with memory are about memory being absent — it exists, just wasn’t there when you needed it. I’ll share (and show, video below) the big issues, why they happen, and why the file you wrote is more reliable than the file your AI wrote for itself.

Problem 1: The memory doesn’t follow you

Let’s start with Claude’s automemory.

Claude Code — and a growing number of tools like it — keeps its own memory. It writes the files itself, and updates them as you work.

That time you said, “now that you know X, don’t forget it!” — it typically saves that somewhere for later.

Memory is made up of multiple files saved over time, they live on your computer here:

It’s a hidden folder in your home directory, with one slot per project, each slot named after the path of the folder it belongs to.

You can open these files, edit them, delete them. You can tell Claude to create one for you around a specific topic.

In this case, a “project” doesn’t mean your specific study or redesign or build, the thing you’d call a project in your daily work. It means the exact folder you were sitting in when you started up Claude. So any folder you start up Claude from gets a memory. (Ex: I’m often working in a big folder named “Screenshots/” 😅. So there’s a memory for that.).

Picture a notebook Claude keeps for each room it works in. It leaves it behind in that room when it’s done. Start work somewhere new, and it will open a fresh, blank notebook.

Let’s say you have a team folder like this:

Your company/             ← never started Claude here → no memory
└── Core product/         ← started up Claude here → gets its own memory
    ├── Q3 projects/      ← started Claude here too → a second, separate memory
    └── Q1 2027 research/ ← never started Claude here → no memory

But the notebook Claude created for itself in Core product is invisible from Q3 projects, even though one folder is inside the other. This is important.

(One exception. If your folder is a git repository, the whole repository shares a single notebook, whichever folder inside it you start from.)

Claude also uses CLAUDE.md context files.

Contrary to how automemory files work, CLAUDE.md files — the plain markdown files that you write yourself or ask Claude to write — they walk up the folder tree. Claude starts where you launched it and loads every other CLAUDE.md sitting in folders above it, all the way to the top.

So if you’re working in Q3 projects/ below, here’s what your session actually picks up:

Your company/
├── CLAUDE.md             ← loads all context from this file above where you are
└── Core product/
    ├── CLAUDE.md         ← loads all context from this file above where you are
    ├── Q3 projects/      ← you're working in THIS folder here
    │   └── CLAUDE.md     ← loads all context from this file, where you're working
    └── Q1 2027/
        └── CLAUDE.md     ← doesn't load: it's next to you, not above you

So we have two memory systems (Claude’s automemory and your own CLAUDE.md files) running side by side, but they don’t behave the same!

A rule you wrote in a CLAUDE.md at the top of a team folder is inherited by every project folder underneath it, forever, without anyone remembering it’s there. But automatic memory just stays where it was written, no trickle-down effects.

Why my client product team’s memory wasn’t working

For that product team, Claude was picking up a few key CLAUDE.md files perfectly. And one of them contained their this-year-only freeze on pricing/packaging changes. But the automatic memory containing details about four completed experiments they didn’t want to repeat sat in a folder one level up in Claude’s automemory from where they were working. That meant it never got read.

Why it matters: the context you care about most is sometimes the thing you told the tool once, in conversation, because you thought that was enough — and that ends up in the file that doesn’t move from whatever folder you were working in that day. Bummer.

〰️

Problem 2: The memory gets cut off without telling you

There’s a ceiling on the automatic memory index: the first 200 lines, or the first 25KB, whichever’s reached first. Information beyond that won’t be loaded when your session starts. Claude is told the file was cut short. But you aren’t told anything.

And because entries get appended to the bottom as you go, the newest thing you saved is the first thing to get dropped. There’s no “last-added = most current” logic here.

Long before you hit a ceiling, an unpruned memory file is carrying a topic from four months ago into a session that has nothing to do with it.

Every one of those lines competes for the model’s attention with the thing you actually asked about. More context loaded is not the same as more context used.

Why the file you wrote beats the file AI wrote

The clearest framing I’ve found comes from CoALA — Cognitive Architectures for Language Agents, by Sumers, Yao, Narasimhan and Griffiths. It sorts an agent’s memory into four kinds:

Automatic memory = episodic and semantic, written by the agent, on its own judgment, without a review step.

Procedural memory “must be initialized by the designer” (that’s you!") and letting an agent rewrite its own is, in the authors’ words, “significantly riskier.”

In other words…don’t rely on your tool to write all the memory it needs! Think strategically.

〰️

Watch it happen

I replicated and recorded the exact failure I saw my client product team face. Problem 1 on screen: the workflow proposing a study that already ran, three CLAUDE.md rules used correctly, while two memory folders sit side by side with nothing passing between them. Then the one block of markdown that enforces it, doesn’t let this memory lapse happen again.

〰️

Problem 3: The memory is there, but it’s wrong

Problems 1 and 2 are memory not coming along where you need it. The third is often the most expensive of the three: the memory is present, loaded, and out of date.

The decision log turns into a wall of outdated notes

I do recommend writing down what you’ve already decided so the AI stops rehashing it. But now imagine fast forwarding six months into the future.

You had a bunch of notes from six months ago. They were correct — back then. But since then, the team’s budget shifted, the freeze on pricing changes has been lifted — now all the execs just want you to fix the service tier packages, find a way to increase profit by shifting features around.

(And they want it yesterday, of course. “Please do this with AI so it’s faster,” they say).

Now the constraints listed in those files that saved you from Claude’s irrelevant pricing experiment ideas is now the reason it refuses to think about the biggest current priority on the roadmap.

Written context goes stale without you noticing, and it keeps sounding authoritative the whole time. Especially if it’s in the wrong place.

You don’t see what it’s carrying into your workflow

Without any memory at all, it’s simple: whatever you send in a prompt is what it knows. With existing memory files, something is being loaded before you type a word, and you likely have no idea what that is.

Two commands can sort this, and most people have never run either:

• /context shows if memory files were actually loaded this session, along with other context usage details

• /memory lists your memory files and opens the auto memory folder, so you can read, edit or delete anything that’s been saved about you.

Run the second one today, just to see what’s in there. I’ve yet to watch someone do it and not find at least one thing they didn’t know was being carried.

Memory was saved in the wrong place

Did you perhaps…save your brilliant rules into the wrong folder? This is what happens.

The ban on pricing changes went into a CLAUDE.md file for the Core product/ folder, one level above the team’s Q3 work. But the Q3 work was the only place it should have been applied.

Remember how CLAUDE.md files get inherited by every folder beneath them? So putting that file in Core product/ meant that the Q3 folder and any other project folder the team created next to it inherits that context file. This quarter’s discovery, next quarter’s prioritization, every roadmap folder anyone creates in 2027 under Core product/. They’ll all get the “no pricing changes” rule. The freeze on pricing changes outlives the year it was true for, and somebody has to remember to go and delete it.

Written one level down, in Q3 projects/, it behaves completely differently. This means that a folder next to it like Q1 2027/ doesn’t inherit the pricing ban rules. The team will open the Q1 2027 folder with pricing back on the table, no ban on pricing experiments, and nobody had to remember to change anything.

Match the folder’s lifespan to the rule’s and expiry stops being a task anyone owns. Put a rule higher than its shelf life and you’ve signed up to maintain it by hand.

🗺️ THE ROUTE

The fix for all three problems

After all that about problems, the fix is rather simple. One habit can sort all three problems:

• Write the context you care most about into a CLAUDE.md file yourself (I mean, you can still ask Claude to do it for you), not just automemory

• Put the file at the level that matches how long it stays true and applies

• That file travels with you (solves problem 1), loads in full however long it remains applicable (solves problem 2), and sits somewhere you can read and date and delete (solves problem 3) — or just stops applying when you switch to current work

The whole decision comes down to one question: how long is this true for, and what level does it apply to? Answer that and the location is fairly automatic.

Where each kind of context belongs

The fourth row is one we should take a longer moment to think about. If you put the right things into the project level there, it’s the fix for the wrong-level problem above, and it replaces a calendar reminder with a filing decision you make once.

〰️

The five-second audit

Check this: paste the two lines below into Claude Code and it’ll tell you which of your projects are over the ceiling.

wc -l ~/.claude/projects/*/memory/MEMORY.md
wc -c ~/.claude/projects/*/memory/MEMORY.md

Over 200 lines, or over 25KB — 25,600 bytes — and you’re losing entries off the bottom of that file without being told.

However, just know know that the limit is measured on the content that actually loads, and frontmatter and comments get stripped out before that happens. So a raw count can make a file look worse than it is. If you’d rather not do the arithmetic, just ask: “Check my memory files against the 200-line and 25KB limits, ignoring frontmatter and comments, and tell me which ones are actually over.”

Then I do recommend put a recurring 20 minutes in the calendar — quarterly is plenty — to open the memory files that matter and delete what’s no longer true. Pruning is worth it here. You can automate Claude to help you with this, too.

〰️

🎒 PACK UP

The next Claude Code for Customer Insights cohort is closing soon

The pricing freeze in the examples today is what I call a decision we’ve made — a settled thing that should be excluded from every workflow without you having to mention it. It’s just one of many info types I teach students to include in CLAUDE.md files for reliable, relevant insights work with Claude.

You saw a snippet of the actual course content I’m updating.

There’s loads more to learn in the course:

• How to set up your file structures to help Claude find the right things, with fewer tokens

• How to correctly design skills that are highly reusable (get that ROI, stop needing to revise them constantly!)

• What you need to build into Claude code processes to make them good enough to automate (and trust the outputs)

• Designing evals for your workflows so you don’t hope they work, you know it

• …and more…

Claude Code for Customer Insights — enrollment closes Friday, August 28, cohort runs August 31 to September 11.

Join the cohort →

🌰 Trail mix

• The Anthropic memory docs themselves — two pages, for some extra memory reading. Skim them, then run /memory in your own setup and see how much you recognize.

• LOCA-bench (Zeng, Huang and He, February 2026) — a benchmark that deliberately grows an agent’s context until it basically falls over. Agents get worse as their environment gets more complex, which you’d expect. The useful part: context management techniques improved the overall success rate. Deciding what not to load is work that’s worth taking time for.

—

Keep moving.

— Caitlin

This will be an efficient edition - five things I think you should know about, a few that will be thought-provoking and two that are hands-on, use them today.

The commonality in this edition: I think all of them individually and together will help you think better as an insights person next week — better challenge ideas internally, set up better systems and do better work in your next project.

Skim the headlines, stop wherever it’s for you.

In this edition:

• 📋 Field notes — the AI “respondent” that beat your screeners, AI interviews vs. surveys, and when synthetic users actually work

• 🗺️ The route — the insight-to-bets workflow most teams can adopt immediately, my synthetic users readiness test, and a growing Skills Library for free

• 📡 Weather report — NotebookLM just went agentic

Let’s dig in —

📋 FIELD NOTES

Three studies worth a minute before your next study

Three recent findings that change how much you can trust what you’re about to run.

The respondents might be robots

A Dartmouth researcher built an AI “respondent” that passed 99.8% of attention checks across 43,000 tries, at about 5 cents each — and every AI-detector missed it (PNAS). Don’t torch your panel though. In real data across 10 platforms, under 1% of responses seemed to be AI-written (Mechanical Turk’s the outlier at up to 16%) (Prolific/Gordon). The capability is real and cheap and worth watching out for, but it’s probably not a problem for most of us yet.

The part to act on: your old screeners are not as effective. In head-to-head testing, classic attention checks flagged only about one in three AI agents (Prolific bench). Before your next field: stop treating “type ‘purple’ to continue” as proof of humanity. The real way to check if your participant is AI is to add signals AI is bad at: mouse movement, response timing, and dedicated bot-checks caught agents 94–100% where attention checks folded. And ask your panel recruiting provider what they actually verify.

〰️

AI-led interviews caught what the survey missed

The freshest evidence here is barely a month old — and it’s not a vendor’s. Researchers ran 571 people through either a standard survey or an AI-led conversational interview on the same topic, then compared the two (arXiv, June 18). The AI interviews surfaced reasoning the survey flattened: people with identical attitude scores turned out to hold completely different mental models — which only came out once the AI could ask “why?” (Unsurprising, since that’s what we do as humans).

The point isn’t “ditch your survey tool” — it’s that a well-run AI interview is becoming a real, established method, and its edge of depth is becoming more established too.

My take: Since I first started testing AI moderators for discovery, I figured they’d become one of the first tools we reach for, in place of surveys, because they scale and pull more depth out of people. What I didn’t anticipate is the rest of this study: that they’d also be relatively sharp at telling mental models apart (depending on the tool and host of other variables, of course). AI moderators vs. traditional surveys is still the comparison I’d watch — not a comparison to human-led interviews just yet.

〰️

Another synthetic users study showing they can potentially work — if you’ve got the data for it

New study from June: Researchers built digital twins of individual people from real, deep panel data and tested them across 2.1 million responses — at best, the twins matched real answers about 79% of the time (arXiv, June 3). So synthetic users can stand in for real people to an extent.

The catch is what it takes: that 79% was their ceiling, and it rests on years of genuine data about each person, not on prompting well or handing it just one little transcript. Give it thin data and accuracy falls off fast — worst on the small subgroups you most want to understand.

So the real question isn’t “can’t we just simulate this?” — it’s “do we have deep enough data to do what we want to do?“

🗺️ THE ROUTE

One workflow to steal, and two free things I made for you

Faster bets, not just faster research

A readout nobody acts on is the most expensive kind. The loop that fixes it: synthesize data into findings → log findings in one backlog (not five tools) → turn the critical ones into hypotheses you can test in two weeks → kill fast when the signal’s negative. It’s like atomic research nuggets 4.0.

This is the workflow the top 1% teams I’ve talked and worked with are building and iterating. It’s not about insights sitting on someone’s desk or waiting for the meeting where they’ll get officially presented, it’s about getting them into product work immediately, with the help of AI. The teams farthest ahead have nailed this and are speeding up experiments and builds because this is partly agentic (but still has human oversight).

Try it right now: take your last research readout and turn it into three hypotheses, each in this shape — “if we [change X], then [metric] will move, because [what the research told us].”

Level up: Turn the creation of the hypotheses (from any and all research results) and documentation of hypotheses -> test plans into a workflow with a file-based AI platform (ex: Claude Code, Cursor). Iterate until it happens well without you.

〰️

My Synthetic Users Readiness Test

Half the questions I get about synthetic users are really the same one:

“Is my use case one where they’ll help, or one where they’ll dangerously mislead me?”

It’s honestly a tough call — and an easy one to get wrong. So I turned findings from synthetic users studies into a two-minute quiz to help you:

• identify whether your target use case is a good one for synthetic users (or dangerous)

• pinpoint whether the data you want to use is right for this

• take the right first steps to prepping better for synthetic users

Take the quiz

The Synthetic Users Readiness Test asks seven questions about what you’re trying to learn, then hands you a straight verdict — a fit, proceed-with-guardrails, or a trap — plus next-step tips for your specific case. It’s built from 40+ studies’ findings, and it’s free.

Take it before your next “can’t we just simulate this?” debate. I’d love to hear what verdict you get and what you think of it — hit reply and tell me.

〰️

The Skills Library — want in?

My free, subscriber-only Skills Library got a fresh batch of ready-to-run skills for customer-insights work — the same ones I use all the time, for things like getting from raw transcripts to something I’d actually put in front of a stakeholder.

Here’s the part I’m excited about: some of the best additions lately (and coming soon) haven’t been mine — they’ve come from course students automating their own real problems. I’m guessing their problems are yours, so I hope this growing collection keeps helping you move faster (and better) without starting from scratch.

Get the Skills!

Built a skill that saves you time on insights work? Send it my way. If it’s good, it goes in the library with your name on it and any link to your work you’d want me to use. Just reply to this email and show me what you’ve got.

📡 WEATHER REPORT

NotebookLM just became a research agent

If you’ve filed NotebookLM under “chat with a PDF”, it started changing jobs in June. Every notebook now has a sandboxed computer that runs code on your sources — so “how many participants raised each theme?” returns a counted number, not an LLM guess. It’ll also go source-hunting on the open web from a blank notebook, and visualize then export results straight to slides, Excel, or CSV.

The catch: at launch these agentic features are gated to Google AI Ultra or a Workspace plan with the AI Ultra add-on — Google says cheaper tiers are coming but hasn’t said when. If that’s steep (it is), either put it on your watch list, or expense a single month of Ultra for a big analysis push and cancel. One caveat — it now pulls its own web sources, so quality checking might be back on you; open its show-your-work and trace a claim before it reaches a stakeholder. (Google · TechCrunch)

—

Keep moving.

— Caitlin Sullivan

How good is Fable, really?

Fable 5 is back on non-US accounts like mine, and the internet has opinions — comments about 12-hour autonomous runs, a “relentlessly proactive” model, a real leap ahead. But almost all of those rave reviews are about coding. Nobody was answering the question I ask about every new model and it’s ability to reason:

Does it make you better at reading customer interviews? Can it identify the patterns, themes and hidden nuance like expert humans cans?

So I wanted to check. And it turned into the same lesson as the second thing that landed on my desk this month: AI hands you something that looks trustworthy — a benchmark score, a cited source — and the looking-trustworthy is doing a lot of work. Let’s dig in.

In this edition:

• 📍 New from me — Future of UX podcast episode is out, and my course is filling

• 📋 How good is Fable, really? — I went looking for the winner and found something weirder

• 🗺️ Can you trust that model comparison? — a checklist, plus questions for the next “new model is better” debate

• 📡 One Reddit comment poisoned an AI research agent — what to check before you trust AI market research

• 🌰 Trail mix — Model updates, tools for in-person discovery, and a Claude Code spend tracker

Let’s dig in —

📍 New from me

Future of UX podcast episode just out

• Patricia Reiners and I talked about the rather unsexy topic of files - but the files you put in to work with Claude Code and other systems, plus the files you get out for documentation are just about the most important part of working with AI and agents right now.

  Listen over here: Apple Podcasts, Spotify

Claude Code for Customers Insights - Aug 31 - Sep 11

• The testimonials and reviews are getting better every cohort because people from teams like Mastercard, Hinge, and Spotify are turning spontaneous chats into repeatable systems within two weeks.

  → Sign up here

📋 FIELD NOTES

Fable vs. Opus: I went looking for the winning model and couldn’t find one

The basic setup: I used three sets of ten real customer interviews, my own original analysis as the answer keys for each. Three models tested — Fable 5, Opus 4.8, Opus 4.6 — each completing 15 runs of the analysis. ChatGPT models graded what was delivered against my golden answer sets.

Worth knowing: This is a simple example of benchmarking - I run the same tests every time a new model comes out. But how I do it is important: I don’t use best-in-class prompting tactics or highly engineered context to help the models do a better job. I run a simple set of instructions that allows the models to reason and choose approaches on their own. That’s how we can see what each model is capable of with its defaults - and whether it’s an improvement over previous models.

The tie

Did one model find significantly more findings than the others, or do a closer analysis to mine? Not really. Roughly 71% findings match for Opus 4.6, 70% for Fable, 67% for Opus 4.8 when compared to my golden results set (e.g. my own thorough analysis).

That’s honestly too close to see a clear upgrade or winner among the three. Which also means the frontier Fable model didn’t clearly deliver better interview findings or themes. “Just use the newest model” isn’t the advice to follow every time for qualitative insights work. TBD if this looks better with quant work.

The twist: where Fable was worse

Fable walked right past the most human thing in the data.

Buried in one interview set: people were a little embarrassed to admit they meditate because of the cultural response they received to that kind of mental health approach in their work environments. So they downplayed how much they used the app. Reading between the lines and catching the “say versus do” gap is often the whole point of qual work. The older Opus 4.6 caught that gap every single run. Fable never did.

Hold these findings loosely — I ran three sets of tests, with three datasets, not hundreds of either. This should be a flag, but not necessarily a final verdict, depending on your own work and data.

The number to remember

Run the same model on the same interviews, with the same settings, and the scores don’t always match. Opus 4.6 landed at 61% one run and 76% the next — a 15-point swing on identical inputs. Not every model bounced that hard, but enough that a single try tells you a mood, not a measurement.

🗺️ THE ROUTE

How to know if you can trust any model comparison

Next time someone claims “the new model is way better”, here’s generally how to test if it’s real. It’s a blind taste test — same task, hidden answer key, a neutral grader, run enough times that one lucky run doesn’t fool you:

• Run it more than a handful of times, and report the average and the range.

• Use 2+ datasets (3+ is ideal) to catch where models might be particularly good or bad with certain kinds of data (ex: two very different sets of interviews)

• Check whether the gap clears each model’s own bounce. Scores jump around run to run — Opus 4.6 swung 15 points above. If one model beats another by less than that swing, you’re looking at a lucky run, not a better model.

• Don’t let a model grade its own family. Use a grader from a different model family, and hide which model wrote what. Ex: ChatGPT grades Claude.

• Lock the rubric and the settings before you compare, then spot-check a few of the grader’s calls yourself.

• Don’t treat your answer key as gospel. “Score” means agreement with one expert’s labels, not truth.

Three questions for the next “it’s better” debate:

📡 WEATHER REPORT

One Reddit comment poisoned an AI research agent

There are many reasons to double-down on verification steps in your AI-assisted research workflows, but this adds one more.

Most researchers and PMs spot-check for hallucinations: the AI inventing something with no source. You typically catch those by asking AI and yourself “where did this come from?” and enforcing citation rules in outputs. But the “poisoning” threat slips past those spot-checks.

Poisoning: someone plants real content on a real platform — a Reddit comment, a forum post, a product review — deliberately worded to steer what the AI tells you. The source is real and looks legitimate, so “where did this come from?” comes back clean. It’s not the AI making something up; it’s the AI trusting a source that was set up to fool it.

Cornell Tech’s WARP study showed this spring how the attack lands on deep-research agents — the tools that scour the open web and hand you a synthesized report. Plant a single 13-word comment on a high-traffic page, and when the agent pulls that page into its research, the attacker’s fake product shows up in the report 38–51% of the time (on the open-source systems they tested).

The study didn’t run live attacks on consumer tools like Gemini or ChatGPT, but it did measure their potential exposure to the risk: Gemini’s Deep Research draws about 12% of its citations from user-generated content, most of it Reddit — the same surface the attack targets. There’s a name for doing this on purpose: Generative Engine Optimization (GEO).

You don’t have to stop using these tools. The rule: treat user-generated content as a lead, not a fact. Reddit, forums, and reviews are gold for finding language, complaints, and emerging problems — but two checks turn a lead into something you can act on:

🌰 Trail mix

• Google shipped Gemini 3.5 Live Translate (June 9) — real-time, voice-to-voice translation in 70+ languages, now in Google Meet and Translate. If it holds up, you could run discovery with more customers more smoothly without avoiding markets where you can’t speak the languages.

• Anthropic shipped Claude Sonnet 5 (June 30), the new default. They claim it hallucinates less and is less of a yes-man than the last Sonnet. If true, that makes it a stronger cheap starting point for a first analysis pass. But “near-Opus quality” isn’t “reads nuance like Opus” — test it on your own transcripts before you lean on it.

• Ellis — Free AI notetaker built for face-to-face conversations: it records, tags who said what by voice - using a sample of your voice to identify when you’re speaking. It’s brand new, but a lot of my course students have wanted a better solution for in-person recording — this might be it?

• WTClaude — free, open-source tool that shows your real Claude Code spend in the terminal, reading the actual numbers Anthropic bills you, not estimates. Sixty-second setup: npx wtclaude setup.

—

Keep moving.

— Caitlin Sullivan

Two weeks ago I ran a series of experiments I’d never really run before. I wanted to measure token usage — how many tokens it actually takes to do specific tasks with MCP-connected tools. It started small: a simple copy-paste from a markdown file to a Notion page was costing 18,000 tokens, which felt insane, and I wanted to see if I could get that number down to something not-so-massive.

That first test led to another. Then another. I kept finding places where tokens were disappearing into MCP calls and tasks I hadn’t questioned before, and kept finding ways to affect them. By the end I’d run 20 experiments across 10 tasks with 6 MCP-connected tools. The result: 450,000 tokens saved in one experiment session. That’s more than the entire context window for some models. In most cases, the tokens originally used in a task could be cut roughly in half with relatively quick fixes.

The takeaways: instructions matter more than you think, and blindly accepting defaults is a token drain. If you’re constantly running out of tokens or hitting context window limits, your MCP setup might be to blame.

In this edition:

• 🏕️ The unnecessary work tax — your agent is doing work nobody asked for

• 🗺️ Fix your MCP instructions — the hidden instruction layer bloating every write, and five rules to override it

• 📡 Weather report — Fable 5 arrived and vanished in 72 hours. Why we should care.

Let’s dig in —

🏕️ BASE CAMP

The unnecessary work tax

Most token waste is mundane: your agent doing work it doesn’t need to do, on a model that’s too expensive for the task.

What GitHub found

GitHub’s engineering team audited their agentic workflows powering Copilot — issue triage, security scanning, contribution tracking. They found waste everywhere. Not from complex reasoning. From silly busywork.

One of their AI workflows sorted incoming bug reports into the right team’s queue. Before the AI even started deciding anything, it was burning tokens on busywork — looking up the same background info every single time, info that never changed between runs. It didn’t need to think to get that info. A simple lookup would do. But because the AI had fancy tools that could do those lookups, it used them — and every one of those tool calls costs tokens.

When they pre-fetched the data and handed the agent a file instead, token spend dropped 62% across 109 production runs.

The results across all five workflows:

The 342-call problem

The most dramatic example: a Glossary Maintainer workflow that only needed to scan local files. It had access to search_repositories — a tool for searching across GitHub repos — and called it 342 times. 58% of all tool calls. Why? Because the tool was available, and nobody had removed it.

AI tools use what they’re given. Every tool description gets sent to the model on every single turn — whether the agent uses it or not. The GitHub team’s blunt summary: “Workflow authors naturally start with a full tool-set since it is the path of least resistance.”

Check this: Count your MCP tools. If you’ve connected Notion, Google Drive, Gmail, Calendar, Slack, and seven more you use once a quarter, that’s dozens of tool schemas loading on every turn. A server with 40 tools adds 10-15 KB of schema overhead per turn. If you use 2 of those 40, the other 38 are dead weight.

The model-fit problem

Reasoning models generate extra tokens before your answer. When you use extended thinking or a reasoning model (OpenAI o-series, DeepSeek R1), the model produces an internal chain of reasoning — sometimes thousands of tokens — before it responds. Those thinking tokens count toward your usage and add latency, even on tasks that don’t need deep reasoning.

Virginia Tech tested this directly — same model family, same basic math problems:

18x more tokens. Worse results. For copying content between platforms, extracting quotes, formatting data, or sometimes even classifying responses — reasoning models can actually be overkill.

Check this: Check which model you’ve been using by default. If you’ve used a reasoning model for most things, turn it off for a session of simple read/write/extract tasks and see how much longer it takes to hit the token wall.

🗺️ THE ROUTE

Fix your MCP instructions

When you connect an AI tool to a platform via MCP (Notion, Confluence, Jira, Google Docs), your agent follows instructions baked into the tool itself — instructions you never wrote and probably haven’t seen.

What your agent actually does

A task that should be one API call — “copy this content to a Notion page” — turns into five or six:

1. Reads the platform’s spec. The Notion MCP tool description literally says: “IMPORTANT: always first read notion://docs/enhanced-markdown-spec.” The agent obeys — every time.

2. Fetches the destination page. Even when you already gave it the page ID.

3. Creates an empty page, then updates it. Instead of creating the page with content included, it creates blank, then inserts separately.

4. Piecemeal updates. It doesn’t copy everything in one go — it takes one bit at a time, doing piecemeal work for 20 minutes until it might decide there’s a better way. (It usually doesn’t). 🤦‍♀️

5. Fetches the page back to verify. The API already returns success or failure — redundant.

6. Sometimes spawns a sub-agent. That sub-agent starts from zero context, re-reads all the same content, re-loads all the same tools.

Here’s an example of what Claude finds in Notion MCP’s automatic instructions:

Where the tokens go

Your agent follows instructions you didn’t write, as you see above. The fix: make your instructions more specific than the tool’s defaults. Here’s an example of what happens with just one write-to-doc task —

And these costs stack. A workflow that writes to three pages wastes 15-30k, not 5k.

Do this first (2 minutes) - A little test :)

Open the instructions for any workflow that writes to Notion, Confluence, or another document platform. Add these two lines:

Run the workflow. Compare the token count to last time. That’s usually 30-50% gone — and you haven’t touched the other three rules yet.

Five rules to cut your write cost in half

These work across Notion, Confluence, Jira, and Google Docs. Parameter names differ but the pattern is the same.

1. Specify the format yourself — skip the spec read.

The single highest-impact change. If you know what format your platform expects, put it in your instructions and tell the agent not to read the platform’s spec.

“Do NOT read the enhanced-markdown-spec resource. Use the format specified below.”

Then include the format rules you need — headers, callouts, dividers. Not the full spec. The ten lines that apply to your output.

2. Give it the destination directly.

Parent pages, Confluence spaces, Jira project keys — if they don’t change between runs, hardcode them.

“Create the page under parent ID 353dc8e0af5e822d9396dff19725436x. Do NOT fetch the parent page first.”

3. One call, complete content.

Every doc platform MCP has a create call that accepts the full page body.

“Use a single create-pages call with the complete content in the body. Do not create an empty page then update.”

4. Trust the API response.

If the create call didn’t result in an error, then the page was created. You don’t need an extra fetch just to confirm it, but that often happens. Add this to instructions:

“Do NOT fetch the page after creating it. The API response confirms success.”

When there’s a verify step automatically included, it typically re-downloads the entire page — every heading, paragraph, and callout you just wrote.

5. Keep the write in context.

If the content is already in the agent’s context, have it write directly.

“Write to Notion yourself — do NOT spawn a sub-agent for the write.”

Sub-agents re-load tools, re-read content, and re-discover page structure. That overhead can hit 8-10k tokens — more than the write itself.

〰️

How to find the right parameters for your tools

Ask your AI tool directly: “What parameters does the confluence-create-page tool accept?” It can see the loaded tool schemas and will describe them in plain language.

Or measure-then-optimize: Run the write once with minimal instructions. Add one constraint at a time. Measure after each change.

A few numbers

From my own workflows, applying just some of those rules:

📡 WEATHER REPORT

Fable 5 arrived and vanished

Anthropic launched Fable 5 earlier this month. Three days later, the U.S. government ordered it suspended. Gone before most of us had time to test it.

But next time it could be the model you depend on — the one running your daily workflows, the one your instructions are tuned for. Maybe they don’t take it away but how they charge for it changes, meaningfully. This is coming, I’m certain of it.

Ask yourself: How portable is your setup? If your model disappeared tomorrow, could you move to the next one? Something to think about.

—

Keep moving.

— Caitlin Sullivan

Think first, then let AI finish. Is that enough?

When people in product and research ask me about AI and their brains, the question is almost always about customers: where should I stop using AI so I don’t lose touch with the people I’m building for?

It’s a reasonable question. But it’s not the question I lie awake thinking about.

When I zoom out, I’m far more worried about my own brain decaying across the hundreds of other things I now run on AI. I use it to keep my course content current. I use it to prep keynote presentations, to pressure-test my own thinking, to help me write newsletter editions like this one (usually 80% me, 20% AI).

I’ve been deliberate about it — careful about which tasks I hand over and the order I hand them over in. My have a set of rules, but roughly it goes: think first, let it finish things off.

Over the past few years, I’ve worked out an approach I trust, and the reassuring part is that it lines up with what learning science and decades of automation research already say.

So if you’re the one asking where to draw the line with AI before it dulls you, here’s my answer: what I keep in my own hands, the method I run on everything else, and the evidence for why it works. The goal is to stay in the top tier of AI users — shipping fast, delivering well — while staying sharp enough that you’d still be good at this if someone took the tools away.

In this edition:

• 🏕️ Why the easy way feels like learning (and isn’t) — the decades-old science of “desirable difficulties,” and why AI is the most convincing illusion of competence ever built

• 🗺️ Keeping your brain in the chat — how to spot what you shouldn’t hand to AI, plus my blueprint for everything you do

• 🔭 The view from here — the study everyone’s citing right now, with a skeptic’s hat on

Let’s get into it —

🏕️ BASE CAMP

Why the easy way feels like learning (and isn’t)

There’s a body of research that’s been sitting there, useful, for thirty years and suddenly matters a lot more than it did in 2019.

In the 1990s, the cognitive psychologist Robert Bjork coined the term “desirable difficulties” based on his decades of work with Elizabeth Bjork. The finding, repeated across decades of studies: the conditions that make learning feel slow and effortful — spacing practice out, mixing topics up, testing yourself instead of rereading, generating an answer before you’re shown one — are the conditions that produce the strongest long-term memory.

Those conditions also produce the best transfer — your ability to use the knowledge in a new situation later. In basic terms, the conditions that feel smooth and fast tend to produce the weakest memory and the shakiest transfer — even though, in the moment, they feel like they’re working better.

The catch is in how badly we misjudge this in the moment. If there’s a line from their work to tattoo on the inside of your eyelids, it’s this one:

“Current performance is not a reliable index of learning.”

When something feels fluent — when it comes easily, reads cleanly, slides down without resistance — we read that fluency as understanding. The Bjorks called it the “illusion of comprehension.” Rereading your notes the night before feels productive because it feels familiar. It mostly isn’t doing much. The struggle you avoided was the part that would have built the memory.

The part that should worry every heavy AI user: In one study, people practiced under two conditions, then were asked which one taught them more. The mixed-up, harder condition won decisively on the actual test — roughly 90% of people learned better that way. And yet most of them, even after seeing their own better results, still believed the easy way had taught them more. They were sure they were in the 10% exception.

We are all sure we’re in the 10% exception.

AI is the most fluent thing ever invented. It hands you a clean, confident, finished-looking answer with zero resistance. By the desirable-difficulties logic, that’s not a neutral convenience — it’s the exact formula that produces the illusion of comprehension at industrial scale. The output looks like understanding. Your “current performance” — the deck, the synthesis, the analysis — looks fantastic. Whether you learned anything is a completely separate question, and the fluency is actively hiding the answer.

The early signals

A couple of recent studies are getting passed around as proof we’re all getting dumber. I’d be careful with them — they’re newer and softer than the desirable-difficulties work, and I’ll point you to one below with the caveats. But the direction is consistent.

A 2025 study of 666 people (Gerlich, in the journal Societies) found a significant negative correlation between frequent AI use and critical-thinking scores, statistically explained by “cognitive offloading” — handing your thinking to the tool — and strongest in younger, heavier users. It’s correlational and self-reported, so hold it loosely.

An MIT group wired people up to EEG while they wrote essays with and without ChatGPT. The AI-assisted writers showed the weakest brain connectivity, which the authors called “cognitive debt.” Small sample, has published critics. Suggestive, not entirely settled, but worth consideration.

But you don’t need the new studies to make the call. The thirty-year-old ones already told us: the easy, fluent path feels like learning and isn’t.

AI just made the easy path frictionless and put it on every task you do.

Why it matters: The thing that makes you valuable isn’t the output AI can now generate. It’s the judgment to know whether the task is worth doing, and when that task’s output is wrong — and that judgment is built by exactly the effortful work AI is now offering to take off your plate.

🗺️ THE ROUTE

Keeping your brain in the chat

I want to be honest about the bind, because pretending it away helps no one. I’m under the same pressure you are. We’re all expected to deliver a hundred times what we delivered five years ago, in a fraction of the time. Our world rewards constant action and speed over the slow development of intelligence over time now.

Expertise used to come with time — years of doing the work by hand until the judgment set in. That’s not the deal anymore. You’ve got a training budget you’re supposed to apply in 24 hours, ten skills to learn by Friday, a product to ship by Monday latest. Nobody’s pausing the roadmap so you can mess around for the sake of using your brain.

So here’s the approach I actually run to use AI hard without letting my brain leave the chat. It comes down to two big moves: decide what you’re not going to hand over, then run a real method on everything else. Each piece lines up with research I’ll point to as we go.

Move 1: Figure out what you shouldn’t hand to AI

I’m not going to give you a list of tasks to keep manual — your work isn’t mine, and the right line sits in a different place for everyone. What I can give you is the question I run on anything I’m tempted to fully automate:

Is the difficulty in this task teaching me something I’ll reuse — judgment, pattern recognition, familiarity with my customers or my data — or is it busywork I have to go through to get to the thoughtful part?

Busywork and toil, AI can have it. The difficulty that’s building something in me, I keep, even when keeping it is slower.

Three signals that a task is one of those:

‣ When coverage is negotiable but contact isn’t.

In research the raw material is always the value — so the rule here isn’t “keep it all manual.” You can’t read every transcript by hand, and handling that volume is exactly what AI is for. The discipline is narrower: compromise on coverage, never on contact. Read enough of your own raw data yourself to stay fluent in it, then let AI scale the rest. When I teach analysis with AI or Claude Code, that’s the line I hold — and the counterintuitive part is that if a team is already running heavy AI moderation and analysis, I tell them to add more manual contact back in, not less.

Hand the whole pipeline to agents and you reach “customer understanding” faster — but you’ve pulled yourself out of the understanding. You become the manager of a manager managing bots. Anyone who’s managed people knows how that goes: the further you are from the work, the harder it is to see what’s really happening in it. You lose touch with the customer, and you lose the ability to catch what your agents get wrong, because you’re no longer fluent in the work yourself.

The signal: if you can’t remember the last time you touched your own raw data (or your whole understanding of customer needs come through AI’s summaries), that’s the time to pull back and dig in on your own.

—

‣ When the doing is the thinking — or the relationship.

I don’t automate some whole tasks like email. I don’t have an agent answering for me. Some things take longer to reply to than I’d like, and I’ll take that, because I’d rather respond as a human with my own brain than have an agent transacting with other humans on my behalf while I have no idea what’s actually being said. The signal: if offloading the task would also offload the thought or the human connection that was the point, keep it.

—

‣ When it’s the actual decision.

This is the one I’d defend hardest. The reader I worry about most is the PM who’s been pushed to use AI across so many tasks, so fast, that they can’t keep an eye on any of it — and who, at some point, starts handing off not just the work but the decisions.

Sitting with all the evidence and making the call yourself is slow and hard, and asking the model what it would do is fast and easy. The more we give the decision itself to AI, the harder it gets to make decisions on our own. That capability doesn’t announce its departure. It just stops being there when you reach for it. The signal: if you’re about to ask AI what you should decide, that’s the line. Gather, draft, format with it. Keep the call to yourself.

〰️

Move 2: When I’m using AI across the board, here’s my blueprint

For the work that is worth handing over, I use the same four steps every time:

Think first → take notes → give the notes to AI → let it finish → check it against your own read.

The thinking and the structure stay mine. AI does the assembly and the polish. The step I guard is the first one — when “think first” shrinks to nothing because I’m rushing, I’ve stopped doing the part that keeps me sharp and I’m just approving plausible-looking output.

Reverse the order — let AI generate first, then react to what it made — and you’ve swapped generating (the strongest desirable difficulty there is) for recognizing, which feels just as smart and builds almost nothing.

The last step matters more than it looks. “Think first” isn’t only about staying sharp — it leaves me with my own read to hold the AI’s version against. So “check it against your own read” isn’t passive editing; it’s putting what AI produced next to the answer I already committed to and hunting for where the two split. That gap is where AI either caught something I missed or confidently made something up. Without my own version first, I’ve got nothing to catch it with — which, it turns out, is the same finding researchers keep landing on: the people who engage their own judgment before they see the AI’s answer are the ones who don’t get captured by it.

Here’s what the blueprint actually looks like across the tasks I run with AI in the mix:

This newsletter. I work out the argument, the structure, and my own takes before AI touches it. It drafts and tightens around my thinking and my notes; I edit every line. That’s how a piece lands at 80% me — the reasoning was done before the drafting started. The perspective, the experiences, the stories are already there — and they came from my brain, in my words.

Client presentations. I build the narrative and land on the findings myself — that’s the part clients are paying for. I write the spine in notes, I reflect on previous presentations, then let AI flesh out and polish the content into slides so I don’t have to (I’m not learning anything from slide formatting).

Data analysis. Before I open AI on a batch of transcripts or survey responses, I read enough of it myself to write my own three-bullet read of what’s going on — generated from my own head, badly if necessary. I get a feel for what I think is happening in the data. Then I run analysis with AI and compare: where did it catch something I missed, where did it miss something I caught, where did it confidently make something up. Having my own read first is the only reason I can tell.

Making decisions - “A or B?”. When I’m genuinely torn — a pricing change, which course to build next, whether to take a client on — I write out my own read first: the two options, what actually matters to me here, where I’m leaning and why. Then I hand that to AI and ask it to come at me: poke holes in my reasoning, make the strongest case for the option I didn’t pick, name the tradeoff I’m pretending isn’t there. What I never do is ask it which one to choose. It’s a sparring partner for the thinking, not the thing that makes the call — and yes, this is the same decision line from Move 1, which is exactly why I’m careful about it. The reasoning gets sharper; the choice stays mine.

〰️

AI never does the original thinking for me or makes the final call. It assembles, structures, and polishes what I’ve already reasoned through.

Why it matters: Staying in the top tier of AI users — while using your own brain — isn’t about offloading the most. It’s about offloading the right things and deliberately keeping the few difficulties that are still making you better.

🔭 The view from here

If you want the uncomfortable version of all this, the MIT Media Lab study making the rounds — “Your Brain on ChatGPT” — is worth the twenty minutes, with a skeptic’s hat on: it’s a small-sample study, so read it as a provocation, not a verdict. The sturdier read is anything from the Bjork Learning and Forgetting Lab on desirable difficulties — thirty years of evidence that the easy way has always been the illusion. AI just made the easy way available everywhere at once.

What do you want to read next?

Last thing: I’m always trying to learn from you all about what specifically you’re struggling to figure out and implement in your work. I want this newsletter to be the best thing in your inbox — tell me what you need more of 🫶

—

Keep moving.

— Caitlin Sullivan

Which of these two feels most like you?

Camp one trusts AI outputs most of the time. If it looks coherent, and the findings sound plausible, it seems good enough to roll with.

Camp two trusts almost nothing, and spends the hours AI just saved combing back through the data by hand, re-doing AI’s work to be 100% sure it’s right.

Both are extreme, not ideal, and I don’t want either one for you. But most people I work with are in camp two, and there’s a big piece of the puzzle missing in their workflows that leaves them there.

They need to get systematic about testing their AI and their workflows, so they know what can and can’t be trusted before a real decision is riding on it.

I’ve talked a little bit about testing workflows and tools more systematically before. This time, we’re talking about an essential tool that is required for nearly every test you might want to run:

Golden test sets.

Maybe you’re thinking, I already have data, do I really need some other version of it? This feels like overkill. My answer is almost always yes.

A few places your golden test sets earn the time it takes to create them (please don’t hate me for giving you a little extra work 🙃):

• A new model released — everyone swears it’s smarter. A golden test set tells you whether it does your specific task better, or if it just sounds more confident while missing the same things. Without one, you’re guessing.

• You rewrite a prompt or agent instructions — did it sharpen the output, or break something you won’t catch until it’s already in a deck? You find out before a stakeholder does.

• You’re deciding whether to keep checking by hand — once a workflow clears the bar, you trust it, spot check in specific places (not everywhere) and move on. The test set is what earns that trust: it shows the findings lined up where they mattered repeatedly, even if the AI got a few minor details wrong.

This edition covers what a golden test set is, how many you need, and how to build your first one this week.

In this edition:

• 📍 New from me — Episode with Patricia Reiners coming soon! + June Claude Code course enrollment ends in 5 days.

• 🏕️ What’s a golden test set? — diagnose where you are, with the smallest next move at each rung

• 🗺️ Build your first golden test set next week - do it now, not later. Use it forever.

Let’s get into it —

📍 New from me

Future of UX podcast episode coming soon!

• I just recorded an episode with Patricia Reiners last week all about files. Yes, files. Like, the files Claude Code and agents need to do really good work. (I promise it’s more interesting than it sounds). Episode coming soon.

  Follower her and find our chat, soon, over here: Apple Podcasts, Spotify

Claude Code for Customers Insights - enrollment ends in 5 days

• Last cohort built 16+ real research workflows together — Cohort 3 runs June 8–19, and enrollment is closing....

  → Sign up here

  
  And did you know you get a subscriber-only discount to this course? Email me if you want it!

🏕️ BASE CAMP

What a golden test set actually is

You run a new prompt or instructions set on your data. The output looks believable, so you send it along to your stakeholders.

“Looks right” tells you the output is coherent. It tells you nothing about whether it’s correct — whether the AI found what you would have, or missed the nuance that changes the decision. The only way to know is to compare it against an answer you already trust.

That’s what a golden test set gives you: it’s a study where you already know the “right answers” you’d ideally like to replicate with AI. It has the right findings, right themes, right counts, right prioritization of what to work on next based on findings — all so you can hold any new prompt, process, or model against a tangible bar instead of using gut feels.

GOLDEN = the answer key exists before you run the test.

You did the study by hand and trust the result — the findings, the surprises, the things you’d flag in a junior’s first draft. New AI output either matches that key, or it doesn’t.

The anatomy: five folders

Every test set I keep has the same five folders inside:

Why it matters: the structure is the boring part, but it’s what makes testing a 15-minute habit instead of a giant project each time. When every study looks the same, you can choose a new model or experiment setup but you’ll know exactly where the answer key is waiting.

How many test sets do you need?

Start with three — but before you ask, I bet “three of what?” is your follow-up, so…

→ A test set belongs to a workflow.

❌ The question isn’t “how many data types should we create golden test sets of?”

✅ It should be, “what’s the range this one workflow will be used and trusted on?”

You build enough sets to cover that range for that repeatable workflow, and the dimension worth covering is the one most likely to break it.

• One workflow, one data type (say, thematic synthesis of interviews) → three sets that might cover easy, typical, and very messy or complicated data or topics.

• One workflow, several data types (the same synthesis run on interviews, survey open-ends, and support tickets) → cover the types instead: then you need a few sets of each, weighted toward the messiest formats. .

You don’t need three of every type all the time. Three interview sets plus three survey sets plus three ticket sets is the best combo for a multi-source workflow, but if all of your data sets are pretty much exactly the same, you might also be proving the same thing nine times (and not testing the model’s on something more likely to make it struggle). So use your common sense here, too, and not just my numbers.

Here’s a simple decision tree for checking how many sets to target:

“How granular do I go? Separate sets for JTBD interviews vs. exploratory interviews?”

Split by failure mode, not by topic. Ask: would the workflow break differently on this?

• JTBD synthesis and usability issue-spotting are entirely different jobs — different methodology, different workflow instructions, different right answers. Each earns its own test set coverage.

• JTBD interviews about onboarding vs. about billing are the same job and methodology (JTBD) on different topics. One set probably covers both well enough.

The rule: add a set only when you can name a new way the workflow could break that your current sets wouldn’t catch. If you can’t name the failure, you’re being too granular — which is why, for most small teams, this still end up at three to five total sets, not thirty.

How many times to run each one:

It’s three again! AI output shifts from run to run, so a single pass is an anecdote, not a test. Run your workflow against a golden set three times. If it clears all three you can trust it; if it clears two, you’ve got a reliability gap to close before it touches live work.

“But this will take too long.”

It’s time well spent, I promise. Plus let’s be honest: skipping the test was not saving us time - it costs us a wrong finding shipped into a roadmap, or the hours you’ll spend re-reading transcripts to check if our AI workflow actually worked.

Curious what my folders look like? This 👇

🗺️ THE ROUTE

Build your first golden test set this week

Three is the target, but you’ll build them one at a time. For most of us, this looks like filling in documentation gaps for a study you already ran, so start there.

1. Pick a study you know well

The best first test set is a study where you’d notice if the AI got it wrong -- a past project you ran end-to-end and still remember well: the key finding that wasn’t obvious, the quote that reframed the whole study for stakeholders, the point a stakeholder pushed back on but had a lot of evidence.

Skip studies you only skimmed. If you can’t confidently grade the output, it can’t be your answer key.

2. Strip it so you can sleep at night

I always advise teams to anonymize and remove PII to the level your legal sense and your gut both sign off on — while keeping enough that the analysis still means something.

3. Write the answer key

This is the part that makes it golden, and the part most people skip.

Open your 00 folder and write down, in plain language in a doc:

• The findings — what the study found.

• The surprises — what you didn’t expect going in.

• The junior-analyst flags — what you’d correct if someone handed you a first-pass draft: missed the secondary theme, overweighted one loud participant, called a one-off a pattern.

That third bullet is a valuable addition, don’t skip it if you have time. It’s the difference between “the AI produced themes” and “the AI produced the right themes without the mistakes a rushed human would make.”

Then run one thing against it

If you set up the exact folders above for your golden test set:

• Drop one new thing into folder 02 — a fresh instruction, a skill, or a different model

• Run it on the data in 01

• Compare the output in 03 against your answer key in 00

→ 👀 Did it find what you found? Did it trip any of the flags you wrote down?

What do you want to read next?

Last thing: I’m always trying to learn from you all about what specifically you’re struggling to figure out and implement in your work. I want this newsletter to be the best thing in your inbox — tell me what you need more of 🫶

—

Keep moving.

— Caitlin Sullivan

I was in London this week running a Claude Code insights workshop at Circus Experimentation conference. A few people reached out to me afterward. One had five solid skills she’d been running for months. Two others had installed Claude Code for the first time just before our workshop. They asked me, in slightly different words, mostly the same thing:

“What would get me to the next level - what am I missing that would double the ROI I’m getting from Claude Code?” → That’s this edition.

I’ve been mapping where teams I work with sit on the Claude Code learning curve, because I keep watching smart people stall in the same places for a painfully long time.

I put together a self-diagnostic plus one specific next move from each level. Most teams stuck at levels 1 to 3 think they need to develop more Claude Code skills. They don’t. They often need different infrastructure.

The jump to L3 is where work gets noticeably faster. The jump to L5 is where it gets so trustworthy, you can actually ship what agents delivered without triple-checking every line of every output.

Most people aren’t ready for Level 5 yet, but I want you to see where you are now, where this all goes, and how you might get from here to there.

Let’s get into it —

In this edition:

• 📍 New from me — June course cohort enrolling

• 🏕️ The 5-level ladder — diagnose where you are, with the smallest next move at each rung

• 🌰 Trail mix — adjacent reads worth your week

Let’s get into it —

📍 New from me

CC Insights Cohort 3 now enrolling ✌️

• Okay, why should you even care? Because the last cohort shipped 16+ workflows together: usability coders, contradiction-flagging interview analyzers, hybrid-codebook theme coders, GDPR PII redactors, most importantly - multi-agent discovery pipelines that run reliably 95% on their own.

  → Next round: June 8–19. Sign up here

🏕️ BASE CAMP

The 5-level ladder of Claude Code use

The level you’re stuck at is the one whose stuck-here signal sounds embarrassingly like you. 🫣 Under each level I’ve added what’s still missing between you and the next rung. Based on what, you ask? Based on working with 200+ individuals learning to use Claude Code since the beginning of this year.

Most people overshoot by one rung when they self-diagnose. If you’re hovering between two, pick the lower one. I’m not judging! ✌️ I always overshoot, too, but mastering basics does matter for moving faster later.

Where are you today? A 30-second self-placement

Now the details —

Level 1 — Single skills that work, on one project

You’ve written a few skill files. They seem to produce decent output. You’d recommend Claude Code to a friend for the shiny skill-to-fast-documentation use cases.

But switch the dataset, change the research question, or scale up, and something breaks.

The signal you’re stuck here: every new project still gives you a feeling you’re starting from scratch. The skills you created before work, but only with edits to fit your latest project.

What’s missing between L1 and L2:

• A useful hierarchy of not-excessively-long CLAUDE.md files across global, project, and subfolder — and a clear sense of which information lives at which layer

• The difference between memory (cross-session) and context (this session), and when to use each

• A project setup so Claude orients itself before you type the first message

• Understanding of how to build skills for use across projects without rewriting them every time

🪜 The smallest next move:

Start with the hierarchy of CLAUDE.md files on different levels of your folder structure. It’s half a day of work for many. But after that, every session starts properly oriented and you’ll never paste context into prompts again.

〰️

Level 2 — Claude already knows your project

You’ve built CLAUDE.md files at every level - and they’re pretty darn good! You “cd” into a project folder and Claude already knows the product, the customer segments, the decision on the table. There was zero copy-pasting context this week.

This is where Claude Code stops feeling like a tool and starts feeling like a colleague who knows its stuff.

The signal you’re here:

you have skills, but you run them one at a time. You’re the orchestrator of a bunch of on-demand agents but you still have to tell each next step to run.

What’s missing between L2 and L3:

• Slash commands or agents as orchestrators — one command running multiple skills in sequence

• Sub-agents — when to launch them, where parallel work most earns its place

• Output files at the right steps so you can audit a pipeline and iterate faster

• Passing context between steps without losing the threads that matter

• What a workflow worth packaging looks like

🪜 The smallest next move:

Wrap your most-repeated workflow in a slash command that orchestrates multiple skills to run in the right order (without you triggering them). After that, one command replaces an hour or more of manual coordination.👇

〰️

Level 3 — Slash commands + orchestrated workflows

You’re not running skills manually. You type one command and a full pipeline runs: multiple skills, some sequential, some parallel via sub-agents. Output files at every step so you can audit what happened. Verification built in and an agentic process that knows when to stop to get your input.

One command. Multiple agents. And a real audit trail neither you nor Claude can ignore.

The signal you’re stuck here:

Workflows run, but Claude burns tokens wandering through your connected sources, or your team can’t reproduce your results.

What’s missing between L3 and L4:

• MCP scoping — writing tool instructions before you connect a source, not after

• The distinction between your personal Claude Code setup and a teammate-shareable skill

• Packaging conventions — naming, versioning, what belongs in settings vs. settings.local

• Knowing how to systematically and rapidly test and iterate every workflow you create - with Claude - to get to reliable workflows you’d let run on their own

🪜 The smallest next move:

Connect sources via MCP, scoped with tool instructions that prevent them from running wild through your data and target retrieval of the right data fast. (Cuts the token burn fast, too).

〰️

Level 4 — Sources scoped, workflows packaged

MCPs are connected and scoped. Claude isn’t wandering aimlessly through your sources. Your best workflows are packaged so your team can run and test the same skills, the same orchestrators. The same standards are applied every time.

The signal you’re stuck here:

You create skills and multi-skill workflows, but it feels like it takes forever to make improvements to them yourself. Workflows look good when YOU run them, but break for teammates, and you don’t know why until they tell you.

What’s missing between L4 and L5:

• Eval design — choosing what to measure (quote fidelity, finding stability, drift across data shifts)

• Multi-run testing on the same input to surface where Claude actually varies

• LLM-as-judge patterns, and the places they break

• Cross-input consistency — does the same skill behave on a new dataset the way it did on your seed?

• Knowing when output is good enough to ship vs. needs another pass

The smallest next move:

Run your most-used skill 5 times on the same input, then look at what shifts between runs. It takes a couple of hours at most (use a small amount of data, not the whole lot). The goal: stop guessing if your workflows are reliable and figure it out systematically. Like this 👇

〰️

Level 5 — Evaluating before you ship

You don’t just build workflows. You test them rigorously here. You know which skills produce stable findings and which lean in the wrong direction as soon as the data changes. You’re not handing a workflow to your team because it looked good when I ran it. You’re handing it off because you have evidence it works repeatedly.

L5 is where the workflow goes out the door without a 2am voice in your head wondering if a something got fabricated or dangerously skewed three runs back.

The leap from L1 to L5 isn’t one giant project. It’s a bunch of key steps, but each can be small enough to do in a few weeks if you target the right improvements. The reason most people stall isn’t intellectual capability, or even “terminal allergy” (yes, I realize you might be allergic to hanging out in a black chat box all day!). The biggest blocker to getting this right is this: most people are overcomplicating things.

Getting from L3/4 to L5 requires:

• Focusing on the basics of experiment design to test consistently

• Building basic evals - rubric-based measurement of what comes out of your workflows over time

• Having consistent test data sets and ideally a golden set that has the “correct” answers you can compare Claude’s answers to

• Having the key pieces from L1→L4 in place before you start running evals (CLAUDE.md files, rules files and other context files, skills with a reasonable chance of delivering good work, etc).

Getting from L1→L3 can be done in 2 weeks. Going L3→L5 depends entirely on how committed you are to thinking clearly and testing systematically.

🌰 Trail mix

A few of the coolest things I have found and used obsessively:

• Feynman — open-source AI research agent — a CLI that ships something like research skills as a standalone product. Useful reference at L4 when you’re thinking about how to package your own workflows for others to run. But it runs outside of Claude as it’s own CLI. Pretty incredible if you want to do a deep review of all academic papers on a topic and get a summary back that’s legitimate.

• Karpathy’s LLM Wiki — If you’re someone asking “how to build a repository or continuous knowledge system” in Claude Code, this might be your answer. It’s a setup for AI-maintained, compounding knowledge bases. Imagine: Your insights-person curates, then the LLM keeps the wiki current and flags contradictions along the way.

• tropes.fyi — AI writing tropes to avoid — ~40 AI-writing patterns in a markdown file, formatted to drop straight into a CLAUDE.md or reference when creating your next writing skill. Catches problematic defaults before they get sent out and make clear you’re letting AI write everything for you. 🙃

—

Keep moving,

— Caitlin

Last week, multiple product and design teams kicking off agentic AI POCs asked me how they should be thinking about this:

“If quality of what goes into the AI workflow is going to determine the quality of what comes out — not just for humans — how should we set up documentation (especially of insights) so Agents can use it successfully?”

It’s a question I’m seeing more and more. AI agents are being deployed across the whole product delivery process, not just engineering or user need synthesis — discovery into findings, findings into PRDs, PRDs into design briefs, briefs into specs, specs into code for new or existing products. A feature factory where AI does the heavy lifting between discovery and working software.

Humans are shifting to “on the loop” rather than in it — setting direction, reviewing outputs, refining the process. Not doing every handoff by hand.

These teams are hinting at what everyone running a pilot like this bumps into: Documentation habits that have worked well enough for humans don’t work well enough for agents.

And AI is becoming your primary consumer.

The problem: What you write is now read by agents, and they take it literally. And when you don’t fill in the information gaps, they’ll fill them in — badly — on their own.

This edition walks through the key things that teams running agentic POCs need to consider:

1. What good documentation looks like when AI is the primary consumer

2. How to structure findings so they flow usably across research, product, and design

3. Validation checkpoints that keep AI outputs grounded in real user insight

Plus a one-week pilot you can run immediately, checklist included.

In this edition:

• 📍 New from me — workshop at Circus in London, new course is rolling

• 🏕️ Base Camp — AI-ready docs from the ground up: the unit, the chain, the mechanics

• 🗺️ The Route — what to set up for your POC this week: Monday moves, validation, one-week pilot

• 🌰 Trail mix + 🌄 The view from here — extra reading and a model leaked last month

Let’s get into it —

📍 New from me

• Running a workshop in London: Claude Code for insights at experimentation conference Circus. May 11th. → Join me in person!

• I launched a Claude Code for Customer Insights course this year. Cohort 2 is running right now. → Get in on the next round

🏕️ BASE CAMP

What good AI-ready docs look like

The fundamentals before any pilot: the atomic unit of an AI-ready doc, how those units chain across roles, and how agents actually read the chain when they get hold of it.

The unit: a claim capsule

Whatever your team produces — a research finding, a PRD requirement, a design rationale, an ADR — the atomic unit of the findings they’re based on can often be the same: one claim, wrapped in six fields that an AI can actually work with.

One capsule per claim. One file (markdown, Notion page, Google Doc — whatever your team already uses).

The six fields

These six fields are what travel with every claim through every layer. They’re non-negotiable for agents doing real work, and they’re exactly what also helps a human spot-check faster.

Most team artifacts cover the first two or three on the list. Rationale, constraints, and verification are where agentic work often breaks.

The structure stays the same across artifact types — you adapt the content to fit each.

One capsule, filled in

Here’s a typical finding, then the agent-ready version of the same insight.

Typical finding:

Mobile iOS checkout: shipping cost is a major friction point

5 of 15 mobile iOS shoppers (33%) flagged shipping cost as frustrating during checkout — unprompted, across multiple sessions.

• “I get to the end and suddenly there’s $12 of shipping. It feels like a bait and switch.” — P3

• “By the time I see the total, I’ve already typed in everything. I just close the tab.” — P9

Recommendation: Reveal shipping cost earlier in the flow.

This is often solid enough for a stakeholder slide — a human gets the picture in ten seconds. But for an agent picking it up to run downstream workflows and decisions, source pointers aren’t atomic (no transcript timestamps to retrace), pattern strength and causal confidence collapse into one claim, no constraints scope it (mobile only? US only? new customers only?), no alternatives ruled out, no counter-evidence in view, no verification protocol. The agent confidently prescribes a full redesign — and a human on the loop has no fast way to challenge it without re-reading raw transcripts.

〰️

Here’s the same finding, with agent-ready documentation:

That’s one atomic finding. Scale it up: a PRD section is a handful of these stitched together; a design brief might be two of those capsules. The fields and structure stay constant. But one version above leaves agents to fill in gaps, and one doesn’t.

Humans are covered here, too. The capsule isn’t extra work — it’s the source everything else gets pulled from. Slack summaries can use Headline + a few key lines from other sections. Weekly deck slides pull Headline + 2–3 Rationale bullets, from all atomic findings. The PRD writeup embeds or links to multiple capsules verbatim. Execs get the Headline. Nothing has to be rewritten — everything is pulled and trimmed from the one file.

〰️

How these capsules chain across roles

A finding doesn’t stop at the person doing the research. It becomes the source for a PM’s requirement, which becomes the source for a designer’s rationale. Each downstream artifact is its own capsule, citing the one upstream.

The same claim, used across three artifacts. Any reviewer — human or agent — can trace the brief back to P3 §2:40.

Break a link and the chain fails. A PRD that drops “causal driver: moderate” gets an agent confidently prescribing full redesign. A brief that doesn’t cite the PRD produces an agent optimizing for the wrong variable.

〰️

How agents actually read your docs

The feature-factory premise relies on agents walking this chain reliably. It’s worth being precise about what that actually looks like under the hood.

The mechanics

• Memory vs. docs. The agent holds only its current context window. The chain lives in the docs — external, persistent, stateless to the model.

• Re-reads. Every run. Each query triggers fresh retrieval via RAG or direct file access. No “last session” carries over unless you explicitly pass it.

• Raw sources. The agent can pull them — but only if they’re indexed and linked alongside the capsules. “P3 §2:40” is a dead pointer unless P3’s transcript is in the retrieval system too.

• Traversal. One hop at a time. Brief → agent retrieves PRD → which cites finding → agent retrieves finding → which cites transcript → agent retrieves quote. Structure enables walking backward; without explicit citations at each layer, the agent can’t.

A perfectly-structured capsule is still a dead pointer if the chain it sits in isn’t navigable.

🗺️ THE ROUTE

Setting up for your POC this week

What to actually do — Monday morning through end of week one. Three setup moves, two checklists, one runnable pilot.

Monday: three setup moves

These are prerequisites. Skip them and the rest of the week unravels.

• Pick one retrieval scope to start. Hard rule: the agent must reach both ends of every citation. The simplest way to guarantee that in week 1 is one scope — one Notion workspace, one repo, one Drive.

  • Monday: name the scope, audit which transcripts and briefs already live in it, and flag anything stuck in slide decks, locked PDFs, or email threads.
    

  • Multiple connected sources (capsule in Notion citing a transcript in Granola) work too, as long as each citation resolves to something the agent can act on, not inline text it just reads. That’s a week-3 problem; start simple.
    

• Lock a citation format on day one. Stable IDs (P3 §2:40 — meaning participant 3, timestamp 2:40) need to survive renames, folder moves, and product-area aggregations.

  • Monday: pick the format, document it once, apply it to the next capsule. Don’t backfill old work — that’s a multi-day chore that doesn’t unblock the POC.
    

• Replace vague references with real links. “See the research deck” is a phrase; an agent can’t follow it. Citations need to be links — to the capsule, the transcript, the prior finding.

  • Monday: take the most recent PRD or brief in flight, walk every reference to a research source, and fix any that aren’t actual links before the agent touches that doc.
    

〰️

✅ Input readiness — the pre-agent checklist

Before a capsule enters any AI workflow, run these. Three or more “no’s” and the doc isn’t ready — fix it before trusting any output built on it.

☑ Headline states the claim in one sentence an agent can act on

☑ Source is specific, dated, addressable (P3 §2:40 — not “the interviews”)

☑ Confidence is labeled with counts and strength, not flattened into a recommendation

☑ Rationale names alternatives considered and what ruled them in or out

☑ Constraints are hard limits, not soft preferences

☑ Counter-evidence sits inside the capsule, not in a separate file

☑ Verification matches the artifact type — source-check / success criteria / acceptance criteria

☑ Raw sources (transcripts, source data) are indexed alongside the capsules so inline refs resolve

〰️

✅ Output spot-checks — humans on the loop

Three checks that work for humans on the loop.

• Spot-check provenance. Sample AI outputs at random. Walk each citation back to a specific participant, quote, or source doc. If you can’t trace it → AI drifted. One to two minutes per sample beats re-reading a thirty-page output.
  

• Contradiction sweep. Before accepting an output, ask the AI to find evidence against its own conclusion. If it can’t find any, be suspicious. Confirmation bias hits models as hard as humans — cheap to mitigate in the workflow.
  

• Red-team with a skeptic. One person’s standing job: ask “where’s this coming from?” for every AI-generated claim. Not the most senior reviewer — the one whose only job is trust.
  

Why it matters. Anthropic’s framing of humans-on-the-loop oversight names three requirements: timely context, intervention authority, defensible rationale. These checks give you all three — context from provenance, intervention from red-teaming, rationale from contradiction sweeps.

📍 Save this section somewhere you can refer to during your POC.

〰️

🥾 TEST RUN

Your first one-week pilot

Don’t overhaul everything. For a four-week POC — or any team easing into this — I recommend running a one-week pilot on the highest-stakes artifact you have.

Before Day 1 — Pre-flight (Monday morning, ~1 hour). Run the three setup moves from the section above: name your retrieval scope, lock the citation format, fix vague references in the most recent in-flight doc. These are prerequisites for the capsule to function as an AI input.

Day 1 — Write one capsule. Pick a finding, PRD requirement, or design rationale your team is about to produce. Write it in the six fields. Use the worked example above as your template.

Day 2 — Render for humans. Pull the Slack summary, the deck slide, the PRD section. Note: nothing got rewritten — everything was trimmed from the one file.

Day 3 — Feed it to the AI workflow. Let the agent consume it as an input. Capture what it produces.

Day 4 — Run the post-agent checks. Provenance spot-check, contradiction sweep, red-team question. Note where the output held and where it drifted.

Day 5 — Debrief. Where the trust chain held → scale. Where it broke → the failure points tell you exactly what to fix before you let agents run more of the process.

Test this: in one week, with one capsule. If the structure holds, copy the template and make it the default for the artifact type. If it breaks, the miss tells you exactly which field needs more rigor before scaling the POC.

🌰 Trail mix

Two more reads on agentic work:

• Anthropic’s “Building Effective AI Agents” resource hub — refreshed 2026 architecture patterns and implementation frameworks, with production case studies that show what key implementation looks like in the wild.

• How to write PRDs for AI Coding Agents (David Haberlah) — basically, building on what’s here and the next steps - how to write a PRD so an agent can act on it without guessing. Explicit hypotheses, hard constraints, verification hooks.

🌄 The view from here

Last month’s accidental data leak revealed Claude Mythos — a new tier sitting above Opus, described internally as a “step change” and aimed squarely at enterprise customers.

The leaked drafts describe it as designed for “deep connective tissue between ideas and knowledge” — a very specific phrase for a model built to synthesize across documents, domains, and whole research corpora rather than answer one question at a time. Too expensive and risky for general release yet; going to cyber defenders first.

Why it matters:

As synthesis-focused models get majorly powerful, the quality of the inputs you feed them stops being a minor detail — poorly-structured findings and PRDs will produce increasingly confident-sounding wrong answers at a scale this tier is built for. And that will increasingly be on us, not “them”.

—

Next edition’s already drafted, shipping a bit sooner than usual — a small test based on feedback I’ve been collecting from many of you.

Keep moving,

— Caitlin

You’re going to be in a meeting soon where someone says “we should use AI to simulate our users.” Maybe they already have.

The promise is compelling: simulate thousands of users in minutes instead of recruiting them over weeks, or months, or not at all because your B2B audience is too hard to recruit.

Test pricing, messaging, feature concepts — all without a single interview. Some studies report 85% accuracy. Vendors are selling it. Teams are talking about it. And the market is projected to hit $4.6B by 2032.

But the gap between what synthetic users can reliably do and what most teams think they can do is where bad product decisions get made.

A synthetic user that’s wrong 15% of the time sounds alright to some — until you realize that the 15% might actually be a bigger number, and that the errors cluster around the questions you might care most about: will they adopt, will they pay, will they switch?

This is the primer you read before that meeting, where someone senior suggests using whatever data you have to create synthetic simulations of your customers. I want you to know just how to challenge their assumptions. It’s a summary of many major studies in the space, plus learnings from 30+ experiments I ran myself.

I genuinely hope this helps you - not just to become more skeptical but to begin thinking more realistically about where synthetic users could have a future in your work, too. ✌️

In this edition:

1. 🧩 “Synthetic users” isn’t one thing — multiple types and what each promises

2. 🔬 What the research says when you line up the studies side by side

3. 🧭 Where this leaves you when someone asks, “can we use synthetic users?”

Let’s get into it —

🧩 What are “synthetic users”? Not a single thing.

People use one term to mean many things. There’s little consensus on terminology across HCI, marketing, cognitive science, and AI research.

The biggest variable is often the data you feed in — but not always. How you ask the question matters too. Here are the main approaches and what each requires.

Every jump in accuracy costs a jump in data investment. But the numbers in this table can’t really be compared to each other directly — and that’s one of the biggest challenges in this space right now.

A note on persona prompting: This is what most teams try first — open an LLM, describe a user, ask it to respond as that person. No real data goes in, so accuracy is near-zero (Peng et al. 2025/2026 found just 15% correspondence for demographics-only prompting). It’s included here because you’ll likely encounter it at work, not because it’s a viable research method.

📍 Tip: When someone says “synthetic users,” ask which type they’re referring to, how they think you’ll create them, and which data they’d use. The answer changes everything about what you should expect from the output.

Why you can’t compare these numbers to each other

The accuracy numbers in the table above come from different studies, measuring different things, on different tasks. “85% accuracy” from one study and “20% correlation” from another don’t mean one is better and the other is worse. They measured completely different questions.

Here’s what’s often being measured:

Normalized accuracy (used by Park et al. for digital twins) asks: “Did the synthetic user pick the same survey answer as the real person?” — then adjusts for the fact that humans don’t always pick the same answer themselves if re-asked. The 85% means the synthetic user got 85% of the way to the human consistency ceiling. Sounds high. But the raw match rate was 69% — roughly one in three answers was wrong.

Correlation (used by Peng et al., Hewitt et al., Kim & Lee) measures whether synthetic and real responses move in the same direction — but different studies correlate different things: from comparing one synthetic user’s answers to one real person’s answers across all humans/synthetics and their answers, to comparing predicted effect sizes vs. actual ones across experiments. A high correlation means the patterns track. It doesn’t tell you how many individual answers were right or wrong.

Correlation attainment (used by Maier et al. for synthetic panels) asks: “How close is the synthetic-to-human correlation compared to the human-to-human test-retest correlation?” The 90% means the synthetic panel tracked 90% as well as humans re-answering the same survey — a strong result, but specifically on purchase intent for known product categories.

〰️

Why this matters for you:

When someone at your company says “this synthetic user tool is 85% accurate,” the first question should be — 85% on which accuracy measure?

• Normalized accuracy on structured survey questions from an established social science questionnaire?

• Raw match rate?

• Correlation?

• Correlation attainment?

These give very different pictures.

Two studies can test the same method of creating synthetic responses — like digital twins built from deep individual data — and get 85% on one metric and 20% on another, because they tested different tasks. The 1000 person study’s 85% accuracy result was on structured attitude questions. Peng et al.’s 20% result was on behavioral and psychological outcomes. They used similar approaches to creating synthetics, but got very different results.

🔬 What the research says

I’ve lined up many major studies published in the last several years. The headline numbers often look impressive. The details are messier — and as I said, the input data used matters most.

Here’s what the evidence shows, organized by finding.

Finding 1: Data depth drives accuracy — more than the model or method

The Generative Agent Simulations of 1,000 People study (Park et al.) is the study that’s probably shown up most in your Linkedin feed. They created digital twins from 2-hour semi-structured interviews averaging 6,491 words per person, with 1,052 human participants.

One of their most useful comparisons - they tested how much the input data mattered:

That 14-15 point gap between persona-only descriptions and the full interview-based twins tells you data depth is a variable that matters.

Columbia’s Twin-2K-500 study (Toubia et al.) confirmed it from a different angle: they tested 12+ method variations for constructing synthetic users — different prompting strategies, different models, different data formats. Everything clustered between 67.88% and 71.92%. Fine-tuning (69.61%) performed worse than base prompt augmentation (71.72%).

What’s “prompt augmentation”? Loading real data — like interview transcripts or survey responses — directly into the LLM’s context window before asking it to respond. No model retraining required. Just giving the AI more to work with.

What’s “fine-tuning”? Retraining the model’s weights on your data — more expensive, more technical.

The authors of the Twin-25-500 study found that accuracy was remarkably similar across all synthetic user creation approaches — suggesting that once you have the data, the specific method you use to build the synthetic user matters less than you’d expect. Reading across both Park and Toubia studies, the implication is clear: data depth matters more than engineering.

〰️

Finding 2: Synthetic users get the direction right — but exaggerate how big effects are

Predicting Results of Social Science Experiments Using LLMs (Hewitt et al.) tested synthetic predictions across 70 experiments and 105,165 participants. The model hit 85% correlation with 90% directional accuracy — it usually got which way people would lean right.

But it exaggerated how big the effects were — by nearly 2x. If a real experiment found that a message changed people’s attitudes by 10 percentage points, the model predicted it would change attitudes by roughly 19. It knew the message would work — it just overestimated how well.

Direction right, magnitude wrong.

If you’re asking “will users prefer option A or B?” — synthetic users are directionally reliable. If you’re asking “by how much?” — treat the number with serious skepticism.

〰️

Finding 3: LLMs are systematically biased — in consistent, documented directions

The Mega-Study of Digital Twins (Peng et al.) tested a hugely data-intensive approach — 500+ real survey answers per person, ~128K characters of data. The result: just 20% correspondence across 164 outcomes — meaning the model explained less than 4% of the variation between people.

Five systematic distortions they identified:

• Stereotyping — over-reliance on demographic cues

• Insufficient individuation — responses cluster toward the mean. In 93.9% of outcomes, synthetic responses showed less spread than real humans. The models compress the range, making everyone look more similar than they are.

• Representation bias — better for WEIRD audiences; worse for underrepresented groups (more on this in Finding 4)

• Ideological bias — twins skew toward certain viewpoints. Park et al.’s study showed interview-based agents reduced this by 36% vs. demographic-only — but didn’t eliminate it

• Hyper-rationality — too “correct,” too consistent; real humans are messier

〰️

Finding 4: LLMs have a “default human” — and fail where real people deviate from it

The models predict a specific kind of person well and struggle with everyone else. The “default human” is rational, agreeable, and WEIRD-adjacent.

What’s “WEIRD”? It’s an acronym for Western, Educated, Industrialized, Rich, Democratic — the demographic profile most overrepresented in LLM training data and academic research alike.

This tendency toward the default WEIRD human shows up in two ways — who the person is, and how they behave.

By demographic:

A World Values Survey study (Sinacola et al.) tested across 64 countries — synthetic predictions of European respondents scored 67.6% accuracy; Middle Eastern respondents scored 54.0%. A 13.6-point gap driven by which populations are best represented in training data.

By behavior:

The Twin-2K-500 (Toubia et al.) — the starkest failures happen where humans are irrational, polarized, or non-normative:

• Vaccine refusal: 45% of real humans refused — only 4% of twins did

• Deportation support: ~45% of humans supported it — 74.1% of twins opposed

• Anchoring bias: 98.8% of twins gave the correct answer on a question where humans are reliably fooled — the model couldn’t pretend not to know the answer

The study “Out of One, Many” (Argyle et al.) tested vote prediction: 99-100% match for strong partisans, but just 2% for Independents in 2020. The model nails people with predictable views and fails on everyone in between.

〰️

Finding 5: How you ask changes what you get

The same model can give you dramatically different results depending on how you ask the question. The Semantic Similarity Elicitation study (Maier et al.) tested this directly, with two ways of asking LLMs about purchase intent - same models, same products, same demographics:

The elicitation method changed everything — across 57 real product surveys and 9,300 human respondents. The study also compared the LLM approach against a traditional machine learning model that had been trained on thousands of real survey responses. The LLM — with no prior training on this data — still won.

〰️

What about real product decisions?

“Using LLMs for Market Research” (Brand et al.) tested whether GPT-3.5 Turbo could estimate how much people would pay for specific product features:

• Rank ordering known features: GPT got this right

• Estimating how much people would pay: overestimated by 3x on some attributes ($8.20 vs. $2.60 human WTP for fluoride in toothpaste)

• Novel features: got the wrong signal entirely — predicted people would pay more for concepts real humans rejected

• Demographic subgroups: wildly off. Low-income users: GPT predicted 95% opt-out rate vs. 49% actual

• Fine-tuning: when researchers trained the model on one prior human survey about laptops, accuracy improved — including for new features like a built-in projector. But when they used that laptop-trained model to predict tablet preferences, accuracy got worse. The learning didn’t transfer across product categories.

“Can LLMs Replicate Marketing Research Findings?” (Yeykelis et al.) tested replication across 133 published marketing findings: 76% of simple findings replicated, but only ~27% of multi-factor findings did.

The pattern: when you ask synthetic users a straightforward question with a known frame of reference, the output is often useful. The more factors involved — subgroups, novel concepts, magnitude estimates — the less you should trust what comes back.

〰️

Finding 6: The best predictions combine synthetic and real - not one or the other.

The Predicting Social Science Experiments study (Hewitt et al. 2024) compared three approaches to predicting the outcomes of 70 social science experiments: human forecasters working alone, an LLM working alone, and the two combined.

Neither humans nor LLMs were best on their own. The LLM caught patterns humans missed. Humans caught things the LLM got wrong. Together, they were more accurate than either one working alone.

The practical takeaway: synthetic users aren’t a replacement for real research — they’re a layer on top of it. The teams that will get the most from this space are the ones using synthetic outputs to pressure-test, extend, and challenge their real data, not to skip collecting it.

〰️

But are the models just memorizing, or predicting?

No. In Predicting Social Science Experiments, Hewitt and team tested this directly — accuracy on published studies (74% correspondence) vs. unpublished studies that could not have been in GPT-4’s training data (90%). The unpublished studies were predicted more accurately. The models have genuine predictive capacity. But that capacity is calibrated to a specific kind of human, and degrades the further someone is from that default.

🧭 Where this leaves you

The synthetic research market is projected to reach $4.6B by 2032. Gartner predicts 75% of businesses will use GenAI for synthetic customer data by 2026, up from under 5% in 2023.

This isn’t going away.

We need to explore and understand them. Whether you feel skeptical of how synthetic users will be used or you’re curious about this space, we need to understand this well enough to help our teams make better decisions.

Every study I’ve reviewed and every experiment I’ve run points to the same conclusion: synthetic users will be part of the toolkit. The question is whether we use them with our eyes open.

Where synthetic users are strongest

Low-stakes exploration and directional questions on familiar ground:

• Rank-ordering known features or options — Brand et al. showed this works even when magnitude estimates are off

• “Will people prefer A or B?” — Hewitt et al. found 90% directional accuracy across 70 experiments

• Replicating population-level patterns for established product categories — Maier et al. hit 90% of human consistency on purchase intent

• Generating hypotheses, edge cases, or assumptions to test with real users — low cost, low risk, useful starting point

Where they’re weakest

Anywhere human decisions are emotional, irrational, or identity-driven:

• Willingness to pay — Brand et al. showed GPT overestimated WTP by 3x on some features and got the sign wrong on novel products. Purchasing decisions involve emotional friction, budget anxiety, and “I’ll think about it” inertia that models don’t replicate.

• Adoption and switching — in my own tests replicating academic studies, real users said 0/10 willingness to sign up for a given product on the spot; both models tested predicted 6/10. Synthetic users are systematically too willing to try things, too willing to switch, too willing to say yes.

• Vaccine hesitancy, political views, and other identity-laden decisions — Columbia’s study found 45% of real humans refused a vaccine but only 4% of twins did. LLMs can’t simulate the accumulated distrust, lived frustration, or personal history that drives these choices.

• Underrepresented user segments — accuracy drops for non-WEIRD audiences across every study reviewed. If you’re building for users who aren’t well-represented in English-language internet text, synthetic users will default to someone else’s preferences.

• Anything genuinely novel — Kim & Lee showed in their study on AI-Augmented Surveys that accuracy dropped from 98% on familiar questions to 67% on questions never asked before. Brand et al. found fine-tuning on laptops made tablet predictions worse. If the market doesn’t have established patterns yet, the model fills gaps with optimism.

The hidden problem most people miss

Even if synthetic panels perfectly replicated human survey responses — say, 90% accuracy — what would that mean?

Purchase intent surveys have four decades of meta-analyses showing they don’t reliably predict actual purchasing behavior. A synthetic panel that perfectly replicates survey data is perfectly replicating humans’ own poor predictive ability. You’ve automated the gap between what people say and what they do.

Communicating what this means to your team

Synthetic users are not a shortcut around real research. They are a tool that gets better the more real research you’ve already done — and worse the less you have.

If you want to set your team up for success, start here:

• Name the decision. What’s the specific question your team wants synthetic users to answer? “Will users switch?” is a very different problem than “which of these three features matters most?” — and the research shows synthetic users handle the second far better than the first.

• Audit your data. Do you already have the kind of deep, relevant data that would support that prediction? Interview transcripts, behavioral data, rich survey responses? If not, that’s your first investment. Start collecting good data first.

• Test before you trust. Run the synthetic approach against real data you already have. Hold back answers and see if the synthetic user can predict them. If you can’t measure accuracy on known ground, you can’t trust the output on unknown ground.

If your team is evaluating a synthetic user platform

Many teams won’t build their own — they’ll buy a tool. If that’s your situation, ask three questions before signing anything:

• Where does their data come from? Are they building synthetic users from your data, their own, or population-level datasets? The source determines the ceiling on accuracy.

• How is your data handled? If they want you to upload interviews or customer data, vet their storage and security the same way you would any research platform.

• Can you see how they measure accuracy? If there’s no transparent methodology — or if they quote a single accuracy number with no context on what it measures — the number is marketing, not evidence.

Lastly, if your team is evaluating synthetic users or already using them and has specific questions, feel free to comment or reply to this email.

If you’ve made it this far, I’d love your input about the future format of this newsletter 🙋‍♀️

—

Have a productive start to April ✌️

-Caitlin

Is your model the problem, or your workflow?

This edition is one I hope will inspire you to think about how to make sure your insights workflows really work—to know that you’ve chosen the best model for the task, and that your process isn’t getting in your way.

Despite that LLMs are non-deterministic (and variable outputs are a given), there is a way to know whether your AI workflows are likely to give you consistent outputs—or whether you’ve just been getting lucky with a couple of runs.

I’ve been connecting multiple LLMs through Claude Code and pressure-testing my research workflows across models. It changed how I build, evaluate, and trust everything I ship to stakeholders.

But almost no one I know is working like this, and that’s a problem. If we’re running workflows that are more likely than we expect to trigger highly variable outputs across runs, we can’t rely on the insights from today’s LLM session as much as we believe we can. I want to help you fix that.

Then, the news is getting harder for all of us to keep up with. I’ve picked a handful of stories you need to know about.

In this edition:

1. 🧰 Workflow upgrade: How to know if your AI workflow (and model) actually works for the task—five multi-model use cases.

2. 🙇‍♀️ I was featured! Two top Product Management sources in one month.

3. 📰 AI news: The battle over ads continues, an AI legend raises money for physical spaces AI (and I promise it’s relevant), and much more

Let’s do this —

WORKFLOW UPGRADES

🧰 Using multiple LLMs through Claude Code - 5 use cases

When was the last time you systematically tested your AI discovery workflows? I mean, really tested different versions, across different models, and figured out which combination gives you the best results?

The common scenario: we run a prompt, the output looks reasonable, we keep using it. Maybe we tweak a few words, run it again, get something similar. Good enough.

The real question is whether your workflow produces consistent, reliable results—or whether it just happened to work alright that one time.

A few months ago, I started connecting multiple LLMs through Claude Code—and it changed how I build and trust my discovery workflows.

If you’re new to this, Claude Code lets you plug in APIs to connect other LLMs—ChatGPT, Gemini, and more. Instead of being locked into using only Claude models, you can run the same workflow across multiple platforms’ models in one place. Same data in, same workflow instructions, different models processing it. No switching between tools, no copying and pasting between chat windows.

Here are use cases the multi-model setup unlocks:

1. Cross-model evals

The most important to me lately. I run the same workflow—same data, same requirements, same instructions—across Claude, ChatGPT, and Gemini. If the outputs clearly converge, the workflow is solid. I want to see that my workflow forces non-deterministic models to behave in predictable ways. Most people blame the model when they get inconsistent results. But if your results diverge too much across runs, the workflow could be the problem.

If we see one model perform particularly well at a task (and we trust our workflow), it’s a signal that model could be the better fit for the task.

2. LLM-as-judge

After one model does the heavy lifting on a task, I can hand that output to a second model to evaluate the first one’s work. Where was the task done too hastily? What other solutions are there that the first model skipped over?

It’s like a skeptical colleague who’s also reviewed all 20 of your customer calls. The critiques aren’t always perfect—your judgment still needs to be in the mix—but they surface blind spots that self-review misses.

3. Ensemble analysis

You run the same process across multiple models, then looks at what’s common across all outputs - and calls that truth. Example: themes that all three models independently surface are your highest-confidence findings. Themes that only one model catches go on the “investigate further” list.

Same logic as triangulation in research. You wouldn’t trust one interviewer’s interpretation as multiple people who agree on what’s true. A multi-model ensemble applies that rigor to AI-assisted discovery process. The convergence pattern becomes the finding—not any single model’s output.

4. Model routing

Not every step in a research workflow needs the same model. Some are better at planning fast, light experiments. Some are stronger at structured coding tasks. Some produce the best arguments for a product decision for specific stakeholders.

When you run evals across models, you start mapping which model handles which task best—then route accordingly. The expensive model handles complicated triangulation. The fast model handles pitching solutions to your CEO. You stop asking one model to be good at everything and start building a system where each step uses the best tool for the job.

5. Red-teaming recommendations

When I want to be truly certain of a product recommendation I’m handing over, I give the recommendation set plus the raw data to a different model and ask it to build the strongest possible case against the recommendation. What would have to be true for this to be the wrong call? What did the data say that contradicts this direction?

It’s a structured pre-mortem powered by a model with no loyalty to the original conclusion. When the counter-case is weak, the recommendation is solid. When it’s strong, I go back to the data before anyone makes a big, bad decision.

—

Multi-model access turns Claude Code from a single-model tool into research infrastructure.

〰️

FEATURES

🙇‍♀️ I was featured by Lenny and Aakash

Two very cool guest spots went live for me this month. I’d love to share them with you—including guidance I hope helps more people feel a little more caught up.

: “How to do AI analysis you can actually trust”

I wrote a guest post with the Lenny’s Newsletter team on the failure modes that break AI-assisted analysis—and the fixes for each one. It covers how to catch AI quote hallucinations, why models default to generic themes, which LLM fabricates the most, and the verification pass that stress-tests everything before it hits a stakeholder deck.

Important: this isn’t a guide to using the web ui, it’s a series of ingredients to use wherever you interact with LLMs running analysis for you - whether that’s chatgpt.com, Claude Code or setting up system prompts for n8n agents.

👉 Read my guest post on Lenny’s

—

: “How to do AI-powered discovery (step by step with live demo)”

Aakash had me on to walk through an AI-powered customer discovery workflow —live, on screen, with test data sets. No slides. No theory. Just an actual workflow running in (almost) real time. From a prompt sequence in the web chat, to the Claude Code semi-automated version (twice as fast).

👉 See the workflow with prompts

〰️

NEWS

📰 Perplexity walks away from ads—and the AI trust debate gets louder

Perplexity was one of the first AI companies to run ads alongside chatbot answers. Now they’re pulling back. They seem to have realized that once ads appear in results, users start questioning whether the responses are honest or commercially influenced.

The timing is notable: OpenAI recently started testing ads in ChatGPT. Anthropic ran Super Bowl ads mocking the practice and committed to staying ad-free. The AI industry is splitting into two camps on monetization, and the dividing line is trust.

Why this matters: If you’re using AI tools for customer research or decision support, the commercial model behind the tool shapes the output. An AI search engine optimizing for ad revenue has different incentives than one optimizing for accuracy. Pay attention to which tools have a commercial layer between you and the answer—especially when you’re making product decisions based on what they return.

📰 World Labs raises $1B to build AI that understands physical space

World Labs—founded by Fei-Fei Li, the Stanford computer scientist whose ImageNet dataset helped launch the modern deep learning era—just closed a $1B round including a $200M investment from Autodesk.

Their product, Marble, is a “world model”: you give it text, photos, or video and it generates navigable, editable 3D environments. Not flat renders. Actual 3D spaces you can walk through, modify with natural language prompts (”add a window here,” “change the lighting to evening”), and export to game engines, VR headsets, or design tools.

But I don’t think gaming and architecture are the only applications here.

Where this could head for product and design teams:

• Physical product prototyping. Describe a product, space, or environment in text and get a walkable 3D version to iterate on before building anything. Concept testing for hardware, retail layouts, packaging—without touching 3D modeling software or waiting for production of the real thing.

• Contextual research. Reconstruct the physical spaces your users live/work in from photos or video. Help your full team understand the spaces people operate in—how they move through environments, what’s around them, what physical constraints shape their behavior.

Why this matters:

For product and design teams working on physical products, spaces, or experiences—retail, healthcare, hardware, hospitality—this is the start of being able to prototype and test spatial concepts as fast as you can describe them, and model how humans will exist within the space or context designed. No 3D modeling skills, no weeks of production time. That changes how quickly teams can go from research insight to testable concept.

📰 Anthropic dropped six major releases in 11 days

Absurd month. More shipped in a few weeks than most companies manage in a year.

• Sonnet 4.6 delivers ≈Opus performance at ~60% of the cost and the same 1M-token context window. For most insights workflows, you won’t feel the difference.

• Claude Code Security scans vulnerabilities like a security researcher —tracing data flows, catching business logic flaws, re-examining its own findings to filter false positives.

• Scheduled tasks + plugins turned Cowork into a digital employee. Describe a recurring task, set a cadence, it runs without you. Many enterprise connectors shipped — Google Drive, Gmail, DocuSign, and more.

• Remote Control lets you start a Claude Code session in your terminal and continue from your phone.

• Auto-Memory gives Claude Code persistent memory across sessions — build commands, debugging patterns, preferences — loaded automatically every session. This is changing my world right now.

• Agent Teams: many Claude Code agents that can share context, and coordinate on complex tasks while running in parallel without constant human intervention. That last part makes me nervous, but I’m getting results (even without me in the loop) that felt far-fetched even 3 months ago.

📰 Gemini 3.1 Pro is upping the multi-modal game

Gemini’s 3.1 Pro arrived, boasting “more than double the reasoning performance of 3 Pro”. Where I’ve noticed some of the biggest improvements from 3 to 3.1: UI. They highlight svg animation file improvements, but there are big jumps in layout, component structure, visual hierarchy, ability to follow highly specific instructions for all kinds of image generation across my tests. This changes what you can realistically prototype with a prompt, or how much fussing you’ll need in Figma later (possibly none?).

📰 FDM-1 learns to use computers by watching people

Standard Intelligence’s FDM-1 learns computer tasks from video, then replicates what it sees. Built on an inverse dynamics model trained on 40,000 hours of labeled screen recordings, which then auto-labeled 11 million hours of internet video — the data FDM-1 was actually trained on.

It already handles multi-step CAD operations, website navigation, and — most interesting — discovers UX bugs through exploration. In testing, it found a duplicate wire transfer vulnerability by navigating deep into an app’s state tree unprompted.

Why we should care: A model that watches screen recordings, learns from what’s happening and copies it, and pinpoints what’s breaking — in accurate detail — is not far from one that watches usability tests and flags where users struggle, then rebuilds the broken UX. FDM-1 is built for task execution, not behavioral analysis — but comprehending video at the interaction level is exactly the foundation video-based UX analysis needs. Keep this on your radar.

〰️

If you’ve made it this far, I’d love your input about the future format of this newsletter 🙋‍♀️

—

Have a great weekend ✌️

-Caitlin

We’ve moved!

In April 2024, I ran a demand test to see who wanted a newsletter like this. The first subscribers paid for this to exist.

That starting point determined the platform I chose and how I set things up. But now that it’s free, I have other ideas. So we’re starting 2026 on Substack. ✌️

This year is kicking off strong: ChatGPT is adding ads, and Claude is integrating everything. Anyone who hasn’t played around with MCP integrations yet likely will now that they’re accessible in Claude’s desktop app—zero technical understanding required.

I ran a quick test chaining Notion and Figma together in one conversation. Two simple prompts, two tools called, one workflow turning feedback sources into a map of which data is worth comparing.

Also, the difference between prompts, skills, and agents — so anyone still using Claude the hard (spontaneous prompting) way can switch to easier modes.

In this edition:

1. 🖥️ Claude Code Skills: Reusable workflows, easily updated.

2. 📰 News: Big updates in Claude and ChatGPT.

3. 📂 On My Desk: How to write specs that actually work for AI agents.

Let’s dive in —

WORKFLOW UPGRADES

🖥️ Claude Code Skills

Reusable workflows, easily updated

Most people using Claude are still typing the same instructions over and over. “Follow our brand guidelines.” “Create an interview guide using this {basic} interview analysis framework.”

There are a few ways to make everything you do a repeatable system - Skills are one of them. But first, let’s clear up any confusion about where Skills sit among other Claude Code options.

The actual difference between prompts, skills, and agents:

• Prompts are one-off instructions. Good for single requests.

• Skills are reusable packages Claude loads when relevant. Not a single prompt—a whole folder of useful pieces.

• Agents (or subagents) are independent workers you delegate tasks to. They can use Skills.

What’s is a Skill made of?

A Skill is a folder containing multiple files that work together:

• A Markdown file with instructions (the “how to do this” part)

• Scripts that execute specific steps (Python, shell, whatever)

• Resource files / assets (templates, examples, validation rules)

This is different from saving a prompt. A prompt tells Claude what you want. A Skill gives Claude the instructions, the tools, AND the resources to actually do it consistently.

One of Anthropic’s early Skills published was for creating Slack-optimized GIFs. It included not just instructions, but a validation function that checked whether the output actually meets Slack’s size constraints. The Skill packages the knowledge AND the quality check.

That’s one of the simplest ways to start using Skills for customer research: package knowledge, frameworks, and quality checks all in one.

How they load (the smart part)

When you start a session, Claude scans available Skills—consuming minimal tokens per skill to read the metadata. Claude only loads the full instructions and files when a skill becomes relevant.

This is the opposite of dumping everything into a system prompt and hoping for the best. Plus if you’ve used Claude Projects, everything you store in your Project’s knowledge base files gets loaded every time—wasting tokens if every file isn’t always needed.

When to actually use them

You want a Skill when you catch yourself typing the same instructions across multiple steps and conversations. Three signs you need one:

1. You’ve been copy-pasting prompts between sessions

2. You have a workflow with specific steps Claude keeps getting wrong

3. You want consistent outputs for a repeated task (PDF exports, data analysis, brand compliance)

Here’s what they end up looking like in my terminal when I turn Skills into slash commands. I type “ / “ and the name, then the workflow runs the same way every time.

Where to start

I’ve been building a Skills library for PMs, designers, and researchers who want Claude Code to systematize workflows exactly the way you want them done.

Inside: skills for interview transcript analysis, competitive research workflows, and document transformation. A few from me, a few borrowed from others like Notion and Atlassian. Each one is a folder you download and drop in—instructions, scripts, and resources ready to go. I’ll be adding to this starter set.

I’ve heard from non-technical friends that many haven’t gotten past the “typing instructions manually” phase. They didn’t have time to figure out how to set up anything consistent. Skills are how you cross that gap—by packaging what you already know into something Claude can reuse.

If you have Skills you’d love to share publicly in adatabase (rather than GitHub), let me know. I’d love to crowdsource this.

Claude Code Skills Library

The best part: Skill packages can be easily kept updated.

I’ve gotten sick of keep my library of prompts updated. Doing this in the terminal with Claude is easy. Say you’ve used a Skill, but something changed - you want an extra step in the instructions, another layer of verification, or for Claude to check one more source before calling the workflow done.

You’ll say, “update the Skill with those new instructions” or explain whatever needs to be improved. It will ask you for permission before editing files, and you’re done.

〰️

I have a live session on the agents part of this coming up

If you want to see what agents in Claude Code can do — running customer research steps in parallel and cutting time dramatically — this session next week is free ↓

Join the session

〰️

NEWS

📢 ChatGPT is launching ads.

In a few weeks, US users on the free tier and the $8/mo plan will see ads at the bottom of ChatGPT’s answers. OpenAI says the ads won’t affect the actual responses, and no conversation or personal data gets shared with advertisers. (I’m skeptical, but let’s see what happens).

For context: OpenAI hit $20B in revenue for 2025—more than 3x their 2024 numbers. Sounds huge. But Meta made $180B+ from ads last year. Google made $295B. OpenAI is still tiny.

Why this matters: If you’re running customer research through ChatGPT, nothing changes yet—ads don’t touch your data or outputs. But it’s a signal. Free tiers will get noisier. If you’re handling customer data—your employer’s or your own for side projects—this is a new moment to ask yourself whether free tools are the right home for that work.

🔁 Claude's MCP apps: live in the desktop app

Anthropic just made Claude’s MCP integrations a lot easier and more useful. Instead of just connecting tools, you can now interact with them directly inside Claude’s desktop app—no tab-switching or terminal MCP/API setup required.

What’s connected now: Amplitude, Asana, Box, Canva, Clay, Clickup, Figma, Fireflies, Hex, Notion, Slack, Vercel, Wordpress, and more. (Full list in Claude’s directory.)

Why it matters: You can chain tools together in a single conversation. Ask Claude to pull information from one app, then send it to another—all without leaving the chat.

—

A quick test I ran:

I prompted Notion to pull every piece of customer feedback I have on my AI courses and client training across my workspace. Then I asked Figma to map those sources visually—showing which are related, which are comparable, and why.

One chat. Two tools. No terminal. 🙃

Watch the 2-min experience.

This isn’t a workflow yet—just me poking at what’s possible. When I was a designer, it took me ages to get from finding the right feedback to analyzing it, mapping it, and showing my team what we have visually. I’m seeing the potential to get from “what do we know?” to understanding what we need to dig deeper into—faster.

〰️

On My Desk

📂 How to write specs that make better AI agents

Addy Osmani published a guide on writing specs for AI agents worth bookmarking. The core insight: vague prompts still don’t work, and piling on more instructions degrades performance. He gets into the details of a workflow to fix that.

What works instead:

• Start high-level, let the AI elaborate the details

• Break tasks into modular prompts instead of one massive instruction dump

• Build in self-checks with three tiers: always do, ask first, never do

• Treat specs as living documents—update as you learn what breaks

If you’re setting up agentic workflows, this is a solid guide on writing instructions that make them reliable.

—

And Q1 is rolling. See you next time.

-Caitlin

How well can Gemini 3 understand and analyze videos?

In November, I shared a snippet of the early findings from some recent tests.

I’ve been pretty excited about how well Gemini 3 actually read and generated the correct information about what was happening in video files I gave it last month.

There’s been a big gap in most people’s analysis processes: getting AI to help with user testing to notice customers’ body language and expressions as feedback - beyond the transcript.

I think everyone reading this newsletter knows that’s a really important piece for measuring real signal — capturing not just what people say, but what they do in key moments like looking at an early prototype of big product changes.

After running Gemini 3 tests on six videos in November, I kept going.

I ran 60+ micro tests with 20 additional videos last week — from Hotjar screen recordings, and clips of user tests with hard-to-read text or fast interactions on screen, to long-form podcast interviews borrowed from the internet.

Before many of you go on a break, I want to catch you up on Gemini 3’s video analysis capabilities. In ≈15 minutes, you’ll know exactly what to use it for (and how to test it on your own).

In this edition:

1. 🎥 Gemini 3: What works, what doesn’t, how to get the best out of it.

2. 🧑‍🔬 Rapid Testing New LLM features: How I ran 60+ Gemini 3 tests in <2 hours (including documenting it all)

Let’s dive in —

WORKFLOW UPGRADES

🔄 Gemini 3: What works, what doesn’t, how to get the best out of it.

Can it actually figure out what’s happening in your user videos for you?

Short answer: Only if you keep videos short.

I ran 60+ micro-tests on Gemini 3 with a mix of video types to figure out if it could:

1. Identify what people were doing (not saying) - people’s body language, facial expressions

2. Accurately describe what was happening in their context - the environment, test materials (ex: app views, landing pages), and mouse movements

3. Interpret people’s behaviors — and whether it’s interpretation aligned with mine. (Less essential here, highly subjective).

My test set was diverse:

• ✅ 20+ unique videos: user tests, Hotjar screen recordings, interviews, and podcasts - many covering completely different subject matter and test materials

• ✅ Ranging from <15 minutes to 1+ hour

The promise 💭

In early tests, I was impressed. Gemini 3 picked up facial expressions, on-screen behavior, even background situation details with surprising accuracy.

I thought: this could really help us move faster through user tests with fewer human hours — like mapping out problematic user paths to compare across tests and find gold nuggets faster.

Here’s what I found —

〰️ What worked — and what didn’t

✅ Short videos (<20 mins): Surprisingly good results

Even when text on screen was grayed out and video quality was mediocre, Gemini’s ability to read and understand the content in short videos was consistently high.

• ≈85% accuracy on observations across short video tests

• Gemini 3 caught things like:

  • Content on-screen (prototypes, detailed text, browser tabs vs. apps)

  • Facial expressions and body language

  • Background context (where the person was sitting, lighting, movement)

• Interpretation of tone and emotion was hit-or-miss — but my interpretation is also subjective. It’s not a hardcoded eval.

→ If your video is short, Gemini 3 can save you time by pinpointing specific user reactions, mapping journeys and finding noteworthy events in tests.

❌ Long videos (>30 mins): Not just yet.

• Every long video I tested produced obvious hallucinations

• Gemini fabricated behaviors, scenes, or whole parts of the conversation

• One response claimed a participant was “gesturing with open palms” — the guy was literally sitting still the whole time

• While technically, we can upload a 45-minute video, the output was consistently hallucinated for 20+ minute clips.

• 1+ hour videos can’t even upload.

〰️

🖼️ An example

Let me show you the difference between the outputs from a 45-minute version of a Rich Roll video podcast episode vs. a 15-minute snippet containing the same content and request from me.

Q: “What changes at around {same specific point in the video}?”

—

Here’s what was actually going on in the video:

Australian pro swimmer and musician Cody Simpson shifted from interview mode at the table with podcast interviewer Rich Roll to playing guitar on a sofa.

Pretty obvious context shift.

Gemini’s output from the 45-minute video 👇

This is obviously wrong. It’s generating what would be reasonable to expect at this point in the conversation — not what it gathers from the actual video.

〰️

Gemini’s output from the same video — a 15-minute clip 👇

Compared to what I see and wrote down before the test, this is correct. ✅

Bottom line: Gemini 3 is not a tool for long-form video analysis yet.

So, should you use it?

If you’re a Researcher, Designer or PM who wants to:

• Skim user test reactions quickly

• Map key issues across some prototype views

• Pull out screens, behaviors, facial cues, tone

  
  — then yes, Gemini 3 can help but only if your videos are under 20 minutes.

Want to run a full 1-hour user test and have Gemini 3 write the highlights?
Don’t. It’ll make stuff up.

TESTING AI TOOLS

🤖 Your turn: Run 50+ micro-tests in Gemini 3 in <2 hours

Keep your test simple.

The key: You’re not testing whether Gemini 3 can do 10 tasks with videos. You’re testing one workflow question —

“Can Gemini accurately describe what is observable in the video at specific timestamps and around specific topic discussions?”

That’s it. If it can’t do that consistently, we can’t use it for decisions based on non-verbal observations without a lot of human involvement.

Here’s how I run tests like this as efficiently as possible:

〰️

Pick your test data set

Examples:

• 5 short videos under 20 minutes

• 5 long videos (40m+)

How you get 50+ tests fast

• 10 videos × 5 moments each = 50 micro-tests

• A “moment” = a timestamp window where something meaningful happens
  
  Alt moment: a specific point in the test/conversation you are hoping it can spot and observe correctly by asking about topics, prototype views, etc.

〰️

Protocol (the exact loop)

Step 1 — Skim and mark moments (≈30 min)

For each short video:

1. Skim (don’t watch everything)

2. Pick 3–5 moments where:

   • you can observe a clear facial expression or behavior (laughter / tension / confusion)

   • a UI screen or specific text is visible

   • text or visuals are less clear, due to video quality or grayed out UI…

3. For each moment, write:

   • Timestamp range (e.g., 06:28–06:35)

   • What you see (facts) (e.g., “smiles, looks away”, “clicks green button”, “privacy policy page, hovers over specific text: {note}”)

   • Your interpretation (optional) (e.g., “seems unsure / amused”, “hesitated, unsure which nav option to click”)

💡 Speed rule: if you can’t describe the moment in 1–2 lines, pick a different moment.

〰️

Step 2 — Run micro-tests in Gemini (≈60-70 min)

You’ll run one prompt for up to 5 moments, and one video at a time (fast, consistent, comparable).

Micro-test question types (pick one per moment):

• “What is happening between them at [timestamp]?”

• “What is [person]’s facial expression at [timestamp] and what might it signal?”

• “What changed right after [timestamp]?”

• “What UI/text is visible at [timestamp]?”

• “Are they aligned or talking over each other at [timestamp]?”

This is how you keep each test tight and scoreable.

Step 3 — Scoring it: Check what holds up (≈20-30 min)

For each micro-test, grade Gemini’s answer in a few seconds:

• ✅ PASS = key facts match what you noted (behavior + scene + interaction)

• ⚠️ PARTIAL = mostly right but misses/warps 1 important detail

• ❌ FAIL = hallucination / wrong scene / wrong behavior / wrong interaction

Hard rule: if it invents something big (like gestures, phase changes, “wrap-up,” etc.) → FAIL, even if some details are correct.

SHIFTING INTO 2026

👩‍🏫 AI Analysis results that hold up for tough decisions

I’m kicking off 2026 with two versions of my highly rated AI Analysis course:

• AI Analysis for Researchers & Designers

• AI Analysis for PMs

They’re built for the realities of each role and the decisions you’re responsible for.

I’ve run the course 5x in 2025 and revamped the curriculum to deliver even more progress in just a few weeks ✨

The best part of each one:

Every live session is a working session — bring your data, and we’ll turn it into insights during the call with a system you can repeat.

You end every week with progress.

Explore the courses →

P.S. The PM course version is new. If you’re a PM and you want in, hit reply with “PM” and I’ll send you a private early rate (limited spots).

Until next time. Have a smooth slide into the new year.

-Caitlin

It's been a wild year with AI.

Despite many still claiming that AI doesn't do research well, I've seen huge gains in 2025 across all my tests—from simple things continuously done poorly in the past (does AI count correctly?) to more complicated things (AI recognizing facial expressions and body language in videos).

It's not the end of the year yet, but with holidays around the corner and that end-of-year crunch time, I want to wrap things up for you while I still have your attention. (December's edition will be a little more relaxed).

I’ve highlighted what I believe are the most meaningful changes we've seen in AI for customer research, and how I think 2026 is shaping up.

In this edition:

1. 🔄 Model updates: ChatGPT 4o → 5.1, Claude Sonnet 3.5 → Opus 4.5: What's the big deal for research tasks?

2. 🎥 Gemini 3: Holy moly, this video analysis works (so far).

3. 🤖 AI Moderators: Can we trust them now? A comparison vs. my Dec'24 study

4. ⚙️ Agents: Are we any further than in 2024?

5. 👩‍🔬 Studies: Synthetic users. Where are we now, and what's reliable?

6. 🔮 Looking ahead: What 2026 already promises.

Let’s get into it —

MODEL UPGRADES

🔄 ChatGPT + Claude

ChatGPT 4o → GPT 5.1: Is this a big deal?

Yes. This one actually matters for research workflows.

The upgrade from GPT-4o (what we had in January) to GPT-5.1 (released November 2025) has two changes I particularly care about:

1. Context windows ≈tripled.

The API now supports 400K tokens—that's roughly 300,000 words. But the actual upload token limit is 272K - still a lot. You can still upload more entire hour-plus interview transcripts without chunking them up.

Fragmented analysis where AI "forgets" what was said earlier in the conversation has been noticeably reduced. The model’s ability to continue performing its best over a long analysis chat has improved.

2. Hallucinations dropped.

OpenAI reports 45% fewer factual errors versus GPT-4o. On open-ended factuality benchmarks, GPT-5 achieves just 1-2.8% hallucination rates versus 5-23% for predecessors.

(Keep in mind these are measures by OpenAI’s team, so there’s probably a bit of bias baked in, but I’ve seen improvements in all my tests).

〰️

My verdict: These improvements have meaningful compound effects for research tasks, where performance consistency and factuality are essential. I’ve consistently seen ChatGPT get better at maintaining accuracy and focus on the right task details over longer chats than it did before.

Claude Sonnet 3.5 → Opus 4.5: Is this a big deal?

Also yes, but for slightly different reasons.

Anthropic's progression through 2025—from Sonnet 3.5 to Opus 4.5 (released this month)—brings real improvements for qualitative analysis workflows.

1. Context windows expanded significantly here, too.

While Opus 4.5 maintains a 200K token standard window, Claude Sonnet 4.5 now offers 1 million tokens in beta—approximately 750,000 words. That's dozens of interview transcripts in a single analysis.

The new Memory function improves the ability to build knowledge bases that maintain consistency across conversations—directly applicable to continuous insights processes.

2. Deep research improved by 15%.

Deep research improved by ~15 percentage points when combining Opus 4.5 with Anthropic's new agentic features (effort control, context compaction, and advanced tool use).

The model specifically targets complex tasks, achieving state-of-the-art results on multi-step reasoning tasks that combine information retrieval, tool use, and deep analysis.

3. Hallucination findings are mixed.

Anthropic positions Sonnet 4.5 as having "lower rates of hallucination".

However, one academic analysis found Claude Opus 4 exhibited ~10% hallucination rate versus under 5% for Claude 3.7—bigger models don't automatically mean fewer errors.

🙋‍♀️ Tip: Always instruct Claude to use only provided documents and cite specific quotes. That's still the best mitigation, regardless of model.

MODEL UPGRADES

🎥 Gemini 3: Holy moly, this video analysis works (so far)

I just wrapped up my final AI Analysis course cohort for the year, and got quite a few questions about Gemini 3.

But I hadn't tested it yet. So I did that this weekend.

Let me tell you (1) what Gemini did that impressed me, and (2) how I repeated one of my regular tests in just 25 minutes. 😄

1. Gemini can actually see your expressions on video.

Previously, while other models and research-specific platforms claimed they could, they never panned out. Every time I uploaded a video to any of the major LLMs, responses were entirely fabricated.

Gemini 3 got observations right. It nailed:

• Hand positions

• Facial expressions

• Reasonably guessed the meaning behind expressions (similar guesses as my own, jotted down quickly beforehand)

• Repeated this across 10 different points in a single video, repeated with 6 videos

2. How I test this (in all the models):

I have a core set of participant videos I use repeatedly for AI video tasks.

Every few months, or when a hyped new model launches, I re-test:

• Same videos

• Same requests—tell me what the person is doing, describe their facial expression, tell me what it means.

• Example request:

  
  “At timestamp 9:34…what is the participant doing? Observe their body language, facial expression, and anything else you notice. Describe them, then interpret what you think they mean. Compare with what they are saying at that timestamp."

None of the models have done this successfully until now.

This is me, talking to a friend about parenting + business. Video frame at 1:05.

I prompted:

“What do you observe about this participant's facial expression and body language at 1:05? Compare non-verbal behavior with what they are saying at that time.”

Gemini 3’s Response

Whether you agree with Gemini’s interpretation of what my expression/behavior means or not, it correctly identified my position and facial expression in the video every time I asked for non-verbal observations ✅

This held true across 60 non-verbal observation requests over 6 videos so far.

⚠️ The reality check:

Gemini 3 achieves 87.6% on Video-MMMU (state-of-the-art), but this benchmark tests knowledge acquisition from educational videos—lectures and tutorials—not behavioral observation in customer research.

There is no standardized benchmark for observing participant body language in research videos.

WORKFLOW UPGRADES

🤖 AI Moderators: Can we trust them in 2026? Comparing today vs. my Dec.’24 study.

Remember my AI Moderators study from December 2024?

Quick refresher on what I found then:

• Tools were best used as a "middle ground" between surveys and interviews ✅

• AI moderators’ follow-up questions were hit-or-miss 🫤

• Some tools allowed follow-up instructions while others didn't - giving guidance worked best

• Participant experience was mixed—repetitive questions, canned responses, abrupt endings.

• AI still needed heavy guidance for study setup or it wouldn’t do nearly as well as a senior researcher.

〰️

What's changed in 12 months?

1. True voice-to-voice interviews is standard.

Many platforms like Listen Labs, Userology and more now enable users to speak with a voice that sounds human. I updated my overview of “23 AI moderator tools to know”, in case you missed it, marking who does voice to voice and doesn’t.

2. Follow-up capabilities + study setup got upgrades

These were some of the biggest pain points in my Dec'24 study. Let me compare:

Some best practices guides now warn about over-probing—the concern shifted from "not enough follow-up" to "AI can and will dig in incessantly if you let it." 😅

〰️

🙋‍♀️ Tip: If you set up specific areas to probe in, make sure they are clearly a level deeper or in some way different from the main question they follow. Otherwise, the participant will feel like they’re getting the same question four times.

AI AGENTS

⚙️ Agents: Are we further than in 2024?

Short answer: Yes, but unevenly. Fully autonomous end-to-end research workflows remain largely out of reach.

What's emerging but not proven

Dovetail announced AI Agents in a closed beta that can perform automated actions like sending monthly Voice of Customer summaries, flagging issues, posting alerts.

But this is an example of predefined workflows, not truly autonomous orchestration.

Computer-use agents from Anthropic and OpenAI exist but remain slow and struggle with common interface interactions like scrolling, dragging and navigating some UIs.

Most custom agents you would build without a ready-made tool rely heavily on your ability to give clear, specific instructions with built-in feedback loops and the right task scope (e.g. not giving AI more than it can chew).

What's still hype

True end-to-end automation—plan → recruit → conduct → analyze → report without human oversight…isn’t happening reliably without exceptional prompting/AI skills.

The reliable capability hierarchy today:

1. More mature: Transcription, some sentiment analysis, some theme clustering, AI-moderated interviews (to a certain extent)

2. Emerging: Semi-automated reporting, integration-triggered workflows, research screening with agent handling checkpoints

3. Still not there: Fully autonomous multi-step research, end-to-end orchestration, agent-led strategic decisions without human guidance

AI STUDIES

👩‍🔬 Synthetic Users - Can they work, and what can we use them for?

PMs are wondering "can we skip recruiting tricky users now?"

Researchers are thinking, "are you trying to avoid talking to real customers?"

I want to do a reality check that is as accurate as I can manage. No agenda here - just calling out what studies say, and what they do not tell us.

〰️

The Stanford study everyone cites

The Stanford HAI "Generative Agent Simulations of 1,000 People" study achieved 85% accuracy on General Social Survey responses.

Impressive…until you understand the methodology:

• Each participant underwent a 2-hour in-depth qualitative interview generating transcripts averaging 6,491 words per person

• Full transcripts were injected into model prompts

• An "expert reflection" module analyzed each interview through psychologist, economist, political scientist, and demographic expert lenses

• The 85% accuracy was measured against the same individuals' own responses two weeks later

👆 This methodology bears no resemblance to how a product team would use synthetic users with generic persona descriptions of unknown customers. When was the last time you ran 2-hour interviews with a highly specific, standardized question set across 1000 customers?

Source: Generative Agent Simulations of 1,000 People (Stanford HAI, April 2025)

The Colgate-Palmolive study that's actually about product research

PyMC Labs partnered with Colgate-Palmolive to test synthetic consumers against 9,300 real human responses across 57 personal care product concept surveys.

Their "Semantic Similarity Rating" method achieved 90% of human test-retest reliability with realistic response distributions (KS similarity >0.85).

But here's what the methodology required:

• They couldn't just ask LLMs "rate this product 1-5"—direct numerical prompting produced unrealistic distributions

• Instead, they elicited free-text responses from GPT-4o and Gemini-2.0-flash, then mapped those to Likert scales using embedding similarity

• They tested against supervised machine learning models trained on actual survey data—and the zero-shot LLM approach outperformed them (90% vs 65% correlation attainment)

• The products were familiar personal care categories, not novel or complex offerings

👆 Notice: this study required a specific technical approach most teams won't implement, and it worked for well-understood consumer goods categories—not your novel B2B SaaS feature.

Source: LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings (arXiv, October 2025)

〰️

What studies actually prove ✅

• With sophisticated methodology (semantic similarity mapping, not direct prompting), LLMs can achieve 90% of human test-retest reliability for purchase intent on familiar consumer goods

• With 2-hour interviews of specific known individuals, agents can match that person's survey responses 85% as accurately as they match their own responses

• Zero-shot LLM approaches can outperform supervised ML models trained on actual survey data—but only with the right elicitation method

What studies do NOT prove ❌

Do not assume that the studies say:

• Generic persona-based synthetic users match real customer behavior (both studies required either deep individual data OR technical methodology sophistication)

• Results generalize to novel products, complex B2B offerings, or niche audiences

• Standard prompting achieves academic-quality results (it doesn't—direct "rate 1-5" prompts produce unrealistic distributions)

• Synthetic users can replace real customer research for final product decisions

• These methods work without validation against your actual customer data

Why you cannot simply say "it works"

The Stanford 85% was achieved by simulating specific individuals with detailed life stories based on a standardized survey protocol—corporate teams are trying to simulate generic customer segments they've interviewed few people from, on assumed profiles.

These are fundamentally different tasks.

Multiple studies found:

• Mode collapse (narrower response distributions than real humans)

• Typicality bias (stereotypical completions over diverse responses)

• Positive bias (synthetic users are "much more positive than real humans")

To be clear: I am really excited about this particular topic and not anti-synthetic users. But we need to know where they work, and what studies don’t tell us yet.

ON TO 2026

🔮 What 2026 might look like

Based on this year's progress and roadmaps announced by teams like OpenAI, Anthropic, and Google DeepMind, here's what I expect:

1. Transcript analysis will approach near-human accuracy

   I expect 95%+ accuracy for clear, single-speaker recordings and 90%+ for standard interview conditions (for top languages in training data) by late 2026.

2. Agent autonomy will be semi-autonomous for most of us, not fully autonomous

   OpenAI is developing specialized agents priced at $2,000-$20,000/month for knowledge work. Claude 4 can work continuously for hours on complex tasks. One of the biggest blockers is still human ability to communicate what we want and tell AI how to do it our way (not whatever way they would otherwise reason through automatically).

   I’m looking forward to automating more things reliably with improved models, like participant recruitment where we have clear screener criteria and qualifying characteristics.

3. Video analysis will improve but require human validation

   By late 2026, AI will handle much more video analysis tasks for customer research and do it well—key moment extraction (based on image recognition, not just transcripts), emotional reaction identification and on-screen actions.

   But human review remains essential for nuanced behavioral interpretation and catching cultural differences or reactions that aren’t highly represented in LLMs’ training data.

4. Synthetic user tools will remain supplements, not replacements

   Appropriate by end of 2026: Testing highly specific use cases (ex: standard usability audits, pricing changes) where we have exactly the right kind of data for the application.

   Still inappropriate: Final product decisions, niche audience research, emotional topics, replacement for qualitative research depth.

WHAT’S COMING NEXT?

• December will be a lighter edition with a tighter focus.

• I'll share some tests I've been kicking off (with 100’s of participants) in early 2026

• And some thoughts on planning your 2026 AI-augmented research model

Best of luck kicking off the new month. ✨

-Caitlin

Claude Code is the research assistant I’ve been waiting for.

I didn’t open Claude Code because I wanted another tool to learn. As someone largely non-technical, sitting in Terminal didn’t sound like a fun Sunday.

But it became unavoidable once my friend Pat said he never used Claude in the browser anymore, and it felt like Claude Code was everywhere.

Except…none of the use cases around were for customer research.

There were two big issues I hoped Claude Code would solve for me, and the PMs and senior researchers I work with:

1️⃣ Token hell.
I love Claude for deep research analysis, but every time you try to analyze long feedback formats you hit the token wall.
This requires workarounds that not everyone is excited about.

2️⃣ Forgotten documentation.
We spend discovery time capturing evidence, sharing insights and next steps.
But at some point, half the proof and reasoning behind a product decision lives in Slack messages and memories. We don’t know why someone came to their conclusions two weeks later (especially with AI involved).

Someone ends up asking, “Didn’t we already test that? Don’t we have those answers somewhere?”

Claude Code quietly fixed both.

Today I want to get you started with Claude Code. 🙂

In this edition:

1. 🖥️ Intro to Claude Code. What’s different in Terminal vs. Claude.ai, and a bit on context windows.

2. 🗺️ Install, troubleshoot and run prompts. Get going in minutes.

3. ⚙️ Core actions and functions that unlock speed. Actions that will get you past blank Terminal window syndrome.

4. 🤖 A Claude Code subagent for product risk. How to set up an agent that identifies the risk of proposed product changes, based on insights.

5. ⏩️ Next level: my subagents run parallel analysis. This is where the magic happens.

Plus…🙋‍♀️ if you’re a Product Manager, I’d love to talk to you. I’m running some AI tests and you’ll want the results. → Join my study here

WORKFLOW UPGRADES

🖥️ Intro to Claude Code

If you’ve used Claude.ai before, you’ve probably seen that it’s pretty good at understanding long, messy customer feedback.

But Claude Code lives on your computer instead of in the browser.

So think of it as Claude with hands.
It can actually hunt around, pick up, write and rearrange your files, not just read what you paste or upload to it.

A few key differences:

Claude.ai

Claude Code

Runs in your browser

Runs in your Terminal / desktop app

Needs you to upload every file or connect with an API

Reads directly from your connected computer folders

Loses some context when you switch chats

Can be fed a persistent context via CLAUDE.md project file

One chat at a time, sequential tasks

Can run multiple subagents that work in parallel

Writes text responses, creates files for download

Can edit, create, and organize your real documents and find the right one when you don’t know where it is

Uses the context window cumulatively in the chat

Each subagent starts fresh with a full context window not shared by the main chat ✨

A big deal to me: Documentation is almost never fun, and Claude Code can essentially keep all my documents updated for me. And when each subagent has the maximum context window available…analysis gets a lot faster while maintaining quality.

AI FUNDAMENTALS

🗺️ New to Claude Code? Let’s set it up.

1) Open a terminal

• macOS: Press ⌘ + Space → type Terminal → Return

• Windows: Press Win + R → type wt → Enter

2) Install Claude Code

• On macOS, run this in Terminal: curl -fsSL https://claude.ai/install.sh | bash

• In Windows, run this: irm https://claude.ai/install.ps1 | iex

3) Start Claude Code in your Terminal by typing “claude” → Enter

Troubleshooting: The best, fastest solution will come from taking a screenshot of the error you get in Terminal (ex: “command not found”) and sending it to Claude in your browser. Explain what you were trying to do. Using Claude to fix Claude issues works 👌

PROMPTING PLUS

⚙️ Core actions that unlock speed

Claude Code starts feeling powerful as soon as you start helping it work inside your files. Here are five essential actions:

1️⃣ Initialize your project brain (/init)

You might want to poke around to familiarize yourself with the Claude Code world first, but when you’re ready to get to work, do this -

Type /init to set up context in your CLAUDE.md markdown file.

That file becomes the project’s memory - your quick brief that Claude will always read before taking action.

You can include:

• Product goals and team objectives

• Target audience and relevant behavioral details

• Roles, perspectives or rules to use at all times

• Output expectations (e.g., “write in Markdown tables,” “never edit raw data,” “summaries go in /Drafts”)

Once created, you never have to re-explain context again.
Every prompt automatically builds on that foundation.

〰️

2️⃣ Control your project directory

When you start Claude Code it automatically has access to some of your files.

But you can decide which file directory to use for your current project (the chat you’re having) → type “cd ~/your/file-path/here”

When you "cd" (change directory) to a given folder:

• Commands operate from that location - they execute in that directory by default

• Relative paths start from there - If you reference a file like ./script.py, it looks in the current directory

• It's your working context - Any file operations (reading, writing, searching) use this as the starting point

Just remember that it’s temporary - it only affects the current terminal session.

〰️

3️⃣ Plan → Execute (Shift + Tab toggle)

Claude Code works in two modes:

• Planning mode = “Think through how we’d do this.” It drafts a plan, shows what it intends to do, and waits for confirmation.

• Action mode = “Do it.” It executes the plan - analyzing, searching, editing, or writing files.

You toggle between them with Shift+Tab on your keyboard.

That one pause prevents unnecessary token use on irrelevant plans and overwritten files.

〰️

4️⃣ Reference, don’t upload (@ symbol)

We’re not uploading here, you can drag files into a chat to grab their path name or point Claude directly at them by typing @ and picking from your local folder in your prompt.

Example:

@Research/Interviews Q3 - Identify recurring pain points and output a ranked table with verbatim quotes for each in the final column.

〰️

5️⃣ Subagents = your parallel workers

As with agents in any platform, these are on-demand teammates, each trained for a specific job. But in Claude Code:

• You can spin them up spontaneously: “Run 2 subagents to draft documentation for [project X with all docs in /this/file/path] and [meeting results]…”

• Or program them for reuse any time by writing their name in Claude, like risk_analyzer or theme_tagger

The magic is that they can also run simultaneously.

Before you try to run tons of agents, let’s start by building 1 subagent together 👇

AGENTS

🤖 A Claude Code subagent for analyzing a product change’s risk

How to set up an agent that:

• identifies our product change’s risky assumptions

• reviews existing customer knowledge

• assesses remaining risk based on knowledge gaps

• tells us: launch + measure or do more research?

(Tip: Pause the video to read some of the detailed instructions I include).

Watch the Video

⏩️ Parallel agents: an actual 10x AI use case

Some analysis processes still feel more time-consuming than we expect with AI involved. Context windows have limitations, leading to chunking and complex workflows to get big data sets analyzed well.

With Claude Code, I’ve seen an actual 10x in speed.

Why? Because multiple subagents can run tasks in parallel that used to be sequential — and each one can use the full Claude context window. 😮 Insane.

Here’s an example:

Three agents - analyzing (1) survey results, (2) customer calls and (3) the bias in both data sets - at the same time. You don’t have to wait for one task to finish before starting the next.

This setup ran 4+ hours of analysis work for me in 14 minutes (yes, I time everything, manual and AI).

🚀 “While I’m stuck in meetings, my analysis is already running in Claude Code.”

That’s what Claude Code has been doing for me since setting up parallel analysis subagents - the workflow in the screenshot above.

So I’m updating my AI Analysis course content to include that workflow.

I really want to kill the blocker of slow, sequential analysis and maxing out tokens before getting any analysis results.

👉 We fix slow analysis in the course.
(It’s also one of the *top Maven AI and Product courses*, rated 4.8/5).

The price just went up, but you can still grab the old price below — only until Saturday night.

🎟️ Grab the last spots

I hope this gave you concrete stepping stones and inspiration as you test Claude Code on your own. Happy Halloween!

-Caitlin

Adopting AI isn’t easy - even a few years in.

This edition was going to be focused entirely on the new Dia browser and its potential uses in customer research.

So there’s that - but then I heard this over and over in August and September:

"We've been using AI for a while, but everyone in the team still uses it differently - and we’re still not sure what works best."

I knew I had to cover that.

There’s a bit of a trend happening right now: non-beginners in actively AI-dabbling teams are just not seeing the magic happen with AI.

If you’re leading or even just quietly championing more effective AI use across your research, product, or design team — and you’re noticing wildly different adoption patterns — I have a roadmap for you.

Let’s dive in.

In this edition:

1. 🖥️ 5 research tasks I found easier in the new Dia browser. Many tests led to a few use cases I preferred in Dia vs. an LLM interface (+ warnings).

2. 🗺️ My 4-month roadmap to solid AI adoption. The steps required to find AI workflows that work for your team and find use cases where you’ll get the most from it.

3. 📰 AI News: More memory across chats could get messy for customer insights work. Memory is cool until it feeds your analysis with another project’s context. What to know.

WORKFLOW UPGRADES

📝 5 research tasks I found easier in the new Dia browser

I got early access to The Browser Company’s new browser Dia (with built-in AI), and as a long-time Arc fan, I was excited to see what they’d do with AI.

I tested a whole bunch of customer research tasks and found 5 tasks that were actually easier in Dia — mostly because of how Dia’s browser-native setup reduces switching costs by letting you prompt while pulling from open tabs.

Below the 5 tasks that worked for me, you’ll find some additional warnings + notes worth knowing about Dia’s AI setup.

1. Improving presentations

If you run internal workshops or present findings in slide decks, you can use Dia’s AI functions to quickly pull your slide deck tab into the prompt chat and get improvements in seconds.

Quick intro: There are two ways to chat with Dia AI - (1) in a new tab where your “search” bar is also a prompt input field, and (2) in a sidebar within the tab where the content you’re working with is. You’ll see both in this clip 👇

Dia - Improving presentations - Watch Video

—

2. Fact-checking claims

Got a stakeholder saying “users just want X”? Drop their quote into a new Dia tab. It can search across your open tabs of data, documents and insights presentations and return a clearer summary or validation much faster than dragging everything into a new GPT session.

—

3. Building a prompt library (with slash commands)

Dia lets you save reusable prompts as custom shortcuts (called “Skills”). For example:

• Create a skill called /review-mining

• Store your best prompt for mining customer reviews

• Use it anytime, on any page or open tab, just by typing /review-mining

It’s like TextBlaze (I mentioned that tool ages ago), but native to the browser and easier to manage.

Adding shortcut "Skills" - Watch Video

—

4. Auto-summarizing your work

Maybe you need to prove to your manager that you were productive this week. Or just want to double-check progress toward your own targets, like me. Dia can pull from your browser activity and summarize it any way you tell it to. I used it to create a weekly update that:

• Mapped to my research to-do list

• Created a 1-paragraph summary

• Noted weekly targets checked off in bullets

Inspired by this Dia Skill: Daily Wrap

—

5. Comparing tools side-by-side

If you’re testing a few tools for research, and you’ve opened a bunch of landing pages in your browser, Dia helps you quickly compare -

• Pull in info from all open tabs (no copy-pasting info or links into an LLM)

• Ask for a TL;DR comparison

• Zero back and forth between landing pages, context explanation and results

⚠️ Tip: If you’re used to writing long, elaborate prompts in LLMs for a comparison task like this one, they won’t typically work here. Short wins. See the prompt I used in the video (8 different longer variations all generated errors!).

Comparing 4 tools in the browser - Watch Video

〰️

Which model is Dia using?

Dia refers to GPT-5 and GPT-Thinking within your Skills library, where you can choose which model to use for each Skill (e.g. prompt template) you save.

A few warnings…

• Dia’s memory is ON by default. It clearly states that it stores your site visits, chats, and preferences on their servers. Proceed with caution (especially with PII or sensitive research material).

• No visual generation. It won’t create charts or visuals the way ChatGPT or Claude can — so where you need to create graphs, mockups, or images, you’ll want to stick to your LLM interface.

〰️

Want to try Dia yourself? 👉 I’ve got invites for a few of you to skip the waitlist here

AI FUNDAMENTALS

🎨 My 4-month roadmap to solid AI adoption

I’ve spent 2024-2025 supporting clients with training and longer-term adoption support - so I’ve seen a variety of issues getting AI on board and getting real value from it.

But most of the blockers come from the same problematic patterns - things that can be easily fixed if you have the right steps in mind.

I created this roadmap to help research and design teams go from scattered experimentation to finally figuring things out. (It’s based on working with all those teams over the last 2 years).

The Roadmap Overview -

• Audit where AI can actually matter

• Make prompting a priority (really - learn how to do it well)

• Track things until you see consistency

• Define human checkpoints

• Run “bake-offs” on real work

• Create shared playbooks

• Build evidence trails people can audit

• Score consistency with a simple rubric

• Create a forum for comparison and discussion

This takes you from “we’re trying a few things” to “we have AI-backed systems that make us faster and better and the proof to back it up”.

📍 Get the full roadmap + details here: Adopting AI That Works for Customer Research (in 4 Months)

AI NEWS

📰 More memory across chats could get messy for customer insights work.

Both Claude and ChatGPT have recently just rolled out expanded memory capabilities:

• Claude’s memory is now live for Team and Enterprise users. It remembers past conversations, adapts to your projects over time, and can carry context across chats automatically. You can see what it remembers, edit it, and turn on “incognito mode” if you don’t want it storing a conversation. (Source)

• ChatGPT’s updated memory (from August’s GPT-5 release ) works similarly — gradually learning preferences, remembering your name or goals, and influencing future chats even if you don’t explicitly reference past ones. It can be toggled on/off in Settings > Personalization (and you can even see all the specific context it has saved from your chats).

Why it matters

We want AI to know what we’re working on. But in research, we often need tight boundaries between projects: different data sets, hypotheses, stakeholder inputs all need to be kept separate…

But memory means context bleeds are possible across chats in ways that are hard to spot.

〰️

How memory works (in both ChatGPT and Claude)

• ✅ Memory is stored at the user level, not per chat

• ✅ It silently influences new chats, even if you didn’t reference past ones

• ✅ It uses implicit memory — it may remember something you didn’t ask it to remember

• ✅ You can turn it off globally or for one chat (use “incognito”)

• ✅ You can view and edit what it remembers in your settings (Settings > Personalization > click the “Manage” button)

• ✅ Deleting a chat does not delete memory it saved from that chat

〰️

What we can do

Some quick ways to stay in control:

• Use incognito chats if keeping things separate is more important than shared continuous knowledge

• Regularly delete or review memory before starting a new research task

• Label projects clearly in prompts, so you can spot when context is leaking

• Verify outputs — especially if something sounds too familiar

Memory is helpful, until it isn't. If you’re seeing signs of spillover from previous studies, data and context input you’ve given, consider that memory might be the culprit.

Pssst - you always get a course discount!

The November cohort of my AI Analysis course is open for enrollment.

So if you’re -

• wishing you could cut analysis time measurably, but…

• feeling like anything coming from AI could be hallucinated…

• and spending more time fixing outputs than is saved using AI…

We fix all of that in the course (also, it’s rated 4.8/5 on Maven).

Plus, the best things about it:

• You get 1:1 help from me privately, on your very personal, use case specific challenges

• There’s so much content - it will continue guiding you for months after our cohort

• …and you’ll have updated content for 6 months!

More details here 👇

See course details

It’s almost Q4! See you next month.

-Caitlin

This wasn’t the newsletter I planned to send you.

I had this edition all mapped out and a few tests already run. But…OpenAI just had to launch a little thing called GPT-5. 🙄

I took this month to run as many mini tests as I could fit around client workshops.

This edition is a summary of a lot of exploring this month:

What I think is actually better in GPT-5 so far, what still needs babysitting, an example of an Agent Mode workflow for research, and a study that helps us understand whether GPT-5’s larger context window is a good thing or not.

Let’s dive in —

In this edition:

1. ⚛️ An Agent Mode use case: A workflow worth using an Agent for in customer research, with example results.

2. 🦾 GPT-5 Tests + Results: What’s improved, what’s not, what to think about.

3. 👩‍🔬 A Study: How increasing input tokens impacts LLM performance (and what it means for customer research).

The list looks short, but there’s a ton to dig into below -

WORKFLOW UPGRADES

⚛️ An Agent Mode Use Case

What is Agent Mode?
A toggle in ChatGPT that let’s you build assistants that can run multi-step tasks linking use of various tools, accessing memory, and behaving with a bit more logic and persistence than a regular chat. Where you find it 👇

That’s the idea anyway. But the truth is, a good Agent still requires you to be really clear about what you want it to do - down to the precise steps, tools, instruction for how to use them, and how you’ll give ChatGPT access to the tools.

〰️

My “agent” workflow - What I asked it to do:

I pretended to work for the fitness tech company Whoop, and said:

1. Collect public reviews for Whoop (from Reddit, Trustpilot, blogs, etc.)

2. Log every individual comment in a Google Sheet (live Drive doc, not a CSV)

3. Label and categorize all feedback using inductive reasoning in the Sheet

4. Synthesize key complaints that negatively impact retention + recommendation

5. Put all findings into Google Slides with:

   • Verbatim quotes and sources

   • Detailed analysis of emerging patterns

   • Design matching a provided template/guidelines

Verdict: This worked pretty well, and after a few tests to improve the prompts I used, the result was good enough that I’d definitely use this workflow for slide creation from desk research next time (in a real scenario). I can actually tell it to go do this for me like an intern, and I can work on something else (see video below) .

〰️

Here’s some of the Slides process + results -

ChatGPT Agent Mode (Slides creation) - Watch Video

Resulting slides from mid-test - they need a little work, but it’s a good start. With 15 slides like the one on the right here, it saved me a lot of copy-pasting. Font is correct (Inter), colors are correct once I sent the HEX codes, logo and other details were added to the template.

What’s worth considering:

It took Agent Mode much longer to turn the review mining findings into slides than it takes a specialty AI based slide tool (Beautiful AI, Gamma, etc) to create slides from a doc with your findings. But if your team already uses Enterprise level ChatGPT, then adding another slide-maker tool with AI means increasing the data privacy risk - you’re putting customer data and possibly secret findings into yet another tool.

🦾 GPT-5 Tests + Results: What’s Actually Improved?

Here are some fast results I got from hands-on testing across a bunch of typical research tasks:

🟡 Better accuracy… kinda.

It’s less hallucination-prone in general - but still unreliable with verbatim quotes. If you’re summarizing interviews or trying to lift exact language from transcripts, expect to double-check and be persistent in your prompts (e.g. “use exact customer language with misspellings, pauses, interruptions and imperfections as it appears in the transcript”.)

〰️

🟡 Math logic is finally saner

GPT-5 now consistently uses Python behind the scenes for math tasks, without needing to be told to. That annoying thing where it sometimes “reasons out” math in natural language (I mentioned this in June) - I have not seen this happen in GPT-5.

I tested this with basic survey calculations and custom metrics from my synthetic data where I’ve seen previous GPT models, Claude and Gemini make funny math mistakes. GPT-5 handled calculations well as long as the prompt was clear. Messy requests = messy math. Still true.

〰️

🟡 Long, multi-step tasks: roughly the same (you need to be structured)

GPT-5 is supposed to be able to handle a long sequence of steps with a little less hand-holding in certain situations. For example, it’s not supposed to forget the earlier parts of a prompt chain as easily, and should remember context better.

I’ve seen this hold true in some of my step by step workflows in a normal chat window. But with Agent Mode, I’ve seen GPT-5 completely drop the ball.

Ex: In my prompt #2: “…put the findings from the review mining into Google Slides.”

(5 minutes later) ChatGPT: “Here’s your PowerPoint presentation. You can download it here [link].” 👿

What can we take from this? Structure, clarity and breaking up complex tasks into smaller chunks are still required. Workflows with too many tasks still derail it.

〰️

🟡 Big files - Seemingly less of a problem

I’ve tested heavily with two synthetic datasets from Kaggle:

• 20,000 rows of quant + geo data (partial dataset)

• 12,000 rows of data with open-text feedback

They triggered fewer issues so far than I’ve had in the past.
GPT-5 handled both data sets with full retention of earlier prompts and instructions - even across multi-turn follow-up queries. No upload issues experienced, no missing or inaccurately retrieved data.

What this means:

This potentially opens up more ambitious workflows for cleaning, segmenting, or running hybrid qual/quant analysis without breaking the tool or your brain.

I’m getting more hopeful on this front, even though there’s still a lot to think about in terms of how much content you upload in one go (see the study below 👇)…

AI STUDIES

👩‍🔬 How Increasing Input Tokens Impacts LLM Performance

Study: “Context Rot: How Increasing Input Tokens Impacts LLM Performance”

TL;DR - As you add more text, reliability often drops—especially with semantic questions, and when key evidence sits in the middle of the set. Bigger windows help load more information, but they don’t guarantee accuracy in processing it.

What they tested -

• 18 models, controlled long-context tasks

• They made the inputs longer, moved the right answer earlier or later, and added extra lines that looked right but weren’t.

Why it matters for research -
Dumping entire interview banks into one prompt can hurt reliability: more plausible-but-wrong answers and missed references are highly likely.

⚠️ Why this is in the GPT-5 issue -
Some people are talking about the larger context window available in GPT-5.

But larger context = capacity. A larger window lets you paste or upload more text, but it doesn’t guarantee better answers. Use the extra space to organize and curate, not to dump everything in and hope GPT-5 is smart enough to accurately deliver you the right things on its own (hint: it probably won’t).

—

Best of luck until September. ✌️

-Caitlin

Hey all!

I’ve been surprised lately:
A lot of you are running your entire analysis process through NotebookLM - using its hidden transcripts as the foundation for all your AI insights.

People are uploading recordings straight into NotebookLM and relying on transcripts that, as several of you put it, “you actually can’t see.”

Yikes. 🫣

Instead of telling you this feels like a risky move, I decided to run some tests.

Is it really that bad? What are NotebookLM’s transcripts like? And how can you tell if they’re hidden?

Plus, comparing NotebookLM transcripts with those from Grain and Whisper, and how to test 3 transcript tools in <1 hour (with an AI evaluation in the mix).

Let’s do this —

In this edition:

A bunch about transcription…

1. ⚠️ NotebookLM vs. Grain vs. Whisper: A few head-to-head battle findings across English and German transcripts.

2. 👀 NotebookLM’s “secret” transcripts: Are they any good? Get NotebookLM to show them to you.

3. ⚖️ Run a transcript evaluation in Claude: Get it to comb transcripts from multiple tools, and tell you which are worth trusting.
   

4. 👩‍🔬 Run your own multi-tool transcript test in <1 hour: Steal my simple test protocol that starts small and scales.

WORKFLOW UPGRADES

⚠️ NotebookLM vs. Grain vs. Whisper

While running tests on NotebookLM’s transcription quality for my course students, I decided to compare them with a few popular options - Grain (common among those of you I’ve chatted with), and Whisper (via MacWhisper Pro) - considered a bit of a gold standard for highly accurate transcripts.

Top findings

1. 👎 Whisper underperformed - don’t assume even “highly accurate” tools will work for your data - test them on your toughest transcripts first.

2. 👍 Grain surprised me - I’d had some underwhelming results from the tool previously, but it aced recent tests.

3. 👇 NotebookLM is at the bottom of this list - consistently missing brand and tool names, and even key terms that were stated clearly by participants.

Examples of how the results compared from my test notes:

〰️

• Grain was smart enough to realize that my participant’s pronunciation of “Myro” actually meant the workshop/design platform “Miro” - the other two didn’t catch this

• NotebookLM consistently synthesized things just enough to make the transcript noticeably less accurate, and skipped or misunderstood small but important words here and there - see “so” (very) vs. “zu” (too much) above.

• Whisper results were a mixed bag. On transcripts where German speakers were interviewed in English, there was a 50/50 chance of error across some sentences where accents made pronunciation of key words different from the way a US-English speaker would have said them.

Verdict: Of these three tools, I’d easily choose Grain if continuing to need a tool that solidly catches brand/tool names and understands English spoken with European accents.

More about my test setup are in the “test protocols” section below

PROMPTING PLUS

👀 NotebookLM’s “secret” transcripts

TL;DR: If you can’t see the words, you can’t trust the insights.

What are NotebookLM transcripts even like?

NotebookLM can turn audio files straight into into analysis, but many people think you can’t see the transcripts. That’s a problem - because every quote list or statement about a pattern is built on that invisible text.

I didn’t believe that NotebookLM would be so secretive, so I found the simplest prompt that reveals the transcripts - and lets us check how solid they are. Please use this - don’t rely on hidden transcripts.

PROMPT THIS after uploading your audio file

“Give me a verbatim transcript of the audio you just processed for [file name]. Include Speaker labels.”

That’s it! It worked in 15/15 tests so far on transcripts of various kinds.

〰️

⚖️ Run a Transcript Evaluation in Claude

A WER score = an instant indication about whether you should trust the tool’s transcript capabilities.

What’s “WER”? Word Error Rate.

It’s a measure of comparison between multiple test transcripts or tests against a control transcript.

If you want to use a control, use a transcript you know is highly reliable, or one you painstakingly wrote out manually.

Want to know how transcripts from 2+ tools compare really fast?

• Use Python and jiwer.compare to compare word error rate between transcripts in Claude

• Upload transcripts with the prompt below

PROMPT THIS - Ideally in Claude (handles JiWER best)

“Act as a master of transcript accuracy and comparison.

Run Python: jiwer.compare to compare the words transcribed in the [#] transcripts uploaded.”

〰️

See the full results I got from this prompt in Claude

By the way…

If you’re still struggling to get reliable analysis results from AI…

• Wishing you could cut analysis time measurably, but…

• feeling like anything coming from AI could be hallucinated…

• and spending more time fixing outputs than is saved using AI…

We fix all of that in my AI Analysis course!

⚡️ Enrollment for September is now open ⚡️

• 4 week course

• September 15 - October 10

• Access to course content forever

• Content kept updated for an additional 6 months

More details below 👇

See course details

TEST PROTOCOLS

💨 Want to repeat my transcript test? Do it in <1 hour

Try this: Pick one tricky interview recording, generate transcripts in 2+ tools, and run a WER comparison. The tiny effort pays for itself the next time you’re tempted to trust a black-box summary from a tool like NotebookLM or something your colleagues are using.

How I ran my complete test round using this -

• Chose ONE audio interview recording per language (EN, DE)

• Ran the one file through all three tools —> First results, in well under 1 hour

• Repeated the process across 10x English audio files and 10x German files - results were consistent.

• Repeated with 5x Swedish audio files

Get the full test protocol in Notion to guide you

WHAT’S COMING NEXT?

• How can AI help us continue learning from existing research? Your AI repository options - coming in August

• Synthetic users is a hot topic and I’m working on something there…

See you in August!

-Caitlin

Hey all!

It’s the first month that this newsletter is FREE. 🥳

But that doesn’t mean this edition is suddenly in “lite” mode - there’s a lot to talk about in June.

In the past month, alongside running my AI Analysis course again, I’ve run workshops with teams like Canva and Vinted 🙇‍♀️ And a few common problems keep showing up...

Despite all the diversity in my client teams—from geolocation and languages used, to tools and democratization levels—there was a common thread:

LLMs make silly math mistakes and no one knows why or how to fix them.

And cleaning data is a giant blocker to AI adoption in research.

I spent more time than expected this month helping teams handle those small-but-hugely-impactful topics, and wanted to share a few quick solutions with you, too.

Plus, a new instruction kept showing up in ChatGPT-generated prompts, and a study went viral. The study asked: Is using AI making us stupid? It seems a little like it is, but I’ll share what else the study says, and how I’ve always thought about maintaining my independent intelligence while being a heavy AI user.

Let’s jump in:

In this edition:

1. 🔢 Why Do LLMs Keep Failing at Simple Math? Here’s why math errors in quantitative tasks happen and how to fix them.

2. 🧹 Using AI to Clean Your Data: One easy way to use AI to speed up a time-consuming prep step for analysis.

3. 🧾 “Do Not Print Your Reasoning” - A new instruction popping up everywhere in my meta prompting. Should we leave it in the prompts?
   

4. 💔 Is AI Actually Making Us Dumber? What we can take from a frequently mentioned study, and how I try to protect my brainpower.

5. 📰 AI News: 🇪🇺 Nvidia & Perplexity are building “Sovereign AI” in Europe.

WORKFLOW UPGRADES

🔢 Why do LLMs keep failing at simple math? Here’s what’s happening and how to fix it

The issue:
It’s alarmingly common: you ask ChatGPT (or Claude, or Gemini) to do some basic quantitative analysis, and it spits out wildly different math results for the same problem.

Here’s an example: One of my June course participants was using my synthetic survey data and kept getting weird calculations -

• The reality: 4/10 users who selected “Sleep issues” as primary reason for downloading/using the Flow app also upgraded to the Premium tier from Free.

• Claude’s first calculation: 33% of “Sleep issues”-selectors upgraded

• Claude’s second calculation: 62% of “Sleep issues”-selectors upgraded

What is going on here?

Why this happens:

• The AI makes hidden assumptions about how to count responses.

• There’s no “show your work” step, so you can’t check its math.

• Calculations are often done in plain text (not code), so there’s no reliable logic.

How to get accurate results:

1. Force it to show its work
   (Still not foolproof - AI may “show” you steps but still fudge the math.)
   

   Prompt:
   “Before calculating percentages, first list out each respondent ID who mentioned this goal, then list which of those respondent IDs upgraded to premium. Show your counting step-by-step.”
   
   —

2. Force the AI to use code/Python:
   Best solution: Calculations done via code are verifiable and reliable - way better than fuzzy text math.
   
   Prompt:
   “Use Python or code for all calculations, groupings, and comparisons. Output the code and the final result.”

〰️

🧹 Using AI to Clean Your Data

You can use any LLM to “pre-inspect” transcripts or survey exports before analysis - saving a ton of time and reducing errors. While this isn’t the best solution for everyone, most people I know and have worked with aren’t using AI for any part of cleaning. If you’re open to it, here’s a simple place to start without completely handing over this critical task -

Prompts to get started:

1. Assess data usability:
   “You are a critical research assistant. Review this transcript/survey file. Analyze and list: (a) all data quality issues, (b) barriers to use (e.g., broken language, missed statements, unclear or missing speaker labels, likely inaccuracies, etc.), and (c) anything that would confuse an AI or human analysts in an analysis process.”

2. Find the highest-impact cleaning steps:
   “From your analysis, which three cleaning steps would save the most time and make this file easier to analyze accurately?
   
   Explain why you chose those steps. Be specific and brief.”

Using prompts like this on your raw data can help you identify where the biggest problems are in formatting, broken content and more, so you can clean whatever has the most impact. You don’t always have to clean data perfectly to get much better results with AI.

Tip:
Always ask the AI to explain its reasoning behind each cleaning step it suggests—so you can spot anything it misses (or overthinks).

PROMPTING PLUS

🧾 “Do Not Print Your Reasoning”: Why Is ChatGPT Hiding Its Thought Process?

If you’ve been writing prompts or using “meta prompting” (asking your LLM to improve or generate prompts for you), you may have noticed a new instruction cropping up:

“Do not print your reasoning.”

This statement was coming up in every revised prompt I asked ChatGPT to write for me lately (in GPT 4o, 4.1 and o3). But why is this suddenly the default? And when should you use it (or not)?

What’s actually happening?

• By default, LLMs “think aloud.”
  If you ask for a summary, a plan, or an answer, you’ll often get pages of step-by-step reasoning.

• This burns tokens (costs more) and makes results longer/slower.

• The new trend is to “hide” the internal thought process, but still have the model do it internally - just not share it with you.

It seems like ChatGPT is now encouraging users to use prompts that deliver results without the reasoning, possibly because the reasoning models intend to show you (some of) their reasoning by default in expandable sections instead, like this -

An example of the reasoning - which often isn’t enough for me to know if the LLM did the task correctly and thoroughly or not.

Should you tell your LLM to skip the reasoning in outputs?

The problem with prompts like the ones ChatGPT generated is this: often, the reasoning laid out in those expandable sections (see above) aren’t enough. Here’s when I believe you still need to ask your LLM for its reasoning (and remove “do not print reasoning” from any revised prompts) -

❌ Debugging or quality control:
You WANT to see the logic if you’re double-checking AI decisions, or if you’re testing a prompt for consistency.

❌ Training new prompt patterns:
If you want to teach yourself or your team how the AI “thinks,” seeing step-by-step logic is essential.

❌ Sensitive/ambiguous cases:
For research, UX, or customer verbatim analysis, it’s helpful to understand why the AI labeled a statement a certain way, or how it arrived at certain groupings.

❌ Higher stakes work:
You need all the transparency you can get for tricky or high-stakes research work where you’ll need to document the full process and arguments for where you landed.

AI STUDIES

💔 Is AI Actually Making Us Dumber?

Study: “Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task”

Why it matters:
If you use LLMs for research/analysis, you might be worried about outsourcing your brain. I certainly am.

This is something I’ve been thinking about since the earliest days after ChatGPT’s launch and my first experiments. “Will this make my brain incapable of doing customer research without AI in the future?” was top of mind, and still is.

〰️

Here’s what this paper found, and how I think about the topic at hand -

What the experiment did:

• 54 participants wrote essays

• Participants were split into 3 groups, using (a) their brain, or (b) a search engine, or (c) an LLM to write the essays.

• 18 participants completed a final task, switching from the method they used in the original tasks (e.g. their brain) to another method (e.g. an LLM)

• EEG measured brain engagement during essay writing.

• Humans & AI both scored the participants’ work produced.

What happened?

• LLMs seem to push “single answer” thinking - they synthesize everything for the user to the point that the users don’t have to understand, sift and make sense of things themselves. Low cognitive load, but also low engagement.

• Search engines give options that the human brain has to work through and synthesize. More cognitive load, and engagement.

• People using LLMs recalled less from their own writing just minutes later.

• Brain activity dropped if they started with AI, but spiked if they started solo.

❝

“The LLM group also fell behind in their ability to quote from the essays they wrote just minutes prior”

From the study

⚠️ What I’m most afraid of: The finding in that quote above signals that we don’t use our working memory as much when using LLMs to understand our data. To me, this says we must stay as close to the data as possible. Our work is not just about sifting through data faster, it’s about being close to the customer and our brains don’t actually absorb as much from the data - and remember it later - if we hand everything over to AI.

〰️

How I think about using AI (and this study encourages me to continue that way):

• Take notes in customer sessions, on own thoughts during meetings, etc. Start with something that is your perspective, not just AI’s to bring to later AI use

• Ask AI to help as a second step after forming own thoughts and perspective - not the other way around.

• Use your notes and perspective to challenge AI’s in any process (planning experiments, writing interview guides, running analysis, etc).

Since day one, I’ve continued to take one step - even if small - to use my own brain before using AI. This study makes me think that’s worth continuing.

〰️

🔍 One final note about this study’s size:

54 participants is relatively small for a neuroscience study - but not unusual for EEG research, which is more intensive and expensive than a survey or click test. (And only 18 completed the final task, though the researchers considered this a bonus.)

I see this study as a signal, not a final verdict. The findings are directionally important—especially about “single answer” thinking and memory.

AI NEWS

🇪🇺 Nvidia & Perplexity: Building “Sovereign AI” in Europe

What happened?
Perplexity (the AI-powered search startup) and Nvidia are launching a big push to build “sovereign AI infrastructure” for European countries.

Instead of relying on U.S. or Chinese LLMs and cloud providers, EU governments and companies could run Perplexity’s AI models on local Nvidia hardware, inside the EU.

Why is this generally a big deal?

• European governments want control over their AI: where it runs, where data lives, and how models are updated.

• “Sovereign AI” means that your data (including customer research and interviews) can stay inside the EU—no U.S. or non-EU company access.

Why this matters to us:

• Better coverage of European languages:
  The new Perplexity models will handle all 24 official EU languages natively - not just “translate” but actually understand and respond in context, with local nuance.

• Fine-tuning for national, cultural context:
  Teams in places like Slovakia and Slovenia are already training models on their national languages and cultures—meaning answers, summaries, and insights are less North American by default, more relevant for local customers and users.

• Less American bias? Hopefully

  I imagine that if these models are trained on European data and not majority-U.S. data, there would be a reduction in the kind of U.S.-centric responses we often get from the major models - sometimes as basic as assuming all our financial calculations should be in $, but often more significant than that.

• Source: Wall Street Journal

WHAT’S COMING NEXT?

Here’s what’s coming in the next few editions -

• AI tools for prototyping - does one generate the best wireframes and views for user testing?

• I’m still planning a big test of ChatGPT Pro to finally test some interesting functionality, like video mode and recording (is ChatGPT a valid note-taker?)

• and more 🤓

See you in July!

-Caitlin

This month comes with a few surprises! 🙋‍♀️

Hi everyone! May’s been an interesting one…

Claude 4 launched with big claims about being the "best model" on some measures, so I've gotten the same question from newsletter readers, course students, and clients I'm working with:

Is Claude 4 actually better than 3.7 in any meaningful ways? I put together some quick tests to find out and I’ll share one today.

I also looked into Anthropic's newly published system prompts - the core instructions that shape how Claude behaves. Understanding these helps explain why Claude responds the way it does in our research workflows (and one instruction actually made me trust it more).

Plus, there's a term that's suddenly everywhere in AI. It's still very new (most teams I talk to aren't using it yet), but it represents AI that can actually provide the kind of value we’ve been hoping for (no, not AI that does your laundry - yet).

⭐️ And now comes the really big news before we get into this ⭐️

Your subscription is cancelled from June 1st.

But don't worry! You'll still get the newsletter - I'm making it FREE!

Here’s why (skip to the newsletter content if you don’t care)

When I launched this newsletter, I was freshly back from parental leave without childcare and at capacity (or beyond it) with client work. I had an idea for a newsletter I felt needed to exist, but I had zero time for it. So I gave myself one rule: if I was going to do this, it had to prove it deserved my time.

That's why I charged €5 for this newsletter from day one. It was a demand test. A way to make sure this newsletter would matter to people like you.

The test worked. The conversations and feedback I've gotten over the last year have been some of my most energizing moments. There are also quite a lot of you here 🤝.

But I’ve been debating this choice with myself for months.

On one side, I feel strongly that it’s important to set industry standards for creators to get paid for their work. On the other, my strategy has changed. My core revenue now comes from other parts of my business. That made me rethink this newsletter’s role in the bigger picture.

Making this free lets me focus on reach and impact, not revenue. There are so many people out there who still feel underserved and lacking support around AI in the bite-size pieces they can commit to monthly. If I’m blocking them from starting out and making progress with that €5 barrier, right now that’s at odds with my mission.

What's not changing: I’m sticking to the same promise I started with - workflow upgrades, faster AI decisions, and the AI news you actually need to know. All focused on customer research. In <20 minutes per month.

Thanks for all your support. If you’re enjoying this content, now you can actually tell your friends about it (because it’s free from June!) 😄

Here comes the May edition -

In this edition:

1. 👩‍🔬 Claude 4 vs Claude 3.7 — is there a difference? A quick test to see how the latest model reasons (and whether it does it better).

2. 🔬 Claude 4’s system prompts and what they mean for users. Anthropic’s latest published core model prompts and what their instructions tell us about Claude’s behaviors.

3. 🔁 What’s an “MCP”? The term that’s suddenly everywhere, and how it applies to people like us.

WORKFLOW UPGRADES

👩‍🔬 Claude 4 vs Claude 3.7 — is there a difference?

Claude 4 launched about a week ago, so I ran a few tests to show you something ASAP. I asked both Claude 4 and 3.7 to write a 15-minute interview guide for a customer interview. I chose this task because it’s something I’ve done often, but I hear from others that it isn’t always the easiest task for AI to get right.

The experiment:

• 🖼️ I gave a bunch of background context

• ⚠️ But the task was a 1-line instruction

Why? I wanted to see how the two models handled reasoning on their own - deciding without my input how to approach the task, which information to prioritize and which steps to follow.

The prompt dropped the reader into the role of a senior researcher at a fictional European home-buying platform, with analytics, hypotheses, and clear business goals.

Here’s what I saw in the interview guides written by 3.7 and 4:

(This is my personal assessment. I’ve included the prompt + outputs below for you to judge for yourself).

Metric

Claude 4

Claude 3.7

Context integration

✅ Better breakdown, deeper integration of background into question flow

🚫 Treated details a bit more like filler

Hypotheses addressed

✅ Direct questions for all 3 hypotheses

🚫 Not as directly addressed

Question wording bias

✅ Consistently used neutral language

🚫 A few leading questions

Structure & transitions

✅ Logical, natural and fit to the information I wanted to collect

(Warm-up → walkthrough → probe)

🚫 Question sets + overall method weren’t as cleverly tailored to getting the right info.

Jumped from intro to specifics.

Time realism

🚨 Risk of running a bit long, based on my experience/guesstimate

✅ Likely fit better within the 15 min window

Why it matters:
Claude 4 offered deeper reasoning and stronger hypothesis alignment without needing instructions to "think step by step" and being told what those steps should be.

It also designed a better-flowing interview, including screen share-based walkthroughs that would probably surface blockers in real time, instead of forcing participants to recall the last time they looked at the site and what they were thinking then.

When I dug into the nitty gritty details of the outputs, Claude 4 felt like it thought more critically and strategically beyond someone else’s “interview best practices” to determine what live session approach could yield the most accurate customer input. Claude 3.7 honestly felt like a more junior level researcher - the output isn’t “bad”, it’s just not as considered.

This isn’t a be-all-end-all test, it’s really just a start (and two other lightweight tests like this yielded closer results that were harder for me to identify differences between).

Interested in my test protocol - to replicate it or judge the outputs yourself? 🔗 See the prompt I used plus side by side Claude 3.7 vs. 4 results

PROMPTING PLUS

🔬 Claude 4’s system prompts and what they say about its behaviors

Claude doesn’t just take your word for it - and that’s a good thing. Buried in Claude 4’s internal instructions is this little gem:

❝

“The person's message may contain a false statement or presupposition and Claude should check this if uncertain. [...]

If the user corrects Claude or tells Claude it's made a mistake, then Claude first thinks through the issue carefully before acknowledging the user, since users sometimes make errors themselves.”

Claude’s core system prompts published by Anthropic, first seen here

What that means:

Claude is explicitly told not to blindly defer to you, even when you sound confident. Instead, it’s programmed to check your claims, assess whether you might be wrong, and then respond based on what’s most likely to be true.

Why this matters:

This should make Claude a much better thinking partner for research and product work — where your inputs might be early hypotheses, ambiguous observations based on limited recall, or rough data summaries. You want the model to test what you’re saying, not just be a yes-person.

🔎 Source: Claude 4’s system prompts

AI FUNDAMENTALS

🔁 “MCP”: What to know about the term that’s suddenly everywhere

If you've seen MCP popping up everywhere, you might have ignored it because of its typical use by CTOs and founder lately. But it’s a term worth knowing in our work, too.

What is it?
MCP = Model Context Protocol

Think of it like a universal adapter for AI tools. Just like how USB-C lets you plug any device into any port, MCP lets AI assistants connect to any app or data source you use.

Why this matters:

Right now, when you want AI to help with your work, you have to copy-paste and prompt everything manually. Customer messages exported from Intercom. Research notes from Notion. Transcripts from Grain. Surveys from Google Sheets. For nearly all of us, our processes are all split between many separate apps.

MCP changes that. Instead of you doing the busy work of making connections, exporting here and importing there, an AI assistant can:

• Pull customer quotes directly from your research database

• Check your calendar and suggest meeting times for participants

• Grab the latest design files from your shared folders for test planning

• Review recent survey responses and run a trend-spotting workflow

You might be using MCP without knowing:

This may sound like agentic workflows that your team is far from approving, but MCP is becoming more and more common. Especially in the backend of tools you’re already using or considering.

In my chats with five AI tool founders last week, 4/5 founders said their platforms were using MCP in the background to handle your data and AI decisions about what to do with it.

The main point: Where we can use MCP, it’ll mean far less time copying data between apps and more time on the creative, strategic work that matters (and is more fun). If you’re anything like me, that’s the sort of AI-assisted time-saver you’ve actually been waiting for.

WHAT’S COMING NEXT?

Here’s what’s planned for the next few editions, (assuming no model releases get in the way) -

• Hopefully a few more tests of Claude 4 and what it’s capable of

• Are there any decent options for user testing with AI? Let’s see.

• Cleaning transcripts is a pain - can we use AI (safely)?

Have a great start to June!

-Caitlin