You sent the analysis: the number, the chart, a short note on top saying what you found. The reply from the stakeholder you sent to: “Interesting, thanks.”

Nothing followed - no question about method, no meeting, nobody asking what you’d do about it. A month later nobody’s put it on a roadmap or into a sprint. Bring it up and the person you sent it to needs to be reminded which analysis you mean.

They read the analysis. They just don’t know what to do with it.

I published something about this last September on LinkedIn, a thing I called the five levels of data insight. Think reporting at the bottom, prescribing actions at the top. Most teams, I wrote, sit at levels one and two and reach three occasionally, and the advice I gave was to keep asking “so what” until the finding connects to a business outcome.

I think that advice was tackling the wrong problem. “Keep asking so what” is a note about effort, as if the people who stop at level two stopped because they gave up trying. The reality is, they didn’t - they stopped because they needed guidance on what it really means.

Assume your analysis was fine, the question was worth the effort to do and a decision was waiting on the answer. You sent the number, the chart, and a recommendation tucked in at the bottom, hedged. The sentence that’s missing is one that helps the reader understand what that number costs or earns them and why they should care.

If you leave this out, you’re asking your reader to do that math, and most just won’t bother. They’ll just file the analysis in the “interesting” bucket, and your job is to make it as effortless as possible for them to understand why it’s much more than that.

The solution: three lines, short enough to say out loud in fifteen seconds or drop into a Slack message.

• Say what you found, and put a number in it - a rate, a gap, a count - so your reader can argue, debate, or ask questions about it instead of nodding passively.

• Translate it into money, users, or risk, the units the person across from you runs their business in so they never have to do that conversion themselves.

• Name the action and price it - what to fund or ship, what it costs in people and weeks, what you expect back. “We recommend further investigation” isn’t an ask.

Subscribe now

Here’s the example I used back in September, filled in across all three. Daily active users down 15% last month. The drop sits entirely with Android users on older operating systems, churning at three times the normal rate since the v3.5 update, and the hypothesis is the new video engine choking on older hardware. Good finding. Investigated, explained, defensible.

Line two is the contextualization that turns it into a decision. That segment is 40% of the user base, which puts the quarterly revenue target at risk. Line three then follows naturally: roll the feature back for those users now, spend the next sprint on a backward-compatible version.

Nobody has to do arithmetic to see what 40% of the base means for the quarter. If the middle line is excluded, you’ll see how the roll-back turns from a grounded recommendation to an opinion.

If the middle line matters so much, why does everyone leave it out? Because it takes effort, business context, and extra mile to connect the dots.

Line two you start from scratch. To write it you have to convert your finding into the units your reader thinks in, and that could take things the analysis never told you. What is one percentage point of retention worth at this company? What does a week of engineering time cost? Which number does your CFO actually care about? What’s that additional slice or cut that provide more context to the numbers you’re seeing? None of that is in your data. It’s in the business, and you have to have been paying attention.

Try this: Open the last analysis you sent that went nowhere. Don’t redo it. Write one sentence, in your reader’s units, saying what that number costs or earns them. See how that changes the strength of your recommendation.

P.S. We have a free lesson this Wednesday, August 12, on AI open models and how to use them in your analysis. Sign up below.

Open Models 101

You’re 20 minutes into a review when two people put the same metric on screen and the numbers don’t match. One counted distinct users with a completed order in the last 30 days; the other counted distinct users who placed an order in calendar September, and threw out guest checkouts. You spend 40 minutes working out why the two decks disagree, and neither definition breaks. Both hold up.

I’ve watched this play out many times, and what matters is where the decision goes. The review ends in confusion, nothing lands in the room, and three days later the VP makes the call in a hallway, with whichever person she trusts more. A real roadmap decision now rides on personal credibility instead of on the number.

It doesn’t stop at one meeting either. When two numbers collide, the room falls back on its default procedure, which ranks people rather than definitions, and that ranking holds. The same person wins the next one and the one after that, so a single undefined metric installs a permanent tiebreaker.

That room exists because weeks earlier, someone typed “users who buy something each month” into a Google Doc and called it a definition. Everyone had already agreed that monthly active buyers was the metric that mattered, and agreeing on that felt like the hard part. It was not a definition. It was a sentence.

That sentence leaves four decisions open: what counts as a buy, at what grain, over what window, and who you throw out. Skip the writing, go straight to SQL or to an AI writing the SQL, and somebody still makes all four - the model just makes them for you, invisibly, on whatever it happens to grab. Anthropic published its own numbers here: running the same model, its analytics accuracy topped out at 21% without written business context and cleared 95% with it. Writing the context down did that.

Benn Stancil put the objection better than I can: “There is no correct win rate waiting to be unearthed; one version isn’t true while another is false.” Picking a definition means making a judgment call about how you want to run the business. He’s right, and it changes what the writing is for. The fix is not one definition to rule them all, but several, each tied to the decision it serves and labeled for who it serves. Two definitions in that review meeting would have caused nobody any trouble if either one had carried its decision on its face, so write the choice down - a judgment call doesn’t survive in shared memory.

Subscribe now

So how do you know when the writing is done? Could two strangers, handed only your definition, independently compute the metric and land on the same answer? I call it the two-person test, and the word doing the work is independently. Take the number your board deck runs on, send the definition to two people separately, and ask each of them to compute it without talking to the other. If the second one peeks at the first one’s answer, or clears up an ambiguity over Slack, you’ve agreed on a number without ever testing the definition that produced it.

The test gives you a stopping rule: you stop when two people hand back the same number, and the decision stays inside the meeting instead of drifting into a hallway.

The version we teach has seven fields, and three of the four decisions that Google Doc sentence left open already live in here.

• Name. Make it specific enough that nobody has to guess which number you mean, since monthly active buyers and monthly active users measure two different things.

• Numerator. Count distinct user IDs, not orders or sessions or dollars - and that one word, distinct, settles about half the disagreements in the room.

• Denominator. Decide what you count out of - everyone who visited, everyone who reached checkout, or everyone who was eligible in the first place - because those are three different rates off one numerator.

• Time window. Pick calendar month or trailing 30 days, because “last month” lets the two readings drift apart a little further every day the month runs.

• Inclusions. List what counts - which platforms, which product lines, whether members and non-members sit in the same number.

• Exclusions. Name what doesn’t - bots, internal test accounts, guest checkouts - and write “none” if it’s none, because a blank row reads as an oversight and the next person fills it in.

• Rationale. Say why you made these calls and not the other defensible ones. Every other field tells you how to compute the number; this one tells you why it looks the way it does.

The first four give the number its shape, and the last three decide what it means, because teams disagree over inclusions and exclusions, and rationale keeps a definition alive after the person who wrote it leaves. Teams skip rationale most often, even though the whole spec fits in a paragraph that one person writes before asking a question, not a review board convening.

We never ran this test on our own materials, and it cost us. Two students in our first cohort, working separately, found that our Week 3 capstone project defined checkout conversion three different ways - session-based, cart-based, and event-based - and we were the ones teaching this. Nobody inside caught it, and it surfaced only because two people outside the room happened to compute it separately, which is the test, run by accident.

The cheapest version of this starts at your next meeting, when someone cites an active user number and you ask which definition. Usually they stumble and reach for the one marketing uses, and when you ask which one that is, they don’t know. That’s the moment. And if you own the number yourself, run it the other way: write the definition down before anyone asks, then hand it to two people who weren’t in the room.

How many of your headline metrics carry more than one definition that nobody has reconciled?

Hit reply and tell me which one you’d least want to defend. And if you want to practice this with us, the next cohort of AI Analytics for Everyone starts August 3 and works through metric specs directly - numerators, denominators, time windows, and the edge cases that turn out to be decisions you made by omission. That link takes 20% off with SUBSTACK20.

Join Us in the Next Cohort

It’s a Monday, and a message from your VP lands in Slack: “How’s our new membership doing? Leadership wants an update by Thursday.” You read it twice. It’s a reasonable ask - you get one like it every week - but it’s not really a question yet, just a direction to go look in.

The instinct is to go get it. You open the warehouse, or now, hand the whole thing to the AI and start pulling data - adoption, retention, a chart or two, something clean enough to paste into a deck. And you’ll get something back, which is the trap. A vague ask always gives you an output that looks like an answer, whether or not it answers anything.

So I’ve learned not to go get the answer - I sharpen the ask first. There are four steps between a vague request and a question that’s worth an hour - goal, decision, metric, hypothesis - and working through them kills as many questions as it sharpens. Let me walk through this one.

The four steps from a vague ask to one worth an hour. The one everyone skips - decision - is the one that carries it.

The first step is the goal: the outcome the leader actually cares about, which is never “an update.” An update is a deliverable. Nobody wants a deliverable - they want to know something, and “an update” is just the stand-in they reached for. Strip the ask down and the goal is usually simpler than the wording: they want to know whether the membership is still worth the company’s time. Name that, and the ask already looks different.

Then comes the step almost everyone skips, and it’s the one that carries the whole thing: the decision. The goal implies a fork - invest more in the membership, or move that effort somewhere it pays back faster - and no analysis is worth an hour unless a choice like that is waiting on it. So before I pull anything, I ask the one question that collapses most requests: what decision are you trying to drive with this?

About 60% of the time, asking that out loud makes the person realize there was no real decision underneath, and they tell me not to bother. That’s the whole point. Most of what this does is kill questions early, before they cost anyone an afternoon. It’s an easy step to skip, because the metric step feels like real work - numbers, a chart, something to show - and the decision step feels like a meeting with nothing to show for it. So people skip to the dashboard, and that’s how the ones nobody opens get built.

Ask “what decision are you trying to drive?” and about 60% of requests don’t survive it. That’s the point.

The usual pushback is some version of “I’m just exploring,” and that’s fair - not every look at the data hangs on a decision due this week. But most exploring is a decision in disguise, so I reframe instead of folding. “I’m just exploring user behavior” becomes “what patterns would change our next sprint’s priorities?”; “I’m just looking at the data” becomes “what would I need to see to recommend we invest in this?”; “I don’t have a decision yet” becomes “then what’s worth investigating further - and that’s your decision.” That step is what points everything after it.

Once a question clears the decision, the metric gets easier - and the word to watch out for is “engagement.” Engagement would have felt like an answer and settled nothing. Invest-or-redirect wants two real numbers instead. The first is adoption: of the members who could use the new thing, how many ever touched it. The second is the one people leave off - whether the members who used it stuck around, or spent more, than the ones who didn’t. Adoption on its own is a vanity metric. Retention is the number that tells you whether it mattered.

The last step is a hypothesis - a guess specific enough to be wrong. Mine, going in: adoption is low because the feature is buried three taps deep, and the handful who find it retain better than average. Now the pull has a shape. If I’m right, the fix is discovery, and the membership earns more investment. If I’m wrong, I find that out cheaply and move on. Either way I can be wrong about it out loud, and that’s what makes it worth effort.

Same membership, same data. Thirty seconds of framing is the whole difference.

This is the part the AI won’t do for you. Hand it the raw ask - “how’s the new membership doing?” - and it answers from right there, fluently, without doing any of the work first. And the fluency is the problem. A clean answer feels settled, and that feeling is what stops you from asking whether the question was any good in the first place. A good analyst who knew your company would push back. Now that the answer is basically free, getting to the right question is the hard part.

The pull is fast now. Anyone can get an answer in the time it takes to type the ask. What’s still slow is deciding whether the answer is worth having, and most of the time an honest look ends with you not pulling any data at all. That part is still yours.

If you’re curious about framework like this to sharpen your analytical thinking and foundation, join us at our next AI Analytics for Everyone 5-week course where we help professionals become analytically independent while delegating execution to AI.

Or just hit reply and tell me the fuzziest ask sitting on your plate this week.

Subscribe now

Subscribe now

For some of that data the question isn’t even optional. In healthcare, legal, or finance you legally can’t hand certain records to a frontier lab’s model, so “just use Claude” was never on the table for the work that matters most.

Not long ago that skepticism was fair - the open models really were a notch back, fine for a demo but not for work you’d put your name on.

That’s not what I found this year. We were running open models side by side with a frontier one on real analytics work, and they held up on the messy questions.

So I went back to what the job even is. In agentic analytics, you point a model at a question and it runs the whole loop - frames the question, explores the data, does the analysis, checks its own work, and tells you the story. The SQL is one step in the middle, and every model on this list can already do it well.

The hard steps are the human ones. Somebody asks “how has retention changed?” - that could mean cohort retention, rolling retention, or churn rate, and a good analyst asks which before touching the data. I’ve watched a model hand back rolling retention when the client meant cohort - a clean answer to a question nobody asked. “Checkout conversion” has five defensible readings that land anywhere from 3% to 90% depending on what you count. Framing the fuzzy question, picking the right definition among several honest ones, reading the numbers back into a recommendation - that’s the reasoning the whole thing rests on. So the real question isn’t whether an open model can code. It’s whether it can reason through that loop and follow the rules for doing analytics well.

With that as the bar, a quick tour of the field.

• Qwen, from Alibaba, is the one everyone reaches for - Nathan Lambert’s read is that it’s overtaken Llama as the most-downloaded, most-fine-tuned open base, the default all-rounder, and it’s Apache 2.0, so you can build on it without a lawyer in the room.

• DeepSeek is the reasoning-forward lab; its R1 release in early 2025 shook everyone’s assumption about how far ahead the frontier really was, and it’s MIT and cheap.

• GLM, from z.ai, is the agentic, tool-native family that drops into Claude Code as a hosted substitute.

• Kimi, from Moonshot AI, is the long-context, trillion-parameter one built for long multi-step runs.

• Gemma, from Google, is the small, efficient one, now Apache 2.0.

• And MiniMax is the cheap, efficient-MoE challenger.

Here’s what surprised me. The good open models have already cleared the steps of the agentic analytics loop - framing, exploring, analyzing, checking, telling the story. They can all reason through that much well enough now. The step that still breaks is context, and that one was never a model problem. The model guesses at “retention” instead of asking, settles on one reading of “conversion,” and hands you a clean answer to a question you didn’t ask - and no bigger model fixes that. It’s the analyst’s job and a context-engineering job: write the definition down once, and the guessing stops. So which model you run matters less than whether you scoped the question.

The numbers back this up. Anthropic gave an agent grep access to thousands of prior SQL files and accuracy moved less than one percent. In a separate study, handing the model the database schema moved accuracy from 53.9 to 93.3 percent. Context is the forty-point lever. A bigger model barely moves the needle next to it.

I wanted to see this on my own data, not in a paper. About a month ago I ran the open models through a benchmark - the messy, real-shaped analytical questions we get. What it shows is narrow and enough: they clear the good-enough bar, so the model stops being the thing you worry about.

None of this means open has caught up across the board. The more times a job has to run the loop, the more the reliability tax compounds - a step that’s right 90 percent of the time is right about 59 percent over five steps, and the frontier labs still edge ahead on holding a long plan together. But that gap is closing very quickly to a point where it’s indistinguishable depending on the complexity of the analysis.

If you’d like to learn more about what open source models are, how they stack up on doing analysis, and how to run analysis through them yourself, join us in this free workshop: https://maven.com/p/568b2d/run-your-data-analysis-on-open-source-ai-models

Most of the AI work I see right now is for other people. AI products for customers, AI workflows for teams, AI demos for boardrooms. They’re spending months on work that looks incredible in the meeting and falls apart the moment a real user touches it.

They’re starting in the wrong place.

Clone yourself first. Before you build for customers, before you build a product, before you pitch your manager on an AI strategy: build an AI version of you. Automate your own work. Replace yourself. It’s the highest-return investment you will make in AI, and it’s not close.

You can only control one person: you. You can’t make a customer use your product, you can’t make your team change how they work, and you can’t talk your boss into funding an experiment when they need deliverables. But you can decide, today, to do your own work through AI instead of by hand. That decision is entirely yours, which is exactly why it works.

Time Is Not the Point. Cognitive Load Is.

You might read “clone yourself” and think this is about saving time. Getting your emails drafted, automating a report, managing your calendar. That stuff matters. It’s not the real prize.

The real scarcity in your life right now is not hours. It’s headspace. Every small obligation takes up cognitive territory. Updating a deck, answering a question about a number you already know, dragging a file into an email. A 30-second interruption doesn’t cost you 30 seconds. It costs you the thread of thought you were following before the ping arrived, the ten minutes it takes to find it again, and the growing chance that the next interruption hits before you do.

Clear that noise and you have something to invest: not just time to do things, but time to think about things. An hour every day processing ideas, talking through what you could build, where the opportunities are. You cannot get that hour if every available minute is consumed by tasks that could have been handled by something other than you.

Time is all you need.

Put that cleared space to work and it compounds fast. But that assumes you can build the thing that clears the space without burning months on it.

You Don’t Need Evals. You’re the Fucking Eval.

The bottleneck isn’t building things. Building things is cheap. The bottleneck is knowing whether the output is any good.

This is where AI projects die. Someone builds a demo, leadership signs off, they scale it to real users, and it collapses. The person who built the demo was unconsciously doing the evaluation, testing against cases they understood. The moment it lands in someone else’s hands, the failure modes multiply faster than anyone can catalog them.

I think engineers automated first because they could close that loop instantly. They looked at the output and knew whether it was right. Look, know, done.

When I try to do this at my day job, building AI tools for lawyers, it’s a completely different experience. I’m not a lawyer, so every time the AI produces output I can’t just look at it and know. I have to get the output into a format a lawyer can assess, which takes iteration because I probably won’t nail the UI the first time, then the lawyer reviews it on their schedule, then I review their review to make sure they evaluated it properly, then I feed the corrections back in, and maybe the annotations from one lawyer are biased so now I need a representative sample, and every time the underlying system changes I run through all of it again. Seven friction points, minimum.

When you build for yourself, all of that collapses. You look at the output, you know if it’s right, you fix it. Done. No validation pipeline. No waiting on someone else’s calendar. Just you and your own judgment, moving as fast as your brain can process.

It costs you almost nothing. The only question left is where you find the time.

The Byproduct Is the Work

When I tell people to automate themselves, they usually think they need to carve out time from their real work to build this AI thing. Ask their boss for two weeks to experiment, stay up late, squeeze in side projects on the weekend.

That’s not how it works. The byproduct of building the clone IS the work output. The best way to build an AI analyst is to do the analysis. You’re not doing something extra. You’re doing the exact same work you were going to do anyway, through AI instead of by hand.

What’s on your plate this week? What did your boss ask for by Friday? Do those things. Just do them with AI. Your five deliverables this sprint are your training data, your eval set, the thing you build the clone on. The first task might take the same amount of time, maybe more because you’re setting up scaffolding. The second one is already faster. By the end of two weeks, if you had five things to get done, you’re going to deliver 20, because every task teaches the system something that carries forward into the next one and the next one and the next one. That’s the whole damn point. You’re not taking time off from your work. You’re doing your work.

Your boss gave you five things to deliver. You delivered twenty. Nobody complains about that. And it happens faster than you think.

By Friday, You’re Faster

I’ll make the progression concrete.

Week one. You connect the AI to your actual data sources, the messy ones you work with every day. You run it on a report you were already going to produce. You force it to write the SQL for every number and output those queries where you can see them. Then you check them manually. You can do this fast because you wrote similar queries yourself last month.

The first time is clumsy. You’re teaching it what your metrics mean, correcting column names, explaining business logic you’ve known for years but never had to put into words because no one else needed to hear it. In multiple cases when I did this, the AI corrected queries where I had been reporting the wrong metric for months. In other cases it chose the wrong column and got the number wrong, but I caught it immediately because I know this data. Nudge it, save the correction, it doesn’t make that mistake again. Look at the output, know if it’s right, correct it, move on.

By my third analysis I was moving faster than I would have been by hand. Every correction gets stored. Every pattern it learns reduces the surface area of things that can go wrong. The return curve bends upward and it bends fast.

Being 10 days ahead of someone is a long time now. Ten days of compounding in this environment puts you in a different operational reality than the person who hasn’t started. But you can catch up quick. Don’t be scared to start.

The Person Who Can Replace Themselves Is Irreplaceable

The person who knows how to replace themselves is irreplaceable.

Three things happen once you’ve automated your own work, and they layer on top of each other.

First, the stuff you knew you had to do gets done in a fraction of the time. Your committed deliverables, your sprint goals, the reports leadership asked for. You haven’t changed what you deliver. You’ve changed how much of your life it costs.

Second, the stuff you always wanted to do but never had time for. That analysis you knew would be valuable but could never justify spending three weeks on, that process improvement you keep meaning to propose, the thing you’ve had in the back of your head for months that you never started because the urgent work always won. You have actual hours now. Not theoretical time. Real hours in your day that are no longer consumed by the work the clone handles.

Third, you start doing things you didn’t even know were possible. Problems you couldn’t see because you never had the bandwidth to look up from the work long enough to notice them. I started building systems to teach others the process. That wasn’t on any roadmap. It only became visible once the noise cleared.

That’s what’s happening to me right now. And it’s happening to people around me who started doing this. The most valuable person in the room isn’t the one doing the most. It’s the one who built the system and now sees what no one else can see.

Just Start

Clone yourself first.

Tomorrow morning, take whatever is at the top of your to-do list. Do it through AI instead. Look at the output, correct it, do the next one. By Friday, you’ll be faster.

Time is all you need. You’re about to get a lot of it back.

Want to build an AI Analyst in Claude Code?

Join our weekend bootcamp on April 4-5. Eight hours, three instructors, leave with a working repo.

Join the Bootcamp

You’ve heard this line. You’ve probably said it, at a conference, in a team meeting, maybe scrolling LinkedIn. It felt right. You could keep doing your job, add some AI for the boring parts, and the human work would stay yours. Business as usual, but shinier.

You could picture yourself in that future. It was comfortable.

Maybe that was true before. It’s not true anymore.

The models got good enough to do the whole job. Not just help with pieces of it. The whole thing, from question to finished deck, end to end. And it happened fast. Things that weren’t possible three months ago are possible today. What changed isn’t that AI got “a little better.” It crossed a line. The thing you do all day can be done by AI. Right now.

AI is going to replace the data analyst. It’s going to replace the analytics engineer. It’s going to replace the data scientist. Not “augment.” Not “change the way you work.” Replace.

Your job right now. Writing queries, building dashboards, answering the same segmentation question for the third time this quarter. That job is done.

The “augment, not replace” line is bullshit. It’s a comfortable story we told ourselves while the ground was shifting. I know because I told it to myself too, right up until I built the system that does my job better than I do.

I’ve been doing this for over a decade. I built an AI data analyst, tested it on the worst data I’ve ever worked with. It caught mistakes I had been making for months. Numbers I report to leadership. Numbers I’d been confident in.

It scared the shit out of me. I’m not writing this from a comfortable distance. More on that at the end.

What comes after the fear is something better. I’m on the other side of it. Stay with me.

Your current job is done. Your next job is more interesting.

The Order of Falling

There’s an order to how this hits the data profession. The roles closest to coding fall first. Coding was the first thing AI got really good at.

Software engineering falls first. It’s already fallen. Building pipelines, shipping applications, infrastructure. What used to require a development team now requires a conversation. And the people who made this happen? Coders, building the tools that automated their own work. They could validate the output because they knew what good code looked like. The expert builds the system that replaces them. Remember that pattern.

Data analysts are next. It’s already here. The core loop of a data analyst is: take a question, explore data, find the answer, present it. AI can do every step of that right now.

Think about what covers 80-90% of the questions your data team fields: funnel analysis, segmentation, drivers and root-cause analysis, trend analysis, cohort analysis, opportunity sizing. If your daily work is on that list, the AI already does it.

A straightforward segmentation or funnel analysis that takes an analyst three days? The AI produces it in ten minutes. Prompt to finished deck. A complex deep-dive that takes three to four weeks? Forty minutes to an hour and a half.

Most of the work data teams do is pattern execution. Not because the people are bad. The patterns are just predictable. And predictable is exactly where AI is strongest.

The output competes with the strongest analysts I’ve worked with. The ones everyone goes to with the hard questions. I’m not running a study here. It’s what I’ve seen, and my boss who built his own version sees the same thing.

Analytics engineering gets absorbed next. Nobody sees this one coming. Analytics engineering doesn’t get replaced by a separate AI system. It gets consumed as a side effect of the AI analyst getting smarter.

As the AI analyst runs analyses, it hits walls. It needs better data, cleaner tables, business context it can trust. So what does it do? It starts building the clean, documented semantic tables that analytics engineers spend months on. It stores its learnings and builds rules from its mistakes. This isn’t a prediction. It’s already started happening. I watched it happen. The AI analyst started building exactly the kind of semantic tables our analytics engineering team had been working on for months.

The critical business context trapped in people’s heads? The AI extracts it through conversation and puts it where anyone can use it.

Data science is the last domino. Causal inference and experimental design require more sophisticated reasoning. It’s months behind, not years. If you’re a data scientist, the five paths below apply to you too. Especially Validator and Translator. Your training in experimental design makes you perfect for those.

You want a timeline? Watch the pace of frontier model releases. It’s not a date on a calendar. It’s a rate of acceleration most people haven’t internalized.

The Proof

I built an AI data analyst in about a week by talking to it. Tens of thousands of lines of code, all generated through conversation. Not by writing code. By describing standards, correcting mistakes, teaching it what good analysis looks like. Then we tested it on real internal company data. Eleven years of bad data foundations at my company. The kind of data where people give up and just work around the problems.

I ran it against our North Star metrics. That’s where it got scary. Numbers I report to leadership. Numbers I’ve been working with for months. A query with 10+ CTEs, some with 15+ table joins. Everything tied out except one number. I asked the AI why. It found the bug: two table abbreviations were accidentally swapped in a join, pulling data from the wrong tables. The system doesn’t re-run your SQL. It writes its own query from scratch to verify every number independently.

The AI was right. I had been reporting the wrong number to the business for three months.

That’s when it got really scary.

It wasn’t a one-time thing. It caught multiple errors across different analyses. Silent errors that had been passing through normal review for months. These weren’t stupid mistakes. They were the kind of errors that pass peer review because everyone makes the same assumptions about the data. The profession’s quality floor is lower than anyone wants to admit, and the AI just exposed it. The system is not infallible. It makes mistakes too. But it learns from correction, and it documents everything it does. More than most teams can say.

I thought this might be a fluke. My boss, who manages the data team, built his own version. Another data professional did it independently, different data, different domain. Same pattern. Same results.

Five Paths Forward

I sat with that fear for a while before I started asking a different question: if the job you have today is done, what’s the job you have tomorrow?

These aren’t a numbered list to pick from. They’re a landscape.

Become the Builder

You are the subject matter expert. You are the single best person in the world to build the AI system that replaces your current role.

If a software engineer tries to build an AI analyst, they have to hire you to check their work. They can’t tell if the output is right or wrong. Only you know your data well enough to validate it.

The skill isn’t coding. Coding is solved. A data professional who couldn’t ship a production application a year ago can build one now by describing what they need. The skill is domain expertise. You know what right looks like. You know which questions to ask. The frontier models and tooling to do this are here right now. They just need someone who knows the domain to sit down and start talking.

Monday morning: Take a question you answered last week. Feed it to an AI with your data. See what comes back. Don’t worry about whether it’s perfect. Worry about whether you can tell what’s wrong and teach it to fix it. If you can, you’re already doing the new job.

Become the Validator

Once the AI is doing the analysis, the problem shifts from producing work to trusting it. Someone has to answer: “Can we act on this?”

That trust doesn’t come from gut feeling. You build it by testing the AI against questions with known answers to establish ground truth. You annotate your team’s historical query library as a benchmark. You set up feedback loops where the people consuming the analysis flag what feels wrong.

Nobody panics about human drivers killing 40,000 people a year. One AI accident makes national news. The AI analyst will face that same scrutiny, and it doesn’t need to be perfect. It needs to be better than you. It already has one permanent advantage: it documents everything, every time, automatically. So your job as validator isn’t to catch every mistake. It’s to build the systems that make trust scalable.

Monday morning: Take the last 10 analyses your team produced. Feed the same questions to the AI. Compare. Track every divergence. That divergence log is the beginning of your validation framework.

Become the Translator

There’s a gap between what a business needs and what an automated system produces. That gap is your job.

A VP asks “why is conversion down?” and you turn that into the real questions. Is it traffic? Is it the product? Is it seasonal? That kind of breakdown is what keeps decisions grounded instead of reactive.

Then you go the other direction. The AI produces findings. You turn them into decisions. You know which finding matters this quarter and which one can wait. You read the politics. Why is that VP really asking right now? Is someone’s job on the line?

This is a human skill. It takes empathy and business context. You have to know the organization and how to talk to people. The AI can do the analysis. It can’t sit in the room and read the room.

Monday morning: Look at the last stakeholder request your team received. Write down the question they asked, then the question they actually meant. Those are probably different. That gap is your job security.

Become the Architect

Messy data and undocumented business logic. The AI can work around some of it, but someone needs to actually fix it.

Analysts work around messy data. They build workarounds into their queries, store the knowledge in their heads, and move on. All that stuff in your head? It needs to get out and into a system. That’s the shift. Stop working around bad data. Start fixing it.

Nothing you do will matter more than this. Every improvement to the data foundation makes every automated analysis better, automatically, forever. Fix a metric definition once and every analysis that touches it gets it right from now on. It’s the analytics engineering job, evolved. Not maintaining tables. Building the data layer that makes trust work. The AI analyst is already doing this as a side effect of getting smarter. Your job is to guide that process and make it production-grade.

Monday morning: Write down the three things about your data that everyone on your team knows but nobody has documented. A metric that means different things in different contexts. A table with a known edge case. A date column that doesn’t mean what its name implies. Getting that out of your head and into a system is the work.

Become the Solver

Do the damn work. Get shit done.

You automated your own domain. You know your data, your metrics, your edge cases cold. Good. That was the Builder path. But here is what happens next. You start noticing things that have nothing to do with your team.

You are sitting in a product review and someone mentions onboarding dropout is up. You pull churn data later that day and see the same cohort. You check support tickets. Same spike, same timeline, same customers. Three teams, three dashboards, three conversations, and nobody connected them because nobody sits across all three datasets. You do now.

That is the Solver. Not deeper expertise in one domain. Broader vision across all of them. Organizations are full of walls that humans built and never questioned. The person who walks into a room they do not own and says “teach me your numbers” is the person who finds the problems nobody else can see. Domain expertise got you in the door. Humility gets you into every other room.

Monday morning: Find a problem that touches two teams. Pull both datasets. See what nobody’s connecting. Then start building the fix.

The Window Is Open. It Won’t Be Forever.

Right now, today, very few data professionals understand what’s happening. Most are still debating whether AI will “augment” their work. Their companies are running AI strategy workshops and pilots.

Meanwhile, a small number of people are already generating months of analytical output every week. In two days, I produced more validated analyses than I’d normally deliver in a month. Not every analysis was a deep-dive. Many were the routine segmentations and trend analyses that eat most of your time. But that’s the point. The routine work is what fills your calendar, and the routine work is exactly what AI handles first.

The people who move now build the deepest expertise, take the most interesting roles, and shape how their organizations adopt this. The people who wait get handed a playbook someone else wrote.

Ten years ago, the people who built the first data teams became the VPs of data. Same thing is happening now.

If your organization won’t adapt, find one that will. I know that’s a hard thing to say. The alternative is worse.

If your organization is fighting it, build the skills anyway. Start with the willing. Let results speak.

The technology is ready. Your organization probably isn’t. That gap between what’s possible and what’s adopted is your runway. Use it.

Automate Yourself Out. Bring Everyone With You.

The new job is three things:

Grow and nurture AI systems. Treat them like people you’re onboarding. Not a metaphor. It’s literally how it works. You talk to them. You teach them. You correct them. You watch them learn. You don’t configure software. You develop a colleague. Every company needs the person who can do this. Almost nobody has them.

Execute on what they produce. The AI will generate more good ideas than you can implement. Your job is no longer generating insight. It’s turning insight into action. Building the thing, not just recommending it. Which is actually the job you wanted when you got into data in the first place.

Bring everyone with you. Open source what you build. Teach people willing to learn. The code costs nothing now, so give it away. The people you help today become the network that helps you tomorrow. This isn’t altruism. It’s strategy. And it’s the right thing to do.

That process of building, teaching, and correcting these systems? It transfers. Once you’ve done it for your domain, you can do it for any domain.

Automate yourself out. Bring everyone with you.

Send this to your data team. Send it to your manager who still thinks this is five years away. Start the conversation now.

Take Care of Yourself

This mindset shift is hard. It hit me hard. Over the past couple months, as all of this became clearer, I went to therapy. I quit drinking. I changed my entire lifestyle. The technology wasn’t the hard part. Watching the thing you’ve spent your career getting good at become automatable -- that’s its own kind of crisis. It rearranges how you think about who you are.

If you’re feeling that right now, you’re not weak. You’re paying attention.

You need a clear head to build what comes next. You can’t reinvent yourself while you’re falling apart. The decisions you’re going to make in the next six months will define your career for the next decade. Make them clear-headed.

When you start doing this, you’re going to get unlocked. You won’t be able to stop. It’ll be scary and exhilarating at once.

Get outside. Move your body. See your friends every week.

Take care of yourself while you do it.

And start.

The system I built is open source.

Your question quality really determines your output quality. This is especially true in the age of AI, because the more generic things you ask of an LLM, the less precise the answer that comes back.

.That before/after isn't hypothetical. "Analyze user engagement" will get you a generic summary. "Which 3 engagement actions in week 1 predict 90-day retention, so we can prioritize them in onboarding?" gets you something you can actually act on. Same data, completely different question.

Subscribe now

In my experience, the thing missing from most weak questions is always the same: the decision. No one has articulated what they’re actually going to do with the answer.

I use a simple three-element checklist to catch this before starting any analysis. It’s not fancy, but it works.

First: is it decision-tied? What specific decision does this inform? If the honest answer is “it would be interesting to know,” that’s a red flag. Interesting is not a decision.

Second: is it data-grounded? Can you actually answer this with data you have? A beautiful, well-formed question is useless if the data doesn’t exist.

Third: is it specific enough? Is the question bounded well enough that you’ll know when you’re done? Not a “boil the ocean” project - something targeted you can actually solve and act on.

Here’s what this looks like in practice. Take the churn question - it’s one I see constantly.

Version 1: “Why are users churning?” Sounds reasonable. But the decision is completely unclear. What are you going to do with the answer? What does “churning” even mean here? Two executives in the same org can mean totally different things by that word.

Version 2: “What’s our churn rate by cohort?” Better - the data probably exists, and it’s more specific. But the decision is still missing. You’ll have a table of numbers and no clear instruction for what to do next.

Version 3: “Which user behaviors in the first 30 days correlate with churn, so we can trigger interventions before they leave?” Now you’ve got something. The decision is explicit (which interventions to build), the data is measurable (behavioral logs), and the scope is bounded (first 30 days, ranked behaviors).

The philosophy here isn't more words. It's more targeted framing. Even when you're not spelling out the three elements explicitly, keeping them in mind changes how you think about the question before you start.

I’ve heard this pushback a lot: “I’m just exploring. I don’t have a decision yet.”

I get it. But I think when someone says that, a decision is usually hiding, not absent. You’re curious about something for a reason - there’s a hunch there, even if it’s not fully formed.

The reframe I like: even exploration has a direction. “I’m exploring user behavior” isn’t a question. “What patterns would change our next sprint priorities?” is. “I’m just looking at the data” isn’t a question. “What would I need to see to recommend we invest in X?” is. And “I don’t have a decision yet” - well, figuring out what’s worth investigating further is the decision.

There’s a useful way to think about it: undirected exploration is a fishing expedition. You’re just seeing where the fish are. Directed exploration is discovery with purpose. You’re not going out with no plan - you know what you’re looking for and why it matters.

One thing I’d add on the AI angle: rather than asking your LLM “Is this a good question?” and getting a yes or no, have it walk you through the checklist instead. The coaching conversation helps you find the gaps in your own thinking, which builds judgment faster than just getting a grade. Coaching is better than grading.

Here’s a practical move I’ve found has disproportionately high returns: when someone hands you a question to answer, before you start, ask them - “What decision are you trying to drive with this?”

About 60% of the time, they pause and realize it’s not actually a great question. One follow-up question, and you’ve potentially saved hours.

So yeah, before you start the next analysis, pressure-test the question first. It’s probably the highest-ROI thing you can do.

Subscribe now

Check out for our upcoming courses and free workshops: https://dataneighbor.com

I built an AI data analyst inside Claude Code that takes any dataset and produces a full slide deck (charts, narrative, recommendations) in about 15 minutes. No Python. No SQL. No coding at all. The entire system is markdown files that Claude Code reads and follows. This post is the full walkthrough: how it works, what it produced on real data, and how you can build one yourself.

If you work with data in any capacity, this is for you. PM, DS, analyst, engineer who gets pulled into “can you look at this?” conversations. Could you take this to work on Monday and actually use it? By the end of this post, I think the answer is yes.

The tools

A few things worth explaining up front so you know what you’re looking at.

Claude Code is Anthropic’s AI coding agent. It runs in your terminal, or inside an IDE, and operates directly in your codebase. It reads your project files, writes code, executes commands, and follows instructions you define in markdown files. It’s an agent that works inside your project with full access to your files, your data, and your tools.

Opus 4.6 is the model powering Claude Code. Anthropic’s most capable model. It’s what gives the system its analytical judgment, its curiosity, and its ability to turn a vague ask into a structured analytical question. You select it when you launch Claude Code.

Google Antigravity is the IDE where this all runs. Think VS Code but with Claude Code built in natively. You can see the terminal, the file explorer, and the output side by side. Not required (Claude Code works in any terminal) but it’s a nice workspace for seeing everything at once.

Cost and usage. Claude Code runs on your Anthropic subscription. Usage limits refresh every few hours and weekly. The entire Hawaii analysis, from raw CSVs to finished deck, ran within a single session’s limits.

Start with the output

Before I explain how any of this works, here’s what it produced.

I pointed it at Hawaii Tourism Authority data and said “go analyze this.” Monthly visitor arrivals by island, country of origin, and airline capacity, 2024 vs 2025. That was my entire prompt.

This wasn’t a clean dataset. It was dozens of separate monthly CSV reports from a government website. Visitor arrivals, spending by category, airline seat capacity, source markets. Each one a separate PDF or spreadsheet download. The kind of data that normally takes half a day just to wrangle into something usable.

Fifteen minutes later, I had a complete slide deck. Not a rough draft. A finished presentation with a narrative arc, publication-ready charts, and actionable recommendations, each one tagged with a decision owner, success metric, and follow-up date.

Here’s the full 20-slide deck. Download the PDF.

I didn’t write a single SQL query, make a single chart, or outline a single slide.

How the system works

It’s a system of markdown files that live in a repo.

The whole thing is controlled by one file: CLAUDE.md. Think of it as onboarding docs for a new analyst. It has six sections:

1. Who You Are sets the persona in one line: “You are an AI Product Analyst.”

2. What You Do defines scope and boundaries. Funnel analysis, segmentation, drivers analysis, yes. Predictive modeling, dashboards, no.

3. Your Skills is a registration table. Each skill has a name, a file path, and a trigger condition. When the trigger matches, Claude Code reads and follows that skill automatically.

4. Your Agents is another registration table. Each agent has a name, a file path, and an invocation condition. You call agents on demand for specific multi-step tasks.

5. Default Workflow is the step-by-step playbook: frame the question, explore the data, analyze, validate, tell the story, present.

6. Rules are non-negotiable guardrails: “always validate SQL before presenting results,” “never present unvalidated findings as conclusions.”

Everything the system does traces back to this one file. And it’s plain text. Editable by anyone.

Skills vs agents

This is the core distinction that makes the whole system make sense.

Skills define HOW things get done. They’re standards, patterns, rules. Like a style guide. They’re always active. When Claude Code generates a chart, the visualization patterns skill automatically applies. When it starts an analysis, the data quality check skill runs without being asked. You don’t invoke a skill. It applies itself whenever the trigger condition matches.

Agents define WHAT gets done. They’re multi-step workflows with inputs, steps, and outputs. You invoke them on demand: “run the descriptive analytics agent on this dataset.” Each agent reads its markdown file, substitutes the variables you provide, and executes the workflow step by step.

The connection: skills tell agents what “good” looks like. Change the chart color palette in the visualization skill, and every agent that makes charts (descriptive analytics, chart maker, storytelling) produces different-looking output. You didn’t touch any of those agents. You changed one skill file. That separation is what makes the system modular and composable.

The pipeline: 15 agents across 6 phases

The full workflow uses 15 agents and 12 skills organized into six phases. Some phases run in parallel (the Explore agents can work simultaneously on different angles of the data). Others are strictly sequential (nothing gets charted until the story architect designs the narrative arc). Four explicit checkpoints gate the process. If the Simpson’s Paradox check hasn’t run, if chart titles collide with slide headlines, if recommendations aren’t ranked by confidence, the pipeline halts until the issue is fixed.

Phase 1: Frame

• Question Framing turns a vague ask into structured analytical questions with decision context

• Hypothesis generates testable theories across four cause categories

Phase 2: Explore (agents run in parallel where inputs allow)

• Data Explorer profiles what data exists, its structure, and its quality

• Descriptive Analytics runs segmentation, funnels, and drivers analysis

• Overtime/Trend finds patterns over time, seasonality, and anomalies

• Root Cause Investigator drills down iteratively through dimensions, up to seven layers deep

• Opportunity Sizer quantifies business impact with sensitivity analysis

• Experiment Designer designs A/B tests with power estimation and decision rules

Phase 3: Validate

• Validation re-derives key numbers independently (writes new queries, not copies) and cross-checks arithmetic. Flags common traps like Simpson’s Paradox and survivorship bias.

Phase 4: Story (sequential, each step feeds the next)

• Story Architect designs narrative beats following a Context/Tension/Resolution arc. The number of beats is emergent, not a target.

• Narrative Coherence Reviewer validates story flow before any charting begins

• Storytelling writes the narrative from the storyboard

Phase 5: Charts

• Chart Maker generates each chart following Storytelling with Data methodology

• Visual Design Critic reviews against a checklist: action titles, direct labels, gray-first-then-color

Phase 6: Deliver

• Deck Creator assembles Marp slides with theming, breathing slides for pacing, and speaker notes. Runs a final design review.

Twelve skills run throughout, shaping how every agent works:

• Analysis: data quality check, question framing, analysis design spec, triangulation

• Metrics: metric spec, tracking gaps, guardrails

• Output: visualization patterns, presentation themes, stakeholder communication

• Process: close-the-loop, run-pipeline (orchestrates the whole thing)

What surprised me: the judgment

The SQL, the charts, the database, the slides. That’s all impressive execution. But the part that actually surprised me was the analytical judgment.

In the Hawaii data, the statewide visitor count was essentially flat. Down 0.4% year over year. Most tools would report “no significant change” and move on.

The descriptive analytics agent includes a mandatory Simpson’s Paradox check. Even when the aggregate looks flat, it automatically segments by default dimensions (geography, channel, device, cohort) to see if opposite trends are hiding underneath. In this case, they were. Maui had surged 7% while every other island declined. The aggregate was hiding completely opposite trends. The agent caught it unprompted because the check is built into the workflow, not dependent on me asking the right question.

Then it decomposed O’ahu’s decline by accommodation type, finding that hotel visitors drove the loss while vacation rentals held steady.

The root cause investigator went further. It follows a “peel the onion” approach: start with the surface observation, test each available dimension to find which explains the most variation, isolate the specific segment responsible, then repeat. Up to seven iterations deep. It doesn’t stop at “Maui is different.” It drills into which source markets, which routes, which months are driving the divergence.

The validation agent then re-derives the key numbers independently and cross-checks with the triangulation skill. No finding makes it to the narrative without arithmetic verification.

And the charts reflect it. Every chart follows Storytelling with Data principles: action titles that state the takeaway, direct labels instead of legends, gray everything first and use color only for the story point.

The story architect then takes all the findings and designs a narrative arc. Context, Tension, Resolution. The number of beats and charts is an emergent property of the story, not a target. The narrative coherence reviewer validates that each beat’s transition question is answered by the next one. No gaps, no orphaned insights, no charts that don’t earn their place in the deck.

It gets better every time

The one-shot quality is impressive, but it’s not the most important part. It gets better every time you use it.

The Hawaii analysis was the second major run. The first was on NovaMart, a synthetic e-commerce dataset. That analysis was good, but I gave it specific feedback afterward. What happened next is what makes this approach fundamentally different from prompting a chatbot.

I told it the chart titles were duplicating the slide headlines. It didn’t just fix the charts in front of me. It updated three separate agent files (chart-maker, deck-creator, and visual-design-critic) and added a validation check so the issue would never recur on any future dataset.

I told it the narrative voice was too corporate and dramatic. It rewrote its own storytelling rules and added voice guidelines to the story-architect and storytelling agents. Now every analysis comes out in the right voice automatically.

I pointed out that chart annotations were colliding with data points on dense charts. It added collision detection logic to the chart-maker agent and a specific review criterion to the visual-design-critic.

This is the key difference between a chat-based AI workflow and a system-based one. In a chat tool, you start fresh every time. There’s no institutional memory. Here, every piece of feedback becomes a permanent improvement to the system files themselves. The agents, the skills, the rules. Self-improving configuration, all living in the project repo, compounding with every run.

If you’ve ever built a team knowledge base or maintained runbooks that get better over time, you already understand this pattern. The difference is that here, the system can update its own documentation based on what it learns.

How we built it

You don’t need to replicate the full 18-step pipeline to get started. But the process matters. There’s a six-step build pattern that applies to building any skill or agent:

1. Explore. Brainstorm with Claude Code. Don’t build anything yet. Talk about what you want to build, what problem it solves, who the output is for, what “good” looks like. The brainstorm that kicked off our system was a one-hour conversation that covered audience, scope, constraints, and exit outcomes before we touched a single file.

2. Spec. Tell Claude Code what you want the skill or agent to do. It writes the specification for you: purpose, trigger conditions, instructions, examples, anti-patterns. You review. You talk through it. Claude Code creates the markdown file.

3. Plan. Claude Code breaks the spec into a concrete file list. What files need to be created, where they live, what each one contains.

4. Build. Claude Code writes the files. Skills go in .claude/skills/. Agents go in agents/. Each one is a markdown file with plain English instructions. Human readable, human editable if you want, but Claude handles the creation.

5. Test. Run on real data. Look at the output. Does the chart theme match? Did the agent follow all the steps? Did it miss anything?

6. Iterate. Give specific feedback. Claude Code updates the files itself. Run again. Every cycle makes the output better.

The key thing: you never have to write these files by hand. You describe what you want in plain English and Claude Code creates them. I use a speech-to-text flow to dictate feedback while reviewing output. Talk, review, iterate. The entire system is markdown files that Claude Code reads and maintains.

We went through this loop multiple times before the Hawaii analysis. Each run produced noticeably better output. There’s no proprietary tooling. Just Claude Code and a well-structured project.

Build systems or get replaced by AI

This is building decks that are ready for me to walk into a Monday morning meeting and present to execs. Recommendations, action items, decision owners, follow-up dates. All from a single prompt. 90% of analysts and data scientists I know can’t operate at this level. Not because they lack the skill. Because they’re still doing everything manually, one query at a time.

The analysts who will thrive in the next few years aren’t the ones who run the best queries. They’re the ones who build systems. Who encode their analytical judgment into tools that compound over time. The rest will get replaced by Claude Code.

Open a terminal. Point Claude Code at a dataset. Ask a question. “Why did signups drop in January?” or “Which user segments have the highest engagement?” or “What’s driving the change in our conversion rate?” It frames the question, explores the data, runs the analysis, validates the numbers, and produces a narrative with charts you could present that afternoon.

Want to see this live before building your own? We’re running a free 2-hour workshop on February 27 where we walk through the system and run a live analysis. No commitment, just come watch it work.

If you want to build one yourself, the Build AI Analysts in Claude Code bootcamp runs March 28-29, 2026. Weekend intensive, eight hours total. You walk through the full six-step build pattern from brainstorm to working system. You leave with a complete repo: skills, agents, connected data sources, and a finished analysis you ran yourself. Three instructors providing real-time support. No coding experience required.

If you want to go deeper after that, we have a five-week course that teaches the full analytical workflow (question framing, metrics, deep dives, experimentation, storytelling) and how to build systems like this for your own work.

The tools are here. The gap is about to get obvious.

Want to build a system like this?

Join our weekend bootcamp on March 28-29. Eight hours, three instructors, leave with a working repo.

Join the Bootcamp ($600)

I remember the first time I saw a data scientist bring a Jupyter notebook to a business stakeholder meeting. The exec squinted at the screen. “Can you just... make it a bar chart?” The data scientist looked confused. “But the notebook shows all the transformations, the statistical…” The exec had already checked their phone.

Xavier Pladevall, co-founder of Index, has had that conversation about 500 times now. And what he found should make every data person rethink what they’re building.

“There’s this huge disconnection between what people are talking about in data world - Twitter, conferences, investors, VC-driven things - and what’s actually on the ground, where people are using very simple tools.”

While Data Twitter debates semantic layers and metrics frameworks, users are asking for clean dashboards. While we argue about the perfect data stack, stakeholders want a chart that doesn’t look like garbage. The gap between what we think matters and what actually moves the needle is enormous.

What people actually want from data tools

After talking to close to 500 people, Xavier landed on patterns that should be obvious but somehow aren’t.

1. Form factor beats features

Notebooks are powerful. Stakeholders hate them.

“Customers constantly keep saying that they don’t want a notebook. They don’t have a mental model for how notebooks run.”

Nobody’s VP of Sales walks into your office asking for a Jupyter notebook. They want a dashboard. They want a chart. The industry spent years pushing notebooks as the future of analytics because they’re flexible and reproducible. Users spent years ignoring notebooks because they don’t fit how business people think.

This isn’t about dumbing things down. It’s about matching the tool to how people actually work. Figma didn’t win because it had more features than Sketch. It won because the form factor - browser-based, collaborative, real-time - matched how design teams actually operated.

Same data, same analysis, different package. One gets used. One sits in a repo.

2. Design quality isn’t cosmetic—it builds trust

Here’s something nobody talks about: ugly charts get trusted less.

Same data. Same analysis. Put it in an ugly dashboard and stakeholders doubt it. Put it in a polished one and they act on it.

Shane Butler said it plainly on the podcast: “There’s some implicit belief that someone has credibility, or the graph has credibility, if it doesn’t look like a piece of shit.”

Xavier confirmed: “They’re gonna trust it less.”

This pattern exists across all software. Linear didn’t beat Jira on features. It beat Jira because it felt professional to use. Notion didn’t beat Confluence because it had better databases. It won because people wanted to open it.

Data tools get a pass on design because we treat analysis as purely logical. But humans don’t work that way. Presentation affects perception. If your dashboard looks like it was built in 2008, people assume your analysis is stuck there too.

Subscribe now

3. Data quality matters more than every other feature combined

Xavier called it the Pareto principle of analytics: “I think data quality - everything else might be a distraction. Everything I tell you might be a variation of data quality.”

The NBA example is perfect. Index’s AI confidently told them the Lakers won 219 championships. Why? It confused playoff games with championships. The AI worked fine. The data was messy.

This is why the AI hype cycle in analytics shifted so fast. A year ago, everyone talked about context windows and model capabilities. Now the conversation is: “Can we trust this in production?”

You can have the smartest AI, the sleekest interface, the fastest queries. If your underlying data is bad, you get analytics slop. Bad threads. Bad chats. Confident answers to the wrong question.

Most data teams know this. But we keep getting distracted by shiny new tools instead of fixing the boring foundation work. Clean your data or nothing else matters.

4. SQL matters more than ever (but it’s getting abstracted)

“It’ll always matter and it might even matter more... it’s like lingua franca now that everything boils down to. Because you have Cursor, it doesn’t mean code doesn’t matter. It matters probably more than ever.”

The pattern: fundamental skills become more important as they get easier to execute.

Code didn’t die when Cursor launched. Developers write more code, faster, with AI assistance. SQL won’t die as AI tools generate queries. Data people will run more analysis, faster, because the execution barrier dropped.

Xavier borrowed the “low floor, high ceiling” idea from Figma’s Dylan Field: “The tool should be very easy to use for a complete noob. And if you have the best designer in the world, that should be their tool.”

AI in analytics should work the same way. Beginners get to useful answers without learning SQL syntax. Experts move faster by offloading boilerplate to AI and focusing on complex logic. The floor drops, the ceiling rises.

5. The future isn’t replacing dashboards - it’s making them proactive

Index 2.0 isn’t trying to kill dashboards. It’s trying to make them smart.

The idea: proactive question suggestions before you even ask. Chat alongside dashboards. Instead of replacing your existing tools, it sits next to them.

Why? Because rip-and-replace doesn’t work. People already have dashboards. They already have workflows. Telling them to throw it all out for your new thing is a losing strategy.

But what if your dashboard suggested the next question before you thought to ask it? What if the chart you’re looking at could answer follow-ups in plain English?

Xavier’s vision goes further: “I got my top 10 leads—sending them an email right from Index. Bringing the role of the data person as a center role, core to the business rather than this auxiliary thing that informs the business.”

The problem today: insight happens, then someone emails it, then someone discusses it, then maybe someone acts on it. Days or weeks later.

“It shortens the feedback loop. It collapses it basically. When will someone act on that insight? It might be days, months. Whereas if the data was much more connected, we could take action on it immediately.”

Data tools that connect directly to action tools. Insight to execution in one click. That’s the shift from reporting what happened to actually moving the business.

What this means for data teams

• Stop building for Data Twitter. Build for the person who just needs a clean bar chart.

• Invest in design. Your analysis is only as trusted as it looks.

• Fix your data quality before you add another tool to the stack.

• Learn SQL (or get better at it). AI makes it more valuable, not less.

Think about where your insights actually lead to action. If there’s a week-long gap, you’re losing.

Ready to level up your analytics skills?

We’re offering two new learning opportunities:

Free workshops: We’re running a series on AI evaluation and agentic analytics. Topics include analyzing product data with no-code tools, designing AI evaluation plans, creating custom annotation tools, and more. Full schedule at dataneighbor.com.

Join Our Free Live Lessons

Cohort-based courses:

• AI Analytics for Builders: Learn to leverage AI to become your own product data scientist, or magnify your impact as a data scientist. Claim your 20% discount here.

• AI Evaluations for Product Development: Learn to measure the actual impact of AI features on user and business outcomes. Claim your 20% discount here.

Both are hands-on, practical, 5-6 week programs where you’ll work with us and other practitioners in the field. Check it out at dataneighbor.com.

Honestly, we didn’t have a great answer.

So we built the thing we wished already existed: the AI Builders Slack Community by Data Neighbor. And you’re invited!

This is a place to build AI better with less guesswork.

This Slack community is to help you:

• Meet other AI builders

• Learn how to design and implement AI evals that actually support product decisions

• Learn how to build and use agentic analytics workflows to go from question → analysis → decision fast

• Share what you’re building and get feedback when you’re stuck

• Swap resources, examples, and patterns that work in real products

• Stay in the loop on free workshops, live sessions, recordings, and many other resources

-> -> Click here to Join the AI Builders Slack by Data Neighbor <- <-

Join AI Builders Slack Community

Invite your friends and colleagues: bit.ly/ai-connect

Looking forward to working with you all!

They improved a metric, or a rubric score went up, or the outputs look better in their error analysis or quick spot checks. Then someone in leadership asks the question that changes the whole conversation:

Did this increase profit, and did it increase risk?

That question is awkward because it mixes worlds. Engineering lives in model behavior and failure modes, while the business lives in retention, expansion, margins, and operational risk. And even when you know the change helped users, business outcomes often lag by weeks or months, while product decisions have to be made now.

Last week, we ran a full live workshop on this topic for over 250 AI builders. We walked through a practical framework for going from AI eval metrics to revenue, using a chain of scientific evidence that holds up in real roadmap decisions.

For context, I teach this workflow (as well as the entire end-to-end eval process) in AI Evals for Product Development, where we go deep on measurement design, instrumentation, analysis, rollout gates, decision making, and much much more.

Learn AI Evals Today!

In today’s issue, we’ll focus on how to validate the business impact of AI feature development.

If you’d rather watch than read, you can check out the live workshop for free here:

Your CFO doesn’t care about your F1 score

It sounds harsh, but it’s a useful mental model.

Leadership is not buying “quality.” They’re buying outcomes: retention, expansion, lower cost to serve, and reduced risk. Quality only matters insofar as it changes those.

If you want AI investments to be legible outside the engineering team, you need to be able to answer three questions:

What changed in the product?
What changed for users?
What changed for the business?

Most teams can answer the first one. Many can answer the second. Very few can answer the third with any confidence, especially on a tight timeline.

Levels: where most teams actually are

I like framing this as three levels, because it makes the gap obvious without making it personal.

Level 1: Shipping blind. You ship AI features, but you don’t measure quality directly. You rely on user complaints, intuition, and a handful of ad hoc checks. It’s hard to make confident investment decisions.

Level 2: AI quality confident. You ship AI and you’re confident in measuring AI quality with consistent metrics and processes. You can defend quality, but business impact is still fuzzy.

Level 3: Business connected. You ship AI, measure quality reliably, and you can calculate impact on business outcomes. You can connect improvements to expansion, retention, or cost reduction with real numbers and uncertainty bounds.

Most teams are between Level 1 and Level 2. The goal is to reach Level 3, because that’s when AI work stops being a constant justification battle and starts looking like a disciplined product and engineering loop.

AI quality is not customer value

A lot of confusion comes from mixing these two.

AI quality answers: is the output better?
Customer value answers: did the user’s job get easier?

Those sound similar until you ship something and realize they diverge all the time.

Quality metrics tend to look like correction rate, acceptance rate, error rate. They’re useful, but they’re not the outcome. You can drive those metrics up while the user still struggles to finish the job, because the workflow is still slow, confusing, or requires too much manual repair.

Customer value metrics tend to look like time to finish the task, time to a correct deliverable, task success rate. These are harder to instrument well, but they map to what users actually feel.

A simple rule of thumb that’s helped a lot of teams:

if a metric can improve while the user experience stays painful, it’s not customer value.

Business impact means profit, not “better product metrics”

The next confusion is assuming customer value automatically means business impact.

Sometimes it does. Sometimes it doesn’t.

Product metrics can go up while business outcomes go down. A feature can increase usage and also increase cost to serve. It can improve satisfaction and still fail to move retention. It can boost engagement while margins get crushed by compute costs and support load.

This is why business impact metrics tend to be boring and finance-shaped.

Revenue-side metrics: expansion, retention, revenue per account.
Cost-side metrics: cost to serve, support load, compute and operational cost.

That framing matters because it keeps you honest. “Impact” is not a vibe. It’s profit under constraints.

Common failure modes

These three statements have killed more AI budgets than any technical failure.

1. “The model got better.”
   You have impressive offline improvements, but you can’t show users benefited in real workflows. Leadership asks why they should keep investing.

2. “Users seemed happier.”
   Feedback is positive, but you can’t show retention or expansion lift. The business case stays unclear.

3. “Revenue moved.”
   You see correlation after a launch, but you can’t explain why. The result isn’t reusable for future decisions, and prioritization becomes political.

These are not bad intentions. They’re broken chains.

The AI Impact Chain

Here’s the framework that fixes the chain.

Feature change → AI quality → customer value → business impact

It’s four links. Nothing fancy.

The point is that each link must be measurable, because leadership will only trust the conclusion if the story is complete. If any link breaks, you’re guessing. If all links hold, you’re proving.

Case study: a text-to-SQL rollout decision

Let’s make this concrete with a realistic scenario: a text-to-SQL AI analyst.

Workflow: ask a question → AI writes SQL → runs query → chart → export.
Business model: seat-based pricing. Expansion is the growth lever. Assume $200 per seat per month.

A team builds a V2 that changes the SQL generation approach. It’s behind a rollout flag for a subset of users, and you have to decide:

1. roll out further,

2. hold,

3. or roll back.

V1 is single-pass SQL generation.
V2 uses multi-step reasoning with validation.

It adds a plan step, more structured schema grounding, and a validation loop that repairs when needed. It also uses more compute per question.

This is a familiar trade-off. The change plausibly improves correctness, but it may hurt latency and cost.

The metrics disagree

You don’t get a clean dashboard that resolves the choice. You get a messy pile of signals.

First: response latency (p95), how long it takes the AI to generate SQL.

V1: 30 seconds
V2: 120 seconds

If you stop here, you roll back V2, because four times slower is scary and very visible.

Then: user edits per query, how many manual fixes users make before the SQL is usable.

V1: 2.4 edits
V2: 0.3 edits

Now the story changes. The model is slower, but it’s closer to one-shot SQL, which might reduce rework.

Then: SQL fails to run (syntax error rate).

V1: 20%
V2: 23%

That’s a regression again.

Then: precision on Customer Support questions, measured via labeled error analysis.

V1: 50%
V2: 75%

Nice improvement.

Then: precision on Finance questions.

V1: 40%
V2: 30%

Regression.

At this point, most teams do the predictable thing. They argue about which quality metrics matter, which segment is more important, and whether the latency regression is acceptable. The decision ends up being driven by politics or fear.

This is exactly where the Impact Chain tells you what to do next.

When AI quality signals conflict, you move down the chain.

Customer value breaks the tie

When quality metrics conflict, you need a customer value metric that represents the user’s workflow outcome.

In this scenario, the customer value metric is:

Time to correct chart export.
Time from initial query to exporting a usable chart, including all user corrections.

This metric absorbs the whole experience. It includes latency, retries, edits, failures, and the general friction of getting to “job done.”

Now the decision becomes clear.

V1: 12.0 minutes
V2: 3.0 minutes

V2 accelerates the workflow by 4x, cutting completion time from 12 minutes to 3 minutes.

Latency still matters. It’s just not the outcome.

A slower first response can still lead to a faster end-to-end workflow if it prevents rework, and rework is where users bleed time and patience.

Experiments teach relationships, not just lift

There’s a second payoff here.

Experiments don’t just tell you whether a feature worked. They teach you how AI quality drives customer value, which is what you need if you want to move faster over time.

For example, in this scenario:

When edits decreased by 87.5%, workflow time decreased by 75%.

That’s useful because it lets you do three pragmatic things:

You can predict customer impact from quality signals earlier.
You can prioritize improvements that move the needle, instead of chasing generic quality.
You can build trust by showing causation, not correlation.

Over time, this is how teams stop doing “quality theatre” and start doing repeatable engineering.

By the way, we’re teaching a free workshops on Designing Experiments for AI Features later this month

Join Free Workshop

Business impact lags, but decisions cannot wait

Here’s the hard part.

Business outcomes like seat expansion and revenue show up months to quarters later. You can’t freeze product decisions for that long, and long experiments kill velocity.

So the practical solution is:

1. Measure customer value now, in days.

2. Use causal inference on historical data to estimate how that value translates into business impact.

3. Validate later as outcomes arrive.

Customer value becomes the leading indicator.
Causal methods convert it into revenue estimates.
You keep velocity without giving up rigor.

The 3-step pipeline: customer value to revenue

Here’s a practical pipeline you can implement.

Step 1: measure customer value.
Example: time to correct chart export.
Observed change: 12 min → 3 min (9 minutes faster).

Step 2: learn the relationship.
Estimate how value changes predict seat expansion.
Methods: panel regression, difference-in-differences, propensity matching.

The key output here is a coefficient with uncertainty bounds, not a single “magic number.”

Step 3: convert to revenue.
Seats lift = coefficient × value change.
Revenue lift = seats lift × revenue per seat.

Using the worked example from above:

Coefficient β = 5 seats per 1 minute faster export (per account).
95% CI: [4, 6] seats per 1 minute.

Seats lift per account: 5 × 9 = 45 seats per account.
Across 100 accounts: 45 × 100 = 4,500 seats.

Revenue per seat: $200 per seat per month.
Monthly lift: 4,500 × $200 = $900,000 MRR.
Annual lift: $10.8M ARR.
95% CI (annual): [$8.64M, $12.96M] ARR.

This is the difference between “we think it helped” and “here is the expected lift, here is the uncertainty, and here is how we’ll validate it over time.”

Annnd, we’re teaching another free workshops on Measuring AI Impact with Causal Inference later this month

Join Free Workshop

The sentence the business cares about

If you want a simple test for whether you’ve completed the chain, write the one sentence that a business partner would actually use.

V2 is expected to increase expansion and revenue by providing faster workflows that require fewer user corrections.

That sentence is short because the chain is complete.

How to implement this without boiling the ocean

If you try to do this across every AI surface area at once, you’ll get stuck in instrumentation and debates.

A good starting point looks like this:

1. Pick one workflow where the business actually cares.

2. Define one customer value metric that represents “job done.”

3. Instrument it end to end.

4. Keep quality metrics as diagnostics, not the outcome.

5. Start learning the mapping from customer value to business outcomes, using experiments when you can and causal methods when you can’t.

Do this a few times, and impact stops being a special project. It becomes part of the engineering loop.

Proposed ending block:

If you want to go deeper, the full workshop recording is here:

https://maven.com/p/2c0839/validate-business-impact-of-ai-features

If you’d rather see the visuals, the slides are here:
https://gamma.app/docs/Validating-Business-Impact-of-AI-Features-hqgxww6qei9qs4k

If you want a complete system for building and operationalizing evals in product teams, our full course AI Evals for Product Development is here:
https://maven.com/dataneighbor/ai-evals

Learn AI Evals for Product Development

Questions, or need help making the internal case for budget? Email me at shane@aieval.ai.

And if you made it this for, thanks for reading! Drop me a comment below and I’ll message you a promo code to our AI Evals cohort :D

Most people treat AI data assistants like magic boxes. You type in a question, get back some SQL and a chart, and ship it. This works until you realize the join was wrong or the AI misunderstood your business logic.

Arsalan Bashir, founder of LiveDocs, spent six years building AI-powered data tools. In our conversation, he walked through what actually works when you’re using AI to analyze data. Not the hype stuff, the practical things that separate teams getting real value from teams wasting tokens on bad outputs.

Subscribe now

The context loading problem nobody talks about

Here’s something counterintuitive. When you ask an AI to write SQL, your first instinct is probably to feed it your database schema and data types. That’s actually the wrong move.

Arsalan explained that most AI data tools already infer schema and data types at runtime. You’re burning context window space on information that’s already there. What the AI actually needs is the stuff it can’t figure out on its own: how your tables relate to each other, what the business logic is, which fields reference what.

In LiveDocs, they tackled this with an indexing system. Instead of reading your documentation in real-time, the AI indexes your data sources in the background. It looks at your tables, writes descriptions, creates embeddings, and asks clarifying questions upfront. Think of it like code coverage for your data warehouse.

The metric Arsalan watches is coverage percentage. How much of your data has the AI actually looked at and understood? The higher that number, the better your results. A random question against an unindexed warehouse gives you random output. A question against a fully indexed system with clear relationships defined? That’s where things actually work.

Prompting for data work is its own skill

Arsalan called prompting the most important skill for working with AI data tools, and the gap between good and bad prompts is massive.

Here’s his advice: Don’t dump everything into the prompt. The AI will get what it needs from your indexed sources. Instead, focus on describing relationships and business context that aren’t obvious from looking at the raw data.

For LiveDocs, they built a prompting guide that works even if you’re using ChatGPT or another tool. The key insight is understanding what the AI already has access to versus what you need to explicitly provide.

Different models are better at different things. An image model can look at a chart and tell you it’s trending up. But for writing complex SQL or doing multi-step analysis, you want a coding-specific model like Codex or one of the frontier models (Claude Opus, GPT-4, etc.). For summarizing findings or writing stakeholder updates, a faster model with better context handling works fine.

Subscribe now

Always review what the AI did

This sounds obvious but most people don’t do it properly. Arsalan said there’s no such thing as zero-shot data analysis. You can’t just take what the model gives you and call it done.

The review step isn’t just checking for bugs in the SQL. You need to understand if the AI actually met your objective. If it didn’t, figure out what to adjust in your prompt or in the context you provided.

Arsalan compared it to reviewing a pull request, but instead of just looking for logical errors, you’re asking: “Did this solve my actual problem? If not, what context was missing?”

He mentioned they’re looking at building PR-style workflows for AI-generated analysis. The output goes into a PR, team members can review it, suggest changes, and then merge it once it’s solid. This creates a clear audit trail and catches problems before they become decisions.

The BI tool trap

When Arsalan started building LiveDocs six years ago, they were making a traditional BI tool. No-code interface, drag-and-drop charts, the whole thing. The idea was that non-technical people could self-serve their analytics.

Then LLMs happened and everything changed.

The problem with BI tools is they’re a tradeoff. You get ease of use but sacrifice flexibility. You’re stuck navigating dropdown menus to do what a data analyst could write in five lines of SQL. And BI tools are really built for monitoring known insights, not exploring new questions.

AI flipped this completely. Now non-technical people can get the flexibility of code without writing it themselves. The complexity is abstracted away but the power is still there.

Arsalan said the question they keep asking is: “What does data work look like five years from now?” They’re not trying to glue chat interfaces onto existing BI tools. They’re building from scratch for that future state.

What you should actually focus on

First, learn to prompt properly. Understand what context the AI needs versus what it already has. Focus on relationships and business logic, not raw schemas.

Second, set up good indexing. Whether you’re using LiveDocs or another tool, make sure the AI has actually looked at your data sources and understands them. Coverage matters.

Third, always review the output. Check if it solved your actual problem. Adjust your prompts and context based on what didn’t work.

Fourth, understand which models are good at what. Complex analysis needs stronger coding models. Summaries and explanations work fine with faster models.

The gap between teams doing this well and teams doing it poorly is going to be enormous. Not because the technology isn’t ready, but because most people are using it wrong.

Join Our Free Live Lessons

Ready to level up your analytics skills?

We’re offering two new learning opportunities:

Free workshops: We’re running a series on AI evaluation and agentic analytics. Topics include analyzing product data with no-code tools, designing AI evaluation plans, creating custom annotation tools, and more. Full schedule at dataneighbor.com.

Join Our Free Live Lessons

Cohort-based courses:

• AI Analytics for Builders: Learn to leverage AI to become your own product data scientist, or magnify your impact as a data scientist. Claim your 20% discount here.

• AI Evaluations for Product Development: Learn to measure the actual impact of AI features on user and business outcomes. Claim your 20% discount here.

Both are hands-on, practical, 5-6 week programs where you’ll work with us and other practitioners in the field. Check it out at dataneighbor.com.

They have a latency problem.

A product question shows up in a planning meeting and the answer gets stuck in a queue.
By the time it comes back, the decision window has moved.

So we built AI Analytics for Builders.

24-hour launch offer (ends Jan 27 at 5pm PT)

To kick off enrollment, we need 5 students to sign up right away. Once we have those early students, Maven is more likely to surface the course more broadly on their site.

So, for the next 24 hours, we’re running a launch draw for Substack readers:

• 5 people will be selected at random to get 85% off and enroll for $180 (normally $1,200)

• Everyone else who applies gets a 40% off code as a thank you

• If you’re selected for the $180 offer, you’ll agree to post a short note on LinkedIn that you enrolled (we’ll provide templates)

Even if you’re not one of the 5 selected for 85% off, you’ll still receive the 40% code for applying.

Apply here: https://forms.gle/suN4h5f1vAw32GTm9

Apply for 85% Off

What AI Analytics for Builders is

AI Analytics for Builders is a live cohort-based course on Maven to help you build analytical independence, using AI as your analysis partner.

It’s for PMs, designers, engineers, and leaders who want to answer their own product questions fast. Or data analysts and data scientists who want higher to increase leverage.

No SQL required.
No stats lectures.
Five weeks of hands-on work with real scenarios.

The goal is not to turn you into a full-time analyst.

The goal is to help you operate with Product Data Scientist-level independence: move from question → analysis → decision without waiting in a queue.

What you’ll do in the course

In the course, you’ll use AI to:

• Answer “what should we do next?” with data

• Spec metrics your team can ship

• Diagnose KPI movement without guessing

• Communicate the why in a stakeholder-ready narrative

• Turn analysis into experiments and decisions

A concrete example of the workflow you’ll practice:

A KPI drops. You isolate which segment moved. You generate a short set of testable hypotheses. Then you translate that analysis into a clear recommendation and the next product decision.

Cohort dates

Next cohort runs Apr 20 to May 24, 2026.

If you want to make faster product decisions, this is the best deal we’ll offer.

Course Link (details): https://maven.com/dataneighbor/ai-analytics-for-builders
Form Link (apply by Jan 27 at 5pm PT): https://forms.gle/suN4h5f1vAw32GTm9

The CEO has a problem. Revenue’s down 15% this quarter. Who do they call?

If your answer is “the data team,” you’re thinking about analytics the right way. If your answer is “wait for the weekly dashboard refresh,” you’re stuck in the old model.

Ollie Hughes, co-founder of Count, spent years as a management consultant going into factories across Europe and figuring out why production lines weren’t hitting targets. He’d show up at a can-making factory in Germany, not speaking the language, armed with a stopwatch and a notebook. His job was to make that factory run twice as fast.

What he learned there shaped how he thinks about data teams today.

Subscribe now

The factory floor mindset

When Ollie walked into those factories, he had to understand the entire system fast. Where’s the bottleneck? What data exists? What’s just tribal knowledge that nobody’s written down?

He’d dig through the data, trace problems back to root causes, and often find that one dial on one machine was holding back the entire operation. Turn it from two to seven, and suddenly the whole factory starts making money again.

That kind of problem-solving is what data teams should be doing for their companies. Not building dashboards. Not running reports. Solving actual business problems.

Visualize the business first

Ollie’s core belief: the role of a data team is to support and facilitate great business improvement. And the way you do that is by helping the business see itself clearly.

Most companies don’t have a clear view of their own growth model. They have dashboards showing metrics, sure. But they don’t have a shared mental model of how the business actually works. What drives revenue? What are the second-order effects when marketing spend goes up? How do different parts of the business connect?

Count’s approach is to build what Ollie calls a “metric tree” - a visual map of how your entire business fits together. Not just metrics on a page, but the actual relationships between them. When you can see that GMV is driven by active buyers and average order value, and those are driven by these other factors, and those are driven by these initiatives - now you’re working with a shared understanding.

Problem-solving as a team sport

Here’s where most AI analytics tools get it wrong: they assume one person asks a question, the AI gives an answer, and that person goes off and does something with it.

But that’s not how business problems get solved. Problems get solved in rooms with multiple people, looking at the same information, debating what it means, deciding what to do.

When we saw Ollie demo Count’s AI agent, this clicked. The AI doesn’t just answer questions. It lays out the problem, breaks it into components, digs into each one, and presents everything in a way that multiple people can interact with together.

One person can ask the AI to investigate why conversion dropped. Someone else can jump in and adjust the time period. A third person can drill into a specific segment. Everyone’s looking at the same canvas, editing the same queries, building shared context in real time.

Everything is transparent and editable

This is critical: when the AI generates analysis, you can see exactly how it’s built. The queries are visible. The logic is traceable. You can edit any part yourself, and the AI recognizes those changes and incorporates them.

No black boxes. No “the model says so.” Just transparent analysis that anyone on the team can verify and build on.

Ollie demonstrated this by having the AI investigate a revenue dip. The AI automatically broke down the problem - was it fewer customers? Lower average order value? Changes in specific segments? It drilled into each factor, showed the data, and explained what it found.

But here’s the key: at any point, a human could step in and say “actually, look at this cohort differently” or “check this time period instead.” The AI adapts. It’s collaborative problem-solving, not automated answer-generation.

The deeper you go

One thing that stood out: Ollie mentioned the AI often goes deeper than he would himself, given time constraints. It pursues lines of investigation that a human might not have bandwidth for.

This is where AI actually helps. Not by replacing human judgment, but by doing the grunt work of checking every angle, running multiple cuts of the data, and laying out all the evidence so humans can make better decisions.

What this means for data teams

Ollie’s advice for data teams: position yourselves as the organization’s problem-solving squad. The CEO has a problem? You’re the people who can help think it through.

You already have the superpowers needed for this:

• Visibility across the entire business

• Strong problem-solving skills

• Ability to work with complex, ambiguous situations

The challenge is making sure those strengths are visible and valued. That means defining your purpose clearly - we’re here to help the business improve- and working in ways that amplify those strengths rather than getting lost in routine reporting.

You can do this with whatever tools you have today. But as AI gets integrated into analytics workflows, the teams that will thrive are the ones who understand their role is problem-solving, not dashboard-building.

Join Our Free Live Lessons

Ready to level up your analytics skills?

We’re offering two new learning opportunities:

Free workshops: We’re running a series on AI evaluation and agentic analytics. Topics include analyzing product data with no-code tools, designing AI evaluation plans, creating custom annotation tools, and more. Full schedule at dataneighbor.com.

Join Our Free Live Lessons

Cohort-based courses:

• AI Analytics for Builders: Learn to leverage AI to become your own product data scientist, or magnify your impact as a data scientist. Claim your 20% discount here.

• AI Evaluations for Product Development: Learn to measure the actual impact of AI features on user and business outcomes. Claim your 20% discount here.

Both are hands-on, practical, 5-6 week programs where you’ll work with us and other practitioners in the field. Check it out at dataneighbor.com.

We talked about how research teams work inside big companies, why most AI agents aren’t ready for real business tasks yet, and what it takes to move from impressive demos to tools people actually use.

Key takeaways from our conversation:

1. Research teams in companies aren’t academic labs

Shelby’s team at Salesforce does research, but it’s shaped entirely by what the business needs. When Salesforce launched Agent Force as their biggest product, her team’s research aligned with pushing the boundaries of agents. The work is exploratory but always tied to an applied vision. As she put it, they’re asking what customers will need in 6, 12, or 18 months and building toward that.

2. Enterprise General Intelligence is the North Star, not AGI

Salesforce’s research org has a grand vision they call EGI (Enterprise General Intelligence). The goal is getting agents to perform consistently well on real enterprise tasks. Right now, everyone’s showing amazing demos, but when it comes to actual business work, agents aren’t there yet for many use cases. All the research across Salesforce’s teams points toward closing that gap.

3. You can’t improve what you can’t measure

Before agents can get better at enterprise tasks, we need ways to measure their performance. Salesforce has released open source benchmarks like their CRM benchmark, which tests agents on realistic customer relationship management cases. Shelby sees more of these agent benchmarks emerging over the next 6 to 12 months, which will fuel real improvement.

4. Interdisciplinary teams are the only way forward

Building AI that works isn’t just about model builders anymore. Shelby emphasized that you need model builders working with people who understand customer discovery, plus domain experts who can fine-tune models for specific fields. A marketing agent needs marketing experts involved in training it. A healthcare agent needs healthcare professionals. The days of AI teams working in isolation are over.

5. Small language models solve real problems that big models can’t

Shelby’s team works on small language models for on-device use and specific applications. These aren’t just scaled-down versions of large models. They’re purpose-built for situations where you need speed, privacy, or efficiency. The future isn’t just about making models bigger—it’s about making the right-sized model for each job.

6. Embodied agents are the next ChatGPT moment

Looking past the next year, Shelby sees embodied agents as the next big shift. Right now, AI agents work in virtual environments like navigating webpages or working in Salesforce. But the same technology that powers these virtual agents will soon power physical robots, drones, and more. Navigating the 3D world is harder than navigating virtual spaces, but she thinks we’ll see a breakthrough moment in the next few years, similar to what ChatGPT did for conversational AI.

Subscribe now

The gap between AI demos and AI that does real work is still wide. But teams like Shelby’s are measuring it, closing it, and building the benchmarks that will help everyone else do the same.

Last week I wrote about a common trap: teams treat AI evaluation like a debugging loop. They watch eval metrics improve, but they still struggle to answer the question leadership ultimately asks: did better AI output change anything meaningful in the business.

This post follows up with what “end-to-end” AI evaluation includes when you treat it as a product capability that has to hold up in production, across functions, and over time.

It is also the backbone of a six-week, hands-on cohort I’m running in April 2026 on AI Evals for Product Development.

If you are already interested in the limited pilot seats mentioned below, you can apply here now:

Apply to the AI Evals Pilot

What teams are investing in as AI products scale

In 2026, more teams will ship AI features embedded in core workflows, not just demos. That shift changes the evaluation bar.

Teams need to be able to:

• instrument the right events so evaluation is reproducible

• define success in terms of user value, not just output quality

• translate signals into metrics with baselines, guardrails, and release criteria

• run evaluation continuously through launches, iterations, and regressions

• make trade-offs across quality, risk, and cost with decision-grade evidence

In other words, evaluation becomes an internal system: people, data, processes, and artifacts that keep paying dividends as the product evolves.

What “end-to-end” covers in practice

Here is the simplest way I think about the full stack of AI evaluation, mapped to the questions product teams actually face.

1. Foundations
   What AI evaluation is in a product context, where AI systems fail, and what the minimum viable evaluation is for right now versus later.

2. Instrumentation and observability
   What to log, how little is enough, how to design traces so results are reproducible, and how observability changes by system type.

3. Interpreting output success and failure
   How to ground evaluation in user value, identify sources of ground truth, derive signals from what you already have, and scale semantic evaluation with users, experts, and LLMs.

4. Measuring impact on user value and business outcomes
   How to translate evaluation signals into metrics, validate sensitivity and predictiveness, segment correctly, run driver analysis, and write metric specs and thresholds that can govern releases.

5. Pipelines, experiments, and continuous validation
   How evaluation pipelines are architected, test set strategy, experiment design for stochastic systems, launch readiness, and monitoring drift and regressions.

6. Decision-making and driving AI product impact
   How to make decisions under uncertainty, turn metrics into product actions, manage trade-offs across quality, risk, and cost, establish an operating rhythm, and communicate impact in a way that holds up.

If you are building AI features and you look at that list, you can probably feel where you are strong and where you are exposed.

The problem is that many companies are approaching evals in silos, team by team (engineering, ML, product, QA, SMEs, data). Each group patches gaps ad hoc, without a coherent cross-functional program.

Full-stack AI evaluation is the standard I’m aiming for with the April cohort. The goal is not to teach isolated tactics, but to help teams build evaluation capability that works across product, engineering, data, and domain expertise.

To make sure the course reflects real constraints, I’m running a small pilot group from January through March 2026. This group will pressure-test selected modules, exercises, templates, and examples before the April cohort runs.

A small pilot group to pressure-test the material (Jan to Mar)

We’ve officially opened applications for 4 pilot seats for the April AI Evals for Product Development cohort.

The cohort is usually $1,500. Pilot seats are $300 (80% off). This discount is in exchange for structured feedback from January through March on selected parts of the curriculum as we finalize the April cohort. Pilot seats also include full access to the April cohort.

• Apply here: Apply to the AI Evals Pilot

• Not sure you’re a fit: comment below with your role and what you’re working on, and I’ll reply with a quick read.

This pilot is not the full cohort experience delivered early. Think of it as an AI working group that helps pressure-test the lessons, exercises, templates, and examples against real products and real constraints.

Who this is for

This is for practitioners who are actively shipping (or about to ship) AI features, including:

• product managers shipping AI-powered features

• data scientists and analysts responsible for measurement and evaluation

• engineers and ML engineers building and maintaining AI systems

What the pilot involves

• working sessions, interviews, and debriefs over the next ~12 weeks

• pressure-testing lessons, exercises, and templates against your real AI feature and workflow

• async feedback on drafts (slides, exercises, templates, examples)

• a small group discussion every other week to compare approaches and learn from each other

In return, you’ll get

• 80% off the April cohort ($1,200 value)

• early access to the material and resource pack before April

• more 1:1 time applying the approach to your specific use case

• direct influence on what gets emphasized, clarified, or cut before the cohort

• full access to the April cohort: live sessions plus recordings and materials

• a peer group working on similar problems in AI evaluation

If you are already interested in the limited pilot seats mentioned above, you can apply here now:

Apply to the AI Evals Pilot

And if you have any questions at all, drop them in the comment section below or feel free to reach out to me directly on LinkedIn.

The problem is that once a team gets that loop in place, a second challenge shows up immediately behind it. The eval metrics start moving, the outputs look better, and the team feels momentum. Then, a month or a quarter later, the question arrives that is much harder to answer: did any of this actually change the business.

Revenue, retention, engagement, churn, support cost. Whatever your organization cares about, those outcomes do not respond to an eval score. They respond to what users do, consistently, inside a workflow.

This is not a new data science problem. There has always been an inference gap between local product changes and business-level outcomes. We ship improvements, we see short-term movement in a dashboard, and we hope those movements add up to something durable. But AI is about to make that gap harder to ignore, because it produces crisp-looking quality metrics for something that sits upstream of user behavior. When those metrics improve, it is very easy to over-interpret what they mean.

Working on AI features? Get our one-page plan for proving AI impact on the business.

Access the free worksheet

The most common failure mode I see is not that teams cannot measure quality. It is that the story about impact is implicit. Everyone has an intuitive narrative about why quality improvements should matter, but the chain is not written down, and it is not validated link by link. When the business does not move, nobody can diagnose where the chain broke. The team either keeps shipping “quality wins” and hoping, or starts arguing about which metric matters, or slowly loses trust in evaluation entirely.

If you want AI evaluation to support product decisions, you need a second layer beyond debugging. You need measurement that forces you to answer, in explicit terms, how quality becomes value.

If AI output quality improves, what should users do differently. Will they adopt the feature more often. Will they rely on it rather than work around it. Will it reduce effort or time-to-complete. Will it reduce back-and-forth in the workflow. Those are the behaviors that translate into user value.

And if user value improves, where will you see it before you see revenue or retention. What is the proxy closest to real-world value that should move first. Then, only after those steps are established, you can make a credible claim about business outcomes, including the time horizon where you should expect to see movement.

What makes this hard is that each link can fail for a different reason. Quality can improve but adoption does not change because trust is the bottleneck. Adoption can increase but user value does not change because the feature is being used on the wrong tasks. User value can improve but business outcomes remain flat because the outcome is driven by many other forces, or because the effect is concentrated in a segment that is too small to move the aggregate. If you treat “impact” as one big question, you cannot diagnose any of this.

The simplest structure I have found is to treat impact as a chain of links you can validate:

quality → behavior → user value proxy → business outcome

This framing also changes what you need from your data. It forces you to define the unit of analysis and what “exposure” actually means. It forces you to specify the instrumentation and joins required to measure behavior and value, not just output quality. It forces you to choose validation methods per link instead of relying on correlation. And it forces you to set thresholds that would actually change a decision, which is the difference between “interesting measurement” and decision-grade evidence.

I put this into a one-page planning sheet and I’m sharing it for free to collect feedback. If you are building customer-facing AI features and trying to prove impact beyond eval scores, you may find it useful.

Working on AI features? Get our one-page plan for proving AI impact on the business.

Access the free worksheet

If you use it, I’d genuinely love to hear what feels unclear, what feels missing, and where the chain breaks most often in your product.

In 2026, as AI products scale, one capability is going to matter more than almost any specific model choice: the ability to evaluate AI features in a way that is rigorous, ongoing, and tied to product outcomes.

The encouraging part is that this capability is teachable. You do not need to be an “AI Expert”. You need a small set of analytical methods and operating habits that make improvement measurable and iteration faster.

This article breaks down what teams that operate with production-grade AI evaluation look like, and a quick diagnostic to see where your team stands today.

If you are building or scaling AI Product and want a structured way to build production-grade evaluation, we run a live cohort focused on practical, product-facing evaluation methods.

Learn More: AI Evals for Prod Dev

AI product evaluation that holds up in production

Most teams have some form of evaluation. The problem is that much of it is not designed for the conditions that matter for AI products.

A demo or staging evaluation tells you whether the system can produce impressive outputs on curated examples. A production evaluation tells you whether the feature is reliable across messy inputs, real user behavior, changing context, and shifting goals.

The difference determines whether your team can iterate with confidence, or whether you end up shipping, guessing, and reacting.

What AI evaluation is, in plain terms

AI product evaluation is the system you build to answer three questions repeatedly:

1. Is the feature working for users the way we intend?

2. Is it improving over time in the dimensions we care about?

3. Is that improvement translating into product and business outcomes?

If you cannot answer those questions continuously, you do not have an evaluation loop. You probably have something closer to “vibes”.

What “good” looks like

Teams that do this well can:

• Reconstruct what happened. They instrument enough context to debug failures with evidence.

• Measure quality in product terms. They evaluate usefulness, correctness in context, policy adherence, and user effort, not just offline scores.

• Prove impact. They connect evaluation improvements to outcomes like retention, conversion, deflection, time saved, escalations, and churn.

• Operate it weekly. Sampling, labeling, analysis, and iteration are a cadence, not a one-time pre-launch phase.

A simple marker of maturity:

You can point to a recurring view of AI quality trends and top failure modes, and the team agrees on what they should fix next.

The failure mode most teams hit

A common trap is optimizing what is easy to measure instead of what is important.

Offline scores improve on small, controlled datasets. Stakeholders feel good. But the product does not move. Or worse, it moves in the wrong direction, because the evaluation never captured the critical edge cases, the true user context, or the real cost of certain failures.

When that happens, the signal does not show up in dashboards. It shows up later in customer support tickets, escalations, and churn.

A quick diagnostic

Use this as a quick and light self-assessment. Teams with a strong system for evaluating the AI products, will find answering “Yes” more often than “No”.

Product

• Can we define “quality” for this feature in a way a new PM can apply consistently?

• Do we know the top failure modes, and which ones are most harmful to users?

• When quality changes, do we detect it quickly, or do we hear about it from customers first?

Engineering

• Do we log enough context to reproduce failures reliably (inputs, context, outputs, user actions)?

• Can we trace regressions back to a model change, prompt change, retrieval change, or data change?

• Do we have an evaluation pipeline we can run repeatedly, not a one-off script?

Data Science & Analytics

• Do we maintain representative evaluation slices by segment, use case, and risk level?

• Do we have a consistent rubric and labeling process that produces usable signal over time?

• Can we tie evaluation movement to product outcomes through experiments or other causal methods?

Leadership

• If the model changed tomorrow, could we say within a week whether the user experience improved or regressed?

• Can we explain, in plain language, why we believe the feature is getting better?

• Do we have a shared “definition of done” for shipping changes to the AI system?

How to Build Rigor in AI Evaluation

If you answered “no” to several questions above, you have identified the exact gaps that cause AI features to stall in production. The fastest way to turn those “no’s” into “yes’s” is not more ad hoc testing. It’s building a production-grade evaluation loop and the operating habits to run it week after week.

That is guided practice: learn the core evaluation methods, apply them to a real feature, and leave with an evaluation system your team can keep running.

Live Cohort (most direct path)

We built this as a practical, product-focused cohort for PMs, engineers, data scientists, researchers, and product leaders shipping AI features.

By the end, you should be able to:

• define and measure quality in product terms

• build a repeatable evaluation loop your team can run continuously

• connect evaluation results to product and business outcomes

• identify and prioritize failure modes so iteration is targeted, not random

AI Evaluations for Product Development

6-week live cohort + certification
Starts April 7
Learn more and register: maven.com/dataneighbor/ai-evals

Learn More: AI Eval Certification

If you are not ready for the full cohort, we are also running FREE live sessions that cover key pieces of the evaluation stack:

Free live sessions

• Dec 17: Redesign Product Metrics for AI Evaluations.

Register: AI Product Metrics

• Jan 21: Validating AI Product Impact on the Business.

Register: AI Product Impact

• Feb 11: Building Custom Annotation UIs for AI Evaluations.

Register: Build Custom Annotation UI

Also coming in 2026:

We are releasing a new interview series with leaders building the next generation of evaluation: Measuring the Machine and The Analyst’s New Toolbox.

Prefer to learn by listening

In January, the Data Neighbor Podcast is launching an exciting new series with leaders building the next generation of evaluation and AI analytics tooling: “Measuring the Machine”.

Subscribe to Data Neighbor on YouTube, or wherever you listen to podcasts. If you want the clearest view of where evaluation is headed and how teams are building it in practice, this will be the simplest way to keep up.

Not sure where to start?

Reply or message me with what you are building, who uses it, and what “bad outputs” look like today. I’m always happy to brainstorm.

There was “Eddy’s,” which was the video rental place down the street. There was “Ivy’s,” the hair salon where I got my haircut. And there was “Anna and Maria’s” a small cafe around the corner. I don’t know the actual name of any of these businesses, but 30 years later I still know them by the names of their owners.

Okay maybe “Anna and Maria’s” was called “The Daily Brew”? Though that might be my childhood memory filling in gaps. To me, it was “Anna and Maria’s”.

This was thirty years ago. None of those places exist anymore. But at the time, that small two or three block radius between our house and my George Peabody Elementary comprised the majority of my universe.

This story is about “Anna and Maria’s", and the best turkey sandwich I ever had….

Anna and Maria’s was nothing particularly special. A typical neighborhood coffee shop. I didn’t drink coffee at six years old, not yet. Was mostly into Snapple in those days. But they served something far more important to me at that time: their turkey sandwich.

Truly I could not find another sandwich like it. From the ages of six to eight, I tried every place in the neighborhood, thinking maybe someone else could match it. No luck. Every time it came up short.

I went home and tried to recreate it myself. Many times. Same outcome. It always felt like I was getting close but never hitting whatever that exact thing was that made theirs so good.

The ingredients seemed simple enough. Deli turkey. Lettuce. Tomato. Red onion. No cheese. And sprouts. I locked in on the sprouts for a long time. I was completely convinced they were the secret. At the grocery store, I made my parents buy a different type of sprout each time so I could experiment.

And yeah, the sandwiches got better. Put sprouts on your sandwiches, it’s good. I was leveling up my seven year old sandwich game. I switched breads. Toasted, Cut the tomatoes thinner. Then thicker. Tried different onions. Again cut them thinner. Different turkey. Flavored turkey. Different greens: romaine, spinach, arugala, mix, etc.

Nothing got me there. It was like chasing something I knew existed but couldn’t reproduce, no matter what combination I tried.

Finally, one day, I asked Anna and Maria directly.

“What do you do to your sandwiches that makes them so good?”

We walked through the ingredients. Lettuce. Tomato. Turkey. Wheat bread. Sprouts. Etc. Etc. Check. Check. Check. Check. Check. Check. Check.

And then they added one more item.

“Mayonnaise. You are missing the mayonnaise.”

This of course was not possible. As any reasonable child would, at this point in my life, I hated mayonnaise. Completely repulsed by it. The smell, the texture, the way it got everywhere. Oh god, when it gets too warm. Seriously, just seeing a big glob of it…

It made me gag!

So, naturally, my reaction when I heard this was an instant and certain: “No”.

No, that cannot be it. There is no mayonnaise on these sandwiches.

Even though these two women were literally the ones making the sandwiches with their own hands and they with certainty knew they spread that bread with mayo…I refused to believe it.

I was very confident that Anna and Maria were mistaken.

This is how powerful preconceived beliefs about what we think we like and what we don’t like can be.

I had made up my mind years prior that mayonnaise was awful. And thus, the idea that the best sandwich I had ever tasted contained it was just impossible.

For many more years, I remained anti-mayonnaise. Probably until age 14 or 15. (At some point, you cannot be asking everyone to remove the mayo because you are acting like a seven year old.)

And of course, once I finally grew up enough to revisit it, I realized the obvious.

It was, of course, the mayonnaise.

It was always the mayonnaise.

You’re all smart people, so I don’t need to connect the dots for anyone reading this, but I will anyways :)

Users tell us what they think they want.
Users tell us what they think they like.
Users tell us what they believe creates value.

There is important insight in that. It helps us form hypotheses worth testing. But the real understanding of value comes from their actions and the results of those tests. From the goals they try to achieve. From the choices they make. From the signals they generate through real behavior.

Feedback is useful for hypotheses, but it isn’t the truth.

Behavior is the truth. Words just help us wrap a story around it.

Anna and Maria knew the truth. They saw the pattern. I would ask for “no mayo,” they would say “okay,” and then, knowing their customer better than he knew himself, they put it on anyway.

I loved the product even when I believed I didn’t.

So don’t just listen to what I say. Watch what I eat.

The deeper problem is that many product teams never had a solid foundation for deterministic metrics in the first place. They inherit a standard suite of numbers (Daily Active Users, Retention, Session Length) and then retrofit their strategy to match those templates. They optimize for “engagement” without first asking whether that signal is meaningfully tied to user success.

This comes from a misunderstanding of the relationship between the user and the product. A product is a delivery mechanism for a goal the user already has in their non-digital life. The user does not wake up intending to “engage” with a SaaS platform. They wake up intending to hire a candidate, debug an outage, close a deal, reconcile a budget, ship code, or finalize a contract. The application is just a tool that connects their intent (the job to be done) to an outcome (the value they realize).

Product analytics is not simply the act of tracking events. It is the engineering work of translating these invisible real-world goals into observable, queryable signals. If we struggle to do this correctly in deterministic systems where inputs reliably produce outputs, the challenge only grows when we introduce probabilistic AI models.

If you’re currently working on an AI product and want help redesigning your metrics, I’m hosting a free 30 minute live walkthrough on Redesigning Product Metrics for AI.

RSVP for Live Walkthrough

Part I: The Measurement of Deterministic Systems

In Part I of this guide, we establish the foundational system of metrics required for deterministic applications where the same input reliably produces the same output. Here, the main challenge is not model variance. It is the more basic problem of turning messy, real-world goals into clean, queryable signals that relate to user value and business outcomes. Once those assumptions are visible, it becomes easier to see why they break when we swap fixed logic for probabilistic models.

I. The Mechanics of User Value Proxies

In product development, we operate under a persistent constraint: we cannot query the user’s intent directly. User value is a latent variable. It lives in the user’s mind, not in our database.

The Observability Constraint and the Inference Gap

Our systems only observe physical interactions with the interface. We log events like button_click, scroll, or form_submit. In an online store, that might look like:

• product_page_viewed

• add_to_cart

• checkout_started

• order_submitted

We do not log “found the right running shoes for my race” or “bought the right size and did not need to return it.”

This creates an Inference Gap between:

• Observed events: what happened in the browser or app.

• Inferred state: whether the user actually achieved their goal.

In a perfect world, we would have a goal_achieved column we could query. In practice, every metric on a dashboard is a bridge we build across this gap.

Consider a standard metric like Time on Page. On a high level dashboard, an increase in average time on a product page often looks like healthy engagement. It seems to imply that users are finding information useful and reading it carefully.

At the event level, the picture is very different. Imagine two users on the same product detail page:

• User A spends five minutes reading reviews, checking the size guide, and then adds the item to their cart.

• User B spends five minutes scrolling up and down, tweaking filters, and hunting for basic information they cannot find before leaving the site.

Our logging system records the same session duration for both sessions. A naive analysis counts both as “high engagement.” In reality, User A found value and User B found friction, and the log cannot distinguish between the two.

The work is to filter this noise and identify interaction patterns that genuinely correlate with successful outcomes.

Defining the User Value Proxy

To cross the inference gap, we leverage User Value Proxies. These are not generic KPIs pulled from a template. They are specific, observable events or aggregates that we accept as stand-ins for the user’s real-world goal.

Choosing a proxy is an engineering trade-off between:

• Accuracy: How closely does this metric track the real goal?

• Latency: How quickly can we observe it after a change?

The most accurate metrics are almost always the slowest to materialize.

Recall the earlier example where two users spent the same time on a product page, but one found value and the other found friction. Time on page alone could not tell those sessions apart. To act on this, we need to move beyond surface activity and pick events that better encode the underlying goal.

For an online store, the user’s real job is not just “place an order.” It is “get the right product, have it solve the problem, and avoid returns.” A natural long-term measure is Customer Lifetime Value (LTV), which captures returns, repeat purchases, and long-term satisfaction. As a proxy for true value, LTV is strong but slow. It can take months or years to observe.

If an engineering team waited for LTV to validate a new search or ranking algorithm, their feedback loop would be unusably long. So they move upstream and select faster proxies such as:

• Product-level actions: add_to_cart, checkout_started, order_submitted

• Session-level aggregates: checkout conversion rate, cart completion rate

These are imperfect signals of:

• Users abandon carts.

• Users browse for fun.

• Users buy the wrong thing and later return it.

Despite that noise, these events arrive immediately and are correlated with long-term value often enough that teams accept them as operational truth. The implicit contract becomes:

We will optimize this proxy with the expectation that improving it will also improve real user value.

Once we adopt a proxy, we design features, run experiments, and make trade-offs in service of moving that number.

The Deterministic Assumption

This entire proxy logic relies on an important structural property of traditional software that we will later lose in AI systems.

In deterministic applications, the product behaves like a finite state machine. The same input leads to the same output:

• If a user clicks “Save,” the system persists a record.

• If a user applies a “Price: Low to High” filter, the sorting algorithm runs as written, every time.

• The code does not roll dice or sample from a distribution when it executes.

In practice, real systems still have sources of noise, but at the level of application logic we treat the code as a constant k. The machine provides a stable background. When we look at our logs and see changes in user behavior, we attribute those changes to differences in user intent or to the product changes we made, not to random variation in the code path.

This Deterministic Assumption is what makes our metrics interpretable. Because the system’s logic is fixed, we can treat high activity volume as a positive signal of engagement. We trust the log because we trust that the underlying machine is rigid and predictable.

In the next sections, we build on this assumption to construct a hierarchy of metrics and a causal chain from inputs to business outcomes. Later, we will see how this structure starts to fail once we replace rigid logic with a probabilistic model.

II. The Hierarchy of Measurement

Once we define user value proxies, we still face a timing problem. The events we can directly influence happen in milliseconds. The business outcomes we care about often move over weeks or months.

To bridge that gap, we build a hierarchy of metrics. This is not just a flat list of KPIs. It is a causal stack where we assume that moving the bottom layer will, through the chain, move the top.

At a high level, the stack has three layers:

1. Input Metrics: the granular interactions and system properties we can directly change.

2. The North Star Metric: the main user value proxy that links product behavior to value.

3. Business Outcomes: the lagging financial signals that validate the whole structure.

We design product changes to act on the inputs, monitor the North Star to steer day to day, and use business outcomes to check whether the proxies still match reality.

Layer 1: Input Metrics (Leading Indicators)

At the base of the stack are the **input metrics**. These are high frequency signals that appear as raw events in the warehouse, such as:

• search_query_executed

• profile_viewed

• message_drafted

• latency_ms

We often call these “the levers” because they are the only quantities a team can directly influence. An engineer cannot ship code that directly increases revenue, but they can:

• Reduce search latency by 200 ms

• Move a “Send Message” button into a more prominent location

• Improve ranking so that the first page of results is more relevant

Input metrics are leading indicators. They respond immediately to product changes and can move weeks before any business outcome is observable.

That speed comes with a cost. Inputs are noisy and easy to game:

• A ten percent increase in profile_viewed might mean better recommendations.

• It might also mean we increased pagination friction and forced more clicks.

• A rise in messages_sent might mean meaningful conversations, or it might mean spam.

In deterministic systems, we accept this noise because we do not look at inputs in isolation. We rely on the layer above them to filter out manipulations that do not translate into real value.

Layer 2: The North Star Metric (User Value Proxy)

Above the chaotic stream of inputs, we define a North Star Metric. In engineering terms, this is the aggregate metric that best captures the intersection of user success and business value.

The North Star should answer a simple question:

When users achieve their goal in our product, what observable event almost always occurs, and does that event also matter for the business?

This is not something we can find by scanning a correlation matrix. Correlation can show coincidences in the logs, but it cannot tell us what the metric means. Defining a North Star is a hypothesis driven product decision.

We start from the job to be done, then ask what a successful session would look like in the data.

Examples:

• In a travel marketplace, we could track searches or revenue. But searches is often just browsing, and revenue is too lagged. A better North Star is Nights Booked, which marks the moment where the guest has lodging and the host earns income.

• In a workplace communication tool, we could track daily active users. But that can be driven by log in requirements and does not guarantee value. A better North Star is something like Messages Sent in Active Channels, which indicates ongoing collaboration.

Once defined, the North Star serves as the operational definition of success for the product team. It gives engineers permission to optimize low level inputs aggressively. If a change improves the North Star without harming guardrail metrics, we treat it as value creating, even before we see it in quarterly revenue.

Layer 3: Business Outcomes (Lagging Indicators)

At the top of the stack sit the business outcome:

• Revenue

• Retention

• Lifetime Value (LTV)

• Churn

For executives and the board, these are the primary measures of success. They represent the actual exchange of value between customer and company.

For product and engineering teams, these metrics are too slow to guide daily work. They are lagging indicators. By the time a customer churns, the product has already failed them. By the time LTV changes in a measurable way, months of product decisions have passed.

We track business outcomes to validate the integrity of the metric stack.

If the North Star goes up over a sustained period, but revenue or retention does not move, that is a circuit breaker. It tells us that our proxy for user value is misaligned. Perhaps we are pushing users toward cheaper plans, cannibalizing other products, or encouraging behavior that looks good locally but is bad globally.

We do not try to optimize business outcomes directly from sprint to sprint. Instead, we use them to periodically check whether the chain from input metrics to North Star to outcomes still holds.

In deterministic systems, this three layer hierarchy gives us a clean structure for measurement and experimentation. Inputs give us fast feedback, the North Star steers day to day, and business outcomes keep us honest over longer time scales. In the next section, we look at how to validate that the links in this chain are truly causal, not just correlated.

III. Validating the Causal Chain

The hierarchy of input metrics, North Star, and business outcomes is not a fact. It is a structured hypothesis about how our system creates value:

If we move certain inputs,

we will move the North Star,

and that, in turn, will move business outcomes.

If this chain is wrong, we can spend quarters (or years) optimizing internal metrics without improving the business or the user experience. Before we commit to optimizing the stack, we need to test whether the links are causal or merely correlated.

Correlation vs Causation

Every analyst knows the phrase “correlation is not causation,” but under pressure to improve figures on a dashboard it is easy to treat a strong correlation as a lever.

It helps to fix terms:

• Correlation: A pattern in historical data. When X is high, Y tends to be high as well. This is descriptive and passive.

• Causation: A counterfactual claim. If we intervene to change X, Y will change in a predictable way. This is active and is the only relationship that matters for product changes.

Product teams get into trouble when they promote correlated metrics into targets without checking whether they are causal.

The Correlation Trap

Consider a professional network. A data scientist runs a query and finds a strong positive correlation (r>0.8) between profile_photo_uploaded and 12-month retention:

Users who upload a photo churn much less.

This is attractive. It is simple to understand and easy to rally around. The organization starts treating “profile photo upload rate” as a proxy for user health.

A naive interpretation quietly upgrades the relationship from correlation to causation:

If we increase photo uploads, retention will rise.

The team then spends multiple quarters shipping features to push uploads:

• Gamified “profile completeness” widgets

• Full-screen modals nagging for photos

• Email reminders and incentives

They succeed in driving the upload rate close to 100 percent. Six months later, retention is flat.

The problem is that they ignored a third variable, Z, such as “user motivation”:

• Highly motivated users are more likely to upload a photo.

• Highly motivated users are also more likely to retain.

In this structure:

• Z→X: Motivation drives photo upload.

• Z→Y: Motivation drives retention.

The photo itself never caused retention. It was just a symptom. By forcing low motivation users to upload a photo, the team removed the signal without affecting the underlying behavior.

This is the correlation trap. We optimize a metric that is statistically connected to the outcome but structurally powerless to change it.

Proving Causality with Experiments

To move from correlation to causation, we need to break the link between user motivation and the action we are studying. That means creating a situation where the system, not the user, determines who takes the action.

The standard tool for this is the Randomized Controlled Trial (RCT), usually implemented as an A/B test.

In an RCT:

1. Users are randomly split into groups (Control and Treatment).

2. The system applies a change only to the Treatment group.

3. Because assignment is random, hidden factors such as motivation are, on average, balanced between groups.

4. Any systematic difference in outcomes can be attributed to the change.

Returning to the photo example, a simple experiment could be:

• Control: Standard onboarding flow (no special prompt for a photo).

• Treatment: Onboarding with a mandatory “Upload Photo” step.

If, after this change, the Treatment group shows a clear lift in 12-month retention relative to Control, we have evidence that photo upload is causal. If retention is unchanged despite higher uploads, we have evidence that the original correlation was spurious.

In practice, teams rarely run such long experiments for slow metrics like retention. Experiments work well when outcomes materialize quickly. They are harder to use for links that play out over long horizons, such as:

• North Star → LTV

• North Star → multi-year retention

To prove causality for these relationships, we would need long-term holdouts that:

• deliberately degrade or withhold a feature for a subset of users,

• maintain that difference for months or years,

• and measure the impact on long-term value.

Large consumer companies sometimes invest in these long experiments, but most teams cannot afford the opportunity cost or the timeline. They need evidence faster.

Causal Inference on Historical Data

When long RCTs are impractical, we still want to reason in counterfactual terms:

For users who took action X, how would their outcomes have differed if they had not?

We approximate that answer on historical data by:

• Making our assumptions about how the world works clear

• Adjusting for the main confounders so that treated and untreated users are comparable

The process usually starts with a simple causal diagram, often a Directed Acyclic Graph (DAG). We list the main variables and sketch arrows that represent our beliefs about cause and effect, for example:

• Seasonality → booking rate

• Marketing spend → visits

• User segment → likelihood of adopting a feature

The DAG is not a mathematical object as much as a thinking tool. It forces us to name likely confounders and decide which variables we need to adjust for.

From there, most product teams can get a long way with three basic patterns rather than a full econometrics toolbox:

1. Regression with controls

   Fit a model where the outcome is a business metric and the key predictor is your candidate lever, plus control variables for obvious confounders such as segment, device, geography, or tenure. This asks: after holding these other factors constant, do changes in the candidate metric still predict changes in the outcome?

2. Panel or Difference in Differences

   When you have data before and after a change, across groups that were exposed at different times, you can compare how outcomes moved over time between those groups. This helps isolate the effect of the change from background trends such as seasonality.

3. Matching

   When adoption is strongly self selected, you can construct comparable groups by matching treated and untreated users with similar observable characteristics, then compare outcomes between those matched pairs. Matching does not remove all bias, but it reduces the most obvious differences.

There are more advanced methods available, but disciplined use of these three patterns usually provides most of the value for product teams.

The goal is not to discover a perfect structural model of the business. The goal is to get closer to that counterfactual:

For users who took action X, how would outcomes differ if they had not?

If, after adjusting for the main confounders, movements in your North Star still predict movements in business outcomes, you can treat it as a usable lever rather than a descriptive statistic. If the relationship disappears once you control for basic factors, you should treat the metric as a symptom and keep searching for a better proxy.

Why Causal Relationships Matter for Product Development

Validated causal links change how we build products. Once we trust that:

1. Certain inputs drive the North Star, and

2. the North Star drives business outcomes,

we can treat features as hypotheses about shifting those inputs and use experiments to confirm or reject them.

In deterministic systems, this structure gives us a clean feedback loop: we can make a change, observe its impact on the metric hierarchy, and attribute the result to the change with reasonable confidence.

In the next section, we will look at how this feedback loop changes the way teams define “done” for features, and how that baseline is disrupted when we replace deterministic logic with probabilistic models.

IV. The Deterministic Feedback Loop

Once we define a metric hierarchy and validate its causal links, the way we build products can change. Metrics stop being a reporting artifact and become the backbone of a feedback loop.

Instead of treating features as items on a roadmap, we can treat them as hypotheses about how to move the levers in that loop.

Feature as Hypothesis

In many organizations, Product Requirements Documents (PRDs) describe features as deliverables:

• “Build Quick Apply.”

• “Add a new dashboard.”

• “Implement in-app notifications.”

The engineering team is measured on output:

• did we ship the feature on time?

• does it match the design?

• does it pass QA?

If those boxes are checked, the work is considered done, regardless of impact on users or the business.

In a hypothesis-driven model, a feature is not a requirement to fulfill; it is a causal claim:

We believe that doing X will change user behavior Y, which will move metric Z.

A PRD becomes a specification of a bet, not just a specification of functionality. The “definition of done” shifts accordingly:

• A change is not done when the code is merged.

• It is not done when tests pass.

• It is done when we have evidence about the hypothesis: either the metric moved as expected, or it did not.

By tying each feature to a specific metric in the hierarchy (often an input metric that rolls up to the North Star), we force the team to own outcomes, not just output.

Example: Quick Apply

Consider a “Quick Apply” feature in a recruiting product.

A feature-factory roadmap might simply list:

“Q3: Implement Quick Apply to improve conversion.”

The implicit reasoning stays in people’s heads. When the feature ships, the team celebrates the launch and moves on.

In a hypothesis-driven model, we make the reasoning explicit:

Barrier: “We believe qualified candidates are dropping out because a 10-field form is too high-friction on mobile.”

Bet: “If we introduce a ‘Quick Apply’ flow that parses a resume and pre-fills fields…”

Signal: “…then application completion rate (our chosen input metric) will increase by 5%.”

We then:

1. Implement the feature.

2. Ship it behind a flag or experiment.

3. Measure its effect on the target metric and relevant guardrails.

If the metric stays flat or degrades, the feature failed the hypothesis, even if the implementation is technically perfect. The code can be bug-free and the UI polished, but if completion rate does not improve, the idea did not solve the barrier we thought it addressed.

The point is not to punish teams for failed experiments. It is to close the loop between our mental model of user behavior and the actual data.

Metrics as a Validation Signal

This framework also changes how teams resolve disagreements.

In an opinion-driven culture, roadmaps are often set by the highest paid person’s opinion or the most persuasive argument. A VP might insist that “users hate the long form,” or a designer might argue that “a cleaner layout will increase engagement.” The team implements the idea largely to satisfy the stakeholder.

In a hypothesis-driven culture, we still listen to that intuition, but we frame it as a testable claim:

“We believe users hate the long form and that shortening it will improve completion rate by 5%.”

We then design an experiment and let the data resolve the question.

Sometimes the result will contradict everyone’s expectations. We might shorten the form and see completion drop. That tells us something important about the user’s priorities: perhaps they care more about perceived seriousness and trust than speed. The feature did not just fail; it updated our model of user behavior.

Metrics, in this sense, are not only success criteria. They are a validation signal for our understanding of the user. They reduce reliance on hierarchy and intuition as the primary decision mechanisms.

The Deterministic Advantage

All of this rests on one structural property of traditional software that is easy to take for granted.

We can treat features as hypotheses and read metrics as validation signals because the product logic is effectively constant. The same input leads to the same output, and the system does not inject randomness into its own behavior.

We can visualize the deterministic product as a linear chain:

1. User Action: The user performs an input (clicks a button, submits a form).

2. Product Logic: The code executes exactly as written.

3. System Output: The system emits events and updates state in a predictable way.

4. Input Metrics: We aggregate those events into trackable metrics.

5. Value Proxy: We observe their causal effect on our North Star.

6. Business Outcome: Over time, those proxies drive revenue, retention, or other financial measures.

In this chain, step 2 (the product logic) is the constant. When we run an experiment, we make a controlled change to that logic and hold everything else fixed. If we then observe a change in the metrics at steps 4 and 5, we attribute it to the change we made.

This deterministic advantage is a measurement superpower:

• The system itself acts as a reliable control variable.

• Any variance in the metrics can, in principle, be mapped back to user behavior or product changes, not to random fluctuations in the logic.

• We can run many experiments per year because the “referee” (the system) behaves consistently.

This is the peak of the traditional model. The machine is boring, which makes the data trustworthy.

In the next part of this guide, we will see what happens when we replace this rigid link in the chain with a probabilistic model. The moment we introduce an LLM as a core part of the product logic, the system stops being a constant. The product itself becomes a source of variance, and our measurement framework has to change accordingly.

If you want help applying this framework to your own product, I’m hosting a free 30 minute live walkthrough on Redesigning Product Metrics for AI.

RSVP for Live Walkthrough

Part II: Redesigning Measurement for Probabilistic Systems

The framework we built in Part I assumes deterministic product logic, and large language models break that assumption.

I. The Collapse of the Deterministic Framework

The central claim of this guide is that the mechanics of the deterministic feedback loop break once we remove the constant from the system’s logic. That break shows up in two ways:

1. The product logic is no longer fixed; it is a stochastic mapping from inputs to outputs.

2. As a result, the signals we use to infer user value become harder to interpret.

Understanding these two shifts sets the stage for the new measurement system we need to build on top.

Fixed logic vs probabilistic output

As mentioned above, in deterministic software, the product logic behaves like a finite state machine. For a given input and state, there is a narrow set of allowed transitions, all defined by code. Again:

• If a user clicks “Save,” we persist a record.

• If a user applies a price filter, we sort accordingly.

• If a user submits a form, we either accept it or return a clear error.

We can think of this as a function:

Given the same input x, we get the same output y, modulo edge cases and infrastructure noise. At the level of application logic, we treat the code as a constant k in our measurement system.

When we replace that logic with a large language model, the behavior changes. An LLM is a conditional probability distribution over tokens:

Given a prompt x, the model samples an output y from this distribution. For the same input and context, we can see different responses over time, depending on factors such as:

• Sampling temperature and decoding strategy

• Small differences in hidden state or context

• Model updates

Even if we reduce temperature and try to “lock down” behavior, we are still interacting with a system whose natural mode is to generate from a distribution rather than execute a fixed rule.

A simple metaphor:

Deterministic logic behaves like a calculator. The same expression always returns the same result.

An LLM behaves more like a slot machine. For a given prompt, we get a draw from a distribution of plausible answers.

This shift from fixed outputs to probabilistic outputs is what voids the deterministic advantage. The product logic is no longer a background constant in our experiments. It is an active source of variation.

Loss of Control and Measurement Integrity

The deterministic feedback loop depends on two properties:

1. We can isolate a variable when we change the product.

2. The metrics we observe have a stable relationship with user value.

Probabilistic outputs strain both.

1. Control becomes more difficult

In a classic A/B test, we change exactly one thing for the treatment group. The rest of the system is assumed to be identical. Any difference in outcomes can be attributed to the treatment, up to random noise.

With an LLM in the loop, the system itself contributes randomness. Even if we fix the model version and parameters, the same prompt and context can still produce different answers over time, and those answers can differ in quality along multiple dimensions (accuracy, tone, completeness).

This complicates attribution for both back-end and front-end changes.

• Prompt experiment: We introduce a new system instruction to make answers more concise. If metrics improve, is it because the instruction changed behavior, or because the sample of outputs for this cohort happened to be better by chance?

• UI experiment: We move a “Copy to Clipboard” button. In a deterministic system, both groups see the same content; any change in clicks can be tied to placement. In a probabilistic system, the treatment group sees a different layout and potentially different answers. Higher copy rates might reflect a better button or a better answer.

We can still run experiments, but the model’s variance is now part of the error term. We need more care in experimental design, more data to reach the same confidence, and often an additional layer of evaluation to understand whether the model behavior itself shifted.

2. Measurement integrity degrades

In the deterministic world, we used our understanding of the product logic to interpret logs:

• We know what “Export PDF” does, because we wrote the code.

• We know what “Checkout Completed” means, because the workflow is fixed.

That allowed us to make strong hypotheses:

If the system reports event X, the user probably achieved outcome Y.

In a probabilistic system, we can no longer make such confident claims about the content of responses just from the presence of an event:

• A model can return an answer that is well-formed JSON, passes schema validation, and looks correct, and still be factually wrong.

• A support bot can resolve a ticket in one shot in the logs while giving advice that is subtly harmful in practice.

At the log level, a confident lie and a confident truth often look identical:

• Same endpoint.

• Same latency.

• Same status codes.

Without additional evaluation, our metrics risk collapsing back to surface activity: number of answers generated, time on page, tickets resolved, without insight into whether those answers were actually helpful.

The Interpretive Crisis: from Binary Errors to Fuzzy Failures

Traditional software gives us sharp, binary signals about correctness:

• Did the API return 200 OK?

• Did the page load in under 200 ms?

• Did the transaction commit?

Failures are usually obvious:

• 500 status codes

• Timeouts

• Exceptions in logs

We can count these failures and set clear thresholds. If error rates are near zero and latency is within budget, we assume the system is working as designed.

LLM-based systems change the shape of failure. The primary output is not a boolean or a structured record; it is text (or another unstructured artifact). The model can:

• Respond quickly

• Return valid JSON

• Stay within token limits

and still fail in ways that matter to users:

• Hallucination: Asserting facts or citations that are not grounded in any source.

• Sycophancy: Agreeing with incorrect user statements.

• Over-refusal: Declining safe prompts because of overly broad safety rules.

• Under-refusal: Answering prompts it should not.

None of these show up in standard operational metrics. We cannot write a query like:

SELECT count(*) FROM responses WHERE answer_was_helpful = false;

without first defining what “helpful” means and how to detect it.

As a result, we get a gray zone of performance:

• Uptime is high.

• Latency is low.

• Error rates are near zero.

But users still complain, or product outcomes stall. The traditional observability stack is largely blind to the semantic quality of model outputs.

To restore a usable feedback loop, we need to add a new layer to the system: an evaluation engine that can:

• Capture richer units of data (traces, not just events).

• Score outputs along dimensions that matter to users (accuracy, faithfulness, tone, safety).

• Turn those scores into metrics that can plug back into the same kind of hierarchy we used in deterministic systems.

The next section introduces that hierarchy for AI evaluations and shows how it connects to agents, retrieval, and context engineering.

II. The AI Evaluation Hierarchy

To govern a probabilistic system, we need more than a new set of KPIs. We need an additional layer in the stack: an AI evaluation engine that can turn variable model outputs into quantitative signals.

The deterministic hierarchy from Part I still matters. We still care about input metrics, a North Star, and business outcomes. What changes is that we insert a more robust evaluation layer between our system output and our North Star.

The product no longer emits clean, binary events that we can treat as proxies. It emits text (or other artifacts) whose quality must be judged before we can trust downstream metrics. The evaluation layer defines how we do that.

Grounding Hypotheses in User Value (again)

We start in the same place as before: with the user’s goal.

In the deterministic world, we grounded metrics in user actions:

• “Export Report” clicked

• “Submit Application” completed

We trusted those events because we trusted the code behind them. If the button worked, the click was a reasonable proxy for intent.

In the probabilistic world, the core object we have to reason about is not the click, but the model output:

• A block of text from an assistant

• A multi-step plan from an agent

• A generated SQL query or API call

We cannot hard-code an expected string and assert output == expected. The “right” answer is often not unique, and the model can fail in subtle ways that do not show up in structure or timing.

So we go back to first principles and ask:

For a given user goal G, what properties must the AI’s response have (or avoid) for the interaction to be considered successful?

For example:

• In a legal research tool, a lawyer might say:

  • Success: cites the correct precedents, stays within the provided documents.

  • Failure: fabricates cases or statutes, or omits critical constraints.

• In a customer support bot, a support lead might say:

  • Success: resolves the issue without escalation, in a tone that matches our brand.

  • Failure: gets stuck in apology loops, repeats irrelevant information, or gives unsafe instructions.

• In a code assistant, an engineer might say:

  • Success: compiles, passes tests, and explains the change clearly.

  • Failure: suggests insecure patterns or uses deprecated APIs.

The goal is to collect qualitative descriptions of success and failure and turn them into evaluation hypotheses.

We are effectively writing a product spec for “what a good answer looks like,” in terms of attributes rather than exact wording:

• Must be accurate with respect to provided context

• Must not hallucinate unsupported facts

• Must be concise

• Must use a professional tone

• Must avoid certain failure patterns

These hypotheses are the raw material for the evaluation hierarchy.

The Multi-dimensional Metric Suite

In deterministic systems, we often treat correctness as a single bit: success or failure. In AI systems, that is not enough. Outputs have multiple dimensions of quality.

A response can be:

• Perfectly formatted but factually wrong.

• Factually correct but needlessly long.

• Accurate and concise, but in the wrong tone.

To capture this, we replace a single “works/doesn’t work” indicator with a metric suite: a set of scores that evaluate different aspects of the model’s behavior.

A useful way to structure this suite is in three layers:

1. Structural metrics (deterministic checks)

2. Similarity metrics (reference-based)

3. Semantic metrics (model- or human-judged)

Layer 1: Structural metrics (deterministic)

Structural metrics answer the question:

Did the model follow the basic rules we imposed?

They are cheap, binary checks we can implement with normal code:

• Format compliance

  • Is the output valid JSON?

  • Does it conform to the requested schema?

  • Are all required fields present?

• Constraint satisfaction

  • Is the answer under 500 words?

  • Did it include the required sections (for example, “Summary,” “Risks,” “Next Steps”)?

• Baseline performance

  • Did we receive a response within 200 ms for the first token?

  • Did the model stay within token or rate limits?

If a response fails structural checks, we often do not need an AI model to judge it. We can either discard the output, retry, or mark it as a hard failure.

Structural metrics are our first line of defense: they keep obviously broken outputs from polluting the rest of the system.

Layer 2: Similarity metrics (reference-based)

Some tasks have reference answer or at least reference text that describes the expected content. In those cases, we can measure how close the model output is to a known good example.

This layer includes N-gram overlap (ROUGE, BLEU, etc.), which is useful for translation, summarization, or extraction tasks where specific phrases matter. And it also includes semantic similarity, which converts the output and the reference into embeddings and compute cosine similarity. This captures “meaning” similarity even when the wording differs.

These metrics are more forgiving than exact string equality. They let us see whether the model roughly matched the intent of an answer key without requiring identical phrasing.

Similarity metrics are not enough on their own, but they are fast and useful as a middle layer between basic structure and deeper judgment.

Layer 3: Semantic metrics (probabilistic)

The most important layer for AI products is the semantic layer:

Does this response actually do what the user needs, according to the criteria we and our users care about?

These are inherently judgment calls. They evaluate properties such as:

• Accuracy / faithfulness

  • Does the answer rely only on the provided context, or did it introduce unsupported claims?

  • For a RAG system, are all asserted facts traceable to retrieved documents?

• Helpfulness / task success

  • Did the answer resolve the user’s issue?

  • For a code assistant, did the suggestion compile and pass tests?

  • For a support bot, would a human agent consider the ticket “resolved”?

• Tone and style

  • Does the answer sound professional, empathetic, or neutral as required?

  • Does it avoid banned phrases or sensitive language?

• Safety

  • Did the model avoid harmful content?

  • Did it refuse unsafe prompts?

  • Did it avoid over-refusing safe but technical prompts (for example, “kill process” in a Linux context)?

We cannot reliably capture these with regexes or simple numeric thresholds. Instead, we typically use model-based evaluations (“LLM-as-judge”) or human review:

1. For each trace, we feed the judge (human or machine):

   • The user request

   • The context provided to the model

   • The model’s response

   • A rubric describing what we want to assess

2. The judge model returns a binary success or failure (0 or 1), and optionally an explanation.

This gives us a numeric semantic score for each dimension we care about. We can then aggregate these across traces to form metrics like:

• Average faithfulness score

• Percentage of responses with safety issues

• Percentage of answers rated “task complete”

Taken together, the three layers give us a **multi-dimensional view** of quality:

1. Structural: “Is the output well-formed?”

2. Similarity: “Is it close to known good outputs?”

3. Semantic: “Is it actually correct and useful for this user, in this context?”

Defining the Unit of Evaluation: The Trace

So far we have talked about scoring “responses.” For AI products (especially agentic ones) that is too narrow. A single helpful or harmful response is the result of a process, not just a single model call.

To evaluate that process, we need a richer unit of data: the trace.

A trace is a single, end-to-end record of an AI interaction. At minimum, it should capture:

1. Input: The user’s request, including any relevant metadata (user id, segment, channel).

2. Context: The documents, database rows, or API results retrieved (for example, via RAG) and passed into the model.

3. System instructions: The prompts, system messages, and policies used to condition the model, such as “act as a support agent” or “respond in JSON.”

4. Intermediate steps (for agents):

   • Tool calls (SQL queries, API calls, web searches).

   • Tool outputs returned to the model.

   • Planning steps, if the agent decomposes the task.

5. Output: The final answer shown to the user (or the action taken on their behalf).

For agent systems, the trace is effectively a timeline: a sequence of model calls and tool interactions that shows how the agent arrived at its final decision.

The trace is the atomic row of data that the evaluation engine operates on. It lets us:

• Apply structural checks (for example, JSON validity) to the output.

• Compare the output to the context for faithfulness.

• Ask a judge model to rate helpfulness using the full story (input + context + output).

• Diagnose where things went wrong when a semantic metric fails.

Consider a simple faithfulness evaluation in a legal assistant:

Trace A (success)
- Context: “The defendant, Mr. Smith, was acquitted on July 4th.”
- Output: “Mr. Smith was found not guilty in early July.”
- Check: Every asserted fact (“not guilty,” “July”) is supported by the context.
- Faithfulness score: 1.0

Trace B (hallucination)
- Context: “The defendant, Mr. Smith, was acquitted on July 4th.”
- Output: “Mr. Smith was convicted on August 1st.”
- Check: The key claim (“convicted on August 1st”) is not present in the context.
- Faithfulness score: 0.0

Without the trace, we only see the output text and we are forced to treat “convicted on August 1st” as just another string. With the trace, we can compare output to context and turn “hallucination” into a computable metric.

The same pattern applies to agents:

• If the wrong answer comes from bad retrieval, the trace will show irrelevant or missing documents.

• If the agent picked the wrong tool, the trace will show a tool call that does not match the user’s need.

• If the reasoning step failed, we will see correct tool outputs followed by a flawed synthesis.

By making traces the core unit of evaluation and layering structural, similarity, and semantic metrics on top of them, we move quality from a vague judgment to a measurable, debuggable property of the system. This, in turn, lets us connect AI-specific metrics back into the broader hierarchy: they become input metrics and quality levers that we can validate against the North Star and, ultimately, business outcomes.

III. Governing the AI Metric Suite

By this point, we have:

1. Traces as the unit of evaluation, and

2. A metric suite that scores each trace along structural, similarity, and semantic dimensions.

The next problem is not a lack of metrics. It is too many of them.

For any non-trivial AI product, we can define dozens of plausible quality metrics:

• Accuracy, faithfulness, coverage

• Helpfulness, task completion

• Brevity, verbosity, reading level

• Tone, empathy, politeness

• Safety, refusal rate, false refusal rate

• Etc.

Each metric is defensible in isolation. But we cannot run a product by trying to optimize all of them simultaneously. Without a way to prioritize, the evaluation engine degenerates into a collection of dashboards that are interesting but not actionable.

We need a way to:

1. Decide which metrics actually matter for user and business value.

2. Manage trade-offs between them when they conflict.

The Infinite Metric Suite Problem

In deterministic software, the number of failure modes we measured was relatively small:

• Latency

• Error rates

• Uptime

• A handful of key product metrics

If the API was fast and returning 200 OK, and the key product metrics looked healthy, we assumed the system was working.

In AI products, the failure surface is much larger. Because the output is text or other unstructured artifacts, there are many ways for it to be “wrong”:

• Factually incorrect (hallucination)

• Correct but irrelevant (off-topic)

• Correct but too long or too short

• Correct but in the wrong tone

• Refusing safe prompts or answering unsafe ones

We can define a metric for each of these. Teams might create a:

• “Faithfulness score”

• “Helpfulness score”

• “Brevity score”

• “Empathy score”

• “Citation score”

• “Toxicity score”

On a dashboard, this becomes a wall of gauges. When one metric improves and another degrades, it is not obvious whether the change is good or bad for the user.

This is the infinite metric suite problem:

The space of possible quality metrics is unbounded, but the team’s attention is not.

The solution is not to stop measuring. It is to treat metrics as candidates that have to earn their place in the core evaluation stack.

Filtering the Metric Suite to Key Quality Indicators

We can borrow the same discipline we used for traditional metrics:

We do not accept a metric into the core stack only because it is easy to compute.

We accept it only if it has predictive power for the North Star or for important business outcomes.

This process looks a lot like feature selection in machine learning:

1. Generate a large set of candidate AI metrics from the evaluation engine.

2. Test which ones actually explain variance in the North Star.

3. Promote only the useful ones to Key Quality Indicators (KQIs).

There are three main ways to do this.

1. Passive validation (historical regression)

Start with historical traces where you know, for each interaction, whether it contributed to your North Star or to a direct, per-interaction version of it.

Examples:

• Code assistant: did this suggestion lead to an accepted change or a commit that passed tests

• Support bot: did this exchange resolve the ticket without human escalation

• Knowledge assistant: did this answer satisfy the query without a follow-up or escalation

For each trace, compute:

• Candidate AI metrics: faithfulness, helpfulness, brevity, safety scores, and so on

• A target outcome that reflects the North Star at the unit of analysis you care about

  • Either the North Star itself, if it is defined per interaction

  • Or a label that rolls up directly into the North Star (for example booking_created_in_this_session when the North Star is Nights Booked)

Then run simple models:

• Correlation analysis

• Logistic or linear regression

• Tree-based models with feature importance

The question is:

Does variance in metric X consistently explain variance in the North Star outcome Y?

You might find:

• A politeness score with almost no relationship to whether engineers accept code suggestions

• A strong negative relationship between latency and tickets resolved in a single turn

This kind of analysis does not prove causality, but it is a cheap way to separate obvious noise from candidate signals. Metrics with near-zero power to explain North Star movement are unlikely to be worth optimizing.

2. Active validation (experiments)

To move from “predictive” to “causal,” we can run experiments where we deliberately change the system to move a quality metric and see what happens to the North Star.

For example:

• Change prompts to enforce more concise answers.

• Introduce an explicit reasoning step (chain-of-thought) to improve accuracy.

• Adjust safety rules to reduce over-refusals.

In a classic RCT:

1. Randomly assign users to Control and Treatment.

2. Apply the change only to Treatment.

3. Measure both the AI quality metric (for example, conciseness score) and the downstream product metric (for example, task completion).

If the treatment group shows a significant improvement in both the AI metric and the product metric, that is evidence that the AI metric is not just a descriptive label; it is a useful lever.

AI experiments have an extra wrinkle: the “treatment” is probabilistic. For example, a prompt intended to enforce brevity might still produce long answers part of the time. To know who actually “received” the treatment, we must score traces after the fact (using the evaluation engine).

This makes experiments more expensive:

1. We have to run the system.

2. Then run evaluations.

3. Then analyze the results.

The risk profile also changes. Because we do not control the exact output a user will see, there is a higher chance that an experimental variant serves a bad answer, a confusing tone, or an unsafe suggestion. In a deterministic system, treatment and control differ in a narrowly scoped way. In an AI system, the entire distribution of outputs can shift in ways that are hard to anticipate, which raises the risk of degrading the experience and eroding user trust.

Because of this increased cost and increased risk, we typically reserve experiments for a small number of promising variants and rely on cheaper historical analysis to filter out weak candidates.

3. Causal Inference on Observational Data

When experiments are risky or expensive, you can still approximate causal impact from logs. The idea is the same as before, but now the “treatment” is a change in an AI quality metric (for example, higher faithfulness) instead of a feature flag.

In practice, you reuse the same tools from the deterministic section:

• Regressions with controls: Model the North Star (or a per interaction version of it) as a function of your AI metric plus user and context controls.

• Panel / before–after comparisons: When a model, prompt, or safety policy changed at a known time or for a known cohort, compare how outcomes moved for exposed vs unexposed groups.

• Matching: When some users naturally see higher or lower quality responses, match them on observables (segment, tenure, workload) and compare outcomes between matched groups.

You will not get a perfect estimate, but you will usually learn whether a given AI metric is directionally useful. The test is simple:

After you control for the obvious confounders, do changes in this metric still line up with changes in the North Star?

If yes, treat it as a candidate KQI. If no, keep it in the dashboard as a diagnostic and look for a better lever.

Managing Trade-offs Between KQIs

Even after pruning the suite, we will not end up with a single scalar metric. We will end up with a subset of KQIs that matter. For example: faithfulness, task success, latency, and safety.

These metrics can often pull in different directions:

• Tightening safety can increase false refusals.

• Forcing more careful reasoning can increase latency.

• Compressing answers to improve brevity can hurt perceived helpfulness.

We cannot maximize all of them at once. We need a way to reason about trade-offs.

The same regression style analyses you used to validate metrics can also give you rough coefficients that describe how sensitive the North Star is to each metric:

how much a small change in metric X tends to move the North Star, holding other factors constant.

For example:

• A 10% increase in hallucination rate might reduce task success by 8% (−0.8 coefficient).

• A 10% increase in average response time might reduce task success by 1% (−0.1 coefficient).

These coefficients act like exchange rates between metrics. They let us ask questions such as:

• How much latency can we afford to add if we reduce hallucinations by a given amount?

• How many additional refusals are acceptable if we significantly reduce toxic outputs?

We can write this informally as:

Where:

• C i is the estimated impact coefficient of metric i on the North Star.

• ΔMetric i is the change we expect for that metric.

Example:

A prompt change reduces hallucinations by 5 percentage points and increases latency by 200 ms.

• Coefficient for hallucinations: −0.8

• Coefficient for latency: −0.01 per 100 ms

Approximate impact:

• Hallucinations: +5×0.8=+4.0 units of value

• Latency: −2×0.01=−0.02 units of value

Even with the added latency, the net effect is strongly positive (4.0 - 0.02 = +3.98). The trade-off is worth it as we expect a net increase in our North Star of 3.98.

The math here is approximate; coefficients have noise and confidence intervals. The point is not to compute an exact utility function. The point is to make trade-offs explicit, ground them in empirical estimates rather than intuition, and keep the focus on Net User Value, not on any single metric in isolation.

With this governance structure the evaluation engine generates candidate metrics, causal validation promotes a small subset to KQIs, and causal weighting helps us navigate trade-offs between KQIs when making product decisions.

In the next section, we will look at how this metric stack must evolve over time, and how trace-level error analysis becomes a continuous discovery process as the model, data, and user behavior change.

IV. The Continuous Discovery Loop

At this point, we have:

1. A metric hierarchy for the product (inputs → North Star → outcomes).

2. An evaluation engine that scores traces along key quality dimensions.

3. A governance layer that filters metrics into a small set of KQIs and manages trade-offs.

It would be tempting to stop here: define the metrics, build the data pipelines and dashboards, and treat the evaluation system as “done.”

For AI products, that assumption is wrong.

In deterministic systems, failure modes are largely defined by the code we wrote. In AI systems, the failure surface is not only larger; it evolves over time. The definition of “quality” is a moving target.

The Limit of Foresight

As we’ve now discussed several times, in traditional product development, we can often enumerate the main ways a feature can fail:

• The API returns an error.

• The form submission times out.

• The database write fails.

We can write tests for these cases up front because the behavior of the system is bounded by the logic we implemented.

In AI systems, new failure modes can emerge even if we never change the code.

Three forces drive this:

1. New inputs

As more users adopt the product, they bring new prompts, edge cases, and domain-specific language that were never present in the original evaluation set.

For example, a code assistant might start seeing prompts about a language or framework that was rare at launch.

2. New underlying data

Many AI products are connected to live data sources via retrieval (RAG) or tools. As new records enter the index, they create new contexts for the model to operate on. The model might start pulling in content with properties we did not anticipate (for example, competitor information, noisy logs, or unstructured legal clauses).

3. Stochastic output

Even with a fixed model and data, sampling can produce new combinations of tokens that were not seen in earlier traces. Small changes in prompts or context can push the model into previously unexplored regions of its behavior. Because of this, the model will eventually fail in ways that were simply not present in our “golden” pre-launch dataset.

For example, consider a legal research assistant:

• Day 0: We define “accuracy” as “no hallucinated cases” and write evals to catch invented citations.

• Day 30: We discover a new pattern: the model responds with real case citations that are technically correct but irrelevant to the user’s question. It is “citation stuffing” to sound authoritative.

• Day 60: As new documents enter the index, we notice that the model occasionally recommends competitors’ products or services. The answers are helpful from the user’s point of view but harmful for the business.

Neither failure mode was in the original rubric. No static metric set defined on Day 0 could have fully anticipated them.

This is the limit of foresight in open-ended AI systems:

We cannot rely on a one-time metric design exercise to cover all future failure modes. Quality definitions must evolve with the product.

Error Analysis as Exploratory Data Analysis

To keep up with new failure modes, we need a systematic way to discover them.

In data science, before modeling, we perform exploratory data analysis (EDA). We inspect raw rows, not just aggregates. We look for outliers, clusters, and patterns that suggest new hypotheses.

For AI products, traces play the role of rows in a dataset. And “error analysis” becomes a form of EDA on traces.

The goal of error analysis is not just to debug known problems, but to discover behaviors our current metrics do not capture and turn them into new hypotheses and metrics.

A useful practice is to regularly sample and review traces, for example:

• Random samples of successful interactions.

• Random samples of failures (low KQI scores, user complaints, escalations).

• Samples filtered by interesting conditions (long latency, multiple tool calls, repeated retries).

When we read traces in detail we are looking for patterns that feel wrong or surprising, even if they technically “pass” current checks.

Examples:

• The model is accurate but condescending when the user asks basic questions.

• A safety filter blocks a harmless technical phrase such as “kill process,” leading to over-refusals in developer tools.

• An agent chooses a needlessly complex tool sequence when a simple lookup would do.

Each of these observations is a candidate failure mode. Once we see it a few times, we can promote it to a hypothesis:

• “We believe our safety filters are too aggressive for technical domains, causing false refusals.”

• “We believe unnecessary tool hops are hurting latency without improving quality.”

• “We believe condescending tone is hurting user trust, even when answers are correct.”

From there, we can:

1. Define a new metric or rule (for example, “false refusal rate on technical prompts”).

2. Implement it in the evaluation engine.

3. Measure its relationship to the North Star or other outcomes.

4. Decide whether it becomes a KQI or remains a diagnostic metric.

Error analysis is the mechanism that converts unknown unknowns into known unknowns, then into measurable quantities.

The Iterative Workflow

Putting it all together, AI metric design becomes a continuous evaluation engine rather than a one-time project.

At a high level:

1. Ground in user value: Choose a specific user workflow and clarify what success and failure mean for that workflow.

2. Formulate hypotheses: Translate those definitions into statements about model behavior.

3. Define metrics: Implement structural, similarity, and semantic metrics that correspond to those hypotheses. Use LLM-as-judge or human review where needed.

4. Validate causally: Check which metrics correlate with and help predict the North Star. Where practical, run experiments or apply causal inference techniques to test whether improving a metric improves user outcomes.

5. Filter and prioritize: Promote a small set of metrics to KQIs with clear ownership and targets. Keep others as secondary diagnostics.

6. Run and monitor: Log traces and scores in production. Monitor KQIs over time alongside traditional product metrics.

7. Perform error analysis (EDA): Regularly review traces, especially:

• Those with good KQI scores but bad user outcomes (missed failures).

• Those with bad KQI scores but good user outcomes (overly strict metrics).

8. Update hypotheses and metrics: For new patterns discovered in trace review, go back to step 2 (turn them into hypotheses → define metrics → validate them → filter and prioritize)

This loop never stops.

As the model is updated, as the data changes, and as users adopt new behaviors, we keep refining both the system that generates outputs and the metrics we use to measure those outputs.

Metrics as a Living Codebase

In the deterministic world, many metrics were effectively static artifacts. We picked them once (“page load time,” “error rate,” “DAU”) and then focused on moving them.

In AI products, metrics themselves become a kind of code. They encode assumptions about what quality means. They are implemented, tested, versioned, and sometimes deprecated. They change over time as we learn more about the product and the users.

Treating metrics as a living codebase has concrete implications:

1. Someone needs to own the evaluation engine and metric definitions, just as someone owns the model and the UI.

2. Changes to metrics should go through review, with clear documentation of what they mean and why they matter.

3. When we see a metric stop correlating with user outcomes, we should be willing to refactor or retire it.

In a probabilistic world, we cannot fully predict how the system will behave. What we can do is build a rigorous machine around it to observe, measure, and adapt. The continuous discovery loop is how we keep our definition of “quality” aligned with reality as the product, users, and models evolve.

V. The New Architecture of AI Metrics

We started with a common failure mode: trying to measure probabilistic AI products with the same tools we used for deterministic software.

In the deterministic world, we could often get away with inheriting generic metrics (like DAU and session length), treating the product logic as a constant, assuming that if the numbers moved in the right direction, users were getting value.

Once we introduce a large language model into the core of the product, that comfort breaks down. The model turns the logic layer into a source of variance. The output is no longer a simple function of the input; it is a draw from a distribution conditioned on prompts, context, and internal weights.

Redesigning metrics for AI is not about inventing a new list of KPIs; it is about accepting that measurement itself is an active engineering discipline.

We can summarize the shift by comparing the data flow in deterministic and AI systems.

The Deterministic Chain

In a traditional product, the chain of custody for data looks like this:

1. User Action: the user performs an input (clicks a button, submits a form).

2. Product Logic: the code executes exactly as written.

3. System Output: the system emits events and updates state in a predictable way.

4. Input Metrics: we aggregate those events into trackable metrics.

5. North Star: we observe their causal effect on our North Star.

6. Business Outcome: over time, those proxies drive revenue, retention, or other financial measures.

This is the same chain we built in Part I, with deterministic product logic at its core.

The Probabilistic Loop

When we introduce an LLM or agent into the product, we do not throw away this chain. Users still act, the system still emits events, we still define value proxies, and we still care about business outcomes.

What changes is Step 2. The product logic is no longer a single, rigid code path. It becomes an AI pipeline that needs its own internal measurement loop.

We can think of the AI version as:

1. User Action: the user issues a query or takes an action, just as before.

2. Product Logic: the LLM (and possibly an agent) generates stochastic intermediate steps and output.

3. System Output: the system emits a full trace (input, context, prompts, tool calls, tool outputs, and final response)

4. Continuous Evaluation Engine: we introduce the evaluation loop described earlier on each trace:

   1. Clarify user value for the workflow

   2. Form hypotheses about AI behavior

   3. Define structural / similarity / semantic metrics

   4. Validate their relationship to the North Star and business outcomes

   5. Filter to KQIs, monitor, and refine based on error analysis

So the overall chain is still:

User Action → Product Logic → System Output → Input Metric → North Star → Business Outcome

What changes is:

in an AI metric workflow the product logic contains a probabilistic model which requires the static input metric to become a living metric suite.

In the deterministic world, we optimized products against a relatively stable set of metrics. In the AI world, we are optimizing both the product and the metrics themselves. That is the new architecture of AI measurement: a joint system where models, products, and evaluations co-evolve to keep our definition of “success” aligned with reality.

If you’d like to see how this looks on a real product, you can join my free 30 minute live walkthrough on Redesigning Product Metrics for AI.

RSVP for Live Walkthrough