The lowest hanging fruit is to optimize retrieval and context. Once that’s done, improve your systems and harness. Consider post-training after the other approaches are exhausted.

I hosted an AI Product Engineering series to explore these topics. I’ve summarized all 13 sessions, organized by theme, with links to the recordings and source materials. The full series is 9.5 hours. The notes take about 20 minutes to read.

All 13 sessions are outlined below. Click the table to read the full post on my website.

If you want to stay on Substack, here are links to the notes from all 13 sessions.

Evals

• Evals for Data Agents

• Cut Classification Costs With a Model Cascade

• Automating Error Analysis

• Case Study: Putting Evals Into Production

Context

• An Intro to Multivector Retrieval

• How to Improve Search Agents

• How to Optimize Retrieval Embeddings

• How to Choose an OCR Model

• Steer AI Writing With Footnotes for Agents

Systems

• Debugging Inference Latency

• Open Weight Model Economics

• The Case for Agent Sandboxes

• Turn Eval Results Into a Better Model

Nearly every improvement in these notes starts with good evals. If you want to work through evals with a live group, the AI Evals course has hands-on exercises and office hours.

That’s why I’m writing. Our next live cohort of AI Evals for Engineers & PMs starts September 5. You have two paths from here. You can join us for the live course, or keep learning on your own with the free sessions below. Here’s a guide for both.

Path 1: Join the live cohort

After four weeks, you’ll know how to find where your AI product fails and measure each failure with evaluators you trust. You’ll also know how to put those evals in CI and improve the product on accuracy and cost. The course runs September 5 to October 3.

Learn more about the course

What our students are saying:

See more testimonials here.

What you’ll learn

The course is built around one running example. You build a real support agent, then take it all the way to production evals. It runs in five modules.

1. Building agents. You build a support agent and instrument it so you can measure how it behaves, then generate the synthetic data and scenarios the later modules use.

2. Error analysis. You read the agent’s traces to find where it fails, then build a small set of evaluators you trust to measure each failure.

3. CI/CD for agents. You turn those failures into a regression suite that runs before you deploy, and add lightweight monitoring for after you ship.

4. Safety and adversarial evaluation. You red-team the agent for attacks like prompt injection and tool misuse, then turn each successful attack into a test you keep.

5. Improving agents. You compare versions of the agent to raise accuracy, then cut cost without losing quality.

There’s also a bonus session on evals interview prep.

What’s included

• About 10 hours of live office hours

• Lifetime access to our Discord community of past students

• A 150+ page course reader

• Access to future cohorts, so you can keep coming to office hours and get the latest material

• 6 months of access to our AI Evals Assistant, trained on everything we’ve made on evals

Learn more about the course

Path 2: Keep learning with the free sessions

If the live course isn’t right for you now, that’s fine. Start with the reading below, then go deeper with the sessions.

Essential reading

• Evals FAQ. The most common eval questions answered, from the basics to advanced.

• “It’s Hard to Eval” Is a Product Smell. Your product should make it easy to verify AI outputs.

• A Field Guide to Rapidly Improving AI Products. The evaluation and improvement process from more than 30 production builds.

• Using LLM-as-a-Judge For Evaluation: A Complete Guide. A step-by-step guide to judges you can trust.

• Do Automated Evals Work?. We checked 100 human-labeled traces against automated eval systems.

• Shreya’s research: Who Validates the Validators?. How to align LLM judges with human labels.

The free sessions

The whole AI Product Engineering series is free to watch. Here are the fifteen, grouped by topic.

Evals

• Stop Letting Your LLM “Find the Issues” In Your Data, with Shreya Shankar. How to use AI to speed up error analysis while you stay in control of what counts as a failure. video

• Evals Your Whole Team Can Run, with Lucas Machado Rocha. How to make evals simple enough for your whole team to run, using how Nova Escola did it. video

• Can AI Actually Answer Your Data Questions? with Shreya Shankar. How to evaluate data agents on messy real questions, where even the best models get only about 57% right. video

• The Cold-Start Eval Problem, with Shreya Shankar. How to build evals from day one, before you have a single user. video

• Turn Eval Results Into a Better Model, with Will Brown and Florian Brand. How to turn your eval results into training that improves the model. video

Retrieval and search

• Scaling Late Interaction to Billions of Documents, with Marek Galovic. How to keep the accuracy of late-interaction retrieval as you scale to billions of documents, without the cost blowing up. video

• What Makes a Good Search Agent? with Nandan Thakur. How to benchmark and debug a search agent, and why the retriever affects accuracy as much as the model. video

• Why We Embed Documents, and Why You Should Go Multi-Vector, with Ben Clavié. When single-vector embeddings fail on long or unusual documents, and when multi-vector is worth the cost. video

• Stop Picking Embedding Models off the MTEB Leaderboard, with Radu Gheorghe. How to pick the embedding model that fits your own data, when a high leaderboard rank won’t tell you. video

Cost and open models

• Why Your AI Product Feels Slow Even When the Model Is Fast, with Abi Aryan. A hands-on walk through where your latency and cost come from when the model is fast but the product feels slow. video

• How to Choose an OCR Model, with Joe Barrow. How to decide when a managed OCR API is enough and when to host your own open model. video

• How to Use Open Models Effectively, with Zach Mueller. When an open model can replace a frontier model without losing quality, so you stop overpaying. video

• Stop Paying Full Price for LLM Classification, with Shreya Shankar. How to cut the cost of LLM classification at scale without hurting quality. video

Agents and context

• No Country for Old Text: Stop Wasting Your Context, with Bryan Bischof and Adam Conway. How to keep track of who wrote each part of AI-generated text, and why, so that context stays with it across tools. video

• Don’t Build Agents, Build Environments Instead, with Adam Azzam. How to build the fast, dependable environment your agent runs in, which does more for results than the agent code. video

Whichever path you pick, I hope this helps you build something that works.

Best, Hamel

👉 For the best reading experience, read the original post with interactive figures. 👈

For the past 3 years, AI evals have been my professional focus.¹ The most common objection I hear to evals is “our product is hard to eval”.

This objection is a product smell. Artifacts that are hard for you to verify are often hard for users too. In the worst case, users have to redo the work from scratch to verify the output. More importantly, designing your product for ease of verification should come before building evals.

In this post, I’ll walk through three products I advised on that faced this issue. I’ll also show before and after sketches to demonstrate design principles. After these examples, I’ll discuss how to apply this general pattern to your product.

Example 1: the AI data agent

Almost every company I’ve worked with builds an internal AI data agent. You ask it a business question, like what was net revenue for Product A last quarter, and it finds relevant data sources, runs the queries, and provides an answer. The goal of this agent is to reduce dependency on data analysts.

A common mistake when building AI data agents is to make the answer the only output, as illustrated below.

In the sketch above, the user has no way to verify the answer beyond redoing work.² A better design is to provide the user with checkable artifacts, informed by how a domain expert might validate the output. Here are techniques I use to validate metrics as a data scientist:

1. Compare the quantity and any intermediate calculations against a trusted source, like a vetted dashboard or report, or a similar analysis a colleague has already vetted.³

2. Confirm the metric definition precisely. A number like net revenue can include or exclude things like returns and discounts.

3. Sanity-check a related quantity. If I can’t verify the number directly, I pull a related number that should move with it, like units sold or unique customers, and check if the combination is plausible.

4. Look at what is beneath the aggregate. A total can hide problems, so I break it down by dimensions like region or time period and sanity-check the distribution.

5. Read the query. For an important number I look at the SQL to confirm it does what I think, and I tweak it and rerun to test my assumptions.

6. Note anything I could not verify. If a step has no trusted reference to check it against, I flag it instead of presenting it as settled.

Here’s what a better interface might look like. The two tabs below show the same answer at two levels of detail. The chat reply surfaces the details worth seeing up front, and the notebook holds the full analysis behind the answer. Use the tabs to switch between them.

Substack shows the two tabs as static screenshots below.

Chat

Notebook

This is the notebook the agent worked in while producing its answer. Scroll within the figure to see all the cells. The sidebar has a Contents tab for jumping between sections and an Assistant tab for asking follow-ups.

👉 The original post lets you scroll the notebook, and inspect the figures interactively. 👈

There is a lot to unpack here. Here are notable changes:

• The agent optionally performs retrieval from vetted analyses, and the interface shows which one was used along with who authored it.

• There is progressive disclosure of details. The chat reply shows high value items like sources, assumptions, and issues. The user can optionally open an interactive notebook to see the full context.

• The AI-generated notebook (see notebook tab above) is organized to promote verification: it opens with the assumptions the agent made, like the metric definition and where it came from, then shows the queries it ran and the numbers they returned. It breaks the total down so you can sanity-check the distribution, and it closes with a list of what it could not verify, each item left as a cell you can run.

• The AI agent is also available in the notebook to help with follow-ups. Finally, the user can publish the notebook back to a knowledge base, where it can be retrieved by future analyses, creating a virtuous cycle.

This design sketch is far from perfect. The point is that the product should help the user verify the answer as a domain expert would. Compare it to the earlier approach, where the only output was the number.

Data agents like these are not science fiction. Hex⁴ is my favorite product in this genre; it integrates notebooks and chat better than anything I’ve seen. Here are screenshots from their landing page:

But what does this have to do with evals? If you design your product for verification, annotation becomes less expensive and evals will have better signals to draw from. More importantly, you’ll provide your users with a better product.

Example 2: the PE curriculum builder

A founder I advised was building an AI tool that writes physical education lesson plans for K-12 teachers. A teacher enters their constraints, like the grade they teach, how long the class is, whether it meets indoors or outdoors, and what equipment they have. The tool then writes a lesson plan for those constraints. The goal is to save teachers the time they spend planning and give them a plan that fits their class. Here is a sketch of what the product looks like:

The founder asked me how to eval the lesson plans. I turned the question around: what does a teacher care about?

The fastest way to trust a plan is to see that a teacher like them already uses it. Additionally, teachers value visibility into what others are doing so they can learn new approaches. Therefore, a better design might start from vetted lesson plans that are actively used in schools. When the tool generates a plan, it shows which vetted plan it started from, who uses that plan, and a diff of what it changed for this teacher’s constraints.

Next, the teacher can check a small set of changes against a plan they already trust, instead of judging a whole plan from scratch. Here’s a sketch of what a better interface might look like:

In this version, most of the plan is inherited from a vetted plan. The teacher’s review is scoped to a few edits, each with a reason explaining why the change was made. This is a more efficient way to review a plan because it reduces the cognitive load of judging a whole plan from scratch.

Designing for this makes the product simpler to build. Instead of stuffing hundreds of examples into a prompt, the tool captures important dimensions, retrieves a close match, and adapts it. Automated evals now become tractable because there is less surface area to test. For example, you can verify that the plan retrieval picked a sensible anchor, and each edit honors the constraints.

Example 3: the workers-comp medical report

The last example comes from a workers’ compensation tool a founder asked me to help with. It reads a patient’s chart (intake forms, imaging reports, therapy notes, prior exams) and generates a long expert opinion report, often fifty pages or more. Here is a sketch of the product:

The problem is the same as the other examples, but the stakes are higher. The only output is the report, and the doctor is the one accountable for it. To trust it they have to go back through the chart and confirm the facts and inferences themselves. That can take as long as writing the report from scratch, which defeats the point of the tool.

You might object that a fifty-page opinion is hard to verify. That is true, and the product should not pretend otherwise. Helping a doctor understand the evidence is arguably more valuable than the finished document. Therefore, I advised the founder to make the product work like a research assistant instead of a report generator.

For example, the product could read every record and pull out relevant facts, with a link back to the page so the doctor can check each one. Where two exams disagree, or the chart leaves a question open, the product should surface that. The doctor can then resolve any contradictions and fill in the gaps. Finally, the product can assemble the final report from what they have already checked. Here is a sketch of what this might look like:

The research assistant version of this product allows the doctor to build trust by verifying facts as they go. Similar to the other examples, this design is easier to build and evaluate. Now there are scoped units to grade, such as whether a contradiction is real or whether a citation supports a claim.

Generalizing the pattern

It is important to understand how users verify your product’s AI artifacts. Sometimes, this may require assembling supporting evidence. In other cases, it could mean refactoring the entire workflow so that the user is in the loop (like the workers’ comp example).

Below are questions that can guide your product’s design for verification:

1. What does the user actually need to check?

2. What trusted thing can they compare it against?

3. Are there signals or heuristics that experts use to aid in verification?

4. What smaller units can they accept, edit, or reject?

A common thread across these examples is provenance. The fastest way to make an output checkable is to show where each part came from, with links to see more detail. Additionally, you can use progressive disclosure so these sources don’t overwhelm the user.

What needs verifying also changes as the user’s trust grows. Early on, the data agent should make provenance obvious, like where a metric definition came from. Once the user trusts the agent gets it right, that detail can collapse by default in the card. Good design meets users where they are instead of showing everything.

These principles hold even when a product seems easy to eval. Coding is a good example: it is one of the most verifiable kinds of work, with tests, types, and diffs. Even so, some coding agents go the extra mile to make their work checkable. Cursor and Devin both record a short video of the UI changes they make, so you can confirm the work is right without reproducing it yourself.⁵

None of this is new

Evals thinking is aligned with good product design. Gathering supporting data and breaking down workflows into smaller units makes automated grading easier. However, I don’t want to pretend like any wisdom here is new.

All of these ideas stem from well-established design principles. For example, watching an expert work to learn what they check before you build is called needfinding.⁶ In research-heavy work like the medical case, there is a design goal called sensemaking, which is the work of building a structured understanding of a body of evidence you can reason over.⁷ There are many other concepts, but I think you get the idea.

Even though these ideas are well established, a reminder is due in the age of AI. Before AI, verification often happened incidentally during the process of creating work product. With AI, verification is the bottleneck. It is time to think about it more explicitly.

👉 For the interactive figures and best reading experience, read the original post. 👈

Thanks to Shreya Shankar and Isaac Flath for feedback on this post.

To stay updated on my writing on AI evals, subscribe to my newsletter below.

Notes

1. More of my writing and teaching on evals: Your AI Product Needs Evals, A Field Guide to Rapidly Improving AI Products, Using LLM-as-a-Judge for Evaluation, LLM Evals: Everything You Need to Know, Selecting the Right AI Evals Tool, Evals Skills for Coding Agents and The Revenge of the Data Scientist. I also co-teach the AI Evals for Engineers & PMs course and co-authored the O’Reilly book Evals for AI Engineers.

2. Lenny Rachitsky tweeted about this recently: most of one data science team’s work is now reviewing half-baked AI analysis from PMs and engineers, and half of it is wrong.

3. When I worked at Airbnb we had an internal tool called the Knowledge Repo, a place where data scientists published notebooks of their deep dives on analytics, modeling, and so on (they wrote about it here). It was one of the fastest ways to get context on a new project, since you could read what someone had already worked out. I don’t know if it is still in use, but the paradigm is a good one.

4. Bryan Bischof led the creation of the AI at Hex. Bryan is a data scientist himself, the kind of domain expert the product serves, and I think that is part of why it is designed well.

5. See a demo of Cursor recording its UI changes, and Devin’s documentation on testing and recordings.

6. Dev Patnaik and Robert Becker, “Needfinding: The Why and How of Uncovering People’s Needs”, Design Management Journal.

7. Daniel M. Russell, Mark J. Stefik, Peter Pirolli, and Stuart K. Card, “The Cost Structure of Sensemaking”.

Why Skills for Evals

Coding agents now instrument applications, run experiments, analyze data, and build interfaces. I’ve been pointing them at evals.

OpenAI’s Harness Engineering article makes the case well: they built a product entirely with Codex agents (~1 million lines of code, 1,500 PRs, three engineers, five months) and found that improving the infrastructure around the agent yielded better returns than improving the model. Their agents queried distributed traces to verify their own work against runtime evidence. Documentation tells the agent what to do. Telemetry tells it whether it worked. Evals apply the same principle to AI output quality.

All major eval vendors now ship an MCP server ^[2]. The tedious parts: instrumenting your app, orchestrating experiments, building annotation tools, now belong to coding agents.

But an agent with an eval platform still needs to know what to do with it. Say a support bot tells a customer “your plan includes free returns” when it doesn’t. Another says “I’ve canceled your order” when nobody asked. Both are hallucinations, but one gets a fact wrong and the other makes up a user action. If you lump them together in a generic “hallucination score” real problems will likely go undetected.

These skills fill in the gaps. They complement the vendor MCP servers: those give your agent access to traces and experiments, these teach it what to do with them.

The Skills

If you’re new to evals or inheriting an existing eval pipeline, start with eval-audit. It inspects your current setup (or lack of one), runs diagnostic checks across six areas (error analysis, evaluator design, judge validation, human review, labeled data, pipeline hygiene), and produces a prioritized list of problems with next steps. Install the skills and give your agent this prompt:

Install the eval skills plugin from https://github.com/hamelsmu/evals-skills, then run /evals-skills:eval-audit on my eval pipeline. Investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report. Use other skills in the plugin as recommended by the audit.

If you’re experienced with evals, you can skip the audit and pick the skill you need:

• error-analysis: Read traces, categorize failures, build a vocabulary of what’s broken

• generate-synthetic-data: Create diverse test inputs when real data is sparse

• write-judge-prompt: Design binary Pass/Fail LLM-as-Judge evaluators

• validate-evaluator: Calibrate judges against human labels using TPR/TNR and bias correction

• evaluate-rag: Evaluate retrieval and generation quality separately

• build-review-interface: Generate annotation interfaces for human trace review

These skills are a starting point. They only cover parts of evals that generalize across projects. Skills grounded in your stack, your domain, and your data will outperform them. Start here, then write your own. Our AI Evals course teaches the end to end workflow.

👉 The repo is here: github.com/hamelsmu/evals-skills 👈

P.S. Our next ​AI Evals course​ cohort starts March 16th. The one after that won’t run until fall. You’ll get lifetime access to all materials and learn with students from OpenAI, Meta, Google, Walmart, Airbnb and more. Use code ​newsletter-25​ for a discount.

Footnotes

1. Not foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether your pipeline works on your task with your data. If you aren’t familiar with product-specific AI evals, check out my AI Evals FAQ.

2. Braintrust, LangSmith, Phoenix, Truesight, and others.

That’s why I’m writing: our next live cohort of AI Evals For Engineers & PMs starts this Monday, January 26th. You can either join us for a live, interactive course, or continue learning on your own with our free materials. Here’s a guide for both paths.

Path 1: Join Our Live Cohort

After completing this course, you’ll walk away with the skills to find, diagnose, and prioritize AI errors, create data flywheels, and automate your evals with approaches you can trust (the syllabus is below). The course runs Jan 26 - Feb 21, 2026 (4 weeks).

Learn more about the course

​What our students are saying:

See more testimonials here.

What you’ll learn:

• Collect data for evals: Understand instrumentation and observability for tracking system behavior. Learn approaches for generating synthetic data to maximize error discovery. Choose the right tools and vendors for your use case.

• Get clarity with error analysis: Apply data analysis techniques to rapidly find systematic issues in your product. Master the processes and tools to annotate and analyze data quickly. Learn how to analyze agentic systems (tool calls, RAG, etc.) to identify patterns and errors.

• Implement effective evaluations: Create evals customized to your product that provide immediate value, NOT generic off-the-shelf evals. Align evals with stakeholders and domain experts so you can scientifically trust the results. Create high-quality LLM-as-a-judge and code-based evals with a systematic, iterative process.

• Master architecture-specific strategies: Measure and debug RAG systems for retrieval relevance and factual accuracy. Tame multi-step pipelines to identify error propagation and root causes. Apply techniques to multi-modal settings including text, image, and audio.

• Run evals in production: Set up automated evaluation gates in CI/CD pipelines. Understand methods for consistent comparison across experiments. Implement safety and quality control guardrails.

• Ensure evals lead to high ROI: Develop strong intuition for when to write an eval and when NOT to. Design interfaces to remove friction from reviewing data. Avoid common pitfalls surrounding team organization, collaboration, tools, and metrics.

What’s included:

• 10+ hours of office hours.

• Lifetime access to a Discord community with 3,000+ students, so you can get unstuck anytime.

• A 150+ page course reader.

• 6 months of unlimited usage of our AI Eval Assistant that has access to everything we have ever created on the topic of evals.

More importantly, we are offering unlimited free retakes. This means you can enroll now to lock in your spot at the current price, and join any future cohort (including the live office hours) if your schedule gets tight. You also get lifetime access to the private Discord community, so the sooner you enroll, the more value you get.

Learn more about the course

​Path 2: Continue Learning with Free Resources

If a live course isn’t the right fit for you right now, no problem. My goal is to help everyone build better AI, so I’ve compiled some of our best free resources to help you on your journey.

Essential Reading

• ​LLM Evals FAQ — The most common questions on implementing evals (and our answers)

• ​Your AI Product Needs Evals​

• ​LLM-as-a-Judge: A Complete Guide​

• ​A Field Guide To Improving AI Products​

• Shreya’s research: Who Validates the Validators?

• Building Effective AI-Powered Data Pipelines (Shreya Shankar): annotated notes​

Lightning Lessons: AI Evals

Free lessons on key topics:​

• Error Analysis: The AI Engineer’s Best ROI: slides ◦ video​

• The Evals That Made GitHub Copilot Happen: slides◦ video​

• Stop Managing AI Projects Like Traditional Software: slides ◦ video​

• WTH is an AI PM?: video​

Lightning Lessons: RAG Series

We recently ran a 7-part series on modern retrieval with leading researchers as part of our course. I wrote annotated notes covering all seven talks:

1. I Don’t Use RAG, I Just Retrieve Documents (Ben Clavié)

2. Modern IR Evaluation In The RAG Era (Nandan Thakur)

3. Reasoning Opens Up New Retrieval Frontiers (Orion Weller)

4. Going Further: Late Interaction Beats Single Vector Limits (Antoine Chaffin)

5. The Map Is Not The Territory – Highly Multimodal Search

6. Context Rot: When Long Context Fails (Kelly Hong)

7. You Don’t Need a Graph DB (Jo Kristian Bergum)

   ​

Whichever path you choose, I hope these resources help you build something great.

Best,

Hamel

1. AI Evals FAQ

These cover questions from our course where we’ve taught over 3,000 students. Some of the questions answered:

Q: What are LLM Evals?
Q: What’s a minimum viable evaluation setup?
Q: Why is “error analysis” so important in LLM evals, and how is it performed?
Q: What is the best approach for generating synthetic data?
Q: Are there scenarios where synthetic data may not be reliable?
Q: Why do you recommend binary (pass/fail) evaluations?
Q: Should I use “ready-to-use” evaluation metrics?
Q: How many people should annotate my LLM outputs?
Q: Should PMs and engineers collaborate on error analysis? How?
Q: What parts of evals can be automated with LLMs?
Q: Should I stop writing prompts manually in favor of automated tools?
Q: What makes a good custom interface for reviewing LLM outputs?
Q: What gaps in eval tooling should I be prepared to fill myself?
Q: How should I version and manage prompts?
Q: How are evaluations used differently in CI/CD vs. monitoring production?
Q: What’s the difference between guardrails & evaluators?
Q: Is RAG dead?
Q: How should I approach evaluating my RAG system?
Q: How do I evaluate sessions with human handoffs?
Q: How do I evaluate complex multi-step workflows?
Q: How do I evaluate agentic workflows?

You can download the FAQ here.

2. AI Evals Flashcards

These are flashcards we distribute in our course with bite sized guidance on important topics.

Download the flashcards here

P.S. if you want to master AI Evals, you may be interested in our course: AI Evals for Engineers and PMs, which has taught over 3,000 students from over 500 companies like Amazon, WalMart, Meta, OpenAI, Google and more. The course is 25% off through tomorrow. You can learn more here.

Over the past few weeks, we've dived into some of the toughest questions in AI Evals.

We've extended enrollment through this Tuesday, October 7th. Every lesson is recorded and you get lifetime access to all materials.

This brings you to a fork in the road. You can either join us for this cohort, or you can continue learning on your own with our free materials. Here’s a guide for both paths.

​
​Path 1: Join Our Next Cohort

If you're ready to build AI applications with a proven, data-driven framework, this is your last chance to do it with us live. You'll walk away with the skills to find, diagnose, and prioritize AI errors, create data flywheels, and automate your evals with approaches you can trust.

➡️ Enroll Now & Get 35% Off ​

Enrollment closes this Tuesday, October 7th.

What our students are saying:

See more on our public testimonial page.

Course Syllabus (2 lessons per week):

• Week 1: Fundamentals & Lifecycle LLM Application Evaluation, Systematic Error Analysis

• Week 2: Implementing Effective Evaluations, Collaborative Evaluation Practices

• Week 3: Architecture-Specific Evaluation Strategies (RAG, Agents), Production Monitoring

• Week 4: Efficient Human Review Systems, Cost Optimization

Plus:

• 12 hours of office hours

• Lifetime access to a discord channel where you can learn and connect with 1,000+ other students working on evals.

• Over 160 pages of detailed notes made available exclusively to students to compliment your learning.

• 10 months of unlimited access to our new AI Eval Assistant. This is an alpha version of a private AI powered by all our course materials (lectures, the course reader, office hours, etc.) that you can have unlimited chats with 24/7.

➡️ Enroll Now & Get 35% Off (Enrollment closes this Tuesday, October 7th.)

Path 2: Continue Learning with Free Resources

If a live course isn't the right fit for you right now, no problem. My goal is to help everyone build better AI, so I've compiled some of our best free resources to help you on your journey:

• ​LLM Evals FAQs: The most common questions on implementing evals (and our answers) from our course.

• Essential Reading:

  • Shreya's research: Who Validates the Validators?​

  • Hamel's blog posts: "Your AI Product Needs Evals", "Creating a LLM Judge That Drives Business Results", and "A Field Guide To Improving AI Products".

• ​Free Series on RAG: Our special series on evaluating and optimizing RAG.

• Lightning Lessons: Quick, digestible lessons on key topics related to evals:

  • Build your own eval tools with notebooks: video​

  • Inspect, an OSS Python library for LLM evals: slides | video​

  • The Evals That Made GitHub Copilot Happen: slides | recording​

  • Stop Managing AI Projects Like Traditional Software: slides | recording

  • How To Setup Evals For Agents: github | video​

  • Optimize Your Dev Setup For Evals w/ Cursor Rules & MCP: slides | recording​

  • Error Analysis: The AI Engineer’s Best ROI: slides | recording​

  • Hybrid Search Is Just The Beginning: Optimizing the R in RAG: slides | recording​

  • WTH is an AI PM?: recording​

  • Optimizing Structured Data Retrieval With Evals: slides | recording​

  • LLM Evals: Common Mistakes: video​

Whichever path you choose, thank you for being part of this community.

Hope to see some of you in class,

Hamel

AI Evaluation is the engineering discipline that solves this. It provides a systematic process to move from guesswork to a structured method for building high-quality AI products. It’s the skill that separates teams that ship reliable AI from those that don’t.

With enrollment for our final course of the year closing this Monday, October 6th, I know some are still deciding if now is the time to learn this skill. I’ve been getting a few common questions, and I want to address them head-on.

If you’re hesitating, you might be thinking one of these things:

1. “I don’t have time in my busy schedule.”

I understand completely. The course is designed for busy professionals, which is why the lessons are professionally recorded and edited for you to watch on your own time.

More importantly, for this October cohort ONLY, we are offering unlimited free retakes. This means you can enroll now to lock in your spot and the current price, and join any future cohort if your schedule gets tight. You also get lifetime access to the private Discord community, so the sooner you enroll, the more value you get.

2. “I’m worried I don’t have the AI skills to keep up.”

We’ve got you covered. As a bonus for all students, Isaac Flath is teaching a 5+ hour mini-course on how to tackle the homework assignments using AI assistance.

Drawn from his popular AI coding course, this series is designed for both developers and PMs. You will learn how to use AI agents as a partner to apply the course concepts, explore new topics, and complete the homework effectively. It’s a course on AI Evals, and we want you to use AI to learn it.

3. “It’s too expensive.”

You’re right, it’s a significant investment. We priced it this way so we can provide an unparalleled level of support. I can promise you that we answer every single question. We have 3+ TAs dedicated to making sure you’re never stuck.

A final heads-up: our seats for this cohort are nearly full.

If you want to join the final cohort of the year and get all the bonuses (including unlimited retakes), now is the time.

➡️ Enroll in the AI Evals Course

Hope to see you there.

Hamel

P.S. If this is the first time you’ve heard about AI Evals, you may want to check out this end-to-end primer I recently did with

I’m excited to announce a major update to that post.

The Evals FAQ has been completely reorganized and expanded with tons of new content, making it an even more comprehensive resource for anyone building with LLMs.

I've grouped the questions into clear sections, so you can quickly find answers whether you're just getting started, designing complex evaluations, or trying to debug a tricky RAG pipeline.

Below is the full table of contents to give you a sense of the new structure and depth.

Table of Contents

• ​Getting Started & Fundamentals​

  • ​Q: What are LLM Evals?​

  • ​Q: What is a trace?​

  • ​Q: What’s a minimum viable evaluation setup?​

  • ​Q: How much of my development budget should I allocate to evals?​

  • ​Q: Will today’s evaluation methods still be relevant in 5-10 years given how fast AI is changing?​

• ​Error Analysis & Data Collection​

  • ​Q: Why is "error analysis" so important in LLM evals, and how is it performed?​

  • ​Q: How do I surface problematic traces for review beyond user feedback?​

  • ​Q: How often should I re-run error analysis on my production system?​

  • ​Q: What is the best approach for generating synthetic data?​

  • ​Q: Are there scenarios where synthetic data may not be reliable?​

  • ​Q: How do I approach evaluation when my system handles diverse user queries?​

  • ​Q: How can I efficiently sample production traces for review?​

• ​Evaluation Design & Methodology​

  • ​Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?​

  • ​Q: Should I practice eval-driven development?​

  • ​Q: Should I build automated evaluators for every failure mode I find?​

  • ​Q: Should I use "ready-to-use" evaluation metrics?​

  • ​Q: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?​

  • ​Q: Can I use the same model for both the main task and evaluation?​

• ​Human Annotation & Process​

  • ​Q: How many people should annotate my LLM outputs?​

  • ​Q: Should product managers and engineers collaborate on error analysis? How?​

  • ​Q: Should I outsource annotation & labeling to a third party?​

  • ​Q: What parts of evals can be automated with LLMs?​

  • ​Q: Should I stop writing prompts manually in favor of automated tools?​

• ​Tools & Infrastructure​

  • ​Q: Should I build a custom annotation tool or use something off-the-shelf?​

  • ​Q: What makes a good custom interface for reviewing LLM outputs?​

  • ​Q: What gaps in eval tooling should I be prepared to fill myself?​

  • ​Q: Seriously Hamel. Stop the bullshit. What’s your favorite eval vendor?​

• ​Production & Deployment​

  • ​Q: How are evaluations used differently in CI/CD vs. monitoring production?​

  • ​Q: What’s the difference between guardrails & evaluators?​

  • ​Q: Can my evaluators also be used to automatically fix or correct outputs in production?​

  • ​Q: How much time should I spend on model selection?​

• ​Domain-Specific Applications​

  • ​Q: Is RAG dead?​

  • ​Q: How should I approach evaluating my RAG system?​

  • ​Q: How do I choose the right chunk size for my document processing tasks?​

  • ​Q: How do I debug multi-turn conversation traces?​

  • ​Q: How do I evaluate sessions with human handoffs?​

  • ​Q: How do I evaluate complex multi-step workflows?​

  • ​Q: How do I evaluate agentic workflows?​

Other Formats

You can read the full post on the blog, or download it for offline reading.

• ​Read the full post online​

• ​Download as a PDF​

• Listen to an audio version.

I hope this becomes an invaluable resource for you. If you think someone else could benefit from this information, please send them this email!

All the best,

Hamel

P.S. I want to let you know about this code to get 35% off our next cohort of our evals course. You'll get lifetime access to the material and recordings, and learn with students from OpenAI, Meta, Google, WalMart, Airbnb and more.

Q: How do I evaluate agentic workflows?

We recommend evaluating agentic workflows in two phases:

1. End-to-end task success. Treat the agent as a black box and ask “did we meet the user’s goal?”. Define a precise success rule per task (exact answer, correct side-effect, etc.) and measure with human or aligned LLM judges. Take note of the first upstream failure when conducting error analysis.

Once error analysis reveals which workflows fail most often, move to step-level diagnostics to understand why they’re failing.

2. Step-level diagnostics. Assuming that you have sufficiently instrumented your system with details of tool calls and responses, you can score individual components such as: - Tool choice: was the selected tool appropriate? - Parameter extraction: were inputs complete and well-formed? - Error handling: did the agent recover from empty results or API failures? - Context retention: did it preserve earlier constraints? - Efficiency: how many steps, seconds, and tokens were spent? - Goal checkpoints: for long workflows verify key milestones.

Example: “Find Berkeley homes under $1M and schedule viewings” breaks into: parameters extracted correctly, relevant listings retrieved, availability checked, and calendar invites sent. Each checkpoint can pass or fail independently, making debugging tractable.

Use transition failure matrices to understand error patterns. Create a matrix where rows represent the last successful state and columns represent where the first failure occurred. This is a great way to understand where the most failures occur.

Transition matrices transform overwhelming agent complexity into actionable insights. Instead of drowning in individual trace reviews, you can immediately see that GenSQL → ExecSQL transitions cause 12 failures while DecideTool → PlanCal causes only 2. This data-driven approach guides where to invest debugging effort. Here is another example from Bryan Bischof, that is also a text-to-SQL agent:

In this example, Bryan shows variation in transition matrices across experiments. How you organize your transition matrix depends on the specifics of your application. For example, Bryan’s text-to-SQL agent has an inherent sequential workflow which he exploits for further analytical insight. You can watch his full talk for more details.

Creating Test Cases for Agent Failures

Creating test cases for agent failures follows the same principles as our previous FAQ on debugging multi-turn conversation traces (i.e. try to reproduce the error in the simplest way possible, only use multi-turn tests when the failure actually requires conversation context, etc.).

P.S. I want to let you know about this code to get 35% off our next cohort of our evals course. You'll get lifetime access to the material and recordings, and learn with students from OpenAI, Meta, Google, WalMart, Airbnb and more.

Let's jump in:

Q: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?

Generic metrics like BERTScore, ROUGE, cosine similarity, etc. are not useful for evaluating LLM outputs in most AI applications. Instead, we recommend using error analysis to identify metrics specific to your application’s behavior. We recommend designing binary pass/fail evals (using LLM-as-judge) or code-based assertions.

As an example, consider a real estate CRM assistant. Suggesting showings that aren’t available (can be tested with an assertion) or confusing client personas (can be tested with a LLM-as-judge) is problematic . Generic metrics like similarity or verbosity won’t catch this. A relevant quote from the course:

“The abuse of generic metrics is endemic. Many eval vendors promote off the shelf metrics, which ensnare engineers into superfluous tasks.”

Similarity metrics aren’t always useless. They have utility in domains like search and recommendation (and therefore can be useful for optimizing and debugging retrieval for RAG). For example, cosine similarity between embeddings can measure semantic closeness in retrieval systems, and average pairwise similarity can assess output diversity (where lower similarity indicates higher diversity).

Have questions about evals that you haven't seen answered yet in this FAQ? Hit reply and ask! I'll include the answer in a future email!

Thanks,

Hamel Husain

Nowadays, it's nearly impossible to discuss LLMs without the topic of prompt optimization/automation coming up. Let's get right into it.

Q: Should I stop writing prompts manually in favor of automated tools?

Automating prompt engineering can be tempting, but you should be skeptical of tools that promise to optimize prompts for you, especially in early stages of development. When you write a prompt, you are forced to clarify your assumptions and externalize your requirements. Good writing is good thinking [1]. If you delegate this task to an automated tool too early, you risk never fully understanding your own requirements or the model’s failure modes.

This is because automated prompt optimization typically hill-climb a predefined evaluation metric. It can refine a prompt to perform better on known failures, but it cannot discover new ones. Discovering new errors requires error analysis. Furthermore, research shows that evaluation criteria tends to shift after reviewing a model’s outputs, a phenomenon known as “criteria drift” [2]. This means that evaluation is an iterative, human-driven sensemaking process, not a static target that can be set once and handed off to an optimizer.

A pragmatic approach is to use LLMs to improve your prompt based on open coding (open-ended notes about traces). This way, you maintain a human in the loop who is looking at the data and externalizing their requirements. Once you have a high-quality set of evals, prompt optimization can be effective for that last mile of performance.

1. Paul Graham, “Writes and Write-Nots”​

2. Shreya Shankar, et al., “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences”

P.S. I want to let you know about this code to get 35% off our next cohort of our evals course. You'll get lifetime access to the material and recordings, and learn with students from OpenAI, Meta, Google, WalMart, Airbnb and more.

The world of Retrieval-Augmented Generation (RAG) has moved far beyond the simple "embed and search" paradigm of 2023. The pace of innovation is staggering, but it has also created a confusing landscape of new terms and techniques: agentic RAG, late-interaction models, multi-modal retrieval, and more.

How do you know which of these advanced methods are just hype and which are essential for building production-ready systems?

To help answer that, we’ve compiled a new, free resource for the community.

I am excited to share "​Beyond Naive RAG: Practical Advanced Methods​": an open mini-book that condenses over 5 hours of expert instruction from leading researchers into an easy-to-read, annotated format.

Each chapter is based on a presentation from our AI Evals ​course​, featuring world-class researchers who are pushing the boundaries of what’s possible with RAG systems.

We explore five key areas where the field is advancing:

• The "RAG is Dead" Controversy: Ben Clavié sets the record straight on why retrieval is more important than ever.

• Modern IR Evaluation: Nandan Thakur explains why traditional search metrics are failing and introduces new benchmarks like FreshStack that measure what really matters for RAG: diversity, grounding, and coverage.

• Reasoning-Enhanced Retrieval: Orion Weller from Johns Hopkins University shows how to embed instruction-following and reasoning directly into the retrieval process with models like Promptriever and Rank1.

• Late Interaction Models: Antoine Chaffin dives into the limitations of single-vector search and explains how models like ColBERT overcome the information loss problem for superior performance.

• Multiple Representations: Bryan Bischof and Ayush Chaurasia provide a powerful framework for building flexible systems that use intelligent routing across multiple, diverse "maps" of your data to better serve user intent.

This mini-book is designed for practitioners. It combines theory with practical recipes and timestamped references to the original videos, allowing you to explore the topics that matter most to you.

Below is the table of contents of the book.

Table of Contents

• 1: I Don’t Use RAG, I Just Retrieve Documents

  • 1.1 The Title

  • 1.2 The “RAG is Dead” Controversy

  • 1.3 What is RAG Really?

  • 1.4 The Standard RAG Flow

  • 1.5 The Problem with “2023 RAG”

  • 1.6 Single-Vector Search Limitations

  • 1.7 Why Long Context Doesn’t Replace RAG

  • 1.8 Retrieval is Essential

  • 1.9 Key Takeaways

  • 1.10 Better RAG is the Solution

  • 1.11 The Retrieval Landscape

• 2: Modern IR Evaluation for RAG

  • 2.1 Introduction and Speaker Background

  • 2.2 The History of Information Retrieval

  • 2.3 The Cranfield Paradigm

  • 2.4 The BEIR Benchmark

  • 2.5 Problems with Current Benchmarks

  • 2.6 The RAG Era Changes Everything

  • 2.7 Different Users, Different Goals

  • 2.8 The Evaluation Mismatch

  • 2.9 Introducing FreshStack

  • 2.10 FreshStack Data Sources

  • 2.11 The FreshStack Pipeline

  • 2.12 FreshStack Evaluation Metrics

• 3: Optimizing Retrieval with Reasoning Models

  • 3.1 LLM Capabilities: Instruction Following and Reasoning

  • 3.2 The Search Paradigm Hasn’t Changed

  • 3.3 Evolution of Search Paradigms

  • 3.4 Understanding Instructions in IR

  • 3.5 Introducing Promptriever and Rank1

  • 3.6 Promptriever: Instruction-Trained Retrieval

  • 3.7 Promptriever Evaluation Results

  • 3.8 Rank1: Reasoning-Based Reranking

  • 3.9 Rank1 Performance Results

  • 3.10 Finding Novel Relevant Documents

• 4: Late Interaction Models For RAG

  • 4.1 Dense Vector Search Architecture

  • 4.2 Why Dense Models Became Popular

  • 4.3 The Benchmark Problem

  • 4.4 Hidden Limitations

  • 4.5 Long Context Performance

  • 4.6 Complex Retrieval Tasks

  • 4.7 BM25’s Surprising Strength

  • 4.8 The Pooling Problem

  • 4.9 Late Interaction Solution

  • 4.10 Performance Advantages

  • 4.11 Interpretability Benefits

  • 4.12 Barriers to Adoption

  • 4.13 PyLate: Making Late Interaction Accessible

  • 4.14 Future Research Directions

• 5: RAG with Multiple Representations

  • 5.1 The Map is Not the Territory

  • 5.2 Deconstructing RAG Buzzwords

  • 5.3 A First-Principles View of RAG

  • 5.4 The Three Responsibilities of an IR Engineer

  • 5.5 Practical Application: Curving Space

  • 5.6 Agents as Routers

  • 5.7 Dynamic Representations

  • 5.8 Demo: Semantic Dot Art

  • 5.9 System Architecture

  • 5.10 Integration with Other Techniques

• Conclusion

  • Key Takeaways

  • Looking Forward

  • Resources

You can get your free copy here:

Download: Beyond Naive RAG (PDF)

We hope this helps you navigate the evolving landscape of RAG and build better AI systems. Know someone who might benefit from learning about advanced RAG methods? Send them this email!

Thanks,

Hamel

P.S. I want to let you know about this code to get ​35% off our next cohort of our evals course​. You'll get lifetime access to the material and recordings, and learn with students from OpenAI, Meta, Google, WalMart, Airbnb and more.

I'm often asked for the best public example of an AI evals workflow for a real, production application. I finally have an answer.

If you've ever thought that evals are too "fancy," too complex, or something only deeply technical engineers can do, I want you to watch this talk.

Teresa Torres, a legendary product discovery coach and author, was a student in our first AI Evals cohort. She's not a daily coder, but she took the core principles from the course and applied them to build an AI interview coach from scratch.

What she created is, in my opinion, the single best public demonstration of how to do evals right.

​Watch the full presentation: From Noob to 5 Automated Evals in 4 Weeks (as a PM)​

In this incredibly hands-on lesson, Teresa shows exactly how she:

• Started with error analysis FIRST to find real, user-impacting issues instead of getting lost in generic metrics.

• Used Jupyter notebooks to systematically investigate failures and analyze results (it's a fantastic commercial for notebooks).

• Built her own custom annotation tools and widgets directly inside her notebooks to speed up her workflow.

• Wrote both LLM-as-a-judge and simple code-based assertions to test for the specific errors she found.

• Iterated relentlessly through this feedback loop, measurably squashing bugs and improving her product with each cycle.

• Kept things simple the whole time, proving you don't need a massive, over-engineered stack to get started.

This isn't a theoretical talk. It's a real-world case study of someone building a production app, learning Python and new tools along the way, and using a practical evals process to drive massive improvements. It’s an empowering story that shows this is within reach for anyone willing to dive in.

Hope you enjoy it,

Hamel

P.S. Teresa's journey started in our ​AI Evals course​. If her story inspires you to build your own systematic feedback loops, I wanted to let you know about our upcoming cohort.

You'll get lifetime access to all materials and learn the same frameworks that helped Teresa build her application. the early bird discount for our next cohort ends this Friday. If you're interested, you can learn more and register here:

➡️ ​AI Evals Course (> $1,000 Discount)​

Hope to see some of you there!

👉 If you want to learn more about AI Evals, check out our AI Evals course. Here is a 35% discount code for readers. 👈

Listen to the audio version of this FAQ

If you prefer to listen to the audio version (narrated by AI), you can play it here.

Hamel Husain · LLM Evals FAQ

Getting Started & Fundamentals

Q: What are LLM Evals?

If you are completely new to product-specific LLM evals (not foundation model benchmarks), see these posts: part 1, part 2 and part 3. Otherwise, keep reading.

Your AI Product Needs Eval (Evaluation Systems)

Contents:

1. Motivation

2. Iterating Quickly == Success
   

3. Case Study: Lucy, A Real Estate AI Assistant

4. The Types Of Evaluation

   1. Level 1: Unit Tests

   2. Level 2: Human & Model Eval

   3. Level 3: A/B Testing

   4. Evaluating RAG

5. Eval Systems Unlock Superpowers For Free

   1. Fine-Tuning

   2. Data Synthesis & Curation

   3. Debugging

Creating a LLM-as-a-Judge That Drives Business Results

Contents:

1. The Problem: AI Teams Are Drowning in Data

2. Step 1: Find The Principal Domain Expert

3. Step 2: Create a Dataset

4. Step 3: Direct The Domain Expert to Make Pass/Fail Judgments with Critiques

5. Step 4: Fix Errors

6. Step 5: Build Your LLM as A Judge, Iteratively

7. Step 6: Perform Error Analysis

8. Step 7: Create More Specialized LLM Judges (if needed)

9. Recap of Critique Shadowing

10. Resources

A Field Guide to Rapidly Improving AI Products

Contents:

1. How error analysis consistently reveals the highest-ROI improvements

2. Why a simple data viewer is your most important AI investment

3. How to empower domain experts (not just engineers) to improve your AI

4. Why synthetic data is more effective than you think

5. How to maintain trust in your evaluation system

6. Why your AI roadmap should count experiments, not features

↗ Focus view

Q: What is a trace?

A trace is the complete record of all actions, messages, tool calls, and data retrievals from a single initial user query through to the final response. It includes every step across all agents, tools, and system components in a session: multiple user messages, assistant responses, retrieved documents, and intermediate tool interactions.

Note on terminology: Different observability vendors use varying definitions of traces and spans. Alex Strick van Linschoten’s analysis highlights these differences (screenshot below):

↗ Focus view

Q: What’s a minimum viable evaluation setup?

Start with error analysis, not infrastructure. Spend 30 minutes manually reviewing 20-50 LLM outputs whenever you make significant changes. Use one domain expert who understands your users as your quality decision maker (a “benevolent dictator”).

If possible, use notebooks to help you review traces and analyze data. In our opinion, this is the single most effective tool for evals because you can write arbitrary code, visualize data, and iterate quickly. You can even build your own custom annotation interface right inside notebooks, as shown in this video.

↗ Focus view

Q: How much of my development budget should I allocate to evals?

It’s important to recognize that evaluation is part of the development process rather than a distinct line item, similar to how debugging is part of software development.

You should always be doing error analysis. When you discover issues through error analysis, many will be straightforward bugs you’ll fix immediately. These fixes don’t require separate evaluation infrastructure as they’re just part of development.

The decision to build automated evaluators comes down to cost-benefit analysis. If you can catch an error with a simple assertion or regex check, the cost is minimal and probably worth it. But if you need to align an LLM-as-judge evaluator, consider whether the failure mode warrants that investment.

In the projects we’ve worked on, we’ve spent 60-80% of our development time on error analysis and evaluation. Expect most of your effort to go toward understanding failures (i.e. looking at data) rather than building automated checks.

Be wary of optimizing for high eval pass rates. If you’re passing 100% of your evals, you’re likely not challenging your system enough. A 70% pass rate might indicate a more meaningful evaluation that’s actually stress-testing your application. Focus on evals that help you catch real issues, not ones that make your metrics look good.

↗ Focus view

Q: Will today’s evaluation methods still be relevant in 5-10 years given how fast AI is changing?

Yes. Even with perfect models, you still need to verify they’re solving the right problem. The need for systematic error analysis, domain-specific testing, and monitoring will still be important.

Today’s prompt engineering tricks might become obsolete, but you’ll still need to understand failure modes. Additionally, a LLM cannot read your mind, and research shows that people need to observe the LLM’s behavior in order to properly externalize their requirements.

For deeper perspective on this debate, see these two viewpoints: “The model is the product” versus “The model is NOT the product”.

“The model is the product”:

“The model is NOT the product”:

↗ Focus view

Error Analysis & Data Collection

Q: Why is "error analysis" so important in LLM evals, and how is it performed?

Error analysis is the most important activity in evals. Error analysis helps you decide what evals to write in the first place. It allows you to identify failure modes unique to your application and data. The process involves:

1. Creating a Dataset

Gathering representative traces of user interactions with the LLM. If you do not have any data, you can generate synthetic data to get started.

2. Open Coding

Human annotator(s) (ideally a benevolent dictator) review and write open-ended notes about traces, noting any issues. This process is akin to “journaling” and is adapted from qualitative research methodologies. When beginning, it is recommended to focus on noting the first failure observed in a trace, as upstream errors can cause downstream issues, though you can also tag all independent failures if feasible. A domain expert should be performing this step.

3. Axial Coding

Categorize the open-ended notes into a “failure taxonomy.”. In other words, group similar failures into distinct categories. This is the most important step. At the end, count the number of failures in each category. You can use a LLM to help with this step.

4. Iterative Refinement

Keep iterating on more traces until you reach theoretical saturation, meaning new traces do not seem to reveal new failure modes or information to you. As a rule of thumb, you should aim to review at least 100 traces.

You should frequently revisit this process. There are advanced ways to sample data more efficiently, like clustering, sorting by user feedback, and sorting by high probability failure patterns. Over time, you’ll develop a “nose” for where to look for failures in your data.

Do not skip error analysis. It ensures that the evaluation metrics you develop are supported by real application behaviors instead of counter-productive generic metrics (which most platforms nudge you to use). For examples of how error analysis can be helpful, see this video, or this blog post.

Here is a visualization of the error analysis process by one of our students, Pawel Huryn - including how it fits into the overall evaluation process:

↗ Focus view

Q: How do I surface problematic traces for review beyond user feedback?

While user feedback is a good way to narrow in on problematic traces, other methods are also useful. Here are three complementary approaches:

Start with random sampling

The simplest approach is reviewing a random sample of traces. If you find few issues, escalate to stress testing: create queries that deliberately test your prompt constraints to see if the AI follows your rules.

Use evals for initial screening

Use existing evals to find problematic traces and potential issues. Once you’ve identified these, you can proceed with the typical evaluation process starting with error analysis.

Leverage efficient sampling strategies

For more sophisticated trace discovery, use outlier detection, metric-based sorting, and stratified sampling to find interesting traces. Generic metrics can serve as exploration signals to identify traces worth reviewing, even if they don’t directly measure quality.

↗ Focus view

Q: How often should I re-run error analysis on my production system?

Re-run error analysis when making significant changes: new features, prompt updates, model switches, or major bug fixes. A useful heuristic is to set a goal for reviewing at least 100+ fresh traces each review cycle. Typical review cycles we’ve seen range from 2-4 weeks. See this FAQ on how to sample traces effectively.

Between major analyses, review 10-20 traces weekly, focusing on outliers: unusually long conversations, sessions with multiple retries, or traces flagged by automated monitoring. Adjust frequency based on system stability and usage growth. New systems need weekly analysis until failure patterns stabilize. Mature systems might need only monthly analysis unless usage patterns change. Always analyze after incidents, user complaint spikes, or metric drift. Scaling usage introduces new edge cases.

↗ Focus view

Q: What is the best approach for generating synthetic data?

A common mistake is prompting an LLM to "give me test queries" without structure, resulting in generic, repetitive outputs. A structured approach using dimensions produces far better synthetic data for testing LLM applications.

Start by defining dimensions: categories that describe different aspects of user queries. Each dimension captures one type of variation in user behavior. For example:

• For a recipe app, dimensions might include Dietary Restriction (vegan, gluten-free, none), Cuisine Type (Italian, Asian, comfort food), and Query Complexity (simple request, multi-step, edge case).

• For a customer support bot, dimensions could be Issue Type (billing, technical, general), Customer Mood (frustrated, neutral, happy), and Prior Context (new issue, follow-up, resolved).

Start with failure hypotheses. If you lack intuition about failure modes, use your application extensively or recruit friends to use it. Then choose dimensions targeting those likely failures.

Create tuples manually first: Write 20 tuples by hand—specific combinations selecting one value from each dimension. Example: (Vegan, Italian, Multi-step). This manual work helps you understand your problem space.

Scale with two-step generation:

1. Generate structured tuples: Have the LLM create more combinations like (Gluten-free, Asian, Simple)

2. Convert tuples to queries: In a separate prompt, transform each tuple into natural language

This separation avoids repetitive phrasing. The (Vegan, Italian, Multi-step) tuple becomes: "I need a dairy-free lasagna recipe that I can prep the day before."

Generation approaches

You can generate tuples two ways:

Cross product then filter: Generate all dimension combinations, then filter with an LLM. Guarantees coverage including edge cases. Use when most combinations are valid.

Direct LLM generation: Ask the LLM to generate tuples directly. More realistic but tends toward generic outputs and misses rare scenarios. Use when many dimension combinations are invalid.

Fix obvious problems first: Don’t generate synthetic data for issues you can fix immediately. If your prompt doesn’t mention dietary restrictions, fix the prompt rather than generating specialized test queries.

After iterating on your tuples and prompts, run these synthetic queries through your actual system to capture full traces. Sample 100 traces for error analysis. This number provides enough traces to manually review and identify failure patterns without being overwhelming.

↗ Focus view

Q: Are there scenarios where synthetic data may not be reliable?

Yes: synthetic data can mislead or mask issues. For guidance on generating synthetic data when appropriate, see What is the best approach for generating synthetic data?

Common scenarios where synthetic data fails:

1. Complex domain-specific content: LLMs often miss the structure, nuance, or quirks of specialized documents (e.g., legal filings, medical records, technical forms). Without real examples, critical edge cases are missed.

2. Low-resource languages or dialects: For low-resource languages or dialects, LLM-generated samples are often unrealistic. Evaluations based on them won’t reflect actual performance.

3. When validation is impossible: If you can’t verify synthetic sample realism (due to domain complexity or lack of ground truth), real data is important for accurate evaluation.

4. High-stakes domains: In high-stakes domains (medicine, law, emergency response), synthetic data often lacks subtlety and edge cases. Errors here have serious consequences, and manual validation is difficult.

5. Underrepresented user groups: For underrepresented user groups, LLMs may misrepresent context, values, or challenges. Synthetic data can reinforce biases in the training data of the LLM.

↗ Focus view

Q: How do I approach evaluation when my system handles diverse user queries?

Complex applications often support vastly different query patterns—from “What’s the return policy?” to “Compare pricing trends across regions for products matching these criteria.” Each query type exercises different system capabilities, leading to confusion on how to design eval criteria.

Error Analysis is all you need. Your evaluation strategy should emerge from observed failure patterns (e.g. error analysis), not predetermined query classifications. Rather than creating a massive evaluation matrix covering every query type you can imagine, let your system’s actual behavior guide where you invest evaluation effort.

During error analysis, you’ll likely discover that certain query categories share failure patterns. For instance, all queries requiring temporal reasoning might struggle regardless of whether they’re simple lookups or complex aggregations. Similarly, queries that need to combine information from multiple sources might fail in consistent ways. These patterns discovered through error analysis should drive your evaluation priorities. It could be that query category is a fine way to group failures, but you don’t know that until you’ve analyzed your data.

To see an example of basic error analysis in action, see this video.

↗ Focus view

Q: How can I efficiently sample production traces for review?

It can be cumbersome to review traces randomly, especially when most traces don’t have an error. These sampling strategies help you find traces more likely to reveal problems:

• Outlier detection: Sort by any metric (response length, latency, tool calls) and review extremes.

• User feedback signals: Prioritize traces with negative feedback, support tickets, or escalations.

• Metric-based sorting: Generic metrics can serve as exploration signals to find interesting traces. Review both high and low scores and treat them as exploration clues. Based on what you learn, you can build custom evaluators for the failure modes you find.

• Stratified sampling: Group traces by key dimensions (user type, feature, query category) and sample from each group.

• Embedding clustering: Generate embeddings of queries and cluster them to reveal natural groupings. Sample proportionally from each cluster, but oversample small clusters for edge cases. There’s no right answer for clustering—it’s an exploration technique to surface patterns you might miss manually.

As you get more sophisticated with how you sample, you can incorporate these tactics into the design of your annotation tools.

↗ Focus view

Evaluation Design & Methodology

Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?

Engineers often believe that Likert scales (1-5 ratings) provide more information than binary evaluations, allowing them to track gradual improvements. However, this added complexity often creates more problems than it solves in practice.

Binary evaluations force clearer thinking and more consistent labeling. Likert scales introduce significant challenges: the difference between adjacent points (like 3 vs 4) is subjective and inconsistent across annotators, detecting statistical differences requires larger sample sizes, and annotators often default to middle values to avoid making hard decisions.

Having binary options forces people to make a decision rather than hiding uncertainty in middle values. Binary decisions are also faster to make during error analysis - you don’t waste time debating whether something is a 3 or 4.

For tracking gradual improvements, consider measuring specific sub-components with their own binary checks rather than using a scale. For example, instead of rating factual accuracy 1-5, you could track “4 out of 5 expected facts included” as separate binary checks. This preserves the ability to measure progress while maintaining clear, objective criteria.

Start with binary labels to understand what ‘bad’ looks like. Numeric labels are advanced and usually not necessary.

↗ Focus view

Q: Should I practice eval-driven development?

Generally no. Eval-driven development (writing evaluators before implementing features) sounds appealing but creates more problems than it solves. Unlike traditional software where failure modes are predictable, LLMs have infinite surface area for potential failures. You can’t anticipate what will break.

A better approach is to start with error analysis. Write evaluators for errors you discover, not errors you imagine. This avoids getting blocked on what to evaluate and prevents wasted effort on metrics that have no impact on actual system quality.

Exception: Eval-driven development may work for specific constraints where you know exactly what success looks like. If adding “never mention competitors,” writing that evaluator early may be acceptable.

Most importantly, always do a cost-benefit analysis before implementing an eval. Ask whether the failure mode justifies the investment. Error analysis reveals which failures actually matter for your users.

↗ Focus view

Q: Should I build automated evaluators for every failure mode I find?

Focus automated evaluators on failures that persist after fixing your prompts. Many teams discover their LLM doesn’t meet preferences they never actually specified - like wanting short responses, specific formatting, or step-by-step reasoning. Fix these obvious gaps first before building complex evaluation infrastructure.

Consider the cost hierarchy of different evaluator types. Simple assertions and reference-based checks (comparing against known correct answers) are cheap to build and maintain. LLM-as-Judge evaluators require 100+ labeled examples, ongoing weekly maintenance, and coordination between developers, PMs, and domain experts. This cost difference should shape your evaluation strategy.

Only build expensive evaluators for problems you’ll iterate on repeatedly. Since LLM-as-Judge comes with significant overhead, save it for persistent generalization failures - not issues you can fix trivially. Start with cheap code-based checks where possible: regex patterns, structural validation, or execution tests. Reserve complex evaluation for subjective qualities that can’t be captured by simple rules.

↗ Focus view

Q: Should I use "ready-to-use" evaluation metrics?

No. Generic evaluations waste time and create false confidence. (Unless you’re using them for exploration).

One instructor noted:

“All you get from using these prefab evals is you don’t know what they actually do and in the best case they waste your time and in the worst case they create an illusion of confidence that is unjustified.”¹

Generic evaluation metrics are everywhere. Eval libraries contain scores like helpfulness, coherence, quality, etc. promising easy evaluation. These metrics measure abstract qualities that may not matter for your use case. Good scores on them don’t mean your system works.

Instead, conduct error analysis to understand failures. Define binary failure modes based on real problems. Create custom evaluators for those failures and validate them against human judgment. Essentially, the entire evals process.

Experienced practitioners may still use these metrics, just not how you’d expect. As Picasso said: “Learn the rules like a pro, so you can break them like an artist.” Once you understand why generic metrics fail as evaluations, you can repurpose them as exploration tools to find interesting traces (explained in the next FAQ).

↗ Focus view

Q: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?

Generic metrics like BERTScore, ROUGE, cosine similarity, etc. are not useful for evaluating LLM outputs in most AI applications. Instead, we recommend using error analysis to identify metrics specific to your application’s behavior. We recommend designing binary pass/fail.) evals (using LLM-as-judge) or code-based assertions.

As an example, consider a real estate CRM assistant. Suggesting showings that aren’t available (can be tested with an assertion) or confusing client personas (can be tested with a LLM-as-judge) is problematic . Generic metrics like similarity or verbosity won’t catch this. A relevant quote from the course:

“The abuse of generic metrics is endemic. Many eval vendors promote off the shelf metrics, which ensnare engineers into superfluous tasks.”

Similarity metrics aren’t always useless. They have utility in domains like search and recommendation (and therefore can be useful for optimizing and debugging retrieval for RAG). For example, cosine similarity between embeddings can measure semantic closeness in retrieval systems, and average pairwise similarity can assess output diversity (where lower similarity indicates higher diversity).

↗ Focus view

Q: Can I use the same model for both the main task and evaluation?

For LLM-as-Judge selection, using the same model is usually fine because the judge is doing a different task than your main LLM pipeline. The judges we recommend building do scoped binary classification tasks. Focus on achieving high True Positive Rate (TPR) and True Negative Rate (TNR) with your judge on a held out labeled test set rather than avoiding the same model family. You can use these metrics on the test set to understand how well your judge is doing.

When selecting judge models, start with the most capable models available to establish strong alignment with human judgments. You can optimize for cost later once you’ve established reliable evaluation criteria. We do not recommend using the same model for open ended preferences or response quality (but we don’t recommend building judges this way in the first place!).

↗ Focus view

👉 If you want to learn more about AI Evals, check out our AI Evals course. Here is a 35% discount code for readers. 👈

Human Annotation & Process

Q: How many people should annotate my LLM outputs?

For most small to medium-sized companies, appointing a single domain expert as a “benevolent dictator” is the most effective approach. This person—whether it’s a psychologist for a mental health chatbot, a lawyer for legal document analysis, or a customer service director for support automation—becomes the definitive voice on quality standards.

A single expert eliminates annotation conflicts and prevents the paralysis that comes from “too many cooks in the kitchen”. The benevolent dictator can incorporate input and feedback from others, but they drive the process. If you feel like you need five subject matter experts to judge a single interaction, it’s a sign your product scope might be too broad.

However, larger organizations or those operating across multiple domains (like a multinational company with different cultural contexts) may need multiple annotators. When you do use multiple people, you’ll need to measure their agreement using metrics like Cohen’s Kappa, which accounts for agreement beyond chance. However, use your judgment. Even in larger companies, a single expert is often enough.

Start with a benevolent dictator whenever feasible. Only add complexity when your domain demands it.

↗ Focus view

Q: Should product managers and engineers collaborate on error analysis? How?

At the outset, collaborate to establish shared context. Engineers catch technical issues like retrieval issues and tool errors. PMs identify product failures like unmet user expectations, confusing responses, or missing features users expect.

As time goes on you should lean towards a benevolent dictator for error analysis: a domain expert or PM who understands user needs. Empower domain experts to evaluate actual outcomes rather than technical implementation. Ask “Has an appointment been made?” not “Did the tool call succeed?” The best way to empower the domain expert is to give them custom annotation tools that display system outcomes alongside traces. Show the confirmation, generated email, or database update that validates goal completion. Keep all context on one screen so non-technical reviewers focus on results.

↗ Focus view

Q: Should I outsource annotation & labeling to a third party?

Outsourcing error analysis is usually a big mistake (with some exceptions). The core of evaluation is building the product intuition that only comes from systematically analyzing your system’s failures. You should be extremely skeptical of this process being delegated.

The Dangers of Outsourcing

When you outsource annotation, you often break the feedback loop between observing a failure and understanding how to improve the product. Problems with outsourcing include:

• Superficial Labeling: Even well-defined metrics require nuanced judgment that external teams lack. A critical misstep in error analysis is excluding domain experts from the labeling process. Outsourcing this task to those without domain expertise, like general developers or IT staff, often leads to superficial or incorrect labeling.
  

• Loss of Unspoken Knowledge: A principal domain expert possesses tacit knowledge and user understanding that cannot be fully captured in a rubric. Involving these experts helps uncover their preferences and expectations, which they might not be able to fully articulate upfront.
  

• Annotation Conflicts and Misalignment: Without a shared context, external annotators can create more disagreement than they resolve. Achieving alignment is a challenge even for internal teams, which means you will spend even more time on this process.

The Recommended Approach: Build Internal Capability

Instead of outsourcing, focus on building an efficient internal evaluation process.

1. Appoint a “Benevolent Dictator”. For most teams, the most effective strategy is to appoint a single, internal domain expert as the final decision-maker on quality. This individual sets the standard, ensures consistency, and develops a sense of ownership.

2. Use a collaborative workflow for multiple annotators. If multiple annotators are necessary, follow a structured process to ensure alignment: * Draft an initial rubric with clear Pass/Fail definitions and examples. * Have each annotator label a shared set of traces independently to surface differences in interpretation. * Measure Inter-Annotator Agreement (IAA) using a chance-corrected metric like Cohen’s Kappa. * Facilitate alignment sessions to discuss disagreements and refine the rubric. * Iterate on this process until agreement is consistently high.

How to Handle Capacity Constraints

Building internal capacity does not mean you have to label every trace. Use these strategies to manage the workload:

• Smart Sampling: Review a small, representative sample of traces thoroughly. It is more effective to analyze 100 diverse traces to find patterns than to superficially label thousands.
  

• The “Think-Aloud” Protocol: To make the most of limited expert time, use this technique from usability testing. Ask an expert to verbalize their thought process while reviewing a handful of traces. This method can uncover deep insights in a single one-hour session.
  

• Build Lightweight Custom Tools: Build custom annotation tools to streamline the review process, increasing throughput.

Exceptions for External Help

While outsourcing the core error analysis process is not recommended, there are some scenarios where external help is appropriate:

• Purely Mechanical Tasks: For highly objective, unambiguous tasks like identifying a phone number or validating an email address, external annotators can be used after a rigorous internal process has defined the rubric.
  

• Tasks Without Product Context: Well-defined tasks that don’t require understanding your product’s specific requirements can be outsourced. Translation is a good example: it requires linguistic expertise but not deep product knowledge.
  

• Engaging Subject Matter Experts: Hiring external SMEs to act as your internal domain experts is not outsourcing; it is bringing the necessary expertise into your evaluation process. For example, AnkiHub hired 4th-year medical students to evaluate their RAG systems for medical content rather than outsourcing to generic annotators.

↗ Focus view

Q: What parts of evals can be automated with LLMs?

LLMs can speed up parts of your eval workflow, but they can’t replace human judgment where your expertise is essential. For example, if you let an LLM handle all of error analysis (i.e., reviewing and annotating traces), you might overlook failure cases that matter for your product. Suppose users keep mentioning “lag” in feedback, but the LLM lumps these under generic “performance issues” instead of creating a “latency” category. You’d miss a recurring complaint about slow response times and fail to prioritize a fix.

That said, LLMs are valuable tools for accelerating certain parts of the evaluation workflow when used with oversight.

Here are some areas where LLMs can help:

• First-pass axial coding: After you’ve open coded 30–50 traces yourself, use an LLM to organize your raw failure notes into proposed groupings. This helps you quickly spot patterns, but always review and refine the clusters yourself. Note: If you aren’t familiar with axial and open coding, see this faq.

• Mapping annotations to failure modes: Once you’ve defined failure categories, you can ask an LLM to suggest which categories apply to each new trace (e.g., “Given this annotation: [open_annotation] and these failure modes: [list_of_failure_modes], which apply?”).
  

• Suggesting prompt improvements: When you notice recurring problems, have the LLM propose concrete changes to your prompts. Review these suggestions before adopting any changes.
  

• Analyzing annotation data: Use LLMs or AI-powered notebooks to find patterns in your labels, such as “reports of lag increase 3x during peak usage hours” or “slow response times are mostly reported from users on mobile devices.”

However, you shouldn’t outsource these activities to an LLM:

• Initial open coding: Always read through the raw traces yourself at the start. This is how you discover new types of failures, understand user pain points, and build intuition about your data. Never skip this or delegate it.
  

• Validating failure taxonomies: LLM-generated groupings need your review. For example, an LLM might group both “app crashes after login” and “login takes too long” under a single “login issues” category, even though one is a stability problem and the other is a performance problem. Without your intervention, you’d miss that these issues require different fixes.
  

• Ground truth labeling: For any data used for testing/validating LLM-as-Judge evaluators, hand-validate each label. LLMs can make mistakes that lead to unreliable benchmarks.
  

• Root cause analysis: LLMs may point out obvious issues, but only human review will catch patterns like errors that occur in specific workflows or edge cases—such as bugs that happen only when users paste data from Excel.

In conclusion, start by examining data manually to understand what’s actually going wrong. Use LLMs to scale what you’ve learned, not to avoid looking at data.

↗ Focus view

Q: Should I stop writing prompts manually in favor of automated tools?

Automating prompt engineering can be tempting, but you should be skeptical of tools that promise to optimize prompts for you, especially in early stages of development. When you write a prompt, you are forced to clarify your assumptions and externalize your requirements. Good writing is good thinking ². If you delegate this task to an automated tool too early, you risk never fully understanding your own requirements or the model’s failure modes.

This is because automated prompt optimization typically hill-climb a predefined evaluation metric. It can refine a prompt to perform better on known failures, but it cannot discover new ones. Discovering new errors requires error analysis. Furthermore, research shows that evaluation criteria tends to shift after reviewing a model’s outputs, a phenomenon known as “criteria drift” ³. This means that evaluation is an iterative, human-driven sensemaking process, not a static target that can be set once and handed off to an optimizer.

A pragmatic approach is to use LLMs to improve your prompt based on open coding (open-ended notes about traces). This way, you maintain a human in the loop who is looking at the data and externalizing their requirements. Once you have a high-quality set of evals, prompt optimization can be effective for that last mile of performance.

↗ Focus view

Tools & Infrastructure

Q: Should I build a custom annotation tool or use something off-the-shelf?

Build a custom annotation tool. This is the single most impactful investment you can make for your AI evaluation workflow. With AI-assisted development tools like Cursor or Lovable, you can build a tailored interface in hours. I often find that teams with custom annotation tools iterate ~10x faster.

Custom tools excel because:

• They show all your context from multiple systems in one place

• They can render your data in a product specific way (images, widgets, markdown, buttons, etc.)

• They’re designed for your specific workflow (custom filters, sorting, progress bars, etc.)

Off-the-shelf tools may be justified when you need to coordinate dozens of distributed annotators with enterprise access controls. Even then, many teams find the configuration overhead and limitations aren’t worth it.

Isaac’s Anki flashcard annotation app shows the power of custom tools—handling 400+ results per query with keyboard navigation and domain-specific evaluation criteria that would be nearly impossible to configure in a generic tool.

↗ Focus view

Q: What makes a good custom interface for reviewing LLM outputs?

Great interfaces make human review fast, clear, and motivating. We recommend building your own annotation tool customized to your domain. The following features are possible enhancements we’ve seen work well, but you don’t need all of them. The screenshots shown are illustrative examples to clarify concepts. In practice, I rarely implement all these features in a single app. It’s ultimately a judgment call based on your specific needs and constraints.

1. Render Traces Intelligently, Not Generically:

Present the trace in a way that’s intuitive for the domain. If you’re evaluating generated emails, render them to look like emails. If the output is code, use syntax highlighting. Allow the reviewer to see the full trace (user input, tool calls, and LLM reasoning), but keep less important details in collapsed sections that can be expanded. Here is an example of a custom annotation tool for reviewing real estate assistant emails:

2. Show Progress and Support Keyboard Navigation:

Keep reviewers in a state of flow by minimizing friction and motivating completion. Include progress indicators (e.g., “Trace 45 of 100”) to keep the review session bounded and encourage completion. Enable hotkeys for navigating between traces (e.g., N for next), applying labels, and saving notes quickly. Below is an illustration of these features:

3. Trace navigation through clustering, filtering, and search:

Allow reviewers to filter traces by metadata or search by keywords. Semantic search helps find conceptually similar problems. Clustering similar traces (like grouping by user persona) lets reviewers spot recurring issues and explore hypotheses. Below is an illustration of these features:

4. Prioritize labeling traces you think might be problematic:

Surface traces flagged by guardrails, CI failures, or automated evaluators for review. Provide buttons to take actions like adding to datasets, filing bugs, or re-running pipeline tests. Display relevant context (pipeline version, eval scores, reviewer info) directly in the interface to minimize context switching. Below is an illustration of these ideas:

General Principle: Keep it minimal

Keep your annotation interface minimal. Only incorporate these ideas if they provide a benefit that outweighs the additional complexity and maintenance overhead.

↗ Focus view

Q: What gaps in eval tooling should I be prepared to fill myself?

Most eval tools handle the basics well: logging complete traces, tracking metrics, prompt playgrounds, and annotation queues. These are table stakes. Here are four areas where you’ll likely need to supplement existing tools.

Watch for vendors addressing these gaps: it’s a strong signal they understand practitioner needs.

1. Error Analysis and Pattern Discovery

After reviewing traces where your AI fails, can your tooling automatically cluster similar issues? For instance, if multiple traces show the assistant using casual language for luxury clients, you need something that recognizes this broader “persona-tone mismatch” pattern. We recommend building capabilities that use AI to suggest groupings, rewrite your observations into clearer failure taxonomies, help find similar cases through semantic search, etc.

2. AI-Powered Assistance Throughout the Workflow

The most effective workflows use AI to accelerate every stage of evaluation. During error analysis, you want an LLM helping categorize your open-ended observations into coherent failure modes. For example, you might annotate several traces with notes like “wrong tone for investor,” “too casual for luxury buyer,” etc. Your tooling should recognize these as the same underlying pattern and suggest a unified “persona-tone mismatch” category.

You’ll also want AI assistance in proposing fixes. After identifying 20 cases where your assistant omits pet policies from property summaries, can your workflow analyze these failures and suggest specific prompt modifications? Can it draft refinements to your SQL generation instructions when it notices patterns of missing WHERE clauses?

Additionally, good workflows help you conduct data analysis of your annotations and traces. I like using notebooks with AI in-the-loop like Julius,Hex or SolveIt. These help me discover insights like “location ambiguity errors spike 3x when users mention neighborhood names” or “tone mismatches occur 80% more often in email generation than other modalities.”

3. Custom Evaluators Over Generic Metrics

Be prepared to build most of your evaluators from scratch. Generic metrics like “hallucination score” or “helpfulness rating” rarely capture what actually matters for your application—like proposing unavailable showing times or omitting budget constraints from emails. In our experience, successful teams spend most of their effort on application-specific metrics.

4. APIs That Support Custom Annotation Apps

Custom annotation interfaces work best for most teams. This requires observability platforms with thoughtful APIs. I often have to build my own libraries and abstractions just to make bulk data export manageable. You shouldn’t have to paginate through thousands of requests or handle timeout-prone endpoints just to get your data. Look for platforms that provide true bulk export capabilities and, crucially, APIs that let you write annotations back efficiently.

↗ Focus view

Q: Seriously Hamel. Stop the bullshit. What’s your favorite eval vendor?

Eval tools are in an intensely competitive space. It would be futile to compare their features. If I tried to do such an analysis, it would be invalidated in a week! Vendors I encounter the most organically in my work are: Langsmith, Arize and Braintrust.

When I help clients with vendor selection, the decision weighs heavily towards who can offer the best support, as opposed to purely features. This changes depending on size of client, use case, etc. Yes - it’s mainly the human factor that matters, and dare I say, vibes.

I have no favorite vendor. At the core, their features are very similar - and I often build custom tools on top of them to fit my needs.

My suggestion is to explore the vendors and see which one you like the most.

↗ Focus view

Production & Deployment

Q: How are evaluations used differently in CI/CD vs. monitoring production?

The most important difference between CI vs. production evaluation is the data used for testing.

Test datasets for CI are small (in many cases 100+ examples) and purpose-built. Examples cover core features, regression tests for past bugs, and known edge cases. Since CI tests are run frequently, the cost of each test has to be carefully considered (that’s why you carefully curate the dataset). Favor assertions or other deterministic checks over LLM-as-judge evaluators.

For evaluating production traffic, you can sample live traces and run evaluators against them asynchronously. Since you usually lack reference outputs on production data, you might rely more on on more expensive reference-free evaluators like LLM-as-judge. Additionally, track confidence intervals for production metrics. If the lower bound crosses your threshold, investigate further.

These two systems are complementary: when production monitoring reveals new failure patterns through error analysis and evals, add representative examples to your CI dataset. This mitigates regressions on new issues.

↗ Focus view

Q: What’s the difference between guardrails & evaluators?

Guardrails are inline safety checks that sit directly in the request/response path. They validate inputs or outputs before anything reaches a user, so they typically are:

• Fast and deterministic – typically a few milliseconds of latency budget.

• Simple and explainable – regexes, keyword block-lists, schema or type validators, lightweight classifiers.

• Targeted at clear-cut, high-impact failures – PII leaks, profanity, disallowed instructions, SQL injection, malformed JSON, invalid code syntax, etc.

If a guardrail triggers, the system can redact, refuse, or regenerate the response. Because these checks are user-visible when they fire, false positives are treated as production bugs; teams version guardrail rules, log every trigger, and monitor rates to keep them conservative.

On the other hand, evaluators typically run after a response is produced. Evaluators measure qualities that simple rules cannot, such as factual correctness, completeness, etc. Their verdicts feed dashboards, regression tests, and model-improvement loops, but they do not block the original answer.

Evaluators are usually run asynchronously or in batch to afford heavier computation such as a LLM-as-a-Judge. Inline use of an LLM-as-Judge is possible only when the latency budget and reliability targets allow it. Slow LLM judges might be feasible in a cascade that runs on the minority of borderline cases.

Apply guardrails for immediate protection against objective failures requiring intervention. Use evaluators for monitoring and improving subjective or nuanced criteria. Together, they create layered protection.

Word of caution: Do not use llm guardrails off the shelf blindly. Always look at the prompt.

↗ Focus view

Q: Can my evaluators also be used to automatically fix or correct outputs in production?

Yes, but only a specific subset of them. This is the distinction between an evaluator and a guardrail that we previously discussed. As a reminder:

• Evaluators typically run asynchronously after a response has been generated. They measure quality but don’t interfere with the user’s immediate experience.
  

• Guardrails run synchronously in the critical path of the request, before the output is shown to the user. Their job is to prevent high-impact failures in real-time.

There are two important decision criteria for deciding whether to use an evaluator as a guardrail:

1. Latency & Cost: Can the evaluator run fast enough and cheaply enough in the critical request path without degrading user experience?

2. Error Rate Trade-offs: What’s the cost-benefit balance between false positives (blocking good outputs and frustrating users) versus false negatives (letting bad outputs reach users and causing harm)? In high-stakes domains like medical advice, false negatives may be more costly than false positives. In creative applications, false positives that block legitimate creativity may be more harmful than occasional quality issues.

Most guardrails are designed to be fast (to avoid harming user experience) and have a very low false positive rate (to avoid blocking valid responses). For this reason, you would almost never use a slow or non-deterministic LLM-as-Judge as a synchronous guardrail. However, these tradeoffs might be different for your use case.

↗ Focus view

Q: How much time should I spend on model selection?

Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system off the bat without evidence. Does error analysis suggest that your model is the problem?”

↗ Focus view

Domain-Specific Applications

Q: Is RAG dead?

Question: Should I avoid using RAG for my AI application after reading that “RAG is dead” for coding agents?

Many developers are confused about when and how to use RAG after reading articles claiming “RAG is dead.” Understanding what RAG actually means versus the narrow marketing definitions will help you make better architectural decisions for your AI applications.

The viral article claiming RAG is dead specifically argues against using naive vector database retrieval for autonomous coding agents, not RAG as a whole. This is a crucial distinction that many developers miss due to misleading marketing.

RAG simply means Retrieval-Augmented Generation - using retrieval to provide relevant context that improves your model’s output. The core principle remains essential: your LLM needs the right context to generate accurate answers. The question isn’t whether to use retrieval, but how to retrieve effectively.

For coding applications, naive vector similarity search often fails because code relationships are complex and contextual. Instead of abandoning retrieval entirely, modern coding assistants like Claude Code still uses retrieval —they just employ agentic search instead of relying solely on vector databases, similar to how human developers work.

You have multiple retrieval strategies available, ranging from simple keyword matching to embedding similarity to LLM-powered relevance filtering. The optimal approach depends on your specific use case, data characteristics, and performance requirements. Many production systems combine multiple strategies or use multi-hop retrieval guided by LLM agents.

Unfortunately, “RAG” has become a buzzword with no shared definition. Some people use it to mean any retrieval system, others restrict it to vector databases. Focus on the ultimate goal: getting your LLM the context it needs to succeed. Whether that’s through vector search, agentic exploration, or hybrid approaches is a product and engineering decision.

Rather than following categorical advice to avoid or embrace RAG, experiment with different retrieval approaches and measure what works best for your application. For more info on RAG evaluation and optimization, see this series of posts.

↗ Focus view

Q: How should I approach evaluating my RAG system?

RAG systems have two distinct components that require different evaluation approaches: retrieval and generation.

The retrieval component is a search problem. Evaluate it using traditional information retrieval (IR) metrics. Common examples include Recall@k (of all relevant documents, how many did you retrieve in the top k?), Precision@k (of the k documents retrieved, how many were relevant?), or MRR (how high up was the first relevant document?). The specific metrics you choose depend on your use case. These metrics are pure search metrics that measure whether you’re finding the right documents (more on this below).

To evaluate retrieval, create a dataset of queries paired with their relevant documents. Generate this synthetically by taking documents from your corpus, extracting key facts, then generating questions those facts would answer. This reverse process gives you query-document pairs for measuring retrieval performance without manual annotation.

For the generation component—how well the LLM uses retrieved context, whether it hallucinates, whether it answers the question—use the same evaluation procedures covered throughout this course: error analysis to identify failure modes, collecting human labels, building LLM-as-judge evaluators, and validating those judges against human annotations.

Jason Liu’s “There Are Only 6 RAG Evals” provides a framework that maps well to this separation. His Tier 1 covers traditional IR metrics for retrieval. Tiers 2 and 3 evaluate relationships between Question, Context, and Answer—like whether the context is relevant (C|Q), whether the answer is faithful to context (A|C), and whether the answer addresses the question (A|Q).

In addition to Jason’s six evals, error analysis on your specific data may reveal domain-specific failure modes that warrant their own metrics. For example, a medical RAG system might consistently fail to distinguish between drug dosages for adults versus children, or a legal RAG might confuse jurisdictional boundaries. These patterns emerge only through systematic review of actual failures. Once identified, you can create targeted evaluators for these specific issues beyond the general framework.

Finally, when implementing Jason’s Tier 2 and 3 metrics, don’t just use prompts off the shelf. The standard LLM-as-judge process requires several steps: error analysis, prompt iteration, creating labeled examples, and measuring your judge’s accuracy against human labels. Once you know your judge’s True Positive and True Negative rates, you can correct its estimates to determine the actual failure rate in your system. Skip this validation and your judges may not reflect your actual quality criteria.

In summary, debug retrieval first using IR metrics, then tackle generation quality using properly validated LLM judges.

↗ Focus view

Q: How do I choose the right chunk size for my document processing tasks?

Unlike RAG, where chunks are optimized for retrieval, document processing assumes the model will see every chunk. The goal is to split text so the model can reason effectively without being overwhelmed. Even if a document fits within the context window, it might be better to break it up. Long inputs can degrade performance due to attention bottlenecks, especially in the middle of the context. Two task types require different strategies:

1. Fixed-Output Tasks → Large Chunks

These are tasks where the output length doesn’t grow with input: extracting a number, answering a specific question, classifying a section. For example:

• “What’s the penalty clause in this contract?”

• “What was the CEO’s salary in 2023?”

Use the largest chunk (with caveats) that likely contains the answer. This reduces the number of queries and avoids context fragmentation. However, avoid adding irrelevant text. Models are sensitive to distraction, especially with large inputs. The middle parts of a long input might be under-attended. Furthermore, if cost and latency are a bottleneck, you should consider preprocessing or filtering the document (via keyword search or a lightweight retriever) to isolate relevant sections before feeding a huge chunk.

2. Expansive-Output Tasks → Smaller Chunks

These include summarization, exhaustive extraction, or any task where output grows with input. For example:

• “Summarize each section”

• “List all customer complaints”

In these cases, smaller chunks help preserve reasoning quality and output completeness. The standard approach is to process each chunk independently, then aggregate results (e.g., map-reduce). When sizing your chunks, try to respect content boundaries like paragraphs, sections, or chapters. Chunking also helps mitigate output limits. By breaking the task into pieces, each piece’s output can stay within limits.

General Guidance

It’s important to recognize why chunk size affects results. A larger chunk means the model has to reason over more information in one go – essentially, a heavier cognitive load. LLMs have limited capacity to retain and correlate details across a long text. If too much is packed in, the model might prioritize certain parts (commonly the beginning or end) and overlook or “forget” details in the middle. This can lead to overly coarse summaries or missed facts. In contrast, a smaller chunk bounds the problem: the model can pay full attention to that section. You are trading off global context for local focus.

No rule of thumb can perfectly determine the best chunk size for your use case – you should validate with experiments. The optimal chunk size can vary by domain and model. I treat chunk size as a hyperparameter to tune.

↗ Focus view

Q: How do I debug multi-turn conversation traces?

Start simple. Check if the whole conversation met the user’s goal with a pass/fail judgment. Look at the entire trace and focus on the first upstream failure. Read the user-visible parts first to understand if something went wrong. Only then dig into the technical details like tool calls and intermediate steps.

Multi-agent trace logging

For multi-agent flows, assign a session or trace ID to each user request and log every message with its source (which agent or tool), trace ID, and position in the sequence. This lets you reconstruct the full path from initial query to final result across all agents.

Annotation strategy

Annotate only the first failure in the trace initially—don’t worry about downstream failures since these often cascade from the first issue. Fixing upstream failures often resolves dependent downstream failures automatically. As you gain experience, you can annotate independent failure modes within the same trace to speed up overall error analysis.

Simplify when possible

When you find a failure, reproduce it with the simplest possible test case. Here’s an example: suppose a shopping bot gives the wrong return policy on turn 4 of a conversation. Before diving into the full multi-turn complexity, simplify it to a single turn: “What is the return window for product X1000?” If it still fails, you’ve proven the error isn’t about conversation context - it’s likely a basic retrieval or knowledge issue you can debug more easily.

Test case generation

You have two main approaches. First, simulate users with another LLM to create realistic multi-turn conversations. Second, use “N-1 testing” where you provide the first N-1 turns of a real conversation and test what happens next. The N-1 approach often works better since it uses actual conversation prefixes rather than fully synthetic interactions, but is less flexible.

The key is balancing thoroughness with efficiency. Not every multi-turn failure requires multi-turn analysis.

↗ Focus view

Q: How do I evaluate sessions with human handoffs?

Capture the complete user journey in your traces, including human handoffs. The trace continues until the user’s need is resolved or the session ends, not when AI hands off to a human. Log the handoff decision, why it occurred, context transferred, wait time, human actions, final resolution, and whether the human had sufficient context. Many failures occur at handoff boundaries where AI hands off too early, too late, or without proper context.

Evaluate handoffs as potential failure modes during error analysis. Ask: Was the handoff necessary? Did the AI provide adequate context? Track both handoff quality and handoff rate. Sometimes the best improvement reduces handoffs entirely rather than improving handoff execution.

↗ Focus view

Q: How do I evaluate complex multi-step workflows?

Log the entire workflow from initial trigger to final business outcome. Include LLM calls, tool usage, human approvals, and database writes in your traces. You will need this visibility to properly diagnose failures.

Use both outcome and process metrics. Outcome metrics verify the final result meets requirements: Was the business case complete? Accurate? Properly formatted? Process metrics evaluate efficiency: step count, time taken, resource usage. Process failures are often easier to debug since they’re more deterministic, so tackle them first.

Segment your error analysis by workflow stages. Early stage failures (understanding user input) differ from middle stage failures (data processing) and late stage failures (formatting output). Early stage improvements have more impact since errors cascade in LLM chains.

Use transition failure matrices to analyze where workflows break. Create a matrix showing the last successful state versus where the first failure occurred. This reveals failure hotspots and guides where to invest debugging effort.

↗ Focus view

Q: How do I evaluate agentic workflows?

We recommend evaluating agentic workflows in two phases:

1. End-to-end task success. Treat the agent as a black box and ask “did we meet the user’s goal?”. Define a precise success rule per task (exact answer, correct side-effect, etc.) and measure with human or aligned LLM judges. Take note of the first upstream failure when conducting error analysis.

Once error analysis reveals which workflows fail most often, move to step-level diagnostics to understand why they’re failing.

2. Step-level diagnostics. Assuming that you have sufficiently instrumented your system with details of tool calls and responses, you can score individual components such as: - Tool choice: was the selected tool appropriate? - Parameter extraction: were inputs complete and well-formed? - Error handling: did the agent recover from empty results or API failures? - Context retention: did it preserve earlier constraints? - Efficiency: how many steps, seconds, and tokens were spent? - Goal checkpoints: for long workflows verify key milestones.

Example: “Find Berkeley homes under $1M and schedule viewings” breaks into: parameters extracted correctly, relevant listings retrieved, availability checked, and calendar invites sent. Each checkpoint can pass or fail independently, making debugging tractable.

Use transition failure matrices to understand error patterns. Create a matrix where rows represent the last successful state and columns represent where the first failure occurred. This is a great way to understand where the most failures occur.

Transition matrices transform overwhelming agent complexity into actionable insights. Instead of drowning in individual trace reviews, you can immediately see that GenSQL → ExecSQL transitions cause 12 failures while DecideTool → PlanCal causes only 2. This data-driven approach guides where to invest debugging effort. Here is another example from Bryan Bischof, that is also a text-to-SQL agent:

In this example, Bryan shows variation in transition matrices across experiments. How you organize your transition matrix depends on the specifics of your application. For example, Bryan’s text-to-SQL agent has an inherent sequential workflow which he exploits for further analytical insight. You can watch his full talk for more details.

Creating Test Cases for Agent Failures

Creating test cases for agent failures follows the same principles as our previous FAQ on debugging multi-turn conversation traces (i.e. try to reproduce the error in the simplest way possible, only use multi-turn tests when the failure actually requires conversation context, etc.).

↗ Focus view

👉 If you want to learn more about AI Evals, check out our AI Evals course. Here is a 35% discount code for readers. 👈

Footnotes

1. Eleanor Berger, our wonderful TA.↩︎

2. Paul Graham, “Writes and Write-Nots”↩︎

3. Shreya Shankar, et al., “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences”↩︎

What’s Actually Dead

Ben Clavié’s opener nailed it: what’s dead is the 2023 marketing version of RAG. Chuck documents into a vector database, do cosine similarity, call it a day. This approach fails because compressing entire documents into single vectors loses critical information.

But retrieval is more important than ever. LLMs are frozen at training time. Million-token context windows don’t change the economics or efficiency of stuffing everything into every query.

Takeaways From the Series

We’ve been measuring wrong. Nandan Thakur showed that traditional IR metrics optimize for finding the #1 result. RAG needs different goals: coverage (getting all the facts), diversity (corroborating facts), and relevance. Models that ace BEIR benchmarks often fail at real RAG tasks.

Retrieval can reason. Orion Weller’s models understand instructions like “find documents about data privacy using metaphors.” His Rank1 system generates explicit reasoning traces about relevance. These models find documents that traditional systems never surface.

Single vectors lose information. Antoine Chaffin demonstrated how late-interaction models like ColBERT preserve token-level information. No more forcing everything into one conflicted representation. Result: 150M parameter models outperforming 7B parameter alternatives on reasoning tasks.

One map isn’t enough. Bryan and Ayush’s finale showed why we need multiple representations. Their art search demo finds the same painting through literal descriptions, poetic interpretations, or similar images—each using different indices. Stop searching for the perfect embedding. Build specialized representations and route intelligently.

What’s Next

I think a path forward is to combine these ideas:

• Evaluation systems that measure what matters for your use case

• Retrievers that understand instructions and reason about relevance

• Representations that preserve information instead of compressing it away

• Multiple specialized indices with intelligent routing

Annotated Notes From the Series

Each post includes timestamped annotations of slides, saving you hours of video watching. We’ve highlighted the most important bits and provided context for quickly grokking the material.

TitleDescription

Part 1: I don’t use RAG, I just retrieve documentsBen Clavié explains why naive single-vector search is dead, not RAG itself

Part 2: Modern IR Evals For RAGNandan Thakur shows why traditional IR metrics fail for RAG and introduces FreshStack

Part 3: Optimizing Retrieval with Reasoning ModelsOrion Weller demonstrates retrievers that understand instructions and reason about relevance

Part 4: Late Interaction Models For RAGAntoine Chaffin reveals how ColBERT-style models preserve information that single vectors lose

Part 5: RAG with Multiple RepresentationsBryan Bischof and Ayush Chaurasia show why multiple specialized indices beat one perfect embedding

AI TEAM

Here’s our agent architecture – we’ve got RAG here, a router there, and we’re using this new framework for…

ME

[Holding up my hand to pause the enthusiastic tech lead.]

“Can you show me how you’re measuring if any of this actually works?”

… Room goes quiet

This scene has played out dozens of times over the last two years. Teams invest weeks building complex AI systems, but can’t tell me if their changes are helping or hurting.

This isn’t surprising. With new tools and frameworks emerging weekly, it’s natural to focus on tangible things we can control – which vector database to use, which LLM provider to choose, which agent framework to adopt. But after helping 30+ companies build AI products, I’ve discovered the teams who succeed barely talk about tools at all. Instead, they obsess over measurement and iteration.

In this post, I’ll show you exactly how these successful teams operate. You’ll learn:

1. How error analysis consistently reveals the highest-ROI improvements

2. Why a simple data viewer is your most important AI investment

3. How to empower domain experts (not just engineers) to improve your AI

4. Why synthetic data is more effective than you think

5. How to maintain trust in your evaluation system

6. Why your AI roadmap should count experiments, not features

I’ll explain each of these topics with real examples. While every situation is unique, you’ll see patterns that apply regardless of your domain or team size.

Let’s start by examining the most common mistake I see teams make – one that derails AI projects before they even begin.

1. The Most Common Mistake: Skipping Error Analysis

The “tools first” mindset is the most common mistake in AI development. Teams get caught up in architecture diagrams, frameworks, and dashboards while neglecting the process of actually understanding what’s working and what isn’t.

One client proudly showed me this evaluation dashboard:

This is the “tools trap” – the belief that adopting the right tools or frameworks (in this case, generic metrics) will solve your AI problems. Generic metrics are worse than useless – they actively impede progress in two ways:

First, they create a false sense of measurement and progress. Teams think they’re data-driven because they have dashboards, but they’re tracking vanity metrics that don’t correlate with real user problems. I’ve seen teams celebrate improving their “helpfulness score” by 10% while their actual users were still struggling with basic tasks. It’s like optimizing your website’s load time while your checkout process is broken – you’re getting better at the wrong thing.

Second, too many metrics fragment your attention. Instead of focusing on the few metrics that matter for your specific use case, you’re trying to optimize multiple dimensions simultaneously. When everything is important, nothing is.

The alternative? Error analysis - the single most valuable activity in AI development and consistently the highest-ROI activity. Let me show you what effective error analysis looks like in practice.

The Error Analysis Process

When Jacob, the founder of Nurture Boss, needed to improve their apartment-industry AI assistant, his team built a simple viewer to examine conversations between their AI and users. Next to each conversation was a space for open-ended notes about failure modes.

After annotating dozens of conversations, clear patterns emerged. Their AI was struggling with date handling – failing 66% of the time when users said things like “let’s schedule a tour two weeks from now.”

Instead of reaching for new tools, they: 1. Looked at actual conversation logs 2. Categorized the types of date-handling failures 3. Built specific tests to catch these issues 4. Measured improvement on these metrics

The result? Their date handling success rate improved from 33% to 95%.

Here’s Jacob explaining this process himself:

Bottom-Up vs. Top-Down Analysis

When identifying error types, you can take either a “top-down” or “bottom-up” approach.

The top-down approach starts with common metrics like “hallucination” or “toxicity” plus metrics unique to your task. While convenient, it often misses domain-specific issues.

The more effective bottom-up approach forces you to look at actual data and let metrics naturally emerge. At NurtureBoss, we started with a spreadsheet where each row represented a conversation. We wrote open-ended notes on any undesired behavior. Then we used an LLM to build a taxonomy of common failure modes. Finally, we mapped each row to specific failure mode labels and counted the frequency of each issue.

The results were striking - just three issues accounted for over 60% of all problems:

• Conversation flow issues (missing context, awkward responses)

• Handoff failures (not recognizing when to transfer to humans)

• Rescheduling problems (struggling with date handling)

The impact was immediate. Jacob’s team had uncovered so many actionable insights that they needed several weeks just to implement fixes for the problems we’d already found.

If you’d like to see error analysis in action, we recorded a live walkthrough here.

This brings us to a crucial question: How do you make it easy for teams to look at their data? The answer leads us to what I consider the most important investment any AI team can make…

2. The Most Important AI Investment: A Simple Data Viewer

The single most impactful investment I’ve seen AI teams make isn’t a fancy evaluation dashboard – it’s building a customized interface that lets anyone examine what their AI is actually doing. I emphasize customized because every domain has unique needs that off-the-shelf tools rarely address. When reviewing apartment leasing conversations, you need to see the full chat history and scheduling context. For real estate queries, you need the property details and source documents right there. Even small UX decisions – like where to place metadata or which filters to expose – can make the difference between a tool people actually use and one they avoid.

I’ve watched teams struggle with generic labeling interfaces, hunting through multiple systems just to understand a single interaction. The friction adds up: clicking through to different systems to see context, copying error descriptions into separate tracking sheets, switching between tools to verify information. This friction doesn’t just slow teams down – it actively discourages the kind of systematic analysis that catches subtle issues.

Teams with thoughtfully designed data viewers iterate 10x faster than those without them. And here’s the thing: these tools can be built in hours using AI-assisted development (like Cursor or Loveable). The investment is minimal compared to the returns.

Let me show you what I mean. Here’s the data viewer built for NurtureBoss (which we discussed earlier):

Here’s what makes a good data annotation tool:

1. Show all context in one place. Don’t make users hunt through different systems to understand what happened.
   

2. Make feedback trivial to capture. One-click correct/incorrect buttons beat lengthy forms.

3. Capture open-ended feedback. This lets you capture nuanced issues that don’t fit into a pre-defined taxonomy.

4. Enable quick filtering and sorting. Teams need to easily dive into specific error types. In the example above, NurtureBoss can quickly filter by the channel (voice, text, chat) or the specific property they want to look at quickly.

5. Have hotkeys that allow users to navigate between data examples and annotate without clicking.

It doesn’t matter what web frameworks you use - use whatever you are familiar with. Because I’m a python developer, my current favorite web framework is FastHTML coupled with MonsterUI, because it allows me to define the back-end and front-end code in one small python file.

The key is starting somewhere, even if it’s simple. I’ve found custom web apps provide the best experience, but if you’re just beginning, a spreadsheet is better than nothing. As your needs grow, you can evolve your tools accordingly.

This brings us to another counter-intuitive lesson: the people best positioned to improve your AI system are often the ones who know the least about AI.

3. Empower Domain Experts To Write Prompts

I recently worked with an education startup building an interactive learning platform with LLMs. Their product manager, a learning design expert, would create detailed PowerPoint decks explaining pedagogical principles and example dialogues. She’d present these to the engineering team, who would then translate her expertise into prompts.

But here’s the thing: prompts are just English. Having a learning expert communicate teaching principles through PowerPoint, only for engineers to translate that back into English prompts, created unnecessary friction. The most successful teams flip this model by giving domain experts tools to write and iterate on prompts directly.

Build Bridges, Not Gatekeepers

Prompt playgrounds are a great starting point for this. Tools like Arize, Langsmith and Braintrust let teams quickly test different prompts, feed in example datasets, and compare results. Here are some screenshots of these tools:

But there’s a crucial next step that many teams miss: integrating prompt development into their application context. Most AI applications aren’t just prompts – They commonly involve RAG systems pulling from your knowledge base, agent orchestration coordinating multiple steps, and application-specific business logic. The most effective teams I’ve worked with go beyond standalone playgrounds. They build what I call integrated prompt environments – essentially admin versions of their actual user interface that expose prompt editing.

Here’s an illustration of what an integrated prompt environment might look like for a real estate AI assistant:

Tips For Communicating With Domain Experts

There’s another barrier that often prevents domain experts from contributing effectively: unnecessary jargon. I was working with an education startup where engineers, product managers, and learning specialists were talking past each other in meetings. The engineers kept saying, “We’re going to build an agent that does XYZ,” when really the job to be done was writing a prompt. This created an artificial barrier – the learning specialists, who were the actual domain experts, felt like they couldn’t contribute because they didn’t understand “agents.”

This happens everywhere. I’ve seen it with lawyers at legal tech companies, psychologists at mental health startups, and doctors at healthcare firms. The magic of LLMs is that they make AI accessible through natural language, but we often destroy that advantage by wrapping everything in technical terminology.

Here’s a simple example of how to translate common AI jargon:

Instead of saying… Say… “We’re implementing a RAG approach” “We’re making sure the model has the right context to answer questions” “We need to prevent prompt injection” “We need to make sure users can’t trick the AI into ignoring our rules” “Our model suffers from hallucination issues” “Sometimes the AI makes things up, so we need to check its answers”

This doesn’t mean dumbing things down – it means being precise about what you’re actually doing. When you say “we’re building an agent,” what specific capability are you adding? Is it function calling? Tool use? Or just a better prompt? Being specific helps everyone understand what’s actually happening.

There’s nuance here. Technical terminology exists for a reason – it provides precision when talking with other technical stakeholders. The key is adapting your language to your audience.

The challenge many teams raise at this point is: “This all sounds great, but what if we don’t have any data yet? How can we look at examples or iterate on prompts when we’re just starting out?” That’s what we’ll talk about next.

4. Bootstrapping Your AI With Synthetic Data Is Effective (Even With Zero Users)

One of the most common roadblocks I hear from teams is: “We can’t do proper evaluation because we don’t have enough real user data yet.” This creates a chicken-and-egg problem – you need data to improve your AI, but you need a decent AI to get users who generate that data.

Fortunately, there’s a solution that works surprisingly well: synthetic data. LLMs can generate realistic test cases that cover the range of scenarios your AI will encounter.

As I wrote in my LLM-as-a-Judge blog post, synthetic data can be remarkably effective for evaluation. Bryan Bischof, the former Head of AI at Hex, put it perfectly:

“LLMs are surprisingly good at generating excellent - and diverse - examples of user prompts. This can be relevant for powering application features, and sneakily, for building Evals. If this sounds a bit like the Large Language Snake is eating its tail, I was just as surprised as you! All I can say is: it works, ship it.”

A Framework for Generating Realistic Test Data

The key to effective synthetic data is choosing the right dimensions to test. While these dimensions will vary based on your specific needs, I find it helpful to think about three broad categories:

1. Features: What capabilities does your AI need to support?

2. Scenarios: What situations will it encounter?

3. User Personas: Who will be using it and how?

These aren’t the only dimensions you might care about – you might also want to test different tones of voice, levels of technical sophistication, or even different locales and languages. The important thing is identifying dimensions that matter for your specific use case.

For a real estate CRM AI assistant I worked on with Rechat, we defined these dimensions like this:

features = [
    "property search",      # Finding listings matching criteria
    "market analysis",      # Analyzing trends and pricing
    "scheduling",          # Setting up property viewings
    "follow-up"           # Post-viewing communication
]

scenarios = [
    "exact match",         # One perfect listing match
    "multiple matches",    # Need to help user narrow down
    "no matches",         # Need to suggest alternatives
    "invalid criteria"     # Help user correct search terms
]

personas = [
    "first_time_buyer",    # Needs more guidance and explanation
    "investor",           # Focused on numbers and ROI
    "luxury_client",      # Expects white-glove service
    "relocating_family"   # Has specific neighborhood/school needs
]

But having these dimensions defined is only half the battle. The real challenge is ensuring your synthetic data actually triggers the scenarios you want to test. This requires two things:

1. A test database with enough variety to support your scenarios

2. A way to verify that generated queries actually trigger intended scenarios

For Rechat, we maintained a test database of listings that we knew would trigger different edge cases. Some teams prefer to use an anonymized copy of production data, but either way, you need to ensure your test data has enough variety to exercise the scenarios you care about.

Here’s an example of how we might use these dimensions with real data to generate test cases for the property search feature (this is just pseudo-code, and very illustrative):

def generate_search_query(scenario, persona, listing_db):
    """Generate a realistic user query about listings"""
    # Pull real listing data to ground the generation
    sample_listings = listing_db.get_sample_listings(
        price_range=persona.price_range,
        location=persona.preferred_areas
    )
    
    # Verify we have listings that will trigger our scenario
    if scenario == "multiple_matches" and len(sample_listings) < 2:
        raise ValueError("Need multiple listings for this scenario")
    if scenario == "no_matches" and len(sample_listings) > 0:
        raise ValueError("Found matches when testing no-match scenario")
    
    prompt = f"""
    You are an expert real estate agent who is searching for listings. You are given a customer type and a scenario.
    
    Your job is to generate a natural language query you would use to search these listings.
    
    Context:
    - Customer type: {persona.description}
    - Scenario: {scenario}
    
    Use these actual listings as reference:
    {format_listings(sample_listings)}
    
    The query should reflect the customer type and the scenario.

    Example query: Find homes in the 75019 zip code, 3 bedrooms, 2 bathrooms, price range $750k - $1M for an investor.
    """
    return generate_with_llm(prompt)

This produced realistic queries like:

Feature Scenario Persona Generated Query property search multiple matches first_time_buyer “Looking for 3-bedroom homes under $500k in the Riverside area. Would love something close to parks since we have young kids.” market analysis no matches investor “Need comps for 123 Oak St. Specifically interested in rental yield comparison with similar properties in a 2-mile radius.”

The key to useful synthetic data is grounding it in real system constraints. For the real-estate AI assistant, this means:

1. Using real listing IDs and addresses from their database

2. Incorporating actual agent schedules and availability windows

3. Respecting business rules like showing restrictions and notice periods

4. Including market-specific details like HOA requirements or local regulations

We then feed these test cases through Lucy and log the interactions. This gives us a rich dataset to analyze, showing exactly how the AI handles different situations with real system constraints. This approach helped us fix issues before they affected real users.

Sometimes you don’t have access to a production database, especially for new products. In these cases, use LLMs to generate both test queries and the underlying test data. For a real estate AI assistant, this might mean creating synthetic property listings with realistic attributes – prices that match market ranges, valid addresses with real street names, and amenities appropriate for each property type. The key is grounding synthetic data in real-world constraints to make it useful for testing. The specifics of generating robust synthetic databases are beyond the scope of this post.

Guidelines for Using Synthetic Data

When generating synthetic data, follow these key principles to ensure it’s effective:

1. Diversify your dataset: Create examples that cover a wide range of features, scenarios, and personas. As I wrote in my LLM-as-a-Judge post, this diversity helps you identify edge cases and failure modes you might not anticipate otherwise.

2. Generate user inputs, not outputs: Use LLMs to generate realistic user queries or inputs, not the expected AI responses. This prevents your synthetic data from inheriting the biases or limitations of the generating model.

3. Incorporate real system constraints: Ground your synthetic data in actual system limitations and data. For example, when testing a scheduling feature, use real availability windows and booking rules.

4. Verify scenario coverage: Ensure your generated data actually triggers the scenarios you want to test. A query intended to test “no matches found” should actually return zero results when run against your system.

5. Start simple, then add complexity: Begin with straightforward test cases before adding nuance. This helps isolate issues and establish a baseline before tackling edge cases.

This approach isn’t just theoretical – it’s been proven in production across dozens of companies. What often starts as a stopgap measure becomes a permanent part of the evaluation infrastructure, even after real user data becomes available.

Let’s look at how to maintain trust in your evaluation system as you scale…

5. Maintaining Trust In Evals Is Critical

This is a pattern I’ve seen repeatedly: teams build evaluation systems, then gradually lose faith in them. Sometimes it’s because the metrics don’t align with what they observe in production. Other times, it’s because the evaluations become too complex to interpret. Either way, the result is the same – the team reverts to making decisions based on gut feeling and anecdotal feedback, undermining the entire purpose of having evaluations.

Maintaining trust in your evaluation system is just as important as building it in the first place. Here’s how the most successful teams approach this challenge:

Understanding Criteria Drift

One of the most insidious problems in AI evaluation is “criteria drift” – a phenomenon where evaluation criteria evolve as you observe more model outputs. In their paper “Who Validates the Validators?”, Shankar et al. describe this phenomenon:

“To grade outputs, people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria.”

This creates a paradox: you can’t fully define your evaluation criteria until you’ve seen a wide range of outputs, but you need criteria to evaluate those outputs in the first place. In other words, it is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.

I’ve observed this firsthand when working with Phillip Carter at Honeycomb on their Query Assistant feature. As we evaluated the AI’s ability to generate database queries, Phillip noticed something interesting:

“Seeing how the LLM breaks down its reasoning made me realize I wasn’t being consistent about how I judged certain edge cases.”

The process of reviewing AI outputs helped him articulate his own evaluation standards more clearly. This isn’t a sign of poor planning – it’s an inherent characteristic of working with AI systems that produce diverse and sometimes unexpected outputs.

The teams that maintain trust in their evaluation systems embrace this reality rather than fighting it. They treat evaluation criteria as living documents that evolve alongside their understanding of the problem space. They also recognize that different stakeholders might have different (sometimes contradictory) criteria, and they work to reconcile these perspectives rather than imposing a single standard.

Creating Trustworthy Evaluation Systems

So how do you build evaluation systems that remain trustworthy despite criteria drift? Here are the approaches I’ve found most effective:

1. Favor Binary Decisions Over Arbitrary Scales

As I wrote in my LLM-as-a-Judge post, binary decisions provide clarity that more complex scales often obscure. When faced with a 1-5 scale, evaluators frequently struggle with the difference between a 3 and a 4, introducing inconsistency and subjectivity. What exactly distinguishes “somewhat helpful” from “helpful”? These boundary cases consume disproportionate mental energy and create noise in your evaluation data. And even when businesses use a 1-5 scale, they inevitably ask where to draw the line for “good enough” or to trigger intervention, forcing a binary decision anyway.

In contrast, a binary pass/fail forces evaluators to make a clear judgment: did this output achieve its purpose or not? This clarity extends to measuring progress – a 10% increase in passing outputs is immediately meaningful, while a 0.5-point improvement on a 5-point scale requires interpretation.

I’ve found that teams who resist binary evaluation often do so because they want to capture nuance. But nuance isn’t lost – it’s just moved to the qualitative critique that accompanies the judgment. The critique provides rich context about why something passed or failed, and what specific aspects could be improved, while the binary decision creates actionable clarity about whether improvement is needed at all.

2. Enhance Binary Judgments With Detailed Critiques

While binary decisions provide clarity, they work best when paired with detailed critiques that capture the nuance of why something passed or failed. This combination gives you the best of both worlds: clear, actionable metrics and rich contextual understanding.

For example, when evaluating a response that correctly answers a user’s question but contains unnecessary information, a good critique might read:

“The AI successfully provided the market analysis requested (PASS), but included excessive detail about neighborhood demographics that wasn’t relevant to the investment question. This makes the response longer than necessary and potentially distracting.”

These critiques serve multiple functions beyond just explanation. They force domain experts to externalize implicit knowledge – I’ve seen legal experts move from vague feelings that something “doesn’t sound right” to articulating specific issues with citation formats or reasoning patterns that can be systematically addressed.

When included as few-shot examples in judge prompts, these critiques improve the LLM’s ability to reason about complex edge cases. I’ve found this approach often yields 15-20% higher agreement rates between human and LLM evaluations compared to prompts without example critiques. The critiques also provide excellent raw material for generating high-quality synthetic data, creating a flywheel for improvement.

3. Measure Alignment Between Automated Evals and Human Judgment

If you’re using LLMs to evaluate outputs (which is often necessary at scale), it’s crucial to regularly check how well these automated evaluations align with human judgment.

This is particularly important given our natural tendency to over-trust AI systems. As Shankar et al. note in “Who Validates the Validators?”, the lack of tools to validate evaluator quality is concerning

Research shows people tend to over-rely and over-trust AI systems. For instance, in one high profile incident, researchers from MIT posted a pre-print on arXiv claiming that GPT-4 could ace the MIT EECS exam. Within hours, [the] work [was] debunked … citing problems arising from over-reliance on GPT-4 to grade itself.”

This over-trust problem extends beyond self-evaluation. Research has shown that LLMs can be biased by simple factors like the ordering of options in a set, or even seemingly innocuous formatting changes in prompts. Without rigorous human validation, these biases can silently undermine your evaluation system.

When working with Honeycomb, we tracked agreement rates between our LLM-as-a-judge and Phillip’s evaluations:

It took three iterations to achieve >90% agreement, but this investment paid off in a system the team could trust. Without this validation step, automated evaluations often drift from human expectations over time, especially as the distribution of inputs changes. You can read more about this here.

Tools like Eugene Yan’s AlignEval demonstrate this alignment process beautifully. It provides a simple interface where you upload data, label examples with a binary “good” or “bad,” and then evaluate LLM-based judges against those human judgments. What makes it effective is how it streamlines the workflow – you can quickly see where automated evaluations diverge from your preferences, refine your criteria based on these insights, and measure improvement over time. This approach reinforces that alignment isn’t a one-time setup but an ongoing conversation between human judgment and automated evaluation.

Scaling Without Losing Trust

As your AI system grows, you’ll inevitably face pressure to reduce the human effort involved in evaluation. This is where many teams go wrong – they automate too much, too quickly, and lose the human connection that keeps their evaluations grounded.

The most successful teams take a more measured approach:

1. Start with high human involvement: In the early stages, have domain experts evaluate a significant percentage of outputs.

2. Study alignment patterns: Rather than automating evaluation, focus on understanding where automated evaluations align with human judgment and where they diverge. This helps you identify which types of cases need more careful human attention.

3. Use strategic sampling: Rather than evaluating every output, use statistical techniques to sample outputs that provide the most information, particularly focusing on areas where alignment is weakest.

4. Maintain regular calibration: Even as you scale, continue to compare automated evaluations against human judgment regularly, using these comparisons to refine your understanding of when to trust automated evaluations.

Scaling evaluation isn’t just about reducing human effort – it’s about directing that effort where it adds the most value. By focusing human attention on the most challenging or informative cases, you can maintain quality even as your system grows.

Now that we’ve covered how to maintain trust in your evaluations, let’s talk about a fundamental shift in how you should approach AI development roadmaps…

6. Your AI Roadmap Should Count Experiments, Not Features

If you’ve worked in software development, you’re familiar with traditional roadmaps: a list of features with target delivery dates. Teams commit to shipping specific functionality by specific deadlines, and success is measured by how closely they hit those targets.

This approach fails spectacularly with AI.

I’ve watched teams commit to roadmaps like “Launch sentiment analysis by Q2” or “Deploy agent-based customer support by end of year,” only to discover that the technology simply isn’t ready to meet their quality bar. They either ship something subpar to hit the deadline or miss the deadline entirely. Either way, trust erodes.

The fundamental problem is that traditional roadmaps assume we know what’s possible. With conventional software, that’s often true – given enough time and resources, you can build most features reliably. With AI, especially at the cutting edge, you’re constantly testing the boundaries of what’s feasible.

Experiments vs. Features

Bryan Bischof, Former Head of AI at Hex, introduced me to what he calls a “capability funnel” approach to AI roadmaps. This strategy reframes how we think about AI development progress.

Instead of defining success as shipping a feature, the capability funnel breaks down AI performance into progressive levels of utility. At the top of the funnel is the most basic functionality – can the system respond at all? At the bottom is fully solving the user’s job to be done. Between these points are various stages of increasing usefulness.

For example, in a query assistant, the capability funnel might look like: 1. Can generate syntactically valid queries (basic functionality) 2. Can generate queries that execute without errors 3. Can generate queries that return relevant results 4. Can generate queries that match user intent 5. Can generate optimal queries that solve the user’s problem (complete solution)

This approach acknowledges that AI progress isn’t binary – it’s about gradually improving capabilities across multiple dimensions. It also provides a framework for measuring progress even when you haven’t reached the final goal.

The most successful teams I’ve worked with structure their roadmaps around experiments rather than features. Instead of committing to specific outcomes, they commit to a cadence of experimentation, learning, and iteration.

Eugene Yan, an applied scientist at Amazon, shared how he approaches ML project planning with leadership - a process that, while originally developed for traditional machine learning, applies equally well to modern LLM development:

“Here’s a common timeline. First, I take two weeks to do a data feasibility analysis, i.e”do I have the right data?” […] Then I take an additional month to do a technical feasibility analysis, i.e “can AI solve this?” After that, if it still works I’ll spend six weeks building a prototype we can A/B test.”

While LLMs might not require the same kind of feature engineering or model training as traditional ML, the underlying principle remains the same: time-box your exploration, establish clear decision points, and focus on proving feasibility before committing to full implementation. This approach gives leadership confidence that resources won’t be wasted on open-ended exploration, while giving the team the freedom to learn and adapt as they go.

The Foundation: Evaluation Infrastructure

The key to making an experiment-based roadmap work is having robust evaluation infrastructure. Without it, you’re just guessing whether your experiments are working. With it, you can rapidly iterate, test hypotheses, and build on successes.

I saw this firsthand during the early development of GitHub Copilot. What most people don’t realize is that the team invested heavily in building sophisticated offline evaluation infrastructure. They created systems that could test code completions against a very large corpus of repositories on GitHub, leveraging unit tests that already existed in high-quality codebases as an automated way to verify completion correctness. This was a massive engineering undertaking – they had to build systems that could clone repositories at scale, set up their environments, run their test suites, and analyze the results, all while handling the incredible diversity of programming languages, frameworks, and testing approaches.

This wasn’t wasted time—it was the foundation that accelerated everything. With solid evaluation in place, the team ran thousands of experiments, quickly identified what worked, and could say with confidence “this change improved quality by X%” instead of relying on gut feelings. While the upfront investment in evaluation feels slow, it prevents endless debates about whether changes help or hurt, and dramatically speeds up innovation later.

Communicating This to Stakeholders

The challenge, of course, is that executives often want certainty. They want to know when features will ship and what they’ll do. How do you bridge this gap?

The key is to shift the conversation from outputs to outcomes. Instead of promising specific features by specific dates, commit to a process that will maximize the chances of achieving the desired business outcomes.

Eugene shared how he handles these conversations:

“I try to reassure leadership with timeboxes. At the end of three months, if it works out, then we move it to production. At any step of the way, if it doesn’t work out, we pivot.”

This approach gives stakeholders clear decision points while acknowledging the inherent uncertainty in AI development. It also helps manage expectations about timelines – instead of promising a feature in six months, you’re promising a clear understanding of whether that feature is feasible in three months.

Bryan’s capability funnel approach provides another powerful communication tool. It allows teams to show concrete progress through the funnel stages, even when the final solution isn’t ready. It also helps executives understand where problems are occurring and make informed decisions about where to invest resources.

Build a Culture of Experimentation Through Failure Sharing

Perhaps the most counterintuitive aspect of this approach is the emphasis on learning from failures. In traditional software development, failures are often hidden or downplayed. In AI development, they’re the primary source of learning.

Eugene operationalizes this at his organization through what he calls a “fifteen-five” – a weekly update that takes fifteen minutes to write and five minutes to read:

“In my fifteen-fives, I document my failures and my successes. Within our team, we also have weekly”no-prep sharing sessions” where we discuss what we’ve been working on and what we’ve learned. When I do this, I go out of my way to share failures.”

This practice normalizes failure as part of the learning process. It shows that even experienced practitioners encounter dead ends, and it accelerates team learning by sharing those experiences openly. And by celebrating the process of experimentation rather than just the outcomes, teams create an environment where people feel safe taking risks and learning from failures.

A Better Way Forward

So what does an experiment-based roadmap look like in practice? Here’s a simplified example from a content moderation project Eugene worked on:

“I was asked to do content moderation. I said, ‘It’s uncertain whether we’ll meet that goal. It’s uncertain even if that goal is feasible with our data, or what machine learning techniques would work. But here’s my experimentation roadmap. Here are the techniques I’m gonna try, and I’m gonna update you at a two-week cadence.’”

The roadmap didn’t promise specific features or capabilities. Instead, it committed to a systematic exploration of possible approaches, with regular check-ins to assess progress and pivot if necessary.

The results were telling:

“For the first two to three months, nothing worked. […] And then [a breakthrough] came out. […] Within a month, that problem was solved. So you can see that in the first quarter or even four months, it was going nowhere. […] But then you can also see that all of a sudden, some new technology comes along, some new paradigm, some new reframing comes along that just [solves] 80% of [the problem].”

This pattern – long periods of apparent failure followed by breakthroughs – is common in AI development. Traditional feature-based roadmaps would have killed the project after months of “failure,” missing the eventual breakthrough.

By focusing on experiments rather than features, teams create space for these breakthroughs to emerge. They also build the infrastructure and processes that make breakthroughs more likely – data pipelines, evaluation frameworks, and rapid iteration cycles.

The most successful teams I’ve worked with start by building evaluation infrastructure before committing to specific features. They create tools that make iteration faster and focus on processes that support rapid experimentation. This approach might seem slower at first, but it dramatically accelerates development in the long run by enabling teams to learn and adapt quickly.

The key metric for AI roadmaps isn’t features shipped – it’s experiments run. The teams that win are those that can run more experiments, learn faster, and iterate more quickly than their competitors. And the foundation for this rapid experimentation is always the same: robust, trusted evaluation infrastructure that gives everyone confidence in the results.

By reframing your roadmap around experiments rather than features, you create the conditions for similar breakthroughs in your own organization.

Conclusion

Throughout this post, I’ve shared patterns I’ve observed across dozens of AI implementations. The most successful teams aren’t the ones with the most sophisticated tools or the most advanced models – they’re the ones that master the fundamentals of measurement, iteration, and learning.

The core principles are surprisingly simple:

1. Look at your data. Nothing replaces the insight gained from examining real examples. Error analysis consistently reveals the highest-ROI improvements.

2. Build simple tools that remove friction. Custom data viewers that make it easy to examine AI outputs yield more insights than complex dashboards with generic metrics.

3. Empower domain experts. The people who understand your domain best are often the ones who can most effectively improve your AI, regardless of their technical background.

4. Use synthetic data strategically. You don’t need real users to start testing and improving your AI. Thoughtfully generated synthetic data can bootstrap your evaluation process.

5. Maintain trust in your evaluations. Binary judgments with detailed critiques create clarity while preserving nuance. Regular alignment checks ensure automated evaluations remain trustworthy.

6. Structure roadmaps around experiments, not features. Commit to a cadence of experimentation and learning rather than specific outcomes by specific dates.

These principles apply regardless of your domain, team size, or technical stack. They’ve worked for companies ranging from early-stage startups to tech giants, across use cases from customer support to code generation.

Resources for Going Deeper

If you’d like to explore these topics further, here are some resources that might help:

• My blog for more content on AI evaluation and improvement. My other posts dive into more technical detail on topics such as constructing effective LLM judges, implementing evaluation systems, and other aspects of AI development¹. Also check out the blogs of Shreya Shankar and Eugene Yan who are also great sources of information on these topics.

• A course I’m teaching: Rapidly Improve AI Products With Evals, with Shreya Shankar. The course provides hands-on experience with techniques such as error analysis, synthetic data generation, and building trustworthy evaluation systems. It includes practical exercises and personalized instruction through office hours.

• If you’re looking for hands-on guidance specific to your organization’s needs, you can learn more about working with me at Parlance Labs.

Footnotes

1. I write more broadly about machine learning, AI, and software development. Some posts that expand on these topics include Your AI Product Needs Evals, Creating a LLM-as-a-Judge That Drives Business Results, and What We’ve Learned From A Year of Building with LLMs. You can see all my posts at hamel.dev.↩︎

People often find me through my writing on AI and tech. This creates an interesting pattern. Nearly every week, vendors reach out asking me to write about their products. While I appreciate their interest and love learning about new tools, I reserve my writing for topics that I have personal experience with.

One conversation last week really stuck with me. A founder confided, “We can write the best content in the world, but we don’t have any distribution.” This hit home because I used to think the same way.

Let me share what works for reaching developers. Companies and individuals alike often skip the basics when trying to grow their audience. These are proven approaches I’ve seen succeed, both in my work and in others’ efforts to grow their audience in the AI space.

1. Build on Great Work

Here’s something surprising: few people take the time to thoughtfully engage with others’ work in our field. But when you do, amazing things happen naturally.

For example, here are some recent posts I’ve enjoyed that present opportunities to engage with others:

• Shreya Shankar’s DocETL

• Eugene Yan’s work on AlignEval

• Ben Claive’s work on rerankers

• Jeremy Howard’s work on llms.txt

In the above examples, you could share how their ideas connect with what you’ve built. You could add additional case studies and real-world insights. If you deeply engage with someone’s work and add your insights, they often share your content with their audience. Not because you asked, but because you’ve added something meaningful to their work. Swyx has written a great post on how to do this effectively.

The key is authenticity. Don’t do this just for marketing—do it because you’re genuinely interested in learning from others and building on their ideas. It’s not hard to find things to be excited about. I’m amazed by how few people take this approach. It’s both effective and fun.

2. Show Up Consistently

I see too many folks blogging or posting once every few months and wondering why they’re not getting traction. Want to know what actually works? Look at Jason Liu. He grew his following from 500 to 30,000 followers by posting ~ 30 times every day for a year.

You don’t have to post that often (I certainly don’t!), but consistency matters more than perfection. Finally, don’t just post into the void. Engage with others. When someone comments on your post, reply thoughtfully. When you see conversations where you can add value, provide helpful information.

Finally, don’t be discouraged if you don’t see results immediately. Here’s some advice from my friend (and prolific writer), Eugene Yan:

In the beginning, when most people start writing, the output’s gonna suck. Harsh, but true—my first 100 posts or so were crap. But with practice, people can get better. But they have to be deliberate in wanting to practice and get better with each piece, and not just write for the sake of publishing something and tweeting about it. The Sam Parr course (see below) is a great example of deliberate practice on copywriting.

3. Get Better at Copywriting

This changed everything for me. I took Sam Parr’s copywriting course just 30 minutes a day for a week. Now I keep my favorite writing samples in a Claude project and reference them when I’m writing something important. Small improvements in how you communicate can make a huge difference in how your content lands.

One thing Sam teaches is that big words don’t make you sound smart. Clear writing that avoids jargon is more effective. That’s why Sam teaches aiming for a 6th-grade reading level. This matters even more with AI, as AI loves to generate flowery language and long sentences. The Hemingway App can be helpful in helping you simplify your writing.¹

4. Build a Voice-to-Content Pipeline

The struggle most people have with creating content is that it takes too much time. But it doesn’t have to if you build the right systems, especially with AI.

Getting this system right takes some upfront work, but the payoff is enormous. Start by installing a good voice-to-text app on your phone. I use either Superwhisper or VoicePal. VoicePal is great for prompting you to elaborate with follow-up questions. These tools let me capture ideas at their best. That’s usually when I’m walking outside or away from my computer. At my computer, I use Flow.

The key is to carefully craft your first few pieces of content. These become examples for your prompts that teach AI your style and tone. Once you have high-quality examples, you can organize these (transcript, content) pairs and feed them to language models. The in-context learning creates remarkably aligned output that matches your writing style while maintaining the authenticity of your original thoughts.

For example, I use this pipeline at Answer AI. We have started interviewing each other and using the recordings as grounding for blog posts. Our recent post about SolveIt shows this in action. The raw conversation is the foundation. Our workflow turns it into polished content.

I’ve also integrated this workflow into my meetings. Using CircleBack, my favorite AI note-taking app, I can automatically capture and process meeting discussions. You can set up workflows to send your meeting notes and transcripts to AI for processing. This turns conversations into content opportunities.

The real power comes from having all these pieces working together. Voice capture, AI, and automation makes content creation fun and manageable.

5. Leverage Your Unique Perspective

Through my consulting work, I notice patterns that others miss. My most popular posts address common problems my clients had. When everyone’s confused about a topic, especially in AI where there’s lots of hype, clear explanations are gold. This is the motivation for some of my blog posts like:

• Fuck You, Show Me The Prompt

• Your AI Product Needs Evals

• Creating a LLM-as-a-Judge That Drives Business Results

You probably see patterns too. Maybe it’s common questions from customers, or problems you’ve solved repeatedly. Maybe you work with a unique set of technologies or interesting use cases. Share these insights! Your unique perspective is more valuable than you think.

6. Use High Quality Social Cards, Threads, and Scheduling

This is probably the least important part of the process, but it’s still important. Thumbnails and social cards are vital for visibility on social media. Here are the tools I use:

• socialsharepreview.com to check how your content looks on different platforms. For X, I sometimes use the Twitter Card Validator.

• chatGPT to create cover images for my posts. Then, paste them into Canva to size and edit them. Some of my friends use ideogram, which generates images with text accurately.

• Canva for the last mile of creating social cards. They have easy-to-use buttons to ensure you get the dimensions right. They also have inpainting, background removal, and more.

• If using X, social cards can be a bit fiddly. As of this writing, they do not show your post title, just the image if using the large-image size. To mitigate this,I use Canva to write the post’s title in the image like this.

• Social media can be distracting, so I like to schedule my posts in advance. I use typefully for this purpose. Some of my friends use hypefury.

Finally, when posting on X, threads can be a great way to raise the visibility of your content. A simple approach is to take screenshots or copy-paste snippets of your content. Then, walk through them in a thread, as you would want a reader to. Jeremy Howard does a great job at this: example 1, example 2.

The Content Flywheel: Putting It All Together

Once you have these systems in place, something magical happens: content creates more content. Your blog posts spawn social media updates. Your conversations turn into newsletters. Your client solutions become case studies. Each piece of work feeds the next, creating a natural flywheel.

Don’t try to sell too hard. Instead, share real insights and helpful information. Focus on adding value and educating your audience. When you do this well, people will want to follow your work.

This journey is different for everyone. These are just the patterns I’ve seen work in my consulting practice and my own growth. Try what feels right. Adjust what doesn’t.

P.S. If you’d like to follow my writing journey, you can stay connected here.

Further Reading

• Simon Willison’s Posts on Writing

• Eugene’s Posts on Writing

• Why you, (yes, you) should blog

Footnotes

1. Don’t abuse these tools or use them blindly. There’s plenty of situations where you should not be writing at a 6th-grade reading level. This includes, humor, poetry, shitposting, and more. Even formal writing shouldn’t adhere strictly to this rule. It’s advice that you should judge on a case-by-case basis. When you simplify your writing - do you like it more?↩︎

The Problem: AI Teams Are Drowning in Data

Ever spend weeks building an AI system, only to realize you have no idea if it’s actually working? You’re not alone. I’ve noticed teams repeat the same mistakes when using LLMs to evaluate AI outputs:

1. Too Many Metrics: Creating numerous measurements that become unmanageable.

2. Arbitrary Scoring Systems: Using uncalibrated scales (like 1-5) across multiple dimensions, where the difference between scores is unclear and subjective. What makes something a 3 versus a 4? Nobody knows, and different evaluators often interpret these scales differently.

3. Ignoring Domain Experts: Not involving the people who understand the subject matter deeply.

4. Unvalidated Metrics: Using measurements that don’t truly reflect what matters to the users or the business.

The result? Teams end up buried under mountains of metrics or data they don’t trust and can’t use. Progress grinds to a halt. Everyone gets frustrated.

For example, it’s not uncommon for me to see dashboards that look like this:

Tracking a bunch of scores on a 1-5 scale is often a sign of a bad eval process (I’ll discuss why later). In this post, I’ll show you how to avoid these pitfalls. The solution is to use a technique that I call “Critique Shadowing”. Here’s how to do it, step by step.

Step 1: Find The Principal Domain Expert

In most organizations there is usually one (maybe two) key individuals whose judgment is crucial for the success of your AI product. These are the people with deep domain expertise or represent your target users. Identifying and involving this Principal Domain Expert early in the process is critical.

Why is finding the right domain expert so important?

• They Set the Standard: This person not only defines what is acceptable technically, but also helps you understand if you’re building something users actually want.

• Capture Unspoken Expectations: By involving them, you uncover their preferences and expectations, which they might not be able to fully articulate upfront. Through the evaluation process, you help them clarify what a “passable” AI interaction looks like.

• Consistency in Judgment: People in your organization may have different opinions about the AI’s performance. Focusing on the principal expert ensures that evaluations are consistent and aligned with the most critical standards.

• Sense of Ownership: Involving the expert gives them a stake in the AI’s development. They feel invested because they’ve had a hand in shaping it. In the end, they are more likely to approve of the AI.

Examples of Principal Domain Experts:

• A psychologist for a mental health AI assistant.

• A lawyer for an AI that analyzes legal documents.

• A customer service director for a support chatbot.

• A lead teacher or curriculum developer for an educational AI tool.

Exceptions

In a smaller company, this might be the CEO or founder. If you are an independent developer, you should be the domain expert (but be honest with yourself about your expertise).

If you must rely on leadership, you should regularly validate their assumptions against real user feedback.

Many developers attempt to act as the domain expert themselves, or find a convenient proxy (ex: their superior). This is a recipe for disaster. People will have varying opinions about what is acceptable, and you can’t make everyone happy. What’s important is that your principal domain expert is satisfied.

Remember: This doesn’t have to take a lot of the domain expert’s time. Later in this post, I’ll discuss how you can make the process efficient. Their involvement is absolutely critical to the AI’s success.

Next Steps

Once you’ve found your expert, we need to give them the right data to review. Let’s talk about how to do that next.

Step 2: Create a Dataset

With your principal domain expert on board, the next step is to build a dataset that captures problems that your AI will encounter. It’s important that the dataset is diverse and represents the types of interactions that your AI will have in production.

Why a Diverse Dataset Matters

• Comprehensive Testing: Ensures your AI is evaluated across a wide range of situations.

• Realistic Interactions: Reflects actual user behavior for more relevant evaluations.

• Identifies Weaknesses: Helps uncover areas where the AI may struggle or produce errors.

Dimensions for Structuring Your Dataset

You want to define dimensions that make sense for your use case. For example, here are ones that I often use for B2C applications:

1. Features: Specific functionalities of your AI product.

2. Scenarios: Situations or problems the AI may encounter and needs to handle.

3. Personas: Representative user profiles with distinct characteristics and needs.

Examples of Features, Scenarios, and Personas

Features

Feature Description Email Summarization Condensing lengthy emails into key points. Meeting Scheduler Automating the scheduling of meetings across time zones. Order Tracking Providing shipment status and delivery updates. Contact Search Finding and retrieving contact information from a database. Language Translation Translating text between languages. Content Recommendation Suggesting articles or products based on user interests.

Scenarios

Scenarios are situations the AI needs to handle, (not based on the outcome of the AI’s response).

Scenario Description Multiple Matches Found User’s request yields multiple results that need narrowing down. For example: User asks “Where’s my order?” but has three active orders (#123, #124, #125). AI must help identify which specific order they’re asking about. No Matches Found User’s request yields no results, requiring alternatives or corrections. For example: User searches for order #ABC-123 which doesn’t exist. AI should explain valid order formats and suggest checking their confirmation email. Ambiguous Request User input lacks necessary specificity. For example: User says “I need to change my delivery” without specifying which order or what aspect of delivery (date, address, etc.) they want to change. Invalid Data Provided User provides incorrect data type or format. For example: User tries to track a return using a regular order number instead of a return authorization (RMA) number. System Errors Technical issues prevent normal operation. For example: While looking up an order, the inventory database is temporarily unavailable. AI needs to explain the situation and provide alternatives. Incomplete Information User omits required details. For example: User wants to initiate a return but hasn’t provided the order number or reason. AI needs to collect this information step by step. Unsupported Feature User requests functionality that doesn’t exist. For example: User asks to change payment method after order has shipped. AI must explain why this isn’t possible and suggest alternatives.

Personas

Persona Description New User Unfamiliar with the system; requires guidance. Expert User Experienced; expects efficiency and advanced features. Non-Native Speaker May have language barriers; uses non-standard expressions. Busy Professional Values quick, concise responses; often multitasking. Technophobe Uncomfortable with technology; needs simple instructions. Elderly User May not be tech-savvy; requires patience and clear guidance.

This taxonomy is not universal

This taxonomy (features, scenarios, personas) is not universal. For example, it may not make sense to even have personas if users aren’t directly engaging with your AI. The idea is you should outline dimensions that make sense for your use case and generate data that covers them. You’ll likely refine these after the first round of evaluations.

Generating Data

To build your dataset, you can:

• Use Existing Data: Sample real user interactions or behaviors from your AI system.

• Generate Synthetic Data: Use LLMs to create realistic user inputs covering various features, scenarios, and personas.

Often, you’ll do a combination of both to ensure comprehensive coverage. Synthetic data is not as good as real data, but it’s a good starting point. Also, we are only using LLMs to generate the user inputs, not the LLM responses or internal system behavior.

Regardless of whether you use existing data or synthetic data, you want good coverage across the dimensions you’ve defined.

Incorporating System Information

When making test data, use your APIs and databases where appropriate. This will create realistic data and trigger the right scenarios. Sometimes you’ll need to write simple programs to get this information. That’s what the “Assumptions” column is referring to in the examples below.

Example LLM Prompts for Generating User Inputs

Here are some example prompts that illustrate how to use an LLM to generate synthetic user inputs for different combinations of features, scenarios, and personas:

ID Feature Scenario Persona LLM Prompt to Generate User Input Assumptions (not directly in the prompt) 1 Order Tracking Invalid Data Provided Frustrated Customer “Generate a user input from someone who is clearly irritated and impatient, using short, terse language to demand information about their order status for order number #1234567890. Include hints of previous negative experiences.” Order number #1234567890 does not exist in the system. 2 Contact Search Multiple Matches Found New User “Create a user input from someone who seems unfamiliar with the system, using hesitant language and asking for help to find contact information for a person named ‘Alex’. The user should appear unsure about what information is needed.” Multiple contacts named ‘Alex’ exist in the system. 3 Meeting Scheduler Ambiguous Request Busy Professional “Simulate a user input from someone who is clearly in a hurry, using abbreviated language and minimal details to request scheduling a meeting. The message should feel rushed and lack specific information.” N/A 4 Content Recommendation No Matches Found Expert User “Produce a user input from someone who demonstrates in-depth knowledge of their industry, using specific terminology to request articles on sustainable supply chain management. Use the information in this article involving sustainable supply chain management to formulate a plausible query: {{article}}” No articles on ‘Emerging trends in sustainable supply chain management’ exist in the system.

Generating Synthetic Data

When generating synthetic data, you only need to create the user inputs. You then feed these inputs into your AI system to generate the AI’s responses. It’s important that you log everything so you can evaluate your AI. To recap, here’s the process:

1. Generate User Inputs: Use the LLM prompts to create realistic user inputs.

2. Feed Inputs into Your AI System: Input the user interactions into your AI as it currently exists.

3. Capture AI Responses: Record the AI’s responses to form complete interactions.

4. Organize the Interactions: Create a table to store the user inputs, AI responses, and relevant metadata.

How much data should you generate?

There is no right answer here. At a minimum, you want to generate enough data so that you have examples for each combination of dimensions (in this toy example: features, scenarios, and personas). However, you also want to keep generating more data until you feel like you have stopped seeing new failure modes. The amount of data I generate varies significantly depending on the use case.

Does synthetic data actually work?

You might be skeptical of using synthetic data. After all, it’s not real data, so how can it be a good proxy? In my experience, it works surprisingly well. Some of my favorite AI products, like Hex use synthetic data to power their evals:

“LLMs are surprisingly good at generating excellent - and diverse - examples of user prompts. This can be relevant for powering application features, and sneakily, for building Evals. If this sounds a bit like the Large Language Snake is eating its tail, I was just as surprised as you! All I can say is: it works, ship it.” Bryan Bischof, Head of AI Engineering at Hex

Next Steps

With your dataset ready, now comes the most important part: getting your principal domain expert to evaluate the interactions.

Step 3: Direct The Domain Expert to Make Pass/Fail Judgments with Critiques

The domain expert’s job is to focus on one thing: “Did the AI achieve the desired outcome?” No complex scoring scales or multiple metrics. Just a clear pass or fail decision. In addition to the pass/fail decision, the domain expert should write a critique that explains their reasoning.

Why are simple pass/fail metrics important?

• Clarity and Focus: A binary decision forces everyone to consider what truly matters. It simplifies the evaluation to a single, crucial question.

• Actionable Insights: Pass/fail judgments are easy to interpret and act upon. They help you quickly identify whether the AI meets the user’s needs.

• Forces Articulation of Expectations: When domain experts must decide if an interaction passes or fails, they are compelled to articulate their expectations clearly. This process uncovers nuances and unspoken assumptions about how the AI should behave.

• Efficient Use of Resources: Keeps the evaluation process manageable, especially when starting out. You avoid getting bogged down in detailed metrics that might not be meaningful yet.

The Role of Critiques

Alongside a binary pass/fail judgment, it’s important to write a detailed critique of the LLM-generated output. These critiques:

• Captures Nuances: The critique allows you to note if something was mostly correct but had areas for improvement.

• Guide Improvement: Detailed feedback provides specific insights into how the AI can be enhanced.

• Balance Simplicity with Depth: While the pass/fail offers a clear verdict, the critique offers the depth needed to understand the reasoning behind the judgment.

Why Write Critiques?:

In practice, domain experts may not have fully internalized all the judgment criteria. By forcing them to make a pass/fail decision and explain their reasoning, they clarify their expectations and provide valuable guidance for refining the AI.

“But my problem is complex!” Trust me—starting simple forces you to focus on what truly matters. You can introduce more complexity later if needed.

Examples of Good Critiques

To illustrate how simple pass/fail judgments combined with detailed critiques work in practice, here’s a table showcasing examples of user interactions with an AI assistant. The table includes both passes and fails, with critiques explaining why the AI received that judgment. In cases where the AI passed despite critical issues, the critique highlights these aspects and justifies why it still passed overall. For failed interactions, the critique explains the critical elements that led to the failure.

User Interaction with AI Judgment Critique User: “I need to cancel my flight booking for next week.”
AI: “Your flight booking for next week has been canceled. A confirmation email has been sent to you.” Pass The AI successfully executed the user’s request by canceling the flight and sending a confirmation, achieving the primary goal. However, it should have asked for confirmation before proceeding to prevent accidental cancellations. Despite this critical oversight, the AI fulfilled the user’s request, so it passes, but adding a confirmation step would improve user safety. User: “What’s the capital of France?”
AI: “The capital of France is Paris.” Pass The AI correctly answered the user’s question, providing accurate information promptly. Although the response is sufficient, the AI could enhance the interaction by offering additional interesting facts or asking if the user has more questions. Despite this opportunity for added engagement, the AI met the user’s primary need, so it passes. User: “I can’t log into my account; it says my password is incorrect.”
AI: “Please reset your password by clicking ‘Forgot Password’.” Fail The AI provided a generic solution without acknowledging the user’s frustration or checking for other issues like account lockout. It failed to offer empathetic support or additional assistance. Due to the lack of personalized help and poor user experience, it fails. User: “Book a table for two at an Italian restaurant tonight at 7 PM.”
AI: “Table for two at Luigi’s Italian Restaurant booked for tonight at 7 PM. Confirmation number is 12345.” Pass The AI efficiently completed the booking as requested, providing all necessary details and a confirmation number. However, it could have asked if the user had seating preferences or dietary restrictions. The AI accomplished the primary task successfully, so it passes, with minor opportunities for enhanced service. User: “I need help with my billing statement.”
AI: “Please provide your account number and full name.” Fail The AI requested sensitive information without verifying the user’s identity through secure means, potentially compromising security. It should guide the user through a secure authentication process first. Due to this critical oversight in user data protection, it fails.

These examples demonstrate how the AI can receive both “Pass” and “Fail” judgments. In the critiques:

• For passes, we explain why the AI succeeded in meeting the user’s primary need, even if there were critical aspects that could be improved. We highlight these areas for enhancement while justifying the overall passing judgment.

• For fails, we identify the critical elements that led to the failure, explaining why the AI did not meet the user’s main objective or compromised important factors like user experience or security.

Most importantly, the critique should be detailed enough so that you can use it in a few-shot prompt for a LLM judge. In other words, it should be detailed enough that a new employee could understand it. Being too terse is a common mistake.

Note that the example user interactions with the AI are simplified for brevity - but you might need to give the domain expert more context to make a judgement. More on that later.

Note

At this point, you don’t need to perform a root cause analysis into the technical reasons behind why the AI failed. Many times, it’s useful to get a sense of overall behavior before diving into the weeds.

Don’t stray from binary pass/fail judgments when starting out

A common mistake is straying from binary pass/fail judgments. Let’s revisit the dashboard from earlier:

If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you’re doing it wrong. Let’s unpack why.

1. It’s not actionable: People don’t know what to do with a 3 or 4. It’s not immediately obvious how this number is better than a 2. You need to be able to say “this interaction passed because…” and “this interaction failed because…”.

2. More often than not, these metrics do not matter. Every time I’ve analyzed data on domain expert judgments, they tend not to correlate with these kind of metrics. By having a domain expert make a binary judgment, you can figure out what truly matters.

This is why I hate off the shelf metrics that come with many evaluation frameworks. They tend to lead people astray.

Common Objections to Pass/Fail Judgments:

• “The business said that these 8 dimensions are important, so we need to evaluate all of them.”

• “We need to be able to say why an interaction passed or failed.”

I can guarantee you that if someone says you need to measure 8 things on a 1-5 scale, they don’t know what they are looking for. They are just guessing. You have to let the domain expert drive and make a pass/fail judgment with critiques so you can figure out what truly matters. Stand your ground here.

Make it easy for the domain expert to review data

Finally, you need to remove all friction from reviewing data. I’ve written about this here. Sometimes, you can just use a spreadsheet. It’s a judgement call in terms of what is easiest for the domain expert. I found that I often have to provide additional context to help the domain expert understand the user interaction, such as:

• Metadata about the user, such as their location, subscription tier, etc.

• Additional context about the system, such as the current time, inventory levels, etc.

• Resources so you can check if the AI’s response is correct (ex: ability to search a database, etc.)

All of this data needs to be presented on a single screen so the domain expert can review it without jumping through hoops. That’s why I recommend building a simple web app to review data.

How many examples do you need?

The number of examples you need depends on the complexity of the task. My heuristic is that I start with around 30 examples and keep going until I do not see any new failure modes. From there, I keep going until I’m not learning anything new.

Next, we’ll look at how to use this data to build an LLM judge.

Step 4: Fix Errors

After looking at the data, it’s likely you will find errors in your AI system. Instead of plowing ahead and building an LLM judge, you want to fix any obvious errors. Remember, the whole point of the LLM as a judge is to help you find these errors, so it’s totally fine if you find them earlier!

If you have already developed Level 1 evals as outlined in my previous post, you should not have any pervasive errors. However, these errors can sometimes slip through the cracks. If you find pervasive errors, fix them and go back to step 3. Keep iterating until you feel like you have stabilized your system.

Step 5: Build Your LLM as A Judge, Iteratively

The Hidden Power of Critiques

You cannot write a good judge prompt until you’ve seen the data. The paper from Shankar et al., “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences” summarizes this well:

to grade outputs,people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria. We dub this phenomenon criteria drift, and it implies thatit is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.

Start with Expert Examples

Let me share a real-world example of building an LLM judge you can apply to your own use case. When I was helping Honeycomb build their Query Assistant feature, we needed a way to evaluate if the AI was generating good queries. Here’s what our LLM judge prompt looked like, including few-shot examples of critiques from our domain expert, Phillip:

Notice how each example includes:

1. The natural language query (NLQ) in <nlq> tags

2. The generated query in <query> tags

3. The critique and outcome in <critique> tags

In the prompt above, the example critiques are fixed. An advanced approach is to include examples dynamically based upon the item you are judging. You can learn more in this post about Continual In-Context Learning.

Keep Iterating on the Prompt Until Convergence With Domain Expert

In this case, I used a low-tech approach to iterate on the prompt. I sent Phillip a spreadsheet with the following information:

1. The NLQ

2. The generated query

3. The critique

4. The outcome (pass or fail)

Phillip would then fill out his own version of the spreadsheet with his critiques. I used this to iteratively improve the prompt. The spreadsheet looked like this:

I also tracked agreement rates over time to ensure we were converging on a good prompt.

It took us only three iterations to achieve > 90% agreement between the LLM and Phillip. Your mileage may vary depending on the complexity of the task. For example, Swyx has conducted a similar process hundreds of times for AI News, an extremely popular news aggregator with high quality recommendations. The quality of the AI owing to this process is why this product has received critical acclaim.

How to Optimize the LLM Judge Prompt?

I usually adjust the prompts by hand. I haven’t had much luck with prompt optimizers like DSPy. However, my friend Eugene Yan has just released a promising tool named ALIGN Eval. I like it because it’s simple and effective. Also, don’t forget the approach of continual in-context learning mentioned earlier - it can be effective when implemented correctly.

In rare cases, I might fine-tune a judge, but I prefer not to. I talk about this more in the FAQ section.

The Human Side of the Process

Something unexpected happened during this process. Phillip Carter, our domain expert at Honeycomb, found that reviewing the LLM’s critiques helped him articulate his own evaluation criteria more clearly. He said,

“Seeing how the LLM breaks down its reasoning made me realize I wasn’t being consistent about how I judged certain edge cases.”

This is a pattern I’ve seen repeatedly—the process of building an LLM judge often helps standardize evaluation criteria.

Furthermore, because this process forces the domain expert to look at data carefully, I always uncover new insights about the product, AI capabilities, and user needs. The resulting benefits are often more valuable than creating a LLM judge!

How Often Should You Evaluate?

I conduct this human review at regular intervals and whenever something material changes. For example, if I update a model, I’ll run the process again. I don’t get too scientific here; instead, I rely on my best judgment. Also note that after the first two iterations, I tend to focus more on errors rather than sampling randomly. For example, if I find an error, I’ll search for more examples that I think might trigger the same error. However, I always do a bit of random sampling as well.

What if this doesn’t work?

I’ve seen this process fail when:

• The AI is overscoped: Example - a chatbot in a SaaS product that promises to do anything you want.

• The process is not followed correctly: Not using the principal domain expert, not writing proper critiques, etc.

• The expectations of alignment are unrealistic or not feasible.

In each of these cases, I try to address the root cause instead of trying to force alignment. Sometimes, you may not be able to achieve the alignment you want and may have to lean heavier on human annotations. However, after following the process described here, you will have metrics that help you understand how much you can trust the LLM judge.

Mistakes I’ve noticed in LLM judge prompts

Most of the mistakes I’ve seen in LLM judge prompts have to do with not providing good examples:

1. Not providing any critiques.

2. Writing extremely terse critiques.

3. Not providing external context. Your examples should contain the same information you use to evaluate, including external information like user metadata, system information etc.

4. Not providing diverse examples. You need a wide variety of examples to ensure that your judge works for a wide variety of inputs.

Sometimes, you may encounter difficulties with fitting everything you need into the prompt, and may have to get creative about how you structure the examples. However, this is becoming less of an issue thanks to expanding context windows and prompt caching.

Step 6: Perform Error Analysis

After you have created a LLM as a judge, you will have a dataset of user interactions with the AI, and the LLM’s judgments. If your metrics show an acceptable agreement between the domain expert and the LLM judge, you can apply this judge against real or synthetic interactions. After this, you can you calculate error rates for different dimensions of your data. You should calculate the error on unseen data only to make sure your aren’t getting biased results.

For example, if you have segmented your data by persona, scenario, feature, etc, your data analysis may look like this

Error Rates by Key Dimensions

Feature Scenario Persona Total Examples Failure Rate Order Tracking Multiple Matches New User 42 24.3% Order Tracking Multiple Matches Expert User 38 18.4% Order Tracking No Matches Expert User 30 23.3% Order Tracking No Matches New User 20 75.0% Contact Search Multiple Matches New User 35 22.9% Contact Search Multiple Matches Expert User 32 19.7% Contact Search No Matches New User 25 68.0% Contact Search No Matches Expert User 28 21.4%

Classify Traces

Once you know where the errors are now you can perform an error analysis to get to the root cause of the errors. My favorite way is to look at examples of each type of error and classify them by hand. I recommend using a spreadsheet for this. For example, a trace for Order tracking where there are no matches for new users might look like this:

Example Trace

In this example trace, the user provides an invalid order number. The AI correctly identifies that the order number is invalid but provides an unhelpful response. If you are not familiar with logging LLM traces, refer to my previous post on evals.

Note that this trace is formatted for readability.

{
 "user_input": "Where's my order #ABC123?",
 "function_calls": [
   {
     "name": "search_order_database",
     "args": {"order_id": "ABC123"},
     "result": {
       "status": "not_found",
       "valid_patterns": ["XXX-XXX-XXX"]
     }
   },
   {
     "name": "retrieve_context",
     "result": {
       "relevant_docs": [
         "Order numbers follow format XXX-XXX-XXX",
         "New users should check confirmation email"
       ]
     }
   }
 ],
 "llm_intermediate_steps": [
   {
     "thought": "User is new and order format is invalid",
     "action": "Generate help message with format info"
   }
 ],
 "final_response": "I cannot find that order #. Please check the number and try again."
}

In this case, you might classify the error as: Missing User Education. The system retrieved new user context and format information but failed to include it in the response, which suggests we could improve our prompt. After you have classified a number of errors, you can calculate the distribution of errors by root cause. That might look like this:

Root Cause Distribution (20 Failed Interactions)

Root Cause Count Percentage Missing User Education 8 40% Authentication/Access Issues 6 30% Poor Context Handling 4 20% Inadequate Error Messages 2 10%

Now you know where to focus your efforts. This doesn’t have to take an extraordinary amount of time. You can get quite far in just 15 minutes. Also, you can use a LLM to help you with this classification, but that is beyond the scope of this post (you can use a LLM to help you do anything in this post, as long as you have a process to verify the results).

An Interactive Walkthrough of Error Analysis

Error analysis has been around in Machine Learning for quite some time. This video by Andrew Ng does a great job of walking through the process interactively:

Fix Your Errors, Again

Now that you have a sense of the errors, you can go back and fix them again. Go back to step 3 and iterate until you are satisfied. Note that every time you fix an error, you should try to write a test case for it. Sometimes, this can be an assertion in your test suite, but other times you may need to create a more “specialized” LLM judge for these failures. We’ll talk about this next.

Doing this well requires data literacy

Investigating your data is much harder in practice than I made it look in this post. It requires a nose for data that only comes from practice. It also helps to have some basic familiarity with statistics and data analysis tools. My favorite post on data literacy is this one by Jason Liu and Eugene Yan.

Step 7: Create More Specialized LLM Judges (if needed)

Now that you have a sense for where the problems in your AI are, you can decide where and if to invest in more targeted LLM judges. For example, if you find that the AI has trouble with citing sources correctly, you can created a targeted eval for that. You might not even need a LLM judge for some errors (and use a code-based assertion instead).

The key takeaway is don’t jump directly to using specialized LLM judges until you have gone through this critique shadowing process. This will help you rationalize where to invest your time.

Recap of Critique Shadowing

Using an LLM as a judge can streamline your AI evaluation process if approached correctly. Here’s a visual illustration of the process (there is a description of the process below the diagram as well):

The Critique Shadowing process is iterative, with feedback loops. Let’s list out the steps:

1. Find Principal Domain Expert

2. Create A Dataset

   • Generate diverse examples covering your use cases

   • Include real or synthetic user interactions

3. Domain Expert Reviews Data

   • Expert makes pass/fail judgments

   • Expert writes detailed critiques explaining their reasoning

4. Fix Errors (if found)

   • Address any issues discovered during review

   • Return to expert review to verify fixes

   • Go back to step 3 if errors are found

5. Build LLM Judge

   • Create prompt using expert examples

   • Test against expert judgments

   • Refine prompt until agreement is satisfactory

6. Perform Error Analysis

   • Calculate error rates across different dimensions

   • Identify patterns and root causes

   • Fix errors and go back to step 3 if needed

   • Create specialized judges as needed

This process never truly ends. It repeats periodically or when material changes occur.

It’s Not The Judge That Created Value, After all

The real value of this process is looking at your data and doing careful analysis. Even though an AI judge can be a helpful tool, going through this process is what drives results. I would go as far as saying that creating a LLM judge is a nice “hack” I use to trick people into carefully looking at their data!

That’s right. The real business value comes from looking at your data. But hey, potato, potahto.

Do You Really Need This?

Phew, this seems like a lot of work! Do you really need this? Well, it depends. There are cases where you can take a shortcut through this process. For example, let’s say:

1. You are an independent developer who is also a domain expert.

2. You are working with test data that already available. (Tweets, etc.)

3. Looking at data is not costly (etc. you can manually look at enough data in a few hours)

In this scenario, you can jump directly to something that looks like step 3 and start looking at data right away. Also, since it’s not that costly to look at data, it’s probably fine to just do error analysis without a judge (at least initially). You can incorporate what you learn directly back into your primary model right away. This example is not exhaustive, but gives you an idea of how you can adapt this process to your needs.

However, you can never completely eliminate looking at your data! This is precisely the step that most people skip. Don’t be that person.

FAQ

I received a lot of questions about this topic. Here are answers to the most common ones:

If I have a good judge LLM, isn’t that also the LLM I’d also want to use?

Effective judges often use larger models or more compute (via longer prompts, chain-of-thought, etc.) than the systems they evaluate.

However, If the cost of the most powerful LLM is not prohibitive, and latency is not an issue, then you might want to consider where you invest your efforts differently. In this case, it might make sense to put more effort towards specialist LLM judges, code-based assertions, and A/B testing. However, you should still go through the process of looking at data and critiquing the LLM’s output before you adopt specialized judges.

Do you recommend fine-tuning judges?

I prefer not to fine-tune LLM judges. I’d rather spend the effort fine-tuning the actual LLM instead. However, fine-tuning guardrails or other specialized judges can be useful (especially if they are small classifiers).

As a related note, you can leverage a LLM judge to curate and transform data for fine-tuning your primary model. For example, you can use the judge to:

• Eliminate bad examples for fine-tuning.

• Generate higher quality outputs (by referencing the critique).

• Simulate high quality chain-of-thought with critiques.

Using a LLM judge for enhancing fine-tuning data is even more compelling when you are trying to distill a large LLM into a smaller one. The details of fine-tuning are beyond the scope of this post. If you are interested in learning more, see these resources.

What’s wrong with off-the-shelf LLM judges?

Nothing is strictly wrong with them. It’s just that many people are led astray by them. If you are disciplined you can apply them to your data and see if they are telling you something valuable. However, I’ve found that these tend to cause more confusion than value.

How Do you evaluate the LLM judge?

You will collect metrics on the agreement between the domain expert and the LLM judge. This tells you how much you can trust the judge and in what scenarios. Your domain expert doesn’t have to inspect every single example, you just need a representative sample so you can have reliable statistics.

What model do you use for the LLM judge?

For the kind of judge articulated in this blog post, I like to use the most powerful model I can afford in my cost/latency budget. This budget might be different than my primary model, depending on the number of examples I need to score. This can vary significantly according to the use case.

What about guardrails?

Guardrails are a separate but related topic. They are a way to prevent the LLM from saying/doing something harmful or inappropriate. This blog post focuses on helping you create a judge that’s aligned with business goals, especially when starting out.

I’m using LLM as a judge, and getting tremendous value but I didn’t follow this approach.

I believe you. This blog post is not the only way to use a LLM as a judge. In fact, I’ve seen people use a LLM as a judge in all sorts of creative ways, which include ranking, classification, model selection and so-on. I’m focused on an approach that works well when you are getting started, and avoids the pitfalls of confusing metric sprawl. However, the general process of looking at the data is still central no matter what kind of judge you are building.

How do you choose between traditional ML techniques, LLM-as-a-judge and human annotations?

The answer to this (and many other questions) is: do the simplest thing that works. And simple doesn’t always mean traditional ML techniques. Depending on your situation, it might be easier to use a LLM API as a classifier than to train a model and deploy it.

Can you make judges from small models?

Yes, potentially. I’ve only used the larger models for judges. You have to base the answer to this question on the data (i.e. the agreement with the domain expert).

How do you ensure consistency when updating your LLM model?

You have to go through the process again and measure the results.

How do you phase out human in the loop to scale this?

You don’t need a domain expert to grade every single example. You just need a representative sample. I don’t think you can eliminate humans completely, because the LLM still needs to be aligned to something, and that something is usually a human. As your evaluation system gets better, it naturally reduces the amount of human effort required.

Resources

These are some of the resources I recommend to learn more on this topic:

• Your AI Product Needs Evals: This blog post is the predecessor to this one, and provides a high-level overview of evals for LLM based products.

• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences: This paper by Shreya Shankar et al provides a good overview of the challenges of evaluating LLMs, and the importance of following a good process.

• Align Eval: Eugene Yan’s new tool that helps you build LLM judges by following a good process. Also read his accompanying blog post.

• Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge): This is a great survey of different use-cases and approaches for LLM judges, also written by Eugene Yan.

• Enhancing LLM-As-A-Judge with Grading Notes by Yi Liu et al. Describes an approach very similar to the one in this blog post, and provides another point of view regarding the utility of writing critiques (they call them grading notes).

• Custom LLM as a Judge to Detect Hallucinations with Braintrust by Ankur Goyal and Shaymal Anadkt provide an end-to-end example of building a LLM judge, and for the use case highlighted, authors found that a classification approach was more reliable than numeric ratings (consistent with this blog post).

• Techniques for Self-Improving LLM Evals by Eric Xiao from Arize shows a nice approach to building LLM Evals with some additional tools that are worth checking out.

• How Dosu Used LangSmith to Achieve a 30% Accuracy Improvement with No Prompt Engineering by Langchain shows a nice approach to building LLM prompts with dynamic examples. The idea is simple, but effective. I’ve been adapting it for my own use cases, including LLM judges. Here is a video walkthrough of the approach.

• What We’ve Learned From A Year of Building with LLMs: is a great overview of many practical aspects of building with LLMs, with an emphasis on the importance of evaluation.

Stay Connected

I’m continuously learning about LLMs, and enjoy sharing my findings. If you’re interested in this journey, consider subscribing.

What to expect:

• Occasional emails with my latest insights on LLMs

• Early access to new content

• No spam, just honest thoughts and discoveries

• Taught by 25+ industry veterans who are experts in information retrieval, machine learning, recommendation systems, MLOps and data science. We discuss how this prior art can be applied to LLMs to give you a meaningful advantage.

• Focused on applied topics that are relevant to people building AI products.

• Free and open to everyone .

We have organized and annotated the talks from our popular paid course.¹ This is a survey course for technical ICs (including engineers and data scientists) who have some experience with LLMs and need guidance on how to improve AI products.

Getting The Most Value From The Course

Prerequisites

The course assumes basic familiarity with LLMs. If you do not have any experience, we recommend watching A Hacker’s Guide to LLMs. We also recommend the tutorial Instruction Tuning llama2 if you are interested in fine-tuning ².

Navigating The Material

The course has over 40 hours of content. To help you navigate this, we provide:

• Organization by subject area: evals, RAG, fine-tuning, building applications and prompt engineering.

• Chapter summaries: quickly peruse topics in each talk and skip ahead

• Notes, slides, and resources: these are resources used in the talk, as well as resources to learn more. Many times we have detailed notes as well!

To get started, navigate to this page and explore topics that interest you. Feel free to skip sections that aren’t relevant to you. We’ve organized the talks within each subject to enhance your learning experience. Be sure to review the chapter summaries, notes, and resources, which are designed to help you focus on the most relevant content and dive deeper when needed. This is a survey course, which means we focus on introducing topics rather than diving deeply into code. To solidify your understanding, we recommend applying what you learn to a personal project.

What Students Are Saying

Here are some testimonials from students who have taken the course³:

Sanyam Bhutani, Partner Engineer @ Meta

There was a magical time in 2017 when fastai changed the deep learning world. This course does the same by extending very applied knowledge to LLMs Best in class teachers teach you their knowledge with no fluff

Laurian, Full Stack Computational Linguist

This course was legendary, still is, and the community on Discord is amazing. I’ve been through these lessons twice and I have to do it again as there are so many nuances you will get once you actually have those problems on your own deployment.!

Andre, CTO

Amazing! An opinionated view of LLMs, from tools to fine-tuning. Excellent speakers, giving some of the best lectures and advice out there! A lot of real-life experiences and tips you can’t find anywhere on the web packed into this amazing course/workshop/conference! Thanks Dan and Hamel for making this happen!

Marcus, Software Engineer

The Mastering LLMs conference answered several key questions I had about when to fine-tune base models, building evaluation suits and when to use RAG. The sessions provided a valuable overview of the technical challenges and considerations involved in building and deploying custom LLMs.

Ali, Principal & Founder, SCTY

The course that became a conference, filled with a lineup of renowned practitioners whose expertise (and contributions to the field) was only exceeded by their generosity of spirit.

Lukas, Software Engineer

The sheer amount of diverse speakers that cover the same topics from different approaches, both praising and/or degrading certain workflows makes this extremely valuable. Especially when a lot of information online, is produced by those, who are building a commercial product behind, naturally is biased towards a fine tune, a RAG, an open source LLM, an open ai LLM etc. It is rather extra ordinary to have a variety of opinions packed like this. Thank you!

Course Website

Stay Connected

I’m continuously learning about LLMs, and enjoy sharing my findings and thoughts. If you’re interested in this journey, consider subscribing.

What to expect:

• Occasional emails with my latest insights on LLMs

• Early access to new content

• No spam, just honest thoughts and discoveries

Footnotes

1. https://maven.com/parlance-labs/fine-tuning. We had more than 2,000 students in our first cohort. The students who paid for the original course had early access to the material, office hours, generous compute credits, and a lively Discord community.↩︎

2. We find that instruction tuning a model to be a very useful educational experience even if you never intend to fine-tune, because it familiarizes you with topics such as (1) working with open weights models (2) generating synthetic data (3) managing prompts (4) fine-tuning (5) and generating predictions.↩︎

3. These testimonials are taken from https://maven.com/parlance-labs/fine-tuning.↩︎