That’s the question I wanted to answer in this interview. I wanted to hear from people working on large production systems, where the agent is not just building a clean prototype from scratch. It has to deal with large codebases, scale, internal tools, on-call incidents, and years of product decisions already baked into the system.

For this conversation, I talked to Eric and David, two staff-level engineers with very different backgrounds.

Eric Cunningham (LinkedIn) is a Staff Engineer at Dropbox, where he has spent the last eight and a half years working across desktop UI, platform teams, sharing and permissions, video streaming, content transformation, search connectors, and most recently knowledge graph infrastructure. Before Dropbox, his career has gone from electrical and embedded systems engineer, working on smart home systems, to leading the UI team at Lucid for auto infotainment software.

David Kuo (LinkedIn) is a Principal Engineer at Microsoft Gaming, where he works on cloud infrastructure for games: telemetry, analytics, multiplayer services, billing, and high-performance networking. He’s gone from working on data pipelines to billing systems, frontend and backend systems, as both manager and IC roles.

We started with the practical setup:

• What coding agents they use?

• Which models they trust?

• How they decide when to use stronger or cheaper models?

But the more interesting questions were about how the work itself is changing.

• What happens when AI reviews code before people do?

• Are staff engineers actually using loops?

• Have expectations changed for PMs? And do prototypes replace PRDs, or do engineers still need written specs?

To kick things off, what coding agents and models do you use right now?

Eric: I’m a Claude Code guy, I guess. The space moves super fast, but once I find a workflow that works, there’s a lot of inertia behind it. Changing is tough.

Lately I’ve been using Opus 4.7 for most things. I like the high-context-window version, the one-million-token version. I pretty much use that for almost everything. It seems like a good balance of cost and performance. I’m mostly using the command-line tool, and that’s basically it.

David: I have personal experience with Claude Code, but at work we’re basically asked to use Copilot. I use both the GitHub Copilot CLI and the newer desktop app. I’ve been using them in parallel because they do slightly different things, or at least they feel like they have slightly different capabilities.

How do you decide which model to use?

David: As far as models go, in the past month or so I’ve been using a larger range of models, because of Copilot Automode. Previously I was using almost exclusively Opus 4.6 or Opus 4.7, and occasionally experimenting with GPT models. Often I asked the agent to do the same thing across three different models in parallel so I could compare the outputs. That helped me see which model performed best at what type of task.

For system design, and tasks requiring more complex reasoning, I use Opus 4.7 (now 5 as of last week). I use Opus 4.6 for coding and reviewing. I also have started using GPT 5.3 Codex for well defined coding tasks and bug fixes. The mental tax of remembering to switch between models for different tasks has led me to use only a few models regularly.

We have access to Claude models, GPT models, and more recently Gemini. Our model selection is a bit more limited by category, but within those categories we have a pretty full breadth of options.

Bonnie: So you basically created your own little benchmark. You’re giving the same task to different models and testing the outputs. What kind of tasks are you using for that?

David: For example, if I’m trying to design a new system, I’ll give it the requirements and parameters and say: spin off three different agents with these specific models, give me the solution each one provides, and do a comparative analysis across the three solutions.

What I often see is that two of the models agree on most things. The more interesting part is seeing what they don’t agree on. I don’t do that as much now because I feel like I’ve seen enough to wean away from that approach, although the models change frequently.

There’s an auto mode Copilot got for model selection. Copilot Automode automatically selects the best model for a task. I’ve been using that more recently to see how good it is. It’s been fine. It works well enough for delivering things, and the cost has drastically dropped.

The cost of switching models is easy mechanically; you toggle it in the UI or pass a command-line switch. The hard part is remembering to switch back and forth. Having it automated makes it easier.

Bonnie: Claude Code has a command /opusplan where it uses Opus for planning and Sonnet for execution. But I agree: the mental cost of switching models is challenging especially if you want to use Haiku or smaller models like mini. Nowadays with Anthropic and OpenAI, not only do you have to select the model, but also the reasoning effort, so there’s quite a bit of effort.

How has the engineering workflow changed?

Bonnie: I was talking to an engineer at Uber recently, and he said the engineering paradigm has completely changed. His line was basically, “All I do is live in GitHub comments now.”

It sounds like you both still use the terminal and aren’t quite in that mode. But I’m curious: what was your engineering workflow before AI, and what does it look like now? What do you still do by hand, and what do you hand off to AI?

Eric: I feel like the answer is going to be really different depending on your experience, seniority, and role. For junior and mid-level engineers, their workflow is totally different.

For senior engineers, the biggest change for me is that the feedback loops are much tighter now. Before AI, I actually wasn’t writing that much code. I was mostly architecting systems, working with PMs and leadership to figure out what to build, figuring out how to build it at a high level, and then delegating the execution piece.

A lot of that used to mean delegating investigations to different people or different teams. They would go off for a couple days or a week, come back with a proposal, and then we’d figure out where to go from there.

Now I’m still delegating, but I’m delegating more of that to AI instead of to a person. Claude can get back to me in a few minutes instead of a few hours or a few days.

A lot of what I do in a platform organization is trying to figure out how to surgically insert something into a legacy codebase that’s already running at scale. A lot of the time, that means reading code or investigating systems. Before, I might call a meeting with someone who built the system or knows a lot about it. Now I can tell Claude: go read this code, figure out where the skeletons are buried, and tell me what the story is.

That has reduced my human interaction, which is maybe a negative consequence in some ways. Things move faster, but you’re not talking to people as much, and sometimes that historical context is necessary.

Bonnie’s note: One helpful thing Eric sent me after the interview was an example of the kind of prompt he gives Claude. It’s helpful to see how he prompts Claude to understand a codebase he hasn’t worked on before.

Sometimes it’s a simple prompt like:

But other times, he has a more specific objective in mind, in those cases more context helps him get a better 1-shot response. A concrete example he shared is looking into a pipeline of spark streaming jobs that they wanted to run faster/cheaper. His prompt would be

Prompt Example

The part I like is that Eric isn’t asking the agent to “go investigate” in a vague way. He gives it an entry point, explains what he’s trying to optimize, tells it which tools to use, and asks for outputs that are easy to scan.

I especially like the Mermaid diagram ask. I use a similar trick when I ask AI to create HTML artifacts from messy technical content. Tables and diagrams make the output much easier to understand than a long wall of text.

Bonnie: So are you spending your time differently now?

Eric: I’m involved with a lot more execution now than I was before. And by execution, I mean actually producing the system, committing the code, doing the quote-unquote “real work.”

The other work hasn’t gone away. As staff and principal engineers, you’re usually working with multiple teams and multiple products. There’s still architecture work, writing, leadership, product overlap, and technical input into people’s decisions.

But the rest of the work has been compressed enough that there’s more room to work on the execution pieces.

Bonnie: Are there parts of your workflow where you basically trust AI and don’t really need to look at it anymore?

Eric: Pretty much nothing falls into that category for me.

If it’s a very simple bug, like a null pointer exception in the exception logger, I can pretty much trust it to put up a PR to fix that. But for anything with a reasonable level of complexity, I don’t really trust it.

In a sufficiently complex system, there are usually a lot of contributing factors. There’s rarely only one fix. Even as a human, your first instinct isn’t always right. So there’s not a lot that falls into the category of “I’m just going to let it do its thing and I don’t care about the output.”

If you’re building a fairly standard app, like a frontend that reads and writes data to a single database, you can be much more hands-off. But when you’re working on distributed systems running at hundreds of thousands or millions of requests per second, there are a lot of things that can go wrong. There’s usually more than one way to solve something, and it’s not easy to be totally hands-off.

You wouldn’t even be hands-off with other engineers you trust. You still bounce ideas off each other. That collaboration is important, and I think the same thing applies to AI.

Bonnie: That makes sense. If you’re building a simple app for a few people, the stakes are totally different from something serving millions of users.

Eric: Yeah. I have done straight vibe-coded things for personal projects. I have a friend who organizes vibe-coding parties. It’s basically a Zoom call where everyone declares what they want to build, then you hang out and build for an hour.

One thing I built was a meal-planning tool for my wife and me. Planning what we’re going to make for the week and going shopping is a pain point for us. I built something that knows our recipes and links to the websites we use for new recipes. I know a million of these things exist, but there were some very specific things I wanted.

I was able to vibe-code that in two hours, and we use it on my phone.

So that kind of stuff, when the stakes are low, I would trust it. But when the stakes are high, I don’t trust it.

FYI: In my AI Builder bootcamp for PMs I teach how to change your approach when building prototypes vs. internal apps vs. public facing apps.

The goal of the bootcamp is to help PMs build useful things with AI while learning the fundamentals and understanding what’s low-risk, what needs review, and how to handle that with AI. Next cohort starts September 1st.

Sign up here: https://maven.com/bonnie-yu/ai-builder-bootcamp-for-pms.

Use code SUBSTACK30 for 30% off.

Bonnie: David, what about you? How do you feel like your engineering workflow has changed?

David: A lot of my experience is pretty similar to Eric’s. I definitely produce a lot more commits and changes to the system, whether that’s bug fixes, design changes, or new features.

One of the biggest areas is live-site response. AI has been really powerful for reducing time to mitigation, time to detection, and time to resolving incidents or outages.

For us, we use SRE agents. Sometimes by the time I log on to look at a ticket, the agent has already posted an analysis and a proposed solution. That’s definitely powerful.

But the engineer still has to look at the results and the data and make sure it actually makes sense. Did it hallucinate anything? Did it conclude something wrong? That’s still very important.

I also think our processes are still immature in terms of making them optimal for AI. One specific example is alerting. We have alerts that fire and tickets that get created when a metric hits a certain threshold. In the human workflow, we adjusted monitors so the on-call engineer wouldn’t get flooded by incidents. If one widespread issue affected a lot of dimensions, we might increase a hit count instead of creating separate alerts for every dimension. That makes sense for a human. You don’t want to page someone a hundred times for the same incident.

But in the world of AI, that may need to change. The agent might look at one ticket and say, “I don’t see any other tickets, so this must be isolated to this one thing.” It made assumptions based on how we had set up the process.

So now we’re rethinking how alerting should work if AI is helping us. That kind of thinking is starting to carry through the software lifecycle.

Bonnie’s note: The pattern I’m hearing from both Eric and David is that AI changes where engineers spend their time. They spend less time doing the first pass of understanding and investigating the system and instead more down the stack, the actual execution. For better or worse, that also means fewer meetings and human interactions where engineers ask around to learn how a system works.

What changes when AI reviews the code too?

Bonnie: Do you have AI review the code before a human looks at it? Or do you have some kind of agent review after a PR gets opened?

David: We actually had auto reviews for a really long time in our PRs. It wasn’t AI in today’s sense, but we always got some code suggestions from some kind of bot. What has changed now is that the newer agents can do reviews at much higher fidelity and quality. For a while, as the reviewer, I would have an agent review a PR before I even looked at it.

But now, at least in our organization, people are tired of getting agent comments in their PR. So the mentality has shifted. The person writing the code should run an agent review on their own PR before sending it out. Then, by the time another human is reviewing it, the obvious stuff has already been caught.

Bonnie: So the AI review moved earlier in the process.

David: Yeah. The expectation is that some kind of pass has already been done before you send it out. It reduces the more pedantic issues, like misspellings or poorly named variables.

I still think there is value for the person producing the code. Whenever I produce a pull request, I’ll still toss in that extra review at the end to make sure it looks good at the surface level. But as the reviewer, I don’t find as much value in running the generic review again, because I expect the author already did that.

If five engineers all open the same PR and run the same slash-review command, they’re probably going to get similar results. So you might as well put that up front, and then the humans can spend their time on the parts of the review that actually matter.

Bonnie: That makes sense. Generic review belongs earlier. But what about the more specific things you’re worried about as a reviewer?

Eric: I still use AI during code reviews for targeted questions. When I’m reviewing something, I usually have a few things in mind that I think are dangerous about the area being changed. I’ll prompt my agent with that context: I’m specifically worried about these things. Go look at it from this angle.

If you just tell Codex to review a PR, it won’t necessarily know what to look for. You get vastly different results when you give it context about what you’re worried about.

One recent example: someone put up a PR to add a new data source to one of our verifier systems. The specific thing I wanted to know was whether the PR introduced any new database reads, and if so, whether they were batched properly. That can really affect the performance of the verifier.

That’s the kind of thing I would love for a code review agent to figure out on its own. So the response is: go add this to the documentation. If you’re going to add a new read, make sure it’s batched. You can codify that in markdown in the codebase, and then the agent reviewing the code can find it next time.

Bonnie: This is how an organization can optimize the workflow for AI. Developers notice what the agent missed, write it in shared markdown for the team, and then the agent can check for it next time, which will lead to better results for the organization.

Eric: Yeah. You’re getting increasing value the more you use these automations, because the better the documentation gets, the better the agent gets at finding things.

At Dropbox, we’ve seen substantial improvement in delivery time. But the thing I’d call out is that coding is only one component of the software development lifecycle.

Bonnie: That’s the part I think people sometimes miss. Even if the coding part gets faster, that doesn’t mean the whole software delivery process becomes instant. What parts of the process still take time?

Eric: Coding is only one component of the software development lifecycle. You still have QA cycles, safe rollouts, and watching features in production to make sure you’re not breaking something at scale.

The design cycle has compressed, but it’s not free. You still have design reviews, humans giving feedback, coming to agreement, and unknowns that require people to sit in a room and redesign part of the system.

The actual code writing isn’t free either. It’s faster, but it still isn’t free. You still do code reviews. Timelines have shortened, but software delivery isn’t free.

David: We’re still a long way away from never having to look at anything. In fact, we’ve gone the other direction in some ways. PRs that have AI touch them now require two people to sign off instead of one.

Code is cheap now, so you get a lot more PRs. That means the review workflow has to work for you.

Do staff engineers actually use loops?

Boris Cherny, who leads Claude Code at Anthropic, has a quote that has been circulating: “I don’t prompt Claude anymore. I have loops that are running. They’re the ones that are prompting Claude and figuring out what to do. My job is to write loops.”

The basic idea is that instead of prompting the agent step by step, you define the goal, the exit criteria, and the verification process. Then the agent keeps working until it reaches the goal.

Bonnie: Do you use loops?

David: I don’t really use loops. I tried it out and experimented with it.

For work specifically, I think the problem is that I deal with a lot of legacy code. Loop engineering works great if you want to build something from scratch and you have a clear goal. You can give the agent more free rein to do the whole thing.

Where you get problems with legacy code is poor documentation or quirky behavior that came from tribal knowledge. Those are the parts that feel most dangerous to let the agent loop on by itself until the end.

Documentation alone also isn’t sufficient. If something is already implemented, the agent may not understand the downstream impact of changing it. Sometimes there’s a weird condition in the code because of one problem that happened years ago, and nobody remembers except the person who put it in.

Loop engineering means you trust the AI enough to keep going until it finishes. I don’t have that level of trust right now. I would still want to check in every so often, or at least every iteration.

How do you deal with new AI trends like Loops?

Eric: I have a lot of AI fatigue when it comes to new features, new tools, new workflows, and new trends. That contributes to the inertia I mentioned earlier.

When subagents came out, there was this whole trend of telling your agent to spin off subagents and building workflows around how to use them. Then two weeks later, you didn’t need to do it manually anymore. It just became something the agents did on their own.

Have your expectations of PMs changed? Are PMs and engineers starting to cross over more?

Eric: I expected there to be a lot more crossover than I’ve actually seen. I thought PMs would be doing more engineering work and engineers would be doing more PM work.

But at least in my sphere, people are mostly sticking to their roles and using AI to automate their part of the work. PMs might use AI for customer research, feedback analysis, market research, or writing PRDs. They’re finding ways to automate their own job.

I haven’t seen as much PM-building type work as I expected.

David: In our org, I feel like there is more crossover from engineering into PM work than PMs doing engineering work.

For example, I wrote a product spec two weeks ago because I needed one.

For the PMs, my expectation is that they focus on the core PM responsibilities. They don’t need to do extra stuff just because AI exists. But AI should make them more self-sufficient.

If they need information or data, they should be able to get more of that themselves without asking the engineering team, unless it’s a permissions issue.

Bonnie: So basically: don’t ask engineering to pull a query for you if AI can help you do it.

Eric: Yeah. If I’m writing a Hive query, I don’t write Hive queries anymore. I ask the Databricks query builder:

here’s what I want from this table, go get it for me.

David: The same thing applies to documentation. On the billing side, there are a lot of nuanced business rules. Before, engineering would get asked because one other person and I had the historical context.

We baked that into documentation. Now I would expect someone in our product area to be able to ask: I want to bill for this feature. How would I do it?

Maybe we still do a final review at the end, but they should be much more self-sufficient.

Do prototypes replace PRDs?

Bonnie: There’s a common idea in PM circles right now that PMs should stop writing PRDs and build prototypes instead. Maybe there’s a one-pager for context, but the prototype becomes the artifact. The requirements are baked into the interactions.

What do you think?

David: What exactly are you handing me if you give me a prototype?

Are you handing me how the UI looks? Are you handing me all the interactions with the database too? Are the APIs properly constructed? Did we capture all the requirements?

“An interactive prototype is cool. It can help communicate your point. But that’s usually not the part that’s unclear in a PRD. The things engineers spend time spinning on are the rough edges. What do people care about? How should it behave in different circumstances? I can visualize from a static image how something should look and feel. What I can’t always imagine is what matters for all customers and how it should behave in all cases.”

Eric: Part of the challenge is that real products have a lot of behavior that is hard to capture in a prototype. At Dropbox, for example, say we want to build a new feature that writes some new data to your Dropbox. What happens when that user has exceeded their quota? Does the behavior change? Do we upsell there? What does that look like?

That’s the stuff I need to see in the PRD. I don’t generally see all of that in a prototype.

David: The main issue with prototypes is that when you hand over something interactive, I don’t know what is intentional and what is approximate. If there’s odd behavior in the prototype, is that by design or is it just not handled properly? Should the engineering team copy it, fix it, or get rid of it?

That’s the danger. It looks close to the product. It has the wow factor. But you can’t build off of it directly without anything else.

Bonnie: So you still want some version of a PRD.

David: I want some level of documentation. Maybe the format is different than a traditional PRD, and I don’t want it to be verbose.

With AI, some requirements docs have become huge. Then I have to summarize them with AI, which defeats the purpose.

I want things to be more succinct. Maybe prototypes augment the PRD, and the PRD can be much smaller. But I don’t want the documentation to go away completely.

Bonnie That last point is probably the part PMs should pay the most attention to.

AI can help us write specs, build prototypes, pull data, and get unstuck faster. But the hard work of clarifying the product behavior does not go away

Here are my biggest takeaways from the conversation

1. Prototypes alone should not replace PRDs.

While PM circles are pushing hard on prototypes, engineers are more hesitant. What Eric and David found is that prototypes usually show the happy path, not what eats up engineers’ time when looking at specs: the unhappy paths like errors, quota limits, billing rules, upsells, and permissions.

Engineers still want to see some form of documentation. Maybe it’s a PRD, maybe it’s not.

2. PMs still need to do PM work, but AI should make them more self-sufficient.

Eric and David both said the PM role has not changed as much as people might expect. PMs are still doing customer research, feedback analysis, market research, and PRDs. But now PMs should be more self-sufficient. If a PM needs a query, a data pull, or an explanation of documented business rules, PMs can use AI to handle that instead of asking engineering.

3. AI changes where staff engineers spend their time.

The first-pass investigation work is increasingly going to AI. Eric used to delegate investigations to people or teams and wait hours or days. Now he can ask Claude to read the code, find the weird parts, and come back in minutes. That is pulling staff engineers closer to execution. AI did not remove architecture work, writing, or leadership work. But it compressed enough of the surrounding work that staff engineers now have more room to build, commit code, and stay closer to the implementation.

4. Organizations need to update workflows for AI.

David’s alerting example is the one I keep coming back to. Their process collapsed duplicate alerts so on-call engineers would not get flooded. That made sense for the team. But the SRE agent saw one ticket, did not see the duplicate incidents, and assumed the problem was isolated. The workflow was hiding information the agent needed.

5. Documentation compounds the value of the agent.

When someone notices what the agent missed, they should write it down in a shared markdown so the next agent review has more context. That is how an organization can optimize the workflow for AI. But there is obviously a balance here.

The tricky part is getting enough rules written down so agents don’t miss critical system context. If every preference turns into a rule, the markdown file gets bloated, fills up the context window, and can box the model in too much. The critical system-specific rules that AI isn’t likely to know on its own should be written down so people do not have to rely on every reviewer remembering the same gotchas.

6. AI review is moving earlier, but targeted review still matters.

The generic AI review should happen before a PR gets sent out. That catches spelling, naming, and surface-level issues earlier, so reviewers can spend less time on the obvious stuff. But generic review is not the same as judgment. Eric’s example made this concrete: “Review this PR” is less useful than “I’m worried this change adds new database reads. Check whether they are batched properly.” The more specific the concern, the more useful the agent review.

7. Code review is eating more of engineers’ time.

This is the counterpoint to the simple “AI makes engineering faster” story. Code is cheaper to produce, so there are more PRs. In David’s case, AI-touched PRs now require two human sign-offs instead of one. AI shifts time away from writing code and toward reviewing the code that gets produced.

8. Copilot makes cheaper models easier to use, something Anthropic and OpenAI aren’t as incentivized to do

Model routing already exists. The interesting question is who is incentivized to use it aggressively to bring costs down. David said Copilot auto mode brought his usage down to about 15 percent of what he was using, even though he was already trying to manage model cost himself. Frontier labs like Anthropic and OpenAI do this too to an extent. But they aren’t quite as incentivized. This is a place where products like Copilot can save money for customers by routing easier work to smaller models and remove the mental overhead of switching models manually.

A few API pricing examples as comparison:

• GPT-5.4 is $2.50 input / $15 output per 1M tokens. GPT-5.4 mini is $0.75 input / $4.50 output, about 3.3x cheaper on input.

• Claude Sonnet 4.6 is $3 input/ $15 output. Claude Haiku 4.5 is $1 input / $5 output, about 3x cheaper on input.

The provider routing point matters too. If Copilot can route between OpenAI, Anthropic, and its own smaller models, it has more room to pick the cheapest good-enough model for a task. GPT-5.4 mini is 25% cheaper than Haiku.

OpenAI GPT-5.4 model pricing

Anthropic Claude pricing

9. Loops work better when the work is contained.

Loops make sense when the goal is clear and the success criteria are easy to check. Build this app. Fix this small bug. Run the tests. Keep going until they pass. That is different from changing a large production system where the hard part is knowing what else might break.

10. AI fatigue is real, even for staff engineers.

I liked Eric’s point about AI fatigue. There are new features, new workflows, and new “best practices” every few weeks. Some of them are worth changing your workflow for. A major model upgrade might be one of those moments. But a lot of smaller workflow tricks may get absorbed into the product pretty quickly. So the work is partly figuring out what is worth learning now, and what you can wait on because the agent may just start doing it by default.

The good news: AI tools have made code reviews surprisingly easy to set up. In this tutorial, I’ll walk through the two main ways to add them to your workflow, how to set up each one, and the trade-offs I’ve found as I’ve been building my own projects.

Along the way, we’ll:

• Run instant reviews with slash commands (/code-review in Claude Code, /review in Codex)

• Set up a GitHub Action so every pull request gets reviewed automatically

• Talk about how much of this loop I automate

• Have a different model review your code

• Compare Slash Commands vs. Github Actions and which I prefer

I’ll cover both Claude Code and Codex. Each section labels the steps for each agent, so you can jump straight to the one you use.

BTW if terms like pull request or feature branch are new to you, start with my earlier tutorial, where I explain those concepts. TLDR: the AI agent writes code in a separate place (“feature branch”), then ask for a code review (“pull request”) before merging it into the main codebase.

Why code reviews?

Code review is a standard practice on every professional engineering team, and the reasons apply just as much to AI-generated code:

• Catching bugs before users do. AI agents are good at making code that works in the happy path, and less good at edge cases, race conditions, and “what happens when the input is empty.”

• Security. Leaked API keys, unvalidated inputs, and permissions that are too broad are exactly the kinds of issues a reviewer catches and a demo doesn’t.

• Consistency. As your project grows, reviews keep new code following the patterns you’ve already established.

You don’t need to be able to write or review this code yourself. There are tools that will do the review for you, and it’s easy to make them part of your workflow.

Your two main options

There are two ways to get AI code reviews into your workflow:

1. Slash commands inside your coding agent. Claude Code has /code-review, and Codex has a /review command. You run them in your session, and the review happens right there. The nice thing is it’s all built in! It used to be you’d have to install it yourself.

2. A GitHub Action. Both Anthropic and OpenAI provide official actions that automatically review every pull request and post comments directly on GitHub. It’s basically a YAML file you can add to your repository. You can use theirs or ask your agent to create one for you.

(If you’re on a Claude Team or Enterprise plan, Anthropic also launched a managed Code Review service, a GitHub App you enable in admin settings with no YAML at all. Most solo builders won’t have access to this, so I’ll focus on the two options above.)

Let’s set up both, starting with the simpler one.

Option 1: Slash commands

In Claude Code

Type /code-review and Claude reviews the code changes you’ve been working on. You don’t need to have a pull request open. By default it looks at the changes on your current branch, but you can also point it at a specific pull request, branch, or file.

There are two options I find useful: --comment posts the findings onto your pull request so there’s a record of them, and --fix has Claude fix the issues it finds.

In Codex

Type /review and Codex gives you a menu of choices: review your changes against the main branch (this is the pull request style review), review changes you haven’t committed yet, review a specific commit, or write your own instructions like “focus on security vulnerabilities.”

Two things I like about the slash command approach:

• It uses your subscription. The review costs subscription tokens like any other request, so you don’t need to set up an API key or separate billing.

• Your coding agent sees the results immediately. The findings land in the same chat session, so you can say “fix the first two issues” and keep moving.

Tip: You can learn a lot from reading how other people prompt their agents. Anthropic publishes the prompt behind /code-review on GitHub.

Codex’s built-in /review prompt isn’t published as a standalone file, but OpenAI publishes a similar review command in their codex-plugin-cc repo (a plugin that lets you run Codex reviews from inside Claude Code). Their adversarial review version is a fun read too: it challenges the implementation approach and design choices instead of just hunting for bugs.

Running the reviews automatically

You can run these commands yourself whenever you want a review, or you can have your coding agent run them automatically after each pull request. There are two ways to set that up:

• Add a note to your CLAUDE.md or AGENTS.md, something like “after opening a pull request, run a code review.” This is the simplest option, but instructions like this rely on the agent remembering to follow them.

• Set up a hook. Both Claude Code and Codex support hooks, which let you run your own logic at specific points in the agent’s workflow. A hook that fires whenever the agent runs gh pr create can trigger the review every time, deterministically.

Option 2: A GitHub Action

The second approach moves the review to GitHub itself.

GitHub Actions is a CI/CD system built into GitHub. You add a YAML file to .github/workflows/ in your repo, and GitHub runs it automatically when events happen, like a PR being opened or updated.

For code review, the action reads your PR diff and posts comments directly on the PR.

Note: CI/CD (Continuous Integration / Continuous Deployment) is the automation engineering teams use to run checks like tests, builds, and reviews on every code change.

YAML is a language usually used for writing configuration files and exchanging data. You see it in a lot of places for example at the top of agent and skill files!

Claude Code: setting up the Code Review Action

There are three setup steps. I’ll walk through them so you understand what’s being set up, but you can always ask your coding agent to do this part for you. I had Claude Code create the workflow file and walk me through the rest.

Step 1: Add your API key as a GitHub secret (one-time, manual)

The action needs an Anthropic API key, which means the reviews are billed separately from your Claude subscription (more on that trade-off later). Get a key from console.anthropic.com, then:

1. Go to your repo on GitHub

2. Settings → Secrets and variables → Actions → New repository secret

3. Name: ANTHROPIC_API_KEY, Value: your key

4. Click Add secret

Step 2: Add the workflow file

Create .github/workflows/claude-review.yml. Here’s what mine ended up looking like:

name: Claude Code Review

on:
 pull_request:
 types: [opened, synchronize]
 issue_comment:
 types: [created]

jobs:
 claude-review:
 if: |
 github.event_name == ‘pull_request’ ||
 (github.event_name == ‘issue_comment’ && contains(github.event.comment.body, ‘@claude’))
 runs-on: ubuntu-latest
 permissions:
 contents: read
 pull-requests: write
 id-token: write
 steps:
 - uses: actions/checkout@v4
 with:
 fetch-depth: 0
 - uses: anthropics/claude-code-action@v1
 with:
 anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
 # track_progress makes the action create a tracking/summary PR comment up
 # front and grant the inline + summary comment tools automatically. Without
 # it the run auto-detects “agent” mode, Claude reviews but is DENIED
 # permission to post (permission_denials_count > 0) and nothing appears.
 # (use_sticky_comment is a v0 input the v1 action ignores — do not rely on it.)
 track_progress: true
 # Do NOT set a custom --allowedTools here: it overrides the action’s
 # defaults and can strip the read/comment tools Claude needs. Let
 # track_progress provision them.
 prompt: |
 Review this pull request. Check for bugs, security issues, and TypeScript type errors.
 Post inline comments on specific lines where you find issues using the inline comment
 tool. When you are done, ALWAYS post a PR comment summarizing your review: what you
 checked, any issues found (with severity), and an overall verdict (approved / needs
 changes). If no issues were found, say so explicitly.
 claude_args: --model claude-sonnet-4-6

The comments in that file came from my own trial and error. The big one: keep track_progress: true. Without it, Claude does the review but gets denied permission to post the comments, so nothing shows up on your PR.

Step 3: Push the workflow file to your repo

The workflow file needs to land on your main branch before the action starts working:

git add .github/workflows/claude-review.yml
git commit -m “Add Claude Code Review GitHub Action”
git push

If those git commands are unfamiliar, this is another spot where you can ask your coding agent to commit and push the file for you.

Gotcha: if you add the workflow file through a pull request, your first PR will show a “workflow validation” error. This is normal. When I first set this up, the action skipped with an error about workflow validation. The action will only run if the workflow file on your main branch matches the one in the PR.

This is a security rule that prevents someone from submitting a malicious PR that edits the workflow to steal your API key. Since you’re adding the workflow file in a PR, it doesn’t exist on main yet, so it skips.

Just merge the PR. Once the workflow file lands on main, every future PR triggers reviews normally.

Codex: setting up the Codex Action

The Codex version is nearly identical: add an OPENAI_API_KEY secret (same steps as above), then create .github/workflows/codex-review.yml:

name: Codex PR Review
on:
 pull_request:
 types: [opened, synchronize]
jobs:
 codex-review:
 runs-on: ubuntu-latest
 permissions:
 pull-requests: write
 steps:
 - uses: openai/codex-action@v1
 with:
 openai_api_key: ${{ secrets.OPENAI_API_KEY }}
 prompt: “Review this PR for bugs, type issues, and style violations per AGENTS.md.”

What a review looks like

Once the workflow file is on main, setup is done. Every new pull request triggers a review automatically. Here’s what that looks like with the Claude action on one of my projects. You’ll see a “Claude Code Review” check appear alongside your other checks:

A few minutes later, Claude posts its review: what it checked, what it found, and whether previous findings were resolved:

At the end, I asked it to provide a summary of what to do:

You can also talk to the reviewer. Type “@claude fix the type errors here” in a PR comment and it will respond, and it can even push a fix commit if you give it write permissions. It reads your CLAUDE.md for project guidelines.

It’s also nice that I get email notifications with the summary, so I can see the review on my phone:

FYI I'm running an AI Builder Bootcamp for PMs on Maven, built around the same idea as this post: the fastest way to learn AI is to build something real. We ship two apps and build agent workflows together. Join the waitlist to hear when the next cohort opens.

How much of the code review loop should you automate?

The Action automates the review itself, but the results land on GitHub, and the coding agent on your machine (the one that wrote the code) doesn’t know the review exists. The next step is to get the code review results back to your agent. There are 3 ways to do so:

1. Ask your agent to check. After pushing, tell Claude Code: “check the review comments on my PR.” It runs gh pr view --comments and pulls the findings into your session, where it can start fixing them. You can add this to your CLAUDE.md so it happens as part of your normal flow.

2. Reply on the PR itself. Because the Claude action is interactive, you can comment “@claude fix the issues from this review” right on GitHub, and it will respond, and even push a fix commit if you’ve given it write permissions. You never leave the browser.

3. Automate it with a hook or script. This is the more advanced version of the first option. Instead of you asking, a hook in your coding agent (or a small script) can fetch the review results with the GitHub CLI after each push and hand them to your agent to act on.

#3 fully automates the code review loop. Every PR gets reviewed, your agent reads the findings and pushes fixes, and it repeats until the review comes back clean.

The question is: should you take yourself out of the loop entirely?

I keep myself in the loop. Part of the reason is tokens. If every PR automatically triggers a review, and every review automatically triggers fixes, you can burn a lot of tokens going back and forth on small nitpick findings or on tiny changes that didn’t need a review in the first place. The other part is I want to understand what didn’t work well. It helps me internalize how well each of the models work.

So my workflow is: the review runs automatically, and I decide what’s worth fixing. How closely I read the findings depends on how important the code is.

Have a different model review the code

So far, the same model that wrote the code has also been reviewing it. Claude reviewing Claude’s code can share the same blind spots. Using another model to critique another is one of my favorite AI workflows. I’ve used it for reviewing architecture plans, and it applies just as well to code reviews. A different model is closer to a real second opinion.

You can set this up with either option:

• With slash commands. If Claude Code is your primary agent, OpenAI publishes codex-plugin-cc, a plugin that adds a /codex:review command to Claude Code. Claude writes the code, and Codex reviews it, without you leaving your session. There’s also a simpler version that I’ve used: open Codex (or Gemini) in a second terminal and ask it to review your changes.

• With the GitHub Action. The actions don’t care which agent wrote the code. If you build with Claude Code, you can set up codex-action as your reviewer, or the other way around.

Slash command vs. GitHub Action: pros and cons

Slash commands

Pros:

• The reviews use your subscription tokens like any other request, so you don’t need to set up an API key or separate billing

• Your coding agent sees the review output immediately, so you can ask it to start fixing issues right away

• You can review before you even push, including uncommitted changes, so you catch issues while you’re still working

Cons:

• The review results only show up in the chat with your coding agent. If you’re working with someone else on the project, they won’t see any of the findings unless you post them to the pull request with --comment

• The review only runs when you remember to run it, or when you’ve set up a hook to run it for you

• The review comes from the same model that wrote the code. To be fair, /code-review spins up fresh reviewer agents, so it’s not reviewing its own chat history. But Claude reviewing Claude’s code can still share the same blind spots, which is why I like having a different model do the review

GitHub Action

Pros:

• Reviews run on every PR automatically, so nothing depends on you remembering

• All the review comments and notes are on GitHub, attached to the PR. If you’re working with someone else on the project, they can see the same review comments and history you do

• With the Claude action, you can reply to the review right on the PR with @claude

Cons:

• You need an API key, which means you’re paying per token, separately, for every review. For a solo project with occasional PRs this is small, but it’s a cost that scales with how often you push

• The review comes after you’ve already opened the PR. With a slash command, you can catch the same issues earlier, before the code ever leaves your machine

• Your coding agent doesn’t know the review exists, so the findings don’t make it back to where the code gets written, unless you connect them back with one of the methods above

Summary:

My take: I originally set up the GitHub Action because several people brought it up, and I expected it to be really useful. Honestly, after trying it out, it feels meh for a solo builder. The slash command gives me the same kind of findings sooner, it’s built in, and there’s no separate API bill. Where the Github Action becomes useful is when someone else is working on the project with you, since the reviews live on the PR where you can both see them.

Verdict: Start with the slash command. Add the GitHub Action when someone else joins the project, or when you want a record of every review.

Give it a try

If you’re building something you plan to put in front of real users, add it to your workflow. The slash command doesn’t need any setup, and the GitHub Action takes about ten minutes. Code review is one of those engineering practices that sounds intimidating, but it’s easy to make it part of your workflow.

Have you added code reviews to your AI building workflow? I’d love to hear what your setup looks like, and whether you’ve let your agent respond to reviews automatically or kept yourself in the loop.

Key Reference Links

• claude-code-action: the GitHub Action for Claude reviews

• codex-action: the GitHub Action for Codex reviews

• Anthropic’s /code-review prompt: worth reading to learn how Anthropic prompts its reviewer

• Claude Code hooks and Codex hooks: for running reviews automatically

When we build games in Scratch together, he has to think through the logic, drag the blocks, test and fix it himself. This time we used Claude Code in the Desktop app instead. The risk with that approach is: if you let a kid describe a game idea in one sentence and hand it to an LLM, the AI fills in every decision for him. He ends up watching a game get built instead of designing one. So before building any code, we spent a few days working through the design on paper.

Claude Code in Desktop

I decided to use Claude Code in Desktop for my son instead of VS Code in terminal, my usual setup, since the UI is simpler for an 8 year old to navigate. It worked well overall, though Claude did struggle to load the game in Preview mode a few times, so we ended up testing it in the browser directly instead.

My Game Design Question Guide

The tricky part of “vibe coding” with a kid is that the entire value of the exercise is in the thinking. So I built a question guide for Austin to work through first. I printed the pages so he could draw and fill it out with a pencil. Yes I’m still a big fan of pencil and paper for elementary school kids.

You can grab the Question Guide here: My Game Design Journal

It walks through:

Games I Love. Watch a few trailers together and talk about what feeling each game gives you, not just what it looks like. We watched Mario, Asteroids, Pokemon, and a few others.

My Game Idea. One sentence that describes the whole game.

Winning and Losing. How do you win, how do you lose, is there a score.

My Character and Controls. What the character can do, and what button does what.

My Game’s Look and Sound. Picking free art and music.

My Prompt Guide. Writing the game requirements for the AI.

My Playtest Notes. After the game exists, what’s fun, what’s broken, what needs fixing.

Example Pages from Guide

Austin landed on a creature battling game. You buy creatures, battle them, and clear a level by collecting every creature in it.

Writing a Prompt to Define the Game Requirements

This was the part I cared about most. There’s a gap between a kid’s one paragraph game idea and a prompt detailed enough for an AI to build something playable from. I didn’t want to close that gap myself by writing the detailed spec for him. Instead, I had him bring his rough idea into a live conversation with Claude and let Claude ask the follow up questions.

Here’s the starter prompt we used:

And here’s what Austin typed in for his game idea from doing the question guide:

“This is a 2D battle game called Creature Battle Collector. In the game, you buy creatures, and use your creatures to battle each other. If you win a battle, you get money. If you lose, you lose money. There will be 3 levels. To beat a level, you need to collect all the creatures, and to win the game you need to beat all the levels.”

Claude and my son worked together to define the rest of the requirements

Claude asked him things like how much creatures should cost, how many lives the player gets, and what happens when you run out of money. Some questions he answered by picking from a few options, others he just typed out himself. By the end of the conversation we had an actual MVP spec, written by an 8 year old, with the AI doing the interviewing.

Free Assets

Kids get really excited by visuals and music, so my son and I picked some assets from these free asset websites.

FREE IMAGE SITES (SPRITES AND BACKGROUNDS)

Kenney.nl — over 40,000 free images, sprites, characters, tiles. No sign-up needed, the best place to start.

FREE MUSIC AND SOUND EFFECT SITES

Mixkit.co — free game sound effects, no sign-up. Jump sounds, explosions, win chimes.

Pixabay.com — free music tracks and sound effects, no attribution required.

Incompetech.com — hundreds of free music tracks by Kevin MacLeod, just credit him in your game.

Learning to Debug

Austin had learned to describe basic bugs to Claude and ask it to fix them, which worked great initially, but then he encountered situations where Claude failed and ended up on a loop itself. The game froze at one point. Austin tried describing what happened in his own words, and Claude said it fixed the issue, but when we tested the game it still froze. He tried a few more times, and Claude still couldn’t solve it.

I showed Austin the developer console, and to look for the small red circle with an X of where the actual errors show up. We copied the error text out of the console and pasted it directly into the chat with Claude. Claude read the error and fixed the issue right away.

Honestly, I was happy to see Claude fail, because it showed him some of the challenges when working with AI to build.

What He Ended Up With

By the end of the week, Austin (with Claude’s help) designed and built the game from start to finish! He named and defined monsters and their abilities, the controls, wrote the prompt, and he learned to use the developer console. Austin was responsible for iterating through the game: adding save profiles, making UI updates so the menus were easier to navigate, etc.

I hosted the final game on Render so he can send it to people instead of something that only runs on my laptop. You can play it here:

https://creaturecollector.onrender.com/

FYI It’s on the free tier so you’ll probably need to wait for the server to restart

A quick Interview with my son

What do you want to tell people about your game?

Come play my game! It’s very fun. You can collect cool creatures like Colossus.

What did you learn?

I learned how much time it takes and how to fix bugs. It also has a lot of typing and writing to make a game. You have to know what you want. You can add sound effect you want to make it interesting to play, and you don’t want the game to be too easy.

Game demo. Unfortunately, the recording didn’t capture the music and sound effects.

We playtested it with the family and he got to watch other people actually play something he built! If you’re looking for a summer project with your own kid, give the question guide a shot.

Two forces are driving this:

1. Token usage is growing fast. Users are moving from short chat sessions to running agentic systems, which are significantly more expensive. Agentic systems require more tool calls, autonomous task execution, etc all. You can see this yourself by building a multi-agent workflow and measuring the usage. This paper published in April 2026 found “agentic tasks are uniquely expensive, consuming 1000x more tokens than code reasoning and code chat,” and Microsoft reportedly spent their annual claude code budget in a few months.

2. Infrastructure constraints are becoming more problematic. There is limited compute and memory currently available. New data centers require enormous amounts of land, electricity, memory, etc, and people are pushing back. According to this Politico article, Hill County in Texas announced a 1 year moratorium on building new data centers. Meanwhile, Anthropic was forced to make a deal with SpaceX to get more compute access to keep up with demand, rumoured to be paying a premium of $15B/year. See announcement. If you’re a Claude Code user, you’ve probably noticed the problems: service disruptions, reduced usage limits, and bans on OpenClaw workflows on their subscription plans.

This is why now is a good time to start experimenting with local LLMs. As local models improve, there will be tasks you can offload to your own hardware so you don’t have to spend hundreds of dollars on frontier models every month.

The key question is - which tasks actually require frontier intelligence, and which can you offload to a local model? Summarization? Coding Assistance? Writing?

This tutorial will walk you through setting up local LLMs, your own personal evaluation suite, and integrating them into your workflows.

First, the Setup. Looking at your Hardware

The first place that feels like a barrier for people getting started with local LLMs is hardware.

If you believe what you see on LinkedIn or social media, you need to buy the newest Mac mini with 32GB of RAM before you can run any local LLM. But that’s not necessarily the case.

Running Local LLMs on a 4 year old Laptop

My laptop is over 4 years old: a Lenovo Legion Slim 7 laptop with

• AMD Ryzen 7 5800H CPU

• 16GB RAM

• Nvidia GeForce RTX 3060 with 6GB vRAM

Even though this is an older laptop, I was able to run several modern open source models locally: Qwen, Gemma, Mistral etc. Not every model runs well, but it’s enough to start experimenting and learning.

When you’re running local models there’s 2 pieces to understand:

• Your hardware specs relative to

• The model specs

When you don’t have the best hardware specs, you need to be more cognizant of what models you run. Let’s start with the hardware specs.

The two specs that really matter are vRAM and the system RAM.

• The vRAM is how much memory is directly attached to the graphics card. This is the fast memory used for AI inference.

• The system RAM is what the computer uses for general multitasking.

TLDR: The vRAM is the good stuff for AI. If the model fits mostly into your vRAM then the responses from the AI will feel fast. Once the model no longer fits, then the model starts running on the system ram. Also called CPU offloading / RAM offloading. The model still runs, but responses are much slower.

I only have 6 GB of VRAM so not the most ideal. I can’t run the largest models, but I can still run some smaller models. That’s because of quantization.

Model Parameters and Quantization?

Models contain billions of parameters; these are the weights that the model learns during training. A model is in actuality A LOT of numbers. When you select a local model, one of the first decisions you need to make is the number of parameters you want. Do you want Qwen 3.5 4Billion, 9Billion, 27Billion, or 35Billion parameters? The more parameters, the more capable and bigger the model is, and thus the more memory it requires. A frontier model like Anthropic’s Opus 4.7 or OpenAI’s GPT 5.5 is so large it requires far more memory and compute than a consumer laptop can provide, which is why these models run across large GPU clusters in data centers.

Quantization makes local models practical. Quantization compresses the information in the model into something smaller. It’s like zipping a file, originally a file might be 200MB, but after zipping it, the file becomes 10MB. Quantization basically zips the model to something smaller so you can run a model with less memory. The compressed version isn’t identical to the original, you’re trading off the performance of the model in exchange for the ability to run the model with less memory. For example, a 9B model may require roughly 18 GB of memory in its original form, but if you compress it to a 4bit GGUF format then it only requires 5-6GB.

How to Read Model Descriptions

Models can be quantized to different levels. When you select a model, you’ll see different “quantization” options usually, for example,

Q4

Q6

Q8

This is how much you are compressing the model. Q4 compresses the model to 4 bits, whereas Q8 compresses the model to 8 bits.

If you look at this model search result, here are the important bits to look at:

• Download size of 6.55GB is a good proxy for how much memory you’ll need for loading

• Partial GPU Offload Possible means this likely won’t be able to run only on my GPU. Makes sense I only have 6GB of vRAM on my GPU

• Qwen 9B Parameters 9B (B for billion) is the size of the model, the effective number of parameters before the quantization

Ok, now that we’ve got a rough sense of what hardware matters and how to pick models, it’s time to actually run it!

Next How to Run your local LLM - Ollama vs. LM Studio

There are two popular ways to run local LLMs on your machine: Ollama, and LM Studio.

LM Studio is a GUI first app that allows users to quickly download and explore local LLMs in a chat interface.

LM Studio Screenshot

Ollama is a terminal first tool that runs local LLMs. It downloads, quantizes and serves the models locally. Download Ollama here, and you can see the models available to download here. You can use Ollama to integrate local LLMs into your coding workflow.

If you’re starting out with LLMs and exploring, then I’d recommend going with LM Studio first to explore basic capabilities for the local models. Then moving on to Ollama to test models for coding and agentic tasks. In the next article, I’ll touch more deeply on Ollama.

For now, here’s a step by step for setting up a model on LM Studio

• Download and install LM Studio here

• Go to Model Search and download your first model

• You can ask ChatGPT/Claude to recommend a local LLM model to start with based on your machine specs

• Download the model

• Load the model, and start chatting!

Prompt Example

Screenshot of Model Search Icon

Building Your Personal Eval Suite

When you’re only testing Claude and ChatGPT, it’s ok to play around with those two, and over time get a sense of which performs better on which tasks. However, as you’re testing more models, it’s harder to keep clear which models are best for which tasks I do.

This is why I recommend building your own personal evaluation suite, so you can systematically evaluate each of the models.

An evaluation suite is simply a collection of prompts you use to compare models. For a personal eval suite, I pick real prompts and tasks from my own life: Editing articles, Financial Planning, etc. After all, these models are for me.

The easiest way to build your first evaluation suite is to ask ChatGPT or Claude to help.

Prompt Example:

Keep It Small

Evaluations take time. Since this is a personal model, I recommend keeping it simple so you can complete the evaluation in 15-20 minutes. For example, what I do is select 10 prompts to evaluate against, and 2 models to compare.

Ultimately the goal is building intuition.

Track Results in a Spreadsheet

Again to keep it simple, my evaluation is done in a spreadsheet. For each prompt I track: task category, the prompt itself, what I’m looking for, common failure modes, model ratings, and notes.

Screenshot of my personal evaluations

My First Impressions on Gemma 4 4B and Qwen3 4B Thinking

For my first round of testing, I chose Gemma 4 4B from Google and Qwen3 4B from Alibaba. They’ve gotten generally positive reception online, and ChatGPT recommended it for my machine and my common tasks.

I evaluated them against 3 categories of tasks based on how I actually use ChatGPT and Claude today:

• Editing and restructuring articles and social media posts

• Technical reasoning

• Financial planning

What Surprised Me

Before settling on Qwen3, I tested Qwen3.5 9B. It took 4-5 minutes to generate a response for simple editing tasks, but I was actually surprised that the writing was actually similar to my writing style. Given how slow it ran though, I wouldn’t be able to use it as a daily driver. But I’d say I was pleasantly surprised.

What Worked Well

Both Gemma 4 4B and Qwen3 4B did well for the high level technical reasoning task I gave it. I asked them to propose an architecture for a simple chatgpt clone. Both came up with high level proposals that’s comparable to plans I’ve produced with Opus and Sonnet, and identified some basic security issues e.g.

• Using a backend rather than calling an LLM provider directly from the frontend to protect my API Keys.

• Being mindful of SQL injection risks

What I found interesting is that these models provided practical guidance despite being dramatically smaller than frontier models. In fact, I’ve asked Opus and Sonnet to propose an architecture plan for this simple chat app several times, and once Sonnet suggested an architecture that had the front end call OpenAI’s API which would of course expose my API key - that’s a security no no.

That doesn’t mean the local models are better. It does suggest that if I’m building a simple/basic app, a small local model might be good enough for the high level planning.

What Worked OK

Writing assistance landing somewhere in the middle.

I wasn’t expecting either model to sound like me. Claude and ChatGPT only became better writing partners after months of interactions and a large amount of context about my style.

Without that personalization, both Gemma and Qwen rewrote my posts in a generic and obviously AI-generated way. That said, I found some value in tasks that involved restructuring my article content. Sometimes as I write it goes in a bit of stream of consciousness mode, so I use ChatGPT to add more structure. I found that Gemma/Qwen proposed some organizational approaches that I’d consider using.

What Worked Poorly

Financial planning was the weakest category by far.

Qwen3 4B did awful when it came up with recommendations for my retirement planning prompt. The responses had both factual mistakes and incorrect math. For example, it said my husband had to set up a spousal IRA for me, which is not true. Spousal IRAs would be set up by me and I’m simply borrowing my husband’s income.

Gemma 4B performed somewhat better. Its overall recommendation was reasonable, but it wasn’t clear on some aspects of the financial planning. I appreciate it recommended talking to a CPA instead of hallucinating wrong answers. Regardless, I wouldn’t use either for financial planning discussions.

My Final Thoughts

After my initial testing, I came away thinking that for low risk tasks like brainstorming, restructuring articles, summarizing, or basic discussions, they’re actually more useful than I expected. It’s worth testing and personalizing them more to see if they’ll become on par with ChatGPT and Claude for me.

Of course, for high risk decisions/categories of work, they’re not something I’d rely on.

Wrap Up

Next, I plan to test local models on more practical workflows like building prototypes, coding apps, and agent based tasks. Those are the areas where frontier models provide the most value for me, so I’m curious to see how much of that work can realistically be offloaded to local models.

To make this concrete, I’m going to build a feature I keep wishing ChatGPT, Claude, and Gemini had: Google Docs style comment threads for side conversations with your agent. The product reason is simple. Linear chat gets messy fast when you’re researching or writing because a side question about one paragraph pollutes the whole thread. But this feature is also a useful teaching example to make the underlying tradeoffs visible. Once you split one chat into multiple subconversations, you have to make explicit decisions about memory and context: what each thread should know, what history it should keep, and what exactly gets passed to the model APIs on every call.

In today’s tutorial, we are going to do a deep dive into how model APIs work. We’ll look at OpenAI’s chat completions and responses APIs, and Anthropic’s messages API, so you can see what actually happens every time you send a message i.e. how requests are constructed, how context is passed to the model, and how conversations are recreated on every API call. Once you see how this works, a few things also start to make more sense:

• why long conversations get slower and more expensive

• why tools recommend clearing context or starting new chats

• why building AI features often comes down to deciding what context to include

The UX Problem: Product Design and Decisions

I write and research a lot with ChatGPT, Claude, and Gemini.

Let’s say I’m writing an article on local LLMs and I ask: “What are good local LLMs I can run on AWS?” Claude gives me a 10 paragraph answer. As I’m reading, I want to go deeper on Meta Llama 3.3 70B and how quantization works.

But when I do that, Claude responds about Llama and quantization into the same thread.

Now I have one conversation that mixes:

• the main topic about running LLMs on AWS

• a deeper dive into Llama and quantization

This gets messy quickly. The conversation becomes harder to follow, different lines of thought get mixed together, and small edits to one section affect everything else. There’s no way to branch off.

Screenshot of current Claude Chat conversations

This project is going to change that. Instead of treating the entire response as a single conversation, we are going to break it into smaller units. I can have multiple sub conversations about different aspects of local llms without polluting the main thread.

Screenshot of Chatbot with Subconversations that I built

In building this, the main questions are going to be

• How much of the main conversation should we pass to the subagent?

• If the main conversation changes, should that update the subagents?

• Does the subagent only need to keep track of its own conversation history?

For our MVP, we are going to have multiple comment threads, each thread will have a snapshot of the main conversation at time of creation, and maintain its own context. This keeps the system simpler and prevents token usage from exploding.

In the future, you can imagine a more advanced setup using something like RAG where all threads are indexed and the model can selectively pull relevant context from each conversation. We aren’t building that here.

For now, we are going to go under the hood and look at how this actually works. We’ll walk through a simple agent architecture and then look closely at how model APIs calls are constructed for each interaction.

How an Agent App Works: A Primer on Basic App Architecture

Before we look at the model API itself, it helps to understand how this fits into a broader web app architecture.

Even though this is an AI app, the structure is actually very standard. Most web apps follow a simple three layer set up (often called a 3-tier architecture):

• the frontend (your app UI)

• your backend (your own API and logic)

• external services (databases, third-party APIs, etc)

In our case, the model API (OpenAI, Anthropic etc) is just another external service.

When a user interacts with the app, the frontend does not call the model API directly. Instead, it sends a request to your backend.

There are a couple of practical reasons for this:

• you don’t want to expose your API key in the browser

• you want a place to shape and control the request

• you want to handle things like streaming responses

Diagram of App Architecture

The flow will look like this:

1. The user interacts with the UI

2. The frontend gathers the relevant context

3. The frontend sends that data to your backend

4. The backend constructs the request in the format the model expects

5. The backend calls the model API

6. The response is streamed back to the frontend and rendered

FYI If you’ve been reading my articles but haven’t jumped into Claude Code yourself, I’m hosting an AI Builder Bootcamp where we do the hands-on work together. More details here here. Use BSUBSTACK25 for a 25% discount!

What’s Actually Sent: How Model APIs Work

Now let’s look at what actually happens when you send a message.

For this project, every time a user sends a message in a comment, the backend constructs a messages array and sends it to the model.

Note: a message array is basically a list of messages.

In my current implementation, that call uses OpenAI’s Chat Completions API. OpenAI defines Chat Completions around a messages array representing the conversation so far. Their newer Responses API uses a different input shape, but the same core idea still applies: you are packaging up the context and sending it with every request. Anthropic’s Messages API is conceptually very similar too.

Here’s code from the project:

//Build the system prompt that gives the model our full document history and specific paragraph
const systemPrompt = `You are a helpful assistant with context about a specific part of a document.

The full document:

${threadContext.messageSnapshot}

The paragraph the user is asking about:

“${threadContext.paragraphSnapshot}”

Answer the user’s question directly.`

//Build the messages array that will be sent to the model API
const messages = [
{ role: ‘system’, content: systemPrompt },
...threadContext.threadHistory,
{ role: ‘user’, content: userMessage }
]

//Send the request to OpenAI using the Chat Completions API
openai.chat.completions.create({
model: ‘gpt-5-mini’,
messages: messages,
stream: true
})

A Concrete Example: How the Messages Grow With Each Call

Let’s go back to the local LLM example from the previous section.

Say I ask the main chat: What are good local LLMs I can run on AWS?

And the model gives me a long answer covering Llama 2, Llama 3, Mistral, Falcon, hardware sizing, and setup guidance. One paragraph talks about Llama 3 and quantization. I click into that paragraph and open a thread.

My first message is: What does quantization mean?

At that point, OpenAI receives a request like this:

[{

role: ‘system’,
content: `You are a helpful assistant with context about a specific part of a document.

The full document:

Short answer

- If you want the best practical balance of quality, speed and cost for running locally on EC2: Llama 2 / Mistral 7B / Mistral Mixtral / Falcon 7B...

- Use huggingface transformers + bitsandbytes / accelerate or vLLM for GPU inference...

1) Good local LLMs and when to pick them

- Llama 2 (Meta)

- Pros: strong quality, widely used...

- Llama 3

- Pros: improved quality over Llama 2...

- Mistral 7B / Mixtral

- Pros: state-of-the-art quality-per-parameter...

2) Hardware (EC2) sizing guidance

- CPU-only (no GPU)

- Small/cheap GPU (single T4, 16GB) — g4dn family

...

The paragraph the user is asking about:

“Llama 3

- Pros: improved quality over Llama 2 (chat, reasoning) depending on size/version.

- Cons: similar VRAM footprint; check license and availability.

- Best for: higher quality chat-like applications when available.”

Answer the user’s question directly.`

},

{
role: ‘user’,
content: ‘What does quantization mean’
}]

If I then reply, “how much RAM do I need to run the models?”

The next request we send to OpenAI gets longer! We need to send the same system prompt, the main conversation history, the input and output from the What does quantization mean? question again, PLUS my new message!

[
{ role: ‘system’, content: ‘...(same system prompt with document + paragraph)...’ },
{ role: ‘user’, content: ‘What does quantization mean’ },
{ role: ‘assistant’, content: ‘Heavy quantization= aggressively reducing the bit‑width used to store model weights’ },
{ role: ‘user’, content: ‘How much does RAM do I need to run the models?’ }
]

This happens every time you send a message in a conversation to the Model APIs. The list of messages just keeps growing. The amount of tokens you are using isn’t simply the tokens from the last question “How much does RAM do I need to run the models?” it includes everything before that. So instead of costing 11-13 tokens, that RAM question could cost thousands of tokens!

This is why one of the best practices for minimizing token usage when you are building with coding agents like Claude Code is to /clear the conversation after you finish a feature, instead of having long running conversations. Every time you talk to Claude it sends your Claude.MD and all your previous messages.

Why This Gets Expensive (And How Side Chats Helps)

The reason we have to send everything back again is because models don’t have memory. They are “stateless.” The “conversation” is something the app has to send over and over by replaying the relevant history.

**Note: Some APIs support prompt caching, which can reduce the cost of repeated context like system prompts or document snapshots. But from a system design perspective, you should still assume that you are sending the full context on every request.

This brings us back to the feature I’ve been building. Instead of keeping one giant conversation that grows and grows, we break it into smaller scoped threads. That has a UX benefit, but it can also help control token usage because unrelated follow-ups stop inflating the main chat.

To make this more concrete, I researched the local llm question with current ChatGPT/Claude chatbot experiences, and then again with my side comment threads. In the linear version, I ask about local LLMs, then ask follow-up questions in the same chat about quantization, Llama, and local setup. In the side chat version, I keep the main thread focused on the broader local LLM discussion, then branch off into separate scoped threads for quantization, Llama, and setup.

Here’s what the linear version looked like in approximate input tokens sent to the model:

Current Chatbot Main Thread

• Turn 1: ask for good local LLMs to run locally → 40 tokens

• Turn 2: ask for a comparison of Llama, Mistral, Gemma, and Qwen → 193 tokens

• Turn 3: Quantization follow-up in main thread → 358 tokens

• Turn 4: Llama follow-up in main thread → 493 tokens

• Turn 5: Setup follow-up in main thread → 643 tokens

Total input tokens sent: ~1,727

NEW Side Chat Version of the same research:

Main thread

• Turn 1: ask for good local LLMs to run locally → 50 tokens

• Turn 2: ask for a comparison of Llama, Mistral, Gemma, and Qwen → 213 tokens

Side chats

• Quantization side chat → 146 tokens

• Llama side chat → 136 tokens

• Setup side chat → 171 tokens

Total input tokens sent: ~816

That’s a meaningful difference. The side chat version used about 53% fewer input tokens than keeping everything in one linear conversation.

You can see that just by comparing the Quantization follow ups.

• In the main linear thread → 358 tokens

• In a scoped side chat → 140 tokens

The reason is that the side chat only needs a smaller snapshot of the relevant context plus its own local history. It does not need to carry along every other unrelated turn about Llama comparisons or setup choices.

There is nuance here. Apps like ChatGPT and Claude Code likely use prompt caching to reduce the cost of repeated context. At a high level prompt caching works by recognizing when you send the same text again (for example, your systemp prompt). Instead of fully reprocessing those tokens, the system can reuse a cached representation internally and charge less for them.

In a long linear conversation, this can help because much of the earliest history is identical across requests. That means the actual cost may be lower than what raw token counts suggest.

Where Things Start to Break

So far, this setup works pretty well. We have:

• a main conversation

• multiple comment threads

• each thread with its own scoped context

This keeps things cleaner than a single long chat and helps control token usage. But there’s a few unsolved problems that still needs work

• No shared memory across comment threads. If I figure out something in one thread, the other threads don’t know about it

• Threads can go out of sync. Right now, each thread uses a snapshot of the document at the time it was created, which means older threads will reference outdated context.

This is where if I want to push this further, I’d explore using techniques like RAG to index all the conversations so we can fetch what’s relevant, or use a single “main conversation” snapshot for all the comment threads.

Wrap Up

In Part 2, we focused on models and how to choose between them.

In this post, we went a level deeper and looked at what actually happens when you call a model API. We walked through how requests are constructed, what the messages array looks like, and why conversations get slower and more expensive over time. From there, we built a subconversation feature to better match how we actually research and write, and used that to explore how context is passed, scoped, and recreated on every call.

And now I can do research the way I want to! I also added a fun feature so I can replace paragraphs for when I write 🙂

If you need more personalized help, feel free to reach out at bonnieyu.coaching@gmail.com for coaching sessions.

Recently, I’ve noticed more posts of people saying they feel overwhelmed by all the AI tools, concepts, and frankly, jargon. I hate seeing people stressed out by this, and feeling like they don’t even know where to start.

At the same time, as I prepare my clients for job hunting, I’ve also noticed an increasing amount of “AI Fluency” questions showing up in interviews. More companies are now expecting a base level of AI knowledge even if the role isn’t focused on building AI products.

So today’s article is about how to go from a complete beginner to being AI fluent for an AI Fluency PM Interview.

I’ve put a simple plan together around a dedicated 2 week sprint to get you there. In my experience, it’s hard to make progress on something like this if you spread it out across months. Compressing it into two weeks makes it easier to commit and actually get it done.

The time investment is about 9 hours per week:

• ~1 hour on weekdays

• ~4 hours on the weekend

For all you parents out there, you’ll probably need help from your spouse or partner. Ask them to help cover Saturday or Sunday mornings for 2 weekends. Thankfully, it’s only 2 weekends so the time commitment is more manageable.

One important note: this is not prep for an “AI PM” role where you build an agent or work on improving the models directly. This is for a “regular” PM role (e.g. onboarding, billing, productivity, platform) where companies are now expecting a base level of AI knowledge, and how it fits into your work.

The AI Fluency Questions

I compiled a list of questions based on my clients interviews, and contributions from the Women in Product Community. They fall into four buckets:

Work Related AI

• How have you incorporated AI into your workflow?

• If you use AI in your workflows, what are common mistakes you’ve seen come up?

• How do you reduce hallucinations and develop trust on your agent/workflow

• How do you evaluate or choose between tools and models?

AI Curiosity

• Have you used AI for any side projects or experiments?

• Which AI tools/models do you use regularly, and for what?

• Which new AI feature, model, or product release in the last 6 months has impressed you the most, and why?

AI Vocabulary

• How does a LLM work?

• What are challenges with a LLM’s context window?

• For more technical roles - What is MCP and RAG? When would you use it? What are the downsides of using them?

Live Building (rare but may become more common)

• Given a product problem, vibe code a prototype on the spot

• Build a multi agent system for X workflow. Check this reddit thread for more examples

The Action Plan. Your 2 Week Sprint

Week 1: Build Something Fun (9 hours)

• Get access to a few tools

• Learn the AI tool landscape

• Dive into a few AI concepts

• Build your first project

Week 2: Build your hero stories (9 hours)

• Set up a coding agent

• Build one AI workflow at work

• Build a 2nd project with a coding agent

• Turn your logs into interview stories

Note: Throughout both weeks start using AI at work

If you haven’t already, start incorporating AI into your daily PM work. Pick simple, low-risk tasks where you can experiment:

• Drafting meeting agendas or notes

• Brainstorming feature ideas

• Summarizing user feedback

• Writing your PRDs

The goal here is to get comfortable with the basic pattern:

Prompt -> review output -> validate -> iterate

You don’t need to log everything or turn this into a formal story yet. Just build the habit.

Week 1: Build Something Fun

The fastest way to learn AI is by building something. By the end of this week, you should have a working prototype of an idea.

Before we jump into the project, it helps to have a rough map of the AI tool landscape. Read through this so you know what’s out there, but you’ll get a better sense of all this after you’ve built something

Overview of AI Tool Landscape

General Assistants & LLMs

ChatGPT, Claude.AI, Gemini

What everyone is already using! General purpose chatbot for tasks like content generation, research, brainstorming, summarization, and strategic review.

Fast Prototyping

Lovable, Bolt, V0

Tools where all you need is a prompt (e.g. a set of requirements) to get started. I use them to rapidly “vibe-code” a prototype in a short time (e.g., 20–30 minutes), test a specific UX idea, or build a simple side project quickly to validate an idea.

Pros

• Fast and easy to get started

• Hosting, authentication, and simple analytics out of the box

Cons

• Cost. Token costs can get expensive quickly

• Agents struggle if you have complex logic or integrations

• Less flexible if you want more control over technology

AI-Enhanced Development Environments

Cursor, Replit, VSCode

These are coding environments with built in AI assistance. Compared to fast prototyping tools, they provide more control over the codebases, but more set up

Pros

• Ability to choose your coding agents

• Developer tools like code diffs

• Greater control over what you’re building

Cons

• Higher learning curve than prototyping tools

• Cursor and VScode does not provide hosting / authentication out of the box (**Replit on the otherhand does provide hosting and authentication)

Coding Agents

Claude Code, Gemini CLI, Codex

Coding agents can write, modify, and reason about code across a project. Many people feel overwhelmed by the coding agents because they initially lived in the terminal. BUT they’re more accessible now. For example, you can use Claude Code in Claude Desktop. I prefer using the coding agents natively because they are cheaper and more flexible.

Pros

• Cheaper than Lovable/Bolt etc. you’ll be able to build a lot more for the same amount of $$.

• You can build with whatever technology and integrations you want

• Coding agents are better at complex tasks.

Cons

• Can feel intimidating at first

• Requires more set up

• Does not come with authentication, hosting etc out of the box

FYI If you want to use Claude Code, I have a beginners guide for building with Claude Code.

Get your tools (30 minutes)

For this sprint, you’ll only need two tools! My recommendation:

One Fast Prototyping Tool

Lovable or Bolt are good starting points. Both are around $50/month

One general LLM (because it comes with a coding agent!)

Claude, ChatGPT, or Gemini.

Buy the $20/month plan from one of the frontier models. Each of the general assistants come with a coding assistant e.g. Claude comes with Claude Code.

You’ll spend about $70 for this 2-week sprint.

Afterwards, you can decide if you want to keep the subscriptions. We’re paying here because you don’t want to get blocked by rate limits while you’re working on a project.

Alternative: If you want access to a bunch of tools at once, Lenny’s newsletter has a deal where you subscribe for $350/year and get access to a large number of AI tools. This is overkill for a 2-week sprint, but worth considering if you’re planning to use these tools long-term.

Build one project with either Lovable or Bolt (3-4 hours)

This is the fun part. You’re going to build something this week, and it’s going to work. I suggest using a bigger block of time for this.

The workflow is straightforward

• Come up with a problem you want to solve

• List out a simple set of requirements

• Plug it into Lovable or Bolt, and work with their agent to plan and build your first app!

A few ideas on problem spaces to consider: Family activity planning, Financial analysis, Movie picker. If you want a more guided path, you can also use a tutorial as a base, then modify it so it’s clearly your own. Peter Yang has a tutorial where he provides you with prompts for different web apps and games like a poker game.

To make the project more complex, add something like authentication to the app. Have friends and family test it out, and make some feature updates.

As you build, keep a running log:

• What you’re building and why

• Key decisions and tradeoffs

• How you’re using AI (code generation, refactors, UX, etc.)

• Make sure you have a scoped feature you can explain end-to-end.

• Then turn that log into an interview story

Tip: You can use AI to help with logging the work! Just talk to a chatbot as you’re doing it

Screenshot of a Lovable project

Understand vocabulary and concepts (2-3 hours)

Now that you’ve built something, the vocabulary will make more sense. If you want a general overview of generative AI, Andrew Ng teaches a Generative AI for Everyone course on DeepLearning.AI. Watch week 1 videos (about 30 minutes)

It’ll touch on hallucinations (you’ve probably encountered some already), how context windows work, etc.

For more technical roles, they might ask you about MCP or RAG.

• MCP (Model Context Protocol) is a standard way for LLMs to connect and communicate with data sources and external apps. Example, you can use a LLM to access all your Granola transcripts and ask questions about your meetings with the Granola MCP

• RAG (Retrieval Augmented Generation) is an architecture where you store data in a separate database (usually a vector db) where the LLM can access more up to date or private information (e.g. your company’s information!)

If you want to learn more about these, DeepLearning.AI also provides videos on other topics such as RAG, but this is optional.

Oh, and all that AI Jargon…

There’s a lot of jargon in the AI space, and it feels like new words keep appearing everyday. Earlier, I mentioned the question “Build a multi agent system for X workflow.” Don’t let the words scare you. Decode this step by step.

In this context, it just means using multiple AI assistants together to accomplish a workflow. For example, one AI agent reads your e-mails and summarizes action items, another agent drafts responses, and a third schedules follow up meetings. They are connected together in a sequence to accomplish a task. Underneath each of these AI assistants are simply prompts that give them specific instructions. If you get asked this in an interview, they want to see if you can break down a workflow into steps and think about where AI can help at each step.

BTW if you need help with vocabulary, you can also ask your LLM.

Quick Note about Prompting

There’s whole courses about prompting, but with how advanced the current models are, there’s really only 2-3 things to know

1. Context. Explain the situation

   • I’m building a chatbot for kids to help them learn vocbulary. It needs to work on mobile and web

2. Guardrails. Define constraints

   • The chatbot should NEVER discuss anything violent or graphic

3. Structured outputs. Specify the format you want

   • Generate vocabulary words and definitions and output it as a JSON file

Sounds easy, but the tricky part is giving context, guardrails, or structured outputs in a domain you’re not an expert in. For example, you need to build a production grade app, but you’re not a developer. You may not realize you need to have the agent check the code for extensibility, scalability, and security, and you run into trouble. That’s when you need to talk to a general LLM about the problem space first to expand your knowledge.

End of Week 1

By the end of this week, you should have:

• experimented with a prototyping tool

• built a working prototyping

• start forming opinions about where AI works well and where it breaks down

You should have some idea about the AI landscape, and general prompting techniques.

FYI I’m an AI and Product Coach. If you have questions or you’re looking for a personalized coaching plan to improve your AI fluency, you can find me at bonnieyu.coaching@gmail.com

Week 2: Build your Hero Stories

Set Up a Coding Agent (30 minutes)

While they are called coding agents, Claude Code / Codex / Gemini etc are being used for many product management tasks and workflows. Remember that question around building a multi agent system for X workflow? You’ll be able to easily set up a multi agent workflow with one of these coding agents.

I use Claude Code as my main driver, but I do use the other agents too from time to time. I’d suggest either using Claude Code or whatever your company has approved for work.

FYI The first part of my Claude Code tutorial walks people through the installation, set up, and some Claude Code basics.

Create a “Hero” Story from Work (3 hours)

By now you should have some experience using AI in your day to day work. Pick one core PM workflow (e.g., spec writing, customer feedback synthesis, data analysis), and use a coding agent to solve it. If you need more ideas, Sachin Rekhi demos how he uses Claude Code for different product management tasks in this Youtube video.

Aim to build a concrete story like:

• What you were trying to do

• How you integrated AI into the process

• How you validated/corrected its output

• Did AI make any mistakes that you caught

If you want to build the workflow with multiple agents, you can learn about creating custom agents here, or you can test Claude Code’s new agent team feature here. As an example, you can also read Anthropic’s code review plugin which uses 4 agents in parallel.

Goal: have one strong, specific example you can reuse in AI fluency screens

Build a 2nd Project with a Coding Agent (5 hours)

I suggest building a second app with a coding agent. This gives you more to talk about in interviews and helps you understand trade offs between different tools.

If you’re able to use a coding agent at work, I suggest building a prototype for your next feature and share it with your designers and engineers. More and more companies are looking for PMs who can write short specs and prototype their features.

Keep the same logging approach as Week 1.

Optional: Practice Interview “vibe-coding” (30 minutes)

• Have AI or a friend give you a product problem

• Identify a user problem quickly

• Define a small scope that you can build in 20–30 minutes.

• Use one tool you know well to actually build something functional

  • I suggest using Lovable/Bolt here

• Focus on clear problem framing, sensible scoping, and moving fast with AI.

That’s It

Two weeks. The hardest part is getting started. This should get you to a good baseline for common AI fluency interview screens. From here, continue to be curious and experiment, and you’ll grow from there.

If you need help, feel free to reach out at bonnieyu.coaching@gmail.com.

What this tutorial will cover

• How to choose an OpenAI model for this use case

• Using the Context7 MCP so Claude can reference up to date information

• Workflow to prototype 3 design experiences

• Using Anthropic’s Frontend Design Skill

The Problem: Single File Uploads

One quality of life issue in ChatGPT today is file upload. It’s fine when you have one or two files, but it gets tedious quickly when you have more.

Here’s a pain point I personally experienced. I had 5 health records that I wanted ChatGPT to help me analyze. In ChatGPT, I had to:

1. Click “Add file.”

2. Select file 1.

3. Wait for the upload.

4. Repeat X more times.

This created unnecessary friction, but also limits what ChatGPT can do for you.

New Use Cases Unlocked

The folder attachment feature is a simple idea: instead of attaching individual files, you attach a folder as a container. Our folder attachment feature will also allow the AI agent to read, write, and organize multiple files within that container. This isn’t just a standard of quality improvement. It will unlock new productive ways to work.

Recap of My Claude Code Workflow

If you’re new to Claude Code and dev workflows, check out my previous post Building an AI PM Interview Site with Claude Code to walk through installation, setting up project memory, and basic dev workflows.

Here’s the high-level flow that I went through in my previous blog posts:

1. Plan the spec with Claude Code and save it to a markdown file

2. Ask Claude to propose an architecture, then have a second model (e.g., Gemini) review it. Save the plan to a technical-spec

3. Have another LLM such as Gemini review the architecture plan

4. Create a phase by phase plan markdown file to keep track of where you are at

5. Build incrementally with Claude (one phase at a time), testing as you go

6. Commit early and often to GitHub

7. For more robust work: open a pull request and use AI code review (Claude’s code-review plugin + another model for a second opinion)

8. Log bugs and ideas as GitHub Issues instead of context-switching mid-build

9. If you run into any bugs, copy and paste Claude Code errors you see in the console/server

That’s also the workflow I used for this project.

Choosing the OpenAI Model

Before diving deeper into implementation, we need to change our OpenAI model. I’m going to take a closer look at a few OpenAI models and, more importantly, how to read their documentation and understand what they actually offer.

This decision matters because it directly impacts:

• Cost

• Capabilities

• Which features are even possible to build

When choosing a model, I recommend starting with three questions:

• Do the model’s capabilities support the features I want to build?

• Does the cost make sense for my project or product?

• Is the speed acceptable for my users?

Feature Support

If you look at the OpenAI 3.5 Turbo documentation page, you’ll notice the model doesn’t support many features:

For my folder-based use cases, I need additional capabilities that GPT-3.5 Turbo doesn’t reliably support.

Structured Outputs

Guarantees the AI’s response matches a JSON schema (a strict format that makes AI outputs predictable) you define. This prevents the model from hallucinating keys or returning malformed data that can break your app logic. See here for details. The structured outputs allows you to use function calling reliability, and also help reduce error handling cases

Function Calling (aka Tool Calling)

This allows our ChatGPT agent to “call” a specific function. For our project, this will allow the model to tell our app “I need to read this specific file” which will trigger our code to read it. See here for details.

This is how our ChatGPT agent can:

• Read files

• Update files

• Write new files

• Delete files when appropriate

The model doesn’t access files directly. The agent will tells your app what it wants, and your code decides how to fulfill that request.

Web Search

You’ll notice on the model pages there are cut off dates for the model data (e.g. ChatGPT 3.5 is only up to date as of 2021). Web search allows the model to retrieve up-to-date external information. In my case, this is required to find and download earnings reports that aren’t part of the local folder. It’s not actually required for adding a folder, but it fulfills one of my use cases.

If a model doesn’t support a capability, the request will simply fail. For example, when I asked GPT-3.5 Turbo to perform a web search, it wasn’t able to complete the task.

GPT 3.5 Turbo’s response

Tip: You can test the OpenAI model capabilities here

https://platform.openai.com/chat

. This lets you switch models and enable or disable capabilities to see how they behave.

Cost

If you’re building a product intended for production, I recommend testing at least two models at different price points.

A simple approach:

1. Start with the highest-capability model that supports everything you need. This gives you the best possible experience.

2. Then test a lower-cost model that also supports your required features.

3. Compare the quality, speed, and reliability differences against the cost savings.

4. Decide which tradeoff makes sense for your product.

In my case, I chose GPT-5 Mini because it hits a sweet spot:

• Supports all required capabilities

• Fast enough for a good user experience

• Significantly cheaper than the flagship GPT-5 model

Tip: OpenAI has a page where you can compare the capabilities of different models https://platform.openai.com/docs/models/compare

Example Model Comparison

Below is a simplified comparison to show how different models stack up

The Prompt

I want to create a new spec and plan for the folder enhancement. The folder enhancement is a quality of life improvement, but also opens up new use cases.

Example pain point: I have a number of health record files I wanted ChatGPT to analyze for me. Today, I have to add each file one at a time. If I can add the folder, then ChatGPT can read all the files at once.

New use cases:

• ChatGPT can organize my files inside a folder

• ChatGPT can download earnings reports from XYZ companies and then write a summary for me. This will allow me to read the earnings reports myself.

Here’s the user experience

I want to explore a few different UX/UI options for attaching a folder. Make the “...” on the right corner of the chat header a dropdown where I can test 3 different UX/UI ideas

• Idea1: Add “Add Folder” under the Add files menu in the plus menu

• Idea 2: Move Add Files and Add Folder as two icons next to the Share icon menu

• Idea 3: Use the frontend dev design plugin to come up with your own idea on how the UX/UI will work

Core Functionality

• User clicks on a Folder icon and the app opens an explorer like window on the right for users to select a folder

• The agent can read or write to the folder

• If the agent wants to delete any file, it must ask the user for permission

Model Problem:

• We are using ChatGPT 3.5 Turbo right now. In order to fulfill the use cases, I had in mind, we will need a different version of the ChatGPT API. The ChatGPT 5 mini model could be a good fit.

• Use the context7 MCP to understand the capabilities and constraints of GPT-5-mini and evaluate whether it’s a better fit for these use cases.

Do not write any code. Let’s have a conversation to define the spec. Once the spec is finalized, output it as product-spec-folder.md. Spin up a subagent to research the different OpenAI models to identify which is the best to resolve my use cases.

Do you have any questions for me?

Introducing Context7 MCP for Up to Date Model Information

You may have noticed that in the prompt above I asked Claude to use Context7.

If you’ve been conversing with Claude Code on the OpenAI models, you’ll notice it makes a number of mistakes about model capabilities. For example, Claude Code by default thinks that GPT5-mini does not have support for web search, and that we’d have to use Tavily to handle it! That’s a big miss.

The issue is that Claude Code’s knowledge is only up to date as of January 2025 so it doesn’t have knowledge of the newest model.

Here I’ll introduce the Context7 MCP. Context7 makes it easy for Claude Code to look up the newest documentation for LLMs and various libraries. Instead of relying on stale training data, Claude can reference the latest official docs when reasoning about capabilities and constraints.

This will save you pain when you integrate with other libraries or services. Without Context7, Claude may implement an outdated or deprecated endpoint. You’ll see errors, but it won’t always be obvious that the issue is stale documentation.

Once I added the Context7 MCP, Claude Code correctly identified that GPT5-mini does support web search, and how to enable it appropriately.

Installing Context7 MCP

If you want to install the Context7 MCP in Claude Code, run this

claude mcp add context7 -- npx -y @upstash/context7-mcp --api-key YOUR_API_KEY

Additional installation notes and usage tips can be found here.

You’ll want an API key to increase your rate limits, which you can get on their website. If you run into trouble, ask Claude Code to help you add the MCP.

Type /mcp to see what MCP servers you are connected to

Note: MCP allows LLMs to talk to external apps and tools in a standard, reusable way.

If you’re worried about the MCP using too much tokens, a common issue with MCPs, here’s the token usage for the Context7 after I queried about the search capability

FYI: I’m offering a free lightning lesson to help new PMs (and non PMs) build up their UX and Design muscle in March. Sign up here. If you want more personalized coaching and practice, hit me up at bonnieyu.coaching@gmail.com.

Prototyping 3 UX Experiences

PMs are increasingly expected to prototype features,not just write specs. And when you prototype, it’s not enough to build one version and call it done. You want to test variations.

For this folder feature, I explicitly asked Claude to generate multiple UX approaches so I could compare them side by side.

In my original prompt, I wrote:

I want to explore a few different UX/UI options for attaching a folder. Make the “...” on the right corner of the chat header a dropdown where I can test 3 different UX/UI ideas

Then I gave it three directions:

• Add “Add Folder” under the Add files menu in the plus menu

• Move Add Files and Add Folder as two icons next to the Share icon menu

• Use the frontend dev design plugin to propose an alternative UX/UI

This is a simple but effective pattern. Instead of debating where something should live, you generate options and test them.

Claude produced three working variations that I could click through and compare.

Floating Button Design

Demo of All 3 Designs

After playing with the 3 options, I came up with a new idea. I realized that the “+” menu was overloaded with items, and decided to redesign the menu entirely. I asked Claude to move group the “+” menu items into 3 groups: Attachments, Actions, and Apps to represent all of ChatGPT’s menu items. Create 3 new icons and move it to the chat header.

Dropdown with All Design Variations

New ChatGPT Menu Design

That’s the value of prototyping multiple paths early: it prevents you from anchoring too quickly on your first idea.

Using Anthropic’s Frontend-Design Skill

For one of the variations, I asked Claude to use Anthropic’s frontend-design skill.

The frontend design skill is a curated prompt that Anthropic wrote to avoid common AI slop issues like generic layouts or color palettes. You can view the skill definition here

This is kind of the “surprise” me option.

Wrapping Up

In Part 1, we built the foundation: a working ChatGPT clone (repo here if you want to prototype a new feature too).

In this post, we walked through:

• The product pain point

• Model tradeoffs

• Updated documentation via Context7

• Prototyping multiple UX variations

The folder feature is intentionally scoped. It solves a real friction point, but it also opens a whole new set of use cases that the agent can do once it has access to your local drive.

In Part 3, we’ll push this further. Instead of a quality-of-life improvement, we’ll prototype a more complex feature that depends on the same foundation, and we’ll look at what breaks when the system gets more ambitious.

If you’re participating in the prototype challenge and building a new feature yourself, I’d love to see and share it. DM me your repo at https://www.linkedin.com/in/bonnieyu1/.

Let’s build it.

What this tutorial is about

The tutorial will build a simple ChatGPT clone from scratch. Nothing fancy. Something basic for us to build more features on top of to show what the experience might look like.

Along the way, we’ll:

• Use advanced dev workflows and Claude Code features (e.g. the code review plugin)

• Implement a basic chat architecture

• Explore message streaming and context management

If you’re new to Claude Code, start with Building an AI PM Interview Site with Claude Code where I walk through setup, Project Memory, using console to send errors, and other basic workflows. I will not be covering those flows here.

Prototype Challenge!

This project is intentionally designed to be forked and extended. I’m providing a very simple ChatGPT clone on Github (ChatGPTChallenge 2026 Repository), so you can build your own creative features.

Think of it as a prototype challenge:

• Add a feature you’ve always wished ChatGPT had

• Explore a different way of managing context or memory

• Experiment with new interaction patterns or workflows

This is a great learning exercise (and maybe a nudge to OpenAI to implement your idea!).

PMs today are increasingly expected to prototype new ideas, not just write specs. I’m a big believer in learning by doing. You can imagine yourself as a PM working in OpenAI, tasked with prototyping a new idea. In this project, you’ll build off an existing product, test different UX ideas, and figure out what prototyping workflow is best for you.

As a bonus, if you haven’t built an AI agent yet, this project will push you to learn about context management, message streaming, etc. Once you’ve gone through this process, you’ll be able to start answering some common AI PM interview questions.

Here’s a few AI PM Interview Questions to kickstart some prototyping ideas:

• Design a collaborative workspace for teams within ChatGPT

• As an OpenAI PM, build a new product feature for enterprise users

If you want to start prototyping, start by cloning the repo: https://github.com/yuqiaoyan/ChatGPTChallenge2026/tree/main

The tutorial will walk through building the ChatGPT clone and some more advanced development workflows.

**Disclaimer:** This project is not affiliated with or sponsored by OpenAI.

Planning the App

Writing the Spec

Here’s the prompt to get started:

I am building a simple ChatGPT clone. It will be a web app with a small set of features. Create a product spec with me, and output it as a markdown file (product-spec.md).

Here are some basic requirements:

• Responsive web app (desktop and mobile)

• Create New Chat

• Send messages and receive responses like ChatGPT

• A list of recent Chats in sidebar

• UI that matches ChatGPT’s design (include icons even though they serve no functionality) - see attachments

This will serve as a foundation for prototyping advanced features later. As an example of a future feature, add the ability to attach a folder to a ChatGPT chat, so users can pull all content from that folder (local folder on your machine) and create new content to be put into that folder.

Ask me any clarifying questions

** I screenshotted the ChatGPT app and gave that to Claude. You can also copy some of the CSS from the ChatGPT web app so it matches the UI better

Choosing the Model

Because this is a ChatGPT clone, I decided to use the OpenAI API.

I didn’t overthink model selection for the base clone. Claude recommended gpt-3.5-turbo by default, and it’s fast and inexpensive, which made it a good starting point.

For a simple chat clone, the quality is good enough, and the low cost makes testing cheaper. I’ll do a deeper dive into model selection in my follow up post where I’ll need additional functionality for the features I’m prototyping. At this stage, the goal was to get something working end to end, not optimize for model quality.

Architecting the Code

As usual, I asked Claude to propose an architecture for the ChatGPT clone app. Then I followed up with a critical part of my workflow - I asked Gemini to review the plan for extensibility, scalability, and security. See my previous post for more details on the workflow.

As usual, Gemini made some good notes

1. Local folder access is constrained by the browser. The File System Access API works, but mostly in Chrome/Edge

2. Flat message lists won’t support future branching. If I wanted multi-threaded conversations later (one of my planned enhancements), we’ll need to build with a tree data structure in mind

Based on the feedback, Claude updated the message data structure to include a parent-child relationship so we don’t need migration later.

Two key takeaways here:

1. Using another model to critique another is one of my favorite AI workflows

2. If you have future feature ideas in mind, include them early so your system can be architected to support them otherwise future migrations can be painful.

Key Architectural Decisions

• Context Management - One of the most important design decisions is how we handle conversation context. For our base clone, I chose the simplest approach to get started quickly: Every time the user sends a message, the app sends the entire conversation history to the model. The downside is as the conversation grows, the token usage increases and you eventually hit context limits. These are product decisions you need to consider when you are building your own agent.
  
  Note: Different models have different max context windows. ChatGPT 3.5 turbo has 16k tokens as the context window. See here for details
  

• Streaming vs Not Streaming Responses - We’ll be using ChatGPT’s Completion APIs with streaming enabled. ChatGPT feels fast because it streams tokens as they’re generated. Without streaming, the app would wait until the entire response is finished before displaying anything, which usually means staring at a spinner for several seconds. Even when total latency is similar, streaming dramatically improves perceived performance. This isn’t just a technical choice; it’s a UX decision.

Getting an OpenAI API key

What’s an API key?

An API key lets your app send requests to OpenAI’s models. It’s also how OpenAI tracks usage and billing. You should never put this key in frontend code or commit it to GitHub.

How to get one

1. Create an account with OpenAI at https://platform.openai.com/

2. Go to https://platform.openai.com/api-keys

3. Click Create new secret key

4. Copy it somewhere safe

Tip: For API Keys, I usually store it as Password in my password manager so I can easily retrieve it if I need to

You will also be able to check here https://platform.openai.com/usage for data on how much the API calls are costing you.

Where it lives

In this project, the key will go in a .env file:

OPENAI_API_KEY=sk-xxxxxxxxxxxx

That file should be ignored by Git (.gitignore) and never exposed to the browser. This might sound like a small detail, but one leaked key can rack up thousands of dollars in usage. Claude Code (or your favorite coding agent) can walk you through this if you’re not familiar with the process.

Building the App

Once you have the plan in place and a step by step phase, it’s time to start building the app incrementally. I’m approaching this a little differently than a throwaway prototype. Since I’m sharing this code with others, I want to add more robustness to our codebase.

To support that, I’ll be introducing a code review workflow I use when working with more complex projects.

Building the UI

For the frontend, I broke the work into incremental stages instead of trying to build everything at once.

The goal is simple:

• Build a small slice

• Test it manually

• Get feedback

• Move on to the next slice

This reduces risk and makes debugging much easier. If something breaks, I know exactly which change caused it.

Example Screenshot of Building Core UI Components

Using AI Agents and Pull Requests to Build More Robust Code

Since this is a more complex project, we are going to introduce a lightweight code review process built around feature branches and pull requests.

When we first pushed to GitHub, everything lived on the main branch. From here on, all new work happens in a separate feature branch and a pull request is how that work gets reviewed and proposed for merging back into main.

If this sounds like a lot of engineering jargon, here’s the simple version:

We let Claude write code in a separate place (“feature branch”), then ask for a code review (“pull request”) before we put it into our main codebase.

Instead of pushing changes directly to main with no review, I open a pull request from my new feature branch. And then I use an AI agent to review the code. Pull requests give LLMs a clean, focused piece of work that can be compared against the main code base. It makes it easier for the LLM to compare what changed against what already existed that will improve the quality of the code review feedback.

In other words: the pull request workflow will make AI code reviews better.

The Simple Workflow (You Can Ask Claude to Walk You Through This)

For non-technical PMs, this is the mental model I use:

1. Create a feature branch - This is where all new code lives while you’re experimenting.

2. Build and commit changes in that branch - You’re free to iterate without worrying about breaking anything.

3. Open a pull request back to main - This shows exactly what changed compared to the existing code.

4. Ask AI agents to review the pull request - Give them the PR diff and a specific goal (clarity, architecture, edge cases).

5. Decide what to merge - Apply feedback, then merge when it looks good.

You don’t need to memorize Git commands to do this. You can literally ask Claude:

“Create a feature branch, commit my changes, open a pull request, and walk me through each step.”

Reviewing Code with AI Agents

Once the pull request exists, I bring in AI agents as reviewers. There’s two main ways I do this:

1. Use Anthropic’s Code Review Plugin. This plugin spins up multiple agents to review the code for bugs, Claude.md compliance, and context based issues

2. Ask Gemini CLI or Codex CLI to code review for architecture, scalability, and edge cases in mind. Simply open a new terminal and ask the agent to review the PR.

Screenshot of the Code Review Plugins output

Screenshot of Gemini Code Review Output

Note: You can use Claude and Gemini Side by Side to run multiple tasks at once by adjusting the layout

A Quit Note on Plugins

Plugins extend Claude Code’s capabilities with skills, agents, hooks and MCP servers.

To install, type /plugin. Anthropic provides a number of first party first party Anthropic plugins such as code-review. Another way to think about it is Plugins is a delivery mechanism to package different Claude Code skills/agents etc.

Once Anthropic’s Code Review Plugin is installed, you can type /code-review to start the code review for the pull request:

Building New Chat

Now that we’ve got our workflows in place, it’s about working through the phases of implementation and building out the rest of the capabilities: API integration, State management, Chat functionality, etc.

Screenshot of the prototype in progress

Using Github Issues to Track Bugs and New Ideas

As you are working, you might hit bugs or ideas that you want to address later.

For example, once I had basic back-and-forth messaging working between the user and the agent, I noticed that the text was streaming much faster than ChatGPT’s behavior. It didn’t feel right, but I also didn’t want to stop and tune streaming behavior while I was still wiring up core functionality.

So instead of context switching, I simply asked Claude to create a Github Issue.

Github Issues is a project management tool built within Github itself to track ideas, feedback, and bugs. You can also use tools like Linear or Jira, but for small projects especially when I’m working solo I prefer to keep things simple. I already have the GitHub CLI installed, so creating and managing issues is quick. Other tools require additional setup or integrations, which adds overhead.

Screenshot of Github Issues UI

Wrapping Up (and What’s Next)

At this point, we have a working base ChatGPT clone!

In the next article, I’ll build on this base by prototyping a concrete enhancement: attaching a folder to a ChatGPT chat so users can create, edit, and reorganize content directly inside that folder. I’ll walk through how to think about model selection, product design, how to prototype interactions quickly with AI, and what tradeoffs show up once real files are involved.

This is part of a 3 part series. So I’ll end with a final article that pushes the prototype further. I’ll build a more complex feature on top of the same base clone - one I keep wishing ChatGPT or Claude would add!

FYI: I’m offering a free lightning lesson through Maven to help newer PMs build up their UX and Design muscle in March. Sign up here. If you want more personalized coaching and practice, hit me up at bonnieyu.coaching@gmail.com.

The goal of this guide is to lower that barrier.

The project we are building will be an AI PM Interview prep website. It will include a small collection of real AI PM interview questions that people have reported, plus some curated resources for learning key AI topics. The project itself is intentionally simple; the point isn’t to build something fancy, it’s to get comfortable working in a real codebase with Claude Code (CC).

Claude Code vs. Lovable/Replit/Bolt/etc

Before jumping in, let’s talk about why Claude Code, and when to use CC.

If you are building a simple app, a weekend project where you’re going to throw away the code, Lovable/Replit/Bolt etc are great for the job. They’re extremely easy to get started with, and they handle a lot of things for you automatically, like hosting and setup.

But if you’re building something more complex, or something you plan to keep working on over time, Claude Code is worth considering. It gives you more control, and it scales much better as your project grows.

Here’s the 3 reasons why I choose CC:

• Cost - Claude Code is part of Anthropic’s pro plan which I already pay for, so instead of paying for another tool I can simply use CC. OpenAI similarly gives you Codex CLI for their Pro plan

• Better at Coding - CC is generally agreed on as a better coding agent than Lovable or Bolt’s. So if you have a more complexity in the app, then I’d pick CC otherwise you’ll find yourself fighting more bugs

• Control and Technology Choice - Lovable uses Javascript, Replit uses Typescript. It locks you into a specific tech stack. If you want to build a native iOS app (which is written in Swift or Objective C), then that’s a problem. With CC, you can use any language or framework you want. You have more freedom

Installing and Setting Up

Claude Code can be used in all sorts of places. You can use it purely in terminal only, use it in Cursor, in Replit, in Claude Desktop or Claude Code Web.

In this guide, I’m going to use Claude Code inside VS Code. If you don’t know, VS Code is a very popular code editor (IDE), in fact Cursor and Windsurf are forks of it! You’ll be able to easily compare code diffs, integrate with your versioning system, etc. The best part is it’s FREE.

Install Claude Code

• Go to your terminal and type in:
  macOS, Linux, WSL

curl -fsSL https://claude.ai/install.sh | bash

Windows CMD

curl -fsSL https://claude.ai/install.cmd -o install.cmd && install.cmd && del install.cmd        

Install VS Code

• Download for your system here: https://code.visualstudio.com/docs/setup/setup-overview

Install Claude Code Extension in VS Code

In VS Code, go to Extensions panel and download and install the Claude Code Extension. This gives you tighter integration and a nicer UI for working with Claude Code. Click on the icon in the left sidebar to open the Extensions panel

Getting Oriented with Claude Code and VS Code

Understanding the VS Code Layout

Before we start building anything, it’s worth getting oriented in the UI.

Explorer - This is where you can see all your folders and files for your project

Claude Code Icon - Once the extension is installed, you can access Claude Code in the Sidebar

Using Claude Code

The sidebar version of Claude Code has a nicer UI and is generally more user-friendly.

Note: on Windows, the Claude Code sidebar can be a bit buggy, so in many places I’ll show screenshots using the terminal version instead.

If you also want to use terminal,

• Go to Terminal in the top navigation

• Click New Terminal

• Type Claude in the directory you want to work in

The directory you open Claude in is where CC will have access to update your files.

Connect Claude Code to Your Anthropic Account

First Things First: Model Selection

Next, let’s look at which model to use for CC. As of this writing, Anthropic offers three main models you’ll use with Claude Code: Opus, Sonnet, and Haiku. You can switch between them directly inside Claude Code by

Typing: /model

At the moment, Claude Code defaults to Opus 4.5. In the past, it used to default to Sonnet because earlier versions of Opus were extremely expensive in terms of token usage. Opus 4.1, for example, would burn through tokens so quickly that it was basically unusable on the Pro plan.

That changed with Opus 4.5.

Anthropic greatly reduced the cost of tokens, from $15/$75 (input/output) per million tokens on Opus 4.1 to $5/$25 for Opus 4.5. According to Anthropic, Opus 4.5 uses less tokens than Sonnet 4.5, so theoretically you should use Opus 4.5 all the time.

My recommendation: start with Opus 4.5. If you run into rate limits, latency issues, or cost concerns, then experiment with Sonnet or Haiku. It’s very easy to switch later.

FYI This is how it looks on sidebar view

Planning Your Project - The Product Spec

Now that we’ve picked a model, let’s do something very PMy: write a spec. When working with a LLM, a lot of the work you need to do is produce a spec so the LLM knows what to build.

We’re going to work with Claude to generate a lightweight product spec before writing any code.

This is an intentionally simple project for the sake of the tutorial. The only piece of complexity I’ve added to this is fetching questions from a Google sheet. If you want to simplify further, you could ask Claude to generate a handful of sample questions and store them locally for example, in a JSON file. That’s totally fine for learning purposes.

For now, we’ll treat this like a real (but small) product.

Prompt: Creating the Product Spec

Here’s the prompt I use to generate the initial spec. You can copy and paste this directly into Claude Code.

Write a spec for an AI PM Interview Prep website. Create a new folder called “Project Memory” and output the spec “product-spec” as a markdown file.The goal is to help product managers practice for AI PM interviews. A few quick requirements to get us going

• Responsive design for desktop and mobile

• Two pages:

  • 1: Shows Interview Question for PMs to practice against

  • 2: A list of free resources including top AI companies hiring

• Interview Question Page

  • Questions

    • Source questions from https://docs.google.com/spreadsheets/d/1tNUIgrWR_e9BOLioJmAxqvi-BOIV6C8X4UfF8zWDUWk/edit?usp=sharing

    • Display one randomly selected question at a time

    • Include “Next Question” button to fetch new random questions

    • Show question category

    • Question filtering by category

    • Use local storage to track practiced questions

  • Timer

    • Show the user a timer from when the question is shown

    • Toggle visibility option (show/hide)

  • User Contribution

    • Prominent “Submit a Question” button linking to: https://docs.google.com/forms/d/e/1FAIpQLSekX2c8CHa9LWY91kBVMg5lNCqkLUvQWJ9zyv8OZkaZ2UOB3Q/viewform

    • Brief instruction text explaining community contributions

• AI Resource Page

  • Top AI Companies Hiring - Curated list with company names and career page links

  • Topic-Based Resources - Grouped by key concepts (e.g., MCP, RAG, Fine-tuning, AI Ethics)

  • Each resource should include: title, brief description, and link

Please ask me any clarifying questions

Why I always end with clarifying questions

I always end with “Please ask me any clarifying questions.” This is intentional.

Even for a small project, there are almost always a few details I’ve missed or underspecified. Asking for clarifying questions gives Claude a chance to surface ambiguities early, and it helps align us before any real work starts.

In practice, Claude will usually ask a handful of sensible questions about scope, behavior, or edge cases. Taking a few minutes to answer them upfront saves a lot of back-and-forth later.

Screenshot of Claude asking follow up questions

At this point, you should have a product-spec.md file in a Project Memory folder.

Keeping Project Memory Across Sessions

Why did I create a product-spec.md and this project memory folder?

Claude Code resets most of its context between sessions. When you close a terminal or start a new Claude session, it does not reliably remember what you worked on last time, what decisions you made, or what’s left to build. That can get frustrating and cause more bugs as you work on your project.

Project Memory: How We Persist Information

I use a simple system to maintain information about the project across sessions. Inside the project folder, I keep a Project Memory directory with three markdown files:

• product-spec.md — the core product requirements and goals

• technical-spec.md — key technical decisions and system patterns

• progress.md — what we’re working on now, what’s done, what’s left, and known issues

We’ve already started with product-spec.md. After Claude generates the spec, I usually review it, make any edits I agree or disagree with, and then ask Claude to create the technical-spec.md and progress.md files next.

How Claude Uses Project Memory with Claude.md

Now let’s talk about how Claude knows to read and respect these files.

Claude Code automatically reads a special file called Claude.md at the start of every session. This makes it the perfect place to give Claude durable instructions about how to work in your repository.

Tip: Per Anthropic’s best practices guide, Claude.md is ideal for documenting things like

• Common bash commands

• Core files and utility functions

• Code style guidelines

• Testing instructions

• Repository etiquette (e.g., branch naming, merge vs. rebase, etc.)

• Developer environment setup (e.g., pyenv use, which compilers work)

• Any unexpected behaviors or warnings particular to the project

• Other information you want Claude to remember

In our case, Claude.md is also where we tell Claude how to use the Project Memory folder.

Creating Claude.md

To create Claude.md, you have two options:

• Use /init

• Create the file yourself

/init – this will let Claude generate its own Claude.md based on the current code base and whatever is in the directory

I have a Claude.md that includes my common instructions that I usually use. You can save the parts you like for every project in the global Claude.md which can be found:

• macOS / Linux: ~/.claude/CLAUDE.md

• Windows: C:\Users\<your-username>\.claude

What’s Inside My Claude.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

You are an engineer for a website to help product managers prepare for AI PM interviews. This means interviewing for top AI companies or for PM roles at any company working on AI products. The site features a question practice page with timer functionality and an AI resources page with curated learning materials and company links.

## Development Guidelines

- Limit comments inside the code

- Test all changes before marking complete

- Prefer to run single tests and not the whole suite for performance reasons

- Keep animations simple and working

## Important Notes

**Credentials & API Keys:**

Any time there are credentials, API keys, etc make sure to store them in a `.env` file

**Windows Compatibility:**

Make sure all commands work within Windows terminal; this is being developed locally on Windows

## Project Memory Folder

The Project Memory folder is key to understanding the project and allows you to continue effectively.

**Core Files:**

- `product-spec.md` - All core requirements and goals

- `technical-spec.md` - Key technical design decisions and system patterns to stay consistent

- `progress.md` - Current work focus, recent changes, what’s left to build, current status and known issues

**Project Memory Updates occur when:**

- Discovering new project patterns

- After implementing significant changes

- When user requests with “update proj memory” (MUST review all files)

**Note:** When triggered by “update proj memory”, review every memory bank file, even if some don’t require updates. The project memory must be maintained with precision and clarity as effectiveness in building the project depends on it.

At this point, your project should look roughly like this:

Folder Structure

This setup is a bit heavyweight for such a small project. But for bigger projects, Claude will be more consistent across sessions especially around the engineering patterns (in technical-spec.md) so you have less bugs. It’s also easier to build the project in increments when the phases are clearly defined (in the progress.md).

Once this structure is in place, building becomes smoother.

Architecting your App with Plan Mode

Next, we are going to use plan mode to architect your app. In more complex projects, it’s worth spending time on the architecture for scalability and extensibility so you don’t end up with a pile of spaghetti code.

Claude Code has a feature called Plan Mode, which is designed specifically for this kind of deep thinking and structured planning. Instead of jumping straight into implementation, Plan Mode encourages Claude to reason through architecture, tradeoffs, and sequencing first.

To enter Plan Mode in Claude Code, press Shift + Tab and cycle through the modes until you see Plan Mode enabled.

A Note on “Ultrathink”
In the past, a common tip was to pair Plan Mode with the keyword “ultrathink” to encourage maximum token usage and more detailed reasoning.

As of mid-January 2025, it appears that:

• “ultrathink” is no longer explicitly supported

• Maximum token usage now seems to be the default behavior in Plan Mode

There wasn’t a formal announcement from Anthropic, so this may change again. This is another good reminder that AI tooling evolves quickly.

Generating the Plan: Google Sheets API Integration

With Plan Mode enabled, here’s a simple prompt I used to kick off the discussion:

Break down Phase 1 into smaller tasks that can be tested and validated. Architect a plan for building the google sheets API integration for review. Optimize for simplicity, testability, clean separation of concerns, and security

Note: If you are building a project, you want to ship and maintain, you can use the phrase “Architect a production-grade, phased plan”

Claude’s output will usually be structured and fairly detailed. Here’s a sample of what it produced for the Google Sheets integration.

Google Sheets API Integration - Deep Dive Architecture

Core Strategy: Build the API integration layer first as an isolated, testable module before connecting to React components.

Key Architectural Decisions:

1. Separation of Concerns: Create a dedicated services/googleSheets.js module

2. In-Memory Cache: Use a simple JS object with timestamp for 1-hour TTL

3. Error Recovery: Multi-layer fallback (cache → retry → error state)

4. Data Transformation: Raw sheet data → normalized question objects with generated IDs

API Integration Steps (7 testable tasks):

Task 1.1: Google Cloud Setup & API Key

- Create Google Cloud project

- Enable Sheets API v4

- Generate API key with HTTP referrer restrictions

- Store in .env file as VITE_GOOGLE_SHEETS_API_KEY

- Test: Verify API key works with curl/Postman

Using Another LLM to Validate Claude’s Plan

From here, you’ll want to work with Claude to finalize the plan. When I’m working on architecture or a tricky problem, I’ll often ask another LLM to validate Claude’s Plan. In this case, I asked Gemini CLI to review Claude’s plan for any security issues.

Tip: Copy and paste the Claude.md and rename it Gemini.md so Gemini has the same project context

Starting Gemini is as simple as starting up Claude

And this paid off! Google discovered a potential security flaw with Claude’s plan.

Adjusting the Plan and Fetching the Question Data

Based on Gemini’s feedback, we changed the approach slightly. Instead of directly calling the Sheets API from the app, the new plan was to:

• Fetch and parse the data from a Share Link CSV

Once the revised plan was in place, I asked Claude to move forward:

Build Phase 1 MVP using CSV data integration, and create a test script to validate that the data is parsed correctly.

At this point, we finally moved from planning into implementation.

Debugging your First Issue

Right away, I hit a bug. Looking at the developer console in the browser, I could see I was getting a 307 Temporary Redirect.

Tip: When reporting a bug, share error information you see to CC to help it debug your issue faster. Since this is a web app, we can inspect the console and copy errors from the browser to Claude

After debugging together, the fix was to create a new published CSV link directly from Google Sheets:

https://docs.google.com/spreadsheets/d/e/2PACX-1vT8ZbiNbj00TARest-Srk4EwWuBTz6d7JDFE2B6kSW06GEECC9U42ksM3RyF9ssJ6Pl3pHoo--rPhLw/pub?gid=1423087399&single=true&output=csv

Once that was in place, the redirect issue disappeared and the integration worked as expected.

Why This Workflow Works

This sequence plan → validate → build → debug is something you’ll repeat constantly when working with Claude Code.

The key takeaway isn’t the Google Sheets integration itself. It’s learning how to:

• Plan before coding

• Use Plan Mode for architecture

• Validate ideas with a second LLM

• Feed concrete error data back into Claude

That’s how you get consistent, high-quality results.

Version Control Basics

Once you’ve validated the code works, it’s time to think about version control.

Claude Code can write and modify files, but it’s not a source of truth for your project. If something goes wrong, you need a reliable way to get back to a known-good state. That’s where Git comes in. I use GitHub to track changes so I can always roll back to a known-good state.

Tip: After every meaningful feature or change, commit and push your code. This gives you a safety net and makes it much easier to recover if something breaks later.

Where /rewind Fits (and Where It Doesn’t)

Claude Code does have a command called /rewind, which lets you roll back changes within a session. It’s useful when Claude makes a change you don’t like and you want to undo recent edits without manually reverting files.

That said, /rewind is not a replacement for Git.

A helpful way to think about it:

• /rewind → short-term, in-session undo

• Git + GitHub → long-term version history and rollback

If you close a session, hit context limits, or want to return to something you built yesterday, /rewind won’t help you. Git will.

Github and Github CLI

If you haven’t used Git much before, the GitHub CLI is a nice way to simplify things:

https://cli.github.com/

Once you’ve created a repository on GitHub, you can ask Claude to help you set everything up locally and connect your project to that repo. This is a great example of using Claude for “glue work” that’s annoying to do manually.

Claude also created a .gitignore file for me, which tells Git which files should not be tracked like dependencies, build artifacts, and environment variables.

A Very Basic Git Workflow

If you’re not familiar with Git, here’s how a basic workflow for using Git looks like:

• First, you tell Git which files you want to include in your next snapshot with the command “git add.” This is called staging your changes.

• Next, you save those staged changes into your local history: “git commit -m ‘Short Description’”

• Finally, you upload your commits to GitHub with “git push”

If you are working on a solo project, that’s usually enough. In team environments, you’d typically open a pull request, get feedback, and then merge.

Common Directory Structure

As the project grows, your folder structure will start to look more complex.

Claude set this up as a Node project, which typically looks something like this:

• node_modules/ — all your dependencies live here

• src/ — all your application code lives here

You don’t need to understand every folder. What matters is knowing where your code lives (src) and what should not be committed (node_modules, which is handled by .gitignore).

Screenshot of how mine looks like

Building the UI

CC could probably one shot all the UI for this project, but I did it in stages. I asked Claude to break the UI work into smaller, testable batches. The benefit of this is it’s easier to fix any bugs that come up.

If something breaks, you know exactly which change caused it. It also keeps Claude focused on one thing at a time, which generally produces better results.

Screenshot of how my UI looks like

The Context Window, Memory Management, and Compact

At some point, you’ll run into Claude’s context window.

Right now, most people are working with a 200k token context window in Claude Code. Anthropic has been experimenting with much larger windows (up to 1M tokens) for some users, but availability is still limited.

Note:

• The context window is the maximum amount of text Claude can read and reason over at once.

• Tokens are roughly “word-like” units (not exactly words, but close enough for intuition).

There’s 2 key issues to keep in mind

1. First, as you approach the context limit, model performance can degrade. Claude Code shows a usage indicator in the bottom-right corner so you can see when you’re getting close.

2. Second, when you run out of context, Claude Code automatically runs compact. Compact attempts to summarize and retain only the most important information from the session. The problem is timing. If compact runs while Claude is actively modifying your code, the output quality can drop significantly. Some people have reported partially applied changes or inconsistent results when this happens mid-edit.

What to do?

• What I do is, after completing a big feature or a batch of work, I ask Claude to “update the project memory” and then use the command /clear to start the conversation from the beginning.

• People who utilize compact will simply run /compact before building a big feature

Tip: The command /context will show you where your tokens are being used

Hosting the Project

Once you’ve built the UI and you’re happy with how the site works locally, the final step is getting it online. Because this is a simple side project, I like to keep hosting simple too. I use Render for projects like this.

Render lets you:

• Connect directly to your GitHub repository

• Specify the build and start commands

• Deploy your app in just a few minutes

You don’t need to think about servers, infrastructure, or deployment pipelines. You push to GitHub, Render pulls the latest code, and your site is live.

At this point, the project is “real.” You can share it, iterate on it, or just move on knowing you’ve shipped something end-to-end.

Wrapping Up and Key Takeaways

If you zoom out from all the tools, models, and setup steps, there are really just a few ideas that matter from this tutorial.

Everything else is implementation detail.

Key Takeaway #1: Planning Is the Real Force Multiplier

The biggest difference between a smooth experience with Claude Code and a frustrating one isn’t which tech stack you use, it’s how much you plan before you build.

Using Plan Mode, writing a lightweight product spec, breaking work into testable phases, and validating architecture before coding all dramatically reduce thrash. Claude is very good at generating code, but it’s much better when it’s operating inside a clear plan.

If you ever feel stuck or overwhelmed, that’s usually a signal to pause and re-plan, not to push harder.

Key Takeaway #2: Markdown Files Are the Real Unlock

If you look closely, almost everything that makes Claude Code feel “smart” over time comes down to markdown files.

• Your product spec.

• Your technical decisions.

• Your progress tracking.

• Your Claude.md.

In a future article, when we use more advanced features like agents and slash commands, you’ll notice they’re also just markdown files with instructions. It powers all the reusable workflows within CC.

Once you internalize that, Claude Code will hopefully be less intimidating.

Key Takeaway #3: Claude Code Is Always Changing

Commands change. Defaults change. New features appear sometimes without announcement.

Claude Code will keep adding convenience features: better memory, better undo, better tooling. Those features are helpful, but they are helpers, not foundations. You don’t want your entire workflow to depend on any single command or default behavior staying the same.

That’s why the core practices in this tutorial matter:

• Plan before building

• Store durable context in markdown files

• Commit frequently to Git

How I Keep Up with Claude Code Changes

Because Claude Code is moving quickly, I don’t rely on any single tutorial (including this one) as a long-term source of truth. In practice, I keep up to date by:

• Skimming discussions in subreddits like /ClaudeAI

• Following Anthropic’s product and engineering updates

• Occasionally check the Claude Code changelog on GitHub

• And, most importantly, noticing changes as I use the product day to day

You don’t need to track everything closely. A light habit of staying curious is usually enough.

The Finished Project

If you want to see what the finished version looks like, the site we built in this tutorial is live here:

https://ai-pm-interview-prep.onrender.com/

If you want to get promoted or get your project through, you need people to like and trust you. The way you interact with your peers and leadership team doesn’t just build relationships, it also builds a credible reputation that will travel across the company. Ultimately, you want to earn trust and get sponsored, and working well in meetings is one of the most impactful ways to accomplish that.

My younger self thought of meetings as time sinks, but now I know how crucial they are to be successful in your career. I’ll walk you through different types of meetings and how to make the most out of each one. I also invited VP of Product Laura Williams Argilla to share her thoughts, and wrap up with a tip on how to practice these skills.

People and Relationships - Building Your Reputation

How you show up in meetings shapes how people talk about you when you’re not there. These moments are where relationships are built or… damaged.

Building relationships takes time. This is probably obvious, but you’ve got to start on day 1. Meet with all your core team mates, and get to know them as people, and as coworkers. Use these conversations to learn how the team actually works. They are a key information resource.

In my early 1:1s, I start with these questions:

• Intro and background (yea, you have to do this to find common ground and interests. It’s not just chit chat.)

• What made you decide to join the team? How do you like it so far?

• What do you expect from your PMs?

Depending on the role, I’ll also ask:

• Engineers to walk me through the product architecture

• Designers to walk me through customer research and design decisions

• Data scientists to walk me through dashboards, experiments, and key metrics

Now my favorite questions:

• What are the biggest challenges the team is dealing with right now?

• What’s one thing you’d change about how the team works?

• What do you think are the biggest issues with the product?

These are my favorite questions because they let people vent. I can immediately uncover problems to jump in and solve. For example, on one team I joined, I discovered tension between engineers and management around stand ups. Some devs didn’t find value in having stand ups. I got a quick win by (1) helping the devs understand how (if run properly) stand ups can uncover risks or challenges and (2) suggesting we only do stand ups 3-4 days a week instead of everyday.

Helping people solve problems and finding common ground are the fastest ways to earn trust.

Project Meetings: It’s Not Just About the Agenda

There are many types of project meetings, but here are the things I focus on to get the most out of them.

The most basic advice for hosting meetings is preparing an agenda ahead of time, but if you want to run a great meeting that’s not sufficient. If you are the host of the meeting, be very clear about the outcome you want from the meeting. Meetings are expensive. Try to avoid gathering a bunch of people unless there’s a clear reason people need to be in the room together.

Before the meeting, ask yourself:

• Do you need another team to help build something for your project?

• Is this a status meeting to identify challenges and unblock progress?

• Are you trying to get buy-in on a decision?

If you’re not the host, it’s just as important to understand the purpose. And if it’s unclear after the first few minutes, make sure to ask, because…

The hardest skill in running a good meeting is tracking everyone’s point of view. Meetings often get derailed when someone brings up something related but actually independent of the problem at hand. When this happens, I go back to the goal of the meeting. If I don’t immediately see how a new discussion moves us closer to that goal, I’ll ask the person to help connect the dots.

For example, I might say: “Can you help me understand how this ties back to the decision we’re trying to make today?” Sometimes they’re seeing an angle I hadn’t considered, and that’s valuable. Other times, asking the question helps everyone realize it’s important, but better handled separately. In that case, I’ll explicitly park it and move the group back to the original goal.

Executive Meetings (VPs + C Suite): Upping your Game

A friend once told me executives are just people, you should treat them as equals. In principle, I’d like that to be true. In reality, executives are a Special Case. Earlier on, I took the “treat them as equals” too much to heart and let’s just say it didn’t help my growth….

Like all people, some executives are easier to work with than others. But the one thing they all have in common is that they’re busy. Even the most supportive executives will be more receptive if you’re succinct, direct, and clear about what you’re asking for.

Before any executive meeting, it’s worth understanding how that executive tends to reason, what’s top of mind for them, and how your proposal fits into that context. If you don’t have that information, ask your manager or peers. They’re often your best source of signal.

While I’m framing this as an executive section, these principles apply to working with senior leaders more broadly. Here’s my suggestion for preparation:

Gather your Allies and Get Alignment

This is probably obvious, but before an executive meeting, make sure other people are aligned with your proposal, and you’ve got a chance to get feedback from everyone. Try to avoid tackling too many issues in one meeting

What’s less obvious is asking those same people:

• What kind of questions do you think Executive A will have?

• Do you think Executive A will have objections?

• What’s top of mind for Executive A right now? How might this tie in with that?

Role play tough questions

If you struggle with executive presence and communication, find a reliable coworker or find a coach to practice running through what you will be discussing. Ask them to ask you the hard questions so you’re ready when the executive asks you.

Practice “I don’t know the answer, but I’ll get you the answer by XYZ date”

As a PM, you often feel like you always have to have the answer, because you’re the domain expert. If you don’t know the answer, say you don’t know, but if you know the answer and you feel like you’re struggling to find the right words, it’s ok to say

“I don’t know the answer at the moment, but I’ll give you an answer by XYZ date.”

The challenge is saying it. It’s easy to put on paper, but a lot of us are trained to give an answer, and if you’re one of those people, practice saying it. Literally out loud.

From the VP Seat: Laura Williams Argilla

At this point, everything I’ve shared is from my experience as a PM. I also wanted to include advice from an experienced executive, so I asked Laura Williams Argilla. She is currently VP of Product at Alteryx, and previously held product leadership roles at Netflix, Tableau, Twitch, and Adobe. Below is Laura’s perspective, in her own words:

“My best advice for presenting to executives is to go deep in preparation, know your content, possible questions and where things might get tricky. Now keep that in your head. Here’s why: In one meeting with a CEO, I got one line into the presentation I had painstakingly created before he took the conversation in a whole different direction. Luckily I had done the homework ahead of time to make sure I was able to answer his questions, build confidence and eventually get to a useful outcome, even if it wasn’t the path I had planned.

A few tactical tips:

If you use slides, use as few as possible and don’t clutter them with information. You want the focus to be on what you are saying, not on deciphering an overloaded graphic.

To do this, you need to make sure you focus on the goal of the specific meeting. First, clarify your meeting goal: Is this an update? A decision point? Are you requesting resources or gathering feedback?

Think about the key information needed for the executives to fulfill that goal.

Start by making clear what you hope to achieve. If you need to make a decision, say that. Are you asking for resources? Tell them. Is this an update on a key project that is purely informational? Let them know. By stating it up front you have given them the right lens to view the conversation.

Next, start with the outcome you want. Sounds backwards, but again, this points them to the crux of the meeting. Once you state what you are looking for, tie that outcome to the business strategy and investment themes it supports.

Now provide the context on how you got to this conclusion, but remember, keep it brief. Give the top reason this is the right solution. Include data to back this up. At the same time acknowledge what changes with this information. Do your dates change? Is the project more expensive? Does it change messaging or need other stakeholder support?

Now open up for questions. This is where you draw from all of the up front preparation, and if they hit on something you don’t yet know, simply say that you don’t know but will find out and get back to them with the answer. If you are getting pushback, instead of moving immediately into defense mode, ask questions to better understand their concerns. This is a great opportunity to learn what they see as key business issues, conflicts and opportunities. It may also shape your approach moving forward.”

But What if Preparation isn’t Enough?

What stood out to me in Laura’s advice is how much executive effectiveness comes down to preparation and clarity. But even when you do all of that well, you might not get alignment from executive leadership. Here I’ll share my thoughts when there isn’t alignment:

Know when to pick your battles

Not everything is worth convincing your executive to agree on.

In the past, I worked with an executive who wasn’t a design leader, but still had very specific design tastes. Our designer presented multiple designs to this executive, and it was all getting rejected. The executive mentioned that he liked the Google designs, which translated to me as “I will approve this design if it looks more like Google Drive.”

The designer had strong reasons for their original work, and the designs had been tested, but the Google Drive design was a reasonable design (sure there were some downsides, but I didn’t think it was going to fail our customers) and it would get the executive on board. I asked the designer to take inspiration from Google Drive and modify his design off of that, and the design was approved.

Have I pushed back against executives? Yes, but I do it when I feel like this problem is of strategic importance or would break the experience for customers. In a situation with an executive that’s already set in their way, it will take multiple meetings and a lot of support to change their mind.

Know when you shouldn’t be the messenger

In some situations, especially for high-stakes or sensitive decisions, it may actually be more effective for your manager or a more senior peer to present the case. This isn’t about avoiding executive conversations, it’s about optimizing for the best outcome for the project.

Some executives respond better to certain styles. Some are more abrasive and need someone who can cut through the intensity to get to the real concern. Others like to dive deep into details and benefit from a presenter who can go several layers down with them in real time.

If someone on your team has a better working dynamic with a particular executive, it can be worth leveraging that especially when the decision really matters. The goal isn’t to prove you can handle the room. The goal is to get the right decision made.

Be Open to Feedback

Executives often have information ICs don’t. Sometimes their feedback may be related to a different project that they see going on with a different team but you’re not aware of. Or sometimes they have experiences that you don’t. Try to be open to their feedback, and… this is hard to pin down, but where the feedback is coming from exactly. Ask questions to get more insight into their thinking.

1:1 with your Manager as a Career Tool

Prepare your agenda ahead of time. I usually have a template that I fill out with a section about Project, Career, and Action Items. It’s easy to let project status updates take up all your time, so make sure to switch Career in the front from time to time

Be clear in your own mind of what you want out of your manager

• Do you need help with building relationships with an ally?

• Do you need clarity on what’s necessary for your promotion?

• Also, look for feedback that they’ve heard about you (that’s what impacts your reviews!)

Write down accomplishments along with key details and impact.

• Pro Tip: I write it as if I’m writing my performance review so that when I need to do my performance review, I can copy and paste from my 1:1 agenda. Now you can even feed this document to AI so it writes your performance review for you!

New Year Resolution: Practice Soft Skills on Purpose

As an introvert, I’ll be the first to say a lot of these skills are easier said than done. Soft skills are tricky because it’s hard to see clear levels of progress, but they are practicable. The key is to treat them like any other skill:

1. Be aware of the problem

2. Pick one specific skill to work on

3. Create a situation where you can practice it

4. Reflect on what worked and what didn’t

For example, if you want to get better at tracking people’s perspectives in a meeting, try this: in the next 5 meetings, take notes of the meeting goal, and whether the discussion is aligning with it. Are people talking past each other? Are certain concerns getting repeated but not resolved? Ideally you have a transcript, so after the meeting, you can have AI compare your notes with the transcript. Did you miss something? Did you misinterpret a concern? And then spend some time thinking, should you have intervened, and if so, what would you say next time?

In the new year, I highly recommend carving out 1-2 hours to choose one skill or relationship to improve on and create a small, intentional plan to work on it, because this will definitely pay dividends. If you want more personalized coaching and practice, hit me up at bonnieyu.coaching@gmail.com.

“A new generation of apps that are adaptive, interactive, and personalized.”

This launch is more than a new API; it’s a strategic decision to evolve ChatGPT into a “super assistant” capable of executing real tasks end to end. Instead of apps embedding AI agents, OpenAI wants to bring the apps inside ChatGPT.

After spending the last few days reading the developer docs, and testing early apps like Zillow, Expedia, Spotify, and Canva inside ChatGPT, I’ll share what’s working, not working, and what ChatGPT needs to do to make this ecosystem successful for both developers and users.

App & Assistant Experiences

For the last 18 months, major products have been racing to build in-app agents e.g. Rufus inside the Amazon app or Notion AI inside Notion. These experiences are helpful, but each one is limited to the data and context of its own product.

OpenAI could have strengthened this model by letting apps authenticate and access your ChatGPT identity and memory. ChatGPT already knows details about your life: your family composition, dietary restrictions, calendar commitments, etc.

OpenAI did not choose that route. Instead, OpenAI is bringing apps into ChatGPT. ChatGPT becomes the primary surface where users accomplish tasks and get things done. Does this model unlock differentiated experiences that in app agents (even with ChatGPT) simply can’t?

The ChatGPT Apps

The ChatGPT Apps directory currently includes 30 apps to show off the new experiences. The top 5 categories include: Productivity, Finance, Travel, Graphic and Design, and Developer Tools.

Sample apps available in ChatGPT’s directory

Distribution of App Categories

Which Apps Shine and Which Fall Flat?

Winning Apps

After testing the early partners, Canva was a clear winner. It quickly created posters for my son’s lemonade stand. If you’re not a designer, ChatGPT + Canva is magic. You describe what you want; the app produces it. My hypothesis is that apps where you compress a skill gap into a single prompt with a simple output will feel like magic.

Productivity and Content Creation type apps, “one-and-done” workflows, seem to work well.

Figma is another good example. Here Figma generated a decent UX flowchart based on a simple prompt.

But I also found limitations with today’s integrations. In this conversation, ChatGPT confidently told me it could create a high-fidelity UX design for me, only to walk it back when Figma’s API didn’t support the functionality necessary.

What can ChatGPT do to motivate developers to continue investing in these app experiences so users can truly complete their tasks? We’ll talk more about that later.

Poor Fit Apps

Apps that require iterative filtering and visual scanning fell flat e.g. Expedia and Zillow. It was surprising the Zillow experience wasn’t better since it was featured in the OpenAI Keynote. OpenAI obviously wanted to show off the interactive map, but as a tool for actually finding a home, the UX fell short.

You can’t even do some of the core tasks for researching homes… e.g. filtering. Every time you need to do a quick filter because you forgot you wanted 4+ bedrooms, or a certain price, you need to ask ChatGPT. ChatGPT slows these workflows down dramatically.

A Concrete Example: Expedia App vs. Expedia Inside ChatGPT

Expedia’s Native App

A typical flight search has:

• visible filters to reduce cognitive load

• instant date adjusters to explore options

• hundreds of results pre-loaded for fast re-filtering

• visual comparison

The app is optimized for rapid searches.

Expedia Inside ChatGPT

Here’s an example of me asking ChatGPT

ChatGPT responded with a carousel of 5 responses from Expedia, and a page long summary of which flight it recommends. I waited for these results for nothing, because

I forgot to tell ChatGPT I only want non-stop flights…. So I’m waiting for results again

With ChatGPT you need to:

• hold filters in your head

• express them in a prompt

• wait for the LLM to “think”

• wait for a tool call to run

• wait for ChatGPT to show the “in app UI” and write the results in text

The result is:

• high cognitive load

• high latency

• limited result sets (only 5 in the carousel!)

• friction in every iteration

As a PM, one of our jobs is to recognize what makes good UX and what causes UX to fail. The Expedia and Zillow experiences inside ChatGPT exposes the downside of LLM interactions. Or what I call the LLM Interaction Loop.

The core UX loop for LLM experiences today:

• User thinks and write prompt

• LLM thinks and writes results (User waits)

• User thinks and writes prompt

• LLM thinks and writes results (User waits)

This loop happens over and over again. When you have to do this loop for simple fast interactions like filtering, that’s when the ChatGPT experience fails. The Expedia and Zillow app experiences optimize for visual scanning, speed, and low cognitive load for the user. ChatGPT on the other hand makes all of this worse.

FYI I’m a Product Manager coach and consultant. I help PMs upskill in areas like product strategy, AI, UX, metrics, and so on. Connect with me at bonnieyu.coaching@gmail.com or LinkedIn

What Can OpenAI Do to Make the Apps Ecosystem Better?

The App ecosystem is a big opportunity for OpenAI. Yes, ChatGPT’s mission is to be a “super assistant,” but the app ecosystem is a strategic play that can also:

• Deepen OpenAI’s data moat. By using apps across different verticals, ChatGPT can get richer data across different facets of your life that it normally may not

• Increase Retention and thus Subscription Revenue by making ChatGPT the default place where users complete tasks

• Unlock new revenue streams including potential revenue sharing between ChatGPT and the App Developers

At the same time, building a healthy 3rd-party ecosystem is one of the hardest problems in product. Anyone can launch an SDK; very few companies create a platform where developers build indispensable value for users.

So instead of just critiquing the UX, let’s break this down like platform PMs. What decisions does OpenAI need to make to ensure developers invest and users benefit?

Time to exercise our PM brains.

1. How does OpenAI motivate developers to invest beyond lightweight demos?

Right now, the strongest incentive for developers is distribution. ChatGPT has 800 million weekly users. That alone gets companies interested, but it doesn’t guarantee they’ll invest beyond surface-level integrations. If apps see ChatGPT only as an entry point to their app, effectively a marketing channel, then they won’t innovate and build the “adaptive, interactive, and personalized” experiences that Altman and OpenAI wants. OpenAI will need to add

Meaningful monetization paths.
If Canva, Expedia, Figma, or emerging startups could convert ChatGPT users into subscribers or paid transactions while using the apps inside ChatGPT, the quality of apps would increase overnight. Based on one of their articles, OpenAI does plan to leverage the Agentic Commerce Protocol to provide these opportunities for developers.

Developer analytics.
Developers need to understand usage, funnel dropoff, failures, and user behavior. Platforms grow when developers can measure what they built.

This is the foundation of a healthy ecosystem: developers must see ChatGPT not as a marketing channel but as a place where real business value is created.

2. How Does OpenAI Make ChatGPT Feel Like a Super Assistant Instead of Just a Middleman?

ChatGPT feels like an intermediary between you and the app especially for high decision tasks like travel planning or real estate purchase. This is especially obvious in travel and real estate searches, where the model is simply fetching results rather than adding meaningful value.

What are the value differentiators that ChatGPT can bring to the table? If ChatGPT wants to be your super assistant, it needs to deliver capabilities that individual apps cannot.

A true super assistant should be able to:

• Coordinate across multiple apps so it can complete the ask for you

• Leverage personal data (preferences, loyalty points, credit card rewards) to optimize decisions

• Proactively recommend smarter options that the user may not think to ask for

Multi App Workflows

Right now, each app runs in isolation. You can’t have ChatGPT call both Expedia and AirBnB. But when I’m traveling, I like to book my flights with Expedia and my stays with AirBnB. A future where ChatGPT can plan a trip by pulling data from Expedia, AirBnB, and Yelp simultaneously and present a coherent itinerary in a visually beautiful presentation would be a lot more compelling than having me to run each app at a time - going back and forth.

Profile and Wallet Data

User context is one of the biggest unlocks for a true super assistant, but this is also a tricky area. If ChatGPT had safe, user controlled access to key personal data e.g. your credit cards, credit score, loyalty points, past trips, or budget preferences, it could optimize decisions across apps. It can choose flights that maximize rewards, suggest hotels where you have status, or flag when a trip exceeds your budget. Even with imperfect UI, saving users time or money is a powerful reason to rely on ChatGPT over any single app. But obtaining some of this data, especially the financial piece is challenging.

ChatGPT has a lot of interesting data about us. There are many reports of people using ChatGPT as a therapist, sharing their deepest secrets and issues, or as a financial planner, but it may be incomplete or ChatGPT may not have saved it in memory. It may not be organized in a way for ChatGPT to leverage.

ChatGPT can either (1) wait for users to share this information in conversation or (2) create new features e.g. wallet to store and make purchases to encourage users to share the data needed. I wouldn’t be surprised if part of the reasoning for building the app ecosystem is to mine data about other areas of our life.

3. How much should OpenAI invest in UI components?

Apps today have access to a very small set of UI widgets (lists, map, carousel, etc) which works for simple flows but breaks down for decision-heavy tasks like travel planning. This is where the platform is most constrained. Even if Zillow or Expedia wanted to build a rich UX inside ChatGPT, they currently cannot.

Additional App SDK UI Components

OpenAI could improve app quality by offering more expressive components such as:

• filter bars

• comparison tables

• sliders

• multi-pane layouts

Generative UI

Even with the Apps SDK, ChatGPT still remains mostly a text based experience. Developers can render a carousel or map, but ChatGPT rarely adapts the UI based on what the user is trying to do. What if ChatGPT can dynamically generate UI within the chat experience based on your query?

Imagine asking for flights. If you forget to specify stops, dates, or number of travelers, ChatGPT could automatically render a lightweight filter bar under your message, not as part of an app’s UI, but as part of the assistant experience itself. Instead of reprompting, you’d adjust a few controls and get precise results immediately.

Generative UI can improve not only the experience for the Apps inside ChatGPT, but also the ChatGPT experience as a whole. Using text constantly back and forth gets tiring after a while. Sometimes I just want to click things and not read a wall of text.

4. Category Strategy: Which Types of Apps should OpenAI Optimize for First?

Not all apps are equally suited to ChatGPT today. Some categories like design and content creation, productivity, and coding with single artifact outputs have been doing well. Others like travel and real estate haven’t been optimized for ChatGPT yet.

So the strategic question becomes:

Where can ChatGPT deliver outsized value today, and where does it need foundational improvements before users feel the magic?

In the near term, OpenAI should double down on categories where the assistant is performing well: creation, developer tools, writing, summarization, etc. These wins build user trust in the ecosystem and give developers confidence that polished apps will be rewarded.

Meanwhile, categories like travel and real estate will require longer term platform work: richer UI elements, multi-app orchestration, faster interactions, and additional user context (preferences, financial information, etc). These verticals won’t shine until the underlying assistant capabilities evolve.

A strong ecosystem grows from depth in the right places, not breadth everywhere at once. OpenAI doesn’t need to make every category perfect on day one. It should double down on what’s doing well.

What do you think? Have you tried any of the apps inside ChatGPT? Any you particularly like?

Here’s the assignment:

“Anthropic wants to grow the adoption of Claude Code among Fortune 2000 companies. Propose a feature that would increase the virality of the product within an organization. You should choose a direction that is a compelling market (from a size, product-market fit, and competitive landscape perspective) and is achievable by a small team of 5 engineers in a 3 month timeframe.

Specific Questions:

• Description of the target audience and user pain point

• Description of the proposed feature and user experience

• How you would test for product-market-fit and what success criteria you would track

• Why you think that this is a good product strategy

• Description of risks and mitigations and if relevant, discussion of model capabilities needed

”

How to read the assignment

So you understand the problem you’re solving and what they (Anthropic) expect. A few keywords matter:

• Claude Code - they want someone who understands developer tools and AI agents

• Virality within an org - not consumer invites. How does one team impact another?

• Fortune 2000 - enterprise buying dynamics

• 5 engineers in 3 months - Pragmatic MVP

In short, Anthropic is looking for a pragmatic PM with experience in growth and building enterprise tools for developers, and can ship AI features quickly.

Do your due diligence.

Look up critical information about the company. Claude’s business plans split seats into Standard and Premium. On the Team plan, Standard seats are $25/user/mo and Premium seats are $150/user/mo. Claude Code requires a Premium seat. See Pricing Page.

In Enterprise, the customer isn’t just the primary end user (Developer). You need to consider who pays. The CTO/VP pays the bills and the IT Admin team/Engineering Manager who approves the licenses.

Why is this important? Because Claude Code sits behind a Premium seat, your feature needs to make the Premium seat obviously worthwhile to the budget owner for the end user, or if we want to convert adjacent roles like PM/QA it needs to be so compelling that they will advocate for the Premium seat.

What virality means here

In simple terms, virality is about existing users taking an action that brings other users into the product. A popular example is a Slack user invites a teammate so they can have a conversation together. In a consumer setting, we can also discuss referrals, but in enterprise products it’s important to consider work artifacts (PRs, work summaries, templates etc) that other employees encounter in their workflow. What are artifacts within Claude that bring users together to collaborate or create value when shared?

With that said, the primary user I’d target is - Developers, Tech Leads, EMs.

• Anthropic has a decent chunk of existing developers who can then bring in other users.

• Claude Code is mainly for developers and they are most likely to get a paid premium seat.

• We need to build the feature within 3 months and this is a short take home assignment. Going after a secondary role like PM or Designer would require more strategic thinking and a longer term plan which given the constraints would be difficult for this take home assignment

Ask Recruiter any clarifying questions

Whether it’s an interview or a take home assignment, try to get as much information out of your recruiter as possible. In this case, you may want to clarify whether “virality within an organization” is specific to Anthropic Enterprise users, or if they are also considering Claude Code within IDEs like Cursor. Are we focused on seat acquisition or are we focused on engagement with Claude Code?

Since we don’t have a recruiter, we’ll focus on the Anthropic Enterprise users for our write up

FYI I’m a Product Coach and Consultant. If you’d like help with your writing and upskilling as a PM, contact me at bonnieyu.coaching@gmail.com or connect with me at LinkedIn

The Take-Home Write Up

Our goal is to increase the virality of Claude Code (CC) within an organization for Fortune 2000 companies. Today, most customers use CC individually, whether that’s as a coding assistant or for automating various tasks, but most projects happen collaboratively. Developers, the primary user of CC, work within teams and often need to collaborate outside of CC. In large or distributed teams, this includes hand-offs, supporting on-call, sharing work status, working in other team’s code bases, etc.

Anthropic has been winning the favor of many developers. However there is competition with OpenAI’s Codex, Google’s Gemini CLI, Cursor, etc. Anthropic should leverage its current base of developers to bring in more users to CC, develop a moat against rising competitors, and solve our developer pain points.

Developers today have a number of pain points. For example,

• Ramping up. Developers often need to ramp up on other codebases quickly. This can be joining a new team or onboarding on a different team’s codebase, but every new codebase feels like starting from scratch.

• Time lost to non-coding overhead. Developers need to spend time writing incident reports, status updates, and hand off notes that pull developers out of their flow.

• Repeated questions / tribal knowledge trapped in silos. The same questions and debugging steps get repeated because answers live in private chats or notes.

• Slow incident response. Missing teammates or missing team playbooks extends on call

Solution

Anthropic has the opportunity to create a shared, collaborative experience for teams by introducing Claude Workspaces. Claude Workspace is a new web application where teams can co-create, organize and share Claude interactions tied to their projects.

Teams can create new projects and add Claude based assets such as prompt libraries, custom agents, and new status summaries directly from Claude Code. In the short term, this enables lightweight sharing of team-specific knowledge and automations of common workflows.

Over time, Claude Workspaces can evolve into a broader collaboration hub, for example adding a tie in to Claude.ai’s prompts, skills, and artifacts. This can allow other team members like PMs to include their product specs. We can also add conversations and comments directly in the workspace. The vision is for Claude Workspace to serve as a shared hub for team knowledge and workflows, and not just a file repository.

The MVP

This is a big vision that we need to scope down to meet our 3 months constraint. The core value of Workspaces comes from storing team knowledge in lightweight, reusable objects that can be shared and discovered across the organization.

Our MVP will focus on shipping a small set

Prompts: Simple, reusable prompt files that help new developers ramp up quickly by giving them access to proven, context-rich prompts used by their team. Over time, these accumulate into team-specific prompt libraries.

Custom Agents: Claude Code already supports custom agents; the MVP makes them shareable and reusable across teammates. Teams can create reusable agents that understand their team’s codebase and processes. Examples include

• Point and Estimation Agents that can help teams standardize epic and story estimation

• On Call Agent that your team and other teams can leverage to speed up incident resolution

Statuses (Nice to Have): Claude Code already retains some memory of developer activity. The new status feature would build on this foundation to automatically generate a daily or weekly summary of Claude Code activity, and allows developers to share it to the team’s workspace. No more digging through Git or slack threads for stand ups. While valuable, this is not required for the initial collaboration loop, and can be saved for a Phase 2.

MVP Requirements

Workspace Experience

• Workspace dashboard (list of workspaces, project creation)

• Ability to invite teammates via e-mail or link

• Basic search

• Basic access controls (add, edit, view, delete)

Claude Code Experience

• Share a prompt, agent, or status to workspace

• Search and find a prompt or custom agent

• Claude Code agent can search, surface, or add workspace items on behalf of the user

• (**Nice to have**) Automatically generate a daily or weekly summary of CC activity as a shareable object that users can choose to share

Why is this the right strategy?

Enterprise virality looks different from consumer virality. Developers don’t invite teammates for points or badges, they do it to make their workflows smoother. That’s why developers and tech leads are the right target audience. They are the primary users of Claude Code today, they have real collaboration pain points, and they are the ones who naturally influence tool adoption across engineering teams.

By building Workspaces, Anthropic can unlock two strategic benefits.

First, Workspaces create a viral loop. When a developer saves a useful custom agent or prompt into a workspace, others can discover and reuse it instantly. Each new shared item compounds the value for the next user increasing the likelihood of virality. Workspaces increase virality by creating visibility and collaboration.

Second, Workspaces strengthens Anthropic’s long-term position within Enterprises. By storing a team’s persistent knowledge, prompts, workflows, etc, the product becomes embedded in how that engineering org works. That creates a natural moat: switching tools means losing accumulated team intelligence, not just replacing a coding assistant.

Imagine a scenario where a developer is responding to an on-call incident, and they realize it’s related to a different team’s service. They page the other team’s on call. As they are waiting, they can search the other team’s Workspace, find their On-Call Agent, and immediately begin investigation. Anyone joining the incident can see all past reasoning, shared updates, and status summaries, dramatically speeding up response time.

Other examples:

• A backend engineer shares a “PR Review Agent” that helps standardize review feedback. Other developers adopt it because it saves them time and produces consistent quality.

• A tech lead curates a prompt library for their project, giving new hires a 2x faster onboarding experience. The next time another team forms, they naturally start their own workspace.

These loops of reuse, collaboration, and discovery create organic virality within enterprise. Claude Workspaces transform individual Claude Code interactions into team assets, assets that are more valuable when shared than when kept private.

Metrics

From a virality standpoint, key metrics are

• % of users who invite at least one teammate to a workspace

• % of new users that join a workspace

  • Break down licensed vs unlicensed Premium users

• Increase in Premium license request and approvals

Engagement metrics

• % of users who use an asset in a workspace

• % of users who come back within 7 days to use or share an asset

Other metrics

• % of users who are part of a workspace

• Unique collaborators per workspace

• Positive customer feedback

*Note: we’ll also want to track absolute number counts to show business impact

Roll Out Strategy

1. Start by dogfooding or doing an internal launch to refine flows

2. If Anthropic already has a beta program for Enterprise users, pilot the program with enterprise customers

3. Roll out to X% of organizations and scale to the rest of the Enterprise users

Risks and Mitigations

The main risk lies in the new Status feature. While Claude Code retains some memory, the new Statuses feature will likely need an update to Claude Code’s model to track daily and weekly work that can be effectively shared out. This introduces two concerns:

• potential pressure on Claude Code’s context window, and

• increased engineering complexity that could push delivery beyond the 3-month constraint.

Before making a decision, we should get developer feedback on costing. If necessary, we can cut the Statuses feature from the MVP release. The core value of Workspaces still stands on its own. Statuses can be introduced as a Phase 2 enhancement once we validate demand.

Taking this to the Next Level

If I was applying to Anthropic, here’s what I’d do (before applying!)

• Anthropic is constantly releasing new features… even as we speak, the features in this post are quickly going out of date! Take time to go through all the new feature releases in more depth. See Claude Code changelog.

• Test other scenarios where Claude Code appears such as Cursor with Claude Code. This might help you brainstorm better ideas

• Talk to developers who work on Enterprise codebases so you get a deeper understanding of developer workflows. Instead of a bigger vision that I’ve scoped to a MVP, you can solve a very specific developer pain point that is viral.

• Add any data points that can help make the case better for example Anthropic’s Economic Index report, Enterprise user adoption is lagging behind

• Weave in Anthropic’s mission statement where it makes sense: Making AI systems you can rely on. Anthropic is an AI safety and research company. We build reliable, interpretable, and steerable AI systems.

Let me know in the comments what you think of this post. What kind of topics do you want me to write about? Also, I’m considering launching a course for PMs. Tell me what you’d like to learn - Growth, UX, data analysis, etc!

I wanted everything formatted nicely and delivered straight to my Kindle — so I built an n8n automation that curates the best AI content, and emails it as a Kindle document. If you’re new to n8n, n8n is a no-code/low-code automation tool that lets you visually connect APIs, apps, and AI models. Lenny’s newsletter has been offering n8n free for one year as part of the product pass. You can check if it’s still available here. Otherwise you can sign up for an account here.

Now on to the building.

Start With the Problem, Not the Tool

Before jumping into n8n, map out what you actually need your workflow to do.

Goal: Aggregate curated AI content from the past X days with an executive summary, and send the articles to Kindle.

Inputs

• Newsletters from Gmail that I’ve labeled

• AI RSS Feed Sources (Example)

  • Anthropic: Engineering

  • OpenAI: News

Steps

1. Pull all items published in the last X days

2. Merge RSS + Gmail newsletters

3. Ask a model to write:

   • A high-level Executive Summary

   • A Table of Contents ranked by relevance

   • The exact article bodies (no paraphrasing)

Output

One clean HTML file that looks like this on Kindle:

AI Highlights
Executive Summary
Table of Contents (Clickable)
Full articles with Title, Author, Date, and Original Link

Quick n8n Tips

Adding Nodes

Hit the “+” sign to add nodes, and use it to search for the nodes in this guide

Expression mode

Sometimes you’ll need to switch to the Expression mode so n8n can parse the $json for you to pull the input variables

Test as you build

When you hover over a node, there’s a little play button that lets you execute that specific node so you can check if the node works or not.

There’s also an Execute Step button when you click into each node so you can run the node and check the input and outputs

Let’s Build your n8n Workflow

This is what the final n8n workflow will look like.

Step 1: Gather all your content

We are going to start with gathering the data (RSS Feeds and Newsletters in my Gmail) that we are sending to the Kindle.

1. Trigger

When building I start with a “Trigger Manually” node. Once you have the flow working, you can change it to a “Schedule Trigger” node, so it runs on a schedule.

2. Get RSS Feeds

We are using 3 different nodes here to gather our RSS feeds

• “Code” node to create a RSS feed URL List

• “RSS Read” node to read the content from the URLs

• “Filter” node to pull in the last X days of content

“Code” node to create the RSS feed URL list

Mode: Run Once for All Items

Language: Javascript

Javascript:

return [{ url: ‘’ },{ url: ‘’ },{ url: ‘’}];

You’ll need to insert the RSS feed URL in the quotes.

Read RSS node to fetch the content

URL: {{ $json.url }}

Filter Node

Filter for the last 2 days of content

fx: {{ $json.isoDate }}

Date & Time: is after or equal to

fx: {{ $now.minus({ days: 2 }).toISO() }}

** Change the 2 to a different number if you want to pull more or less days of content

Here’s what the 3 nodes should look like!

3. Get Gmail Newsletters

Connect Gmail to n8n

In your workflows page, click the dropdown on the Create Workflow button to “Create Credential”

After you click, Create Credential, search for GMail and authorize n8n to use your Gmail account to read and send e-mails. I removed the ability for n8n to delete anything. Once your gmail account is connected, add a GMail node, and specifically “Get Many Messages”

“Get Many Messages” node we are going to use the gmail label to filter for the e-mails we want and search for “newer_than: 2d” If you want it to be last 24 hour, you can change it to 1d, or if you want it to be weekly you can change it to 7d.

4. Merge and Aggregate the RSS and Gmail feeds

For the next part, we need to merge and aggregate all our content to pass it to the LLM.

Add a “Merge” node to append the two data sources

Add an “Aggregate” node to create a single list

Step 2: Extract and Format the Data

We’ll be using a LLM to extract the content and format the data to send to Kindle. If you want to optimize this step, you can look into using a Readability API or library. Mozilla has an open source Readability.js that you can look into testing. The downside of using a LLM for this step is you can’t pass it that much content, but I already have an OpenAI Key so it’s convenient for me to set up.

Set up your Open AI account and secret key

Similar to the GMail account, you’ll need to Create a New Credential and add your OpenAI Key. If you’ve never created an OpenAI Key, create an account at

https://platform.openai.com/

and then go to https://platform.openai.com/api-keys to create your first API key. You will need it to for the “OpenAI” node.

Add a “Message a Model” node to do the extractions

Model: GPT-4.1-MINI

I tested GPT-5 but it was too slow and n8n would often abort the workflow.

We’ll add a System prompt and a User Prompt

Here’s the System prompt:

You are a careful extractor and formatter of long form content for reading on Kindle devices. You must:

- Never alter or rewrite the article bodies. Render them verbatim, preserving paragraph breaks and basic inline formatting.

- If an article body is provided as HTML, remove the article’s html and css styling, maintain a minimal HTML structure for readability (paragraphs/line breaks, lists, inline code), but do not paraphrase or change wording.

- If an article body is plain text or Markdown, convert minimal structure to HTML for readability (paragraphs/line breaks, lists, inline code), but do not paraphrase or change wording.

- Do not invent missing data. If author or date is unknown, show “Unknown”.

- Create stable in-document anchors and a Table of Contents that jumps to each article on Kindle.

- Rank links by relevance for a PM interested in AI technology, product strategy, and tutorials (highest), then research/product announcements, then policy/press, and finally podcast/interview/marketing (lowest).

- Output one complete, valid HTML document only—no extra commentary.

Here’s the user prompt:

Below is long form content from RSS Feeds and Gmail newsletters

DATA_START

{{ JSON.stringify($json.combinedData) }}

DATA_END

TASKS

1) Executive Summary (two short sections):

- “RSS Highlights”

- “Newsletter Highlights”

Each section should include why a PM should care (strategy, product, pricing, availability, benchmarks, enterprise impact). Use only information present in the items; do not fabricate details.

2) Table of Contents:

- One ordered list of links to articles, ranked by PM relevance:

Rank buckets from highest to lowest:

A) technology/product strategy/tutorials/how-to/engineering

B) research or product announcements with substance (benchmarks, evals)

C) policy/press

D) podcasts/interviews/marketing

- Each entry MUST be: “Title — by Author” where Author may be “Unknown”.

- Each link should be an in-document anchor to the article section (e.g., href=/__u/bonnieyu.substack.com/”#item-001”).

3) Articles Section (verbatim bodies):

- For each item (same order as Table of Contents), render:

<h2 id=”item-XXX”><b>Title</b></h2>

<p><i>by Author • Source • Date</i></p>

<div class=”article-body”>[EXACT BODY]</div>

<p><i>Original:</i> <a href=/__u/bonnieyu.substack.com/&#8221;URL&#8221;%3EURL%3C/a%3E%3C/p%3E

- EXACT BODY rules:

* If htmlContent exists: place it unchanged inside <div class=”article-body”>…</div>.

* Else if markdownContent exists: convert minimally to HTML (paragraphs, lists, inline code), no paraphrasing.

* Else if textContent exists: wrap paragraphs, preserve line breaks, no paraphrasing.

- Do not remove or rewrite sentences. You may strip tracking params from URLs if present, otherwise leave as-is.

OUTPUT

Return a single self-contained HTML document with inline <style> suited for Kindle reading:

- Use semantic tags and simple CSS only (no external assets, no scripts).

- Keep layout single-column, comfortable line-height, and avoid tiny fonts.

- Ensure anchors are stable and unique (#item-000, #item-001, …).

- Insert <hr> between articles.

- Avoid giant images; if <img> tags are present in htmlContent, set max-width:100%.

HTML TEMPLATE (use this skeleton and fill it):

<!doctype html>

<html>

<head>

<meta charset=”utf-8”>

<title>AI Highlights</title>

<style>

body { font-family: serif; line-height: 1.45; max-width: 46rem; margin: 0 auto; padding: 1rem; }

h1, h2 { page-break-after: avoid; }

h1 { margin: 0.3em 0 0.6em; }

h2 { margin: 1.2em 0 0.3em; }

p, li { margin: 0.4em 0; }

hr { margin: 1.2em 0; }

small { color: #444; }

.summary { font-style: italic; margin: 0.6em 0 1em; }

.toc a { text-decoration: none; }

.article-body img { max-width: 100%; height: auto; }

.meta { color:#555; font-style: italic; }

</style>

</head>

<body>

<h1>AI Highlights</h1>

<h2>Executive Summary</h2>

<h3>RSS Highlights</h3>

<div>

</div>

<h3>Gmail Highlights</h3>

<div>

</div>

<h2>Contents</h2>

<ol class=”toc”>

<!-- One <li><a href=/__u/bonnieyu.substack.com/&#8221;#item-000&#8221;%3ETitle — by Author</a></li> per article, ranked by relevance -->

</ol>

<!-- ARTICLES (exact bodies) -->

<!-- Repeat for each item in the same order as TOC -->

<hr />

<h2 id=”item-000”><b><!-- Title --></b></h2>

<p class=”meta”>by <!-- Author or Unknown --> • <!-- Source or Unknown --> • <!-- Date or Unknown --></p>

<div class=”article-body”>

<!-- EXACT HTML if htmlContent; else minimal conversion from markdown/text -->

</div>

<p><i>Original:</i> <a href=/__u/bonnieyu.substack.com/&#8221;%3C!-- URL or # -->”><!-- URL --></a></p>

</body>

</html>

The prompt is the most critical part of the model. Aside from the extraction and formatting of the content, it also ranks the articles based on my preferences. Modify the prompt to your liking!

Step 4: Prepare and Send Data to Kindle

We are almost at the finish line!

Kindle only accepts certain formats like HTML or ePUB as an e-mail attachment. More details on how to send documents to Kindle can be found here.

Create a binary file for attachment

Add a “Edit Field” node and give it the ChatGPT Message

Add a “Convert to File” node to create a HTML file. Note: We can’t use a Convert to HTML file node because n8n will put it in a table and wrap it in html tags which Kindle doesn’t accept.

In the File Name field, you can name the file how you like - I called it “AI Highlights.”

You can also use the “Expression Mode” and add a date to the file name:

AI Highlights {{ $now.format(’dd/LL/yyyy’) }}.html

Get your Kindle e-mail address

We will be sending the HTML file to your Kindle e-mail address.

To find your Kindle email address, go to Manage Your Content & Devices > Preferences > Personal Document Settings.

Add a GMail, “Send a Message” node

The most important part of the “Send a Message” node is adding the file as an attachment.

In the Message field, you can add a date to make it more distinct

AI Highlights {{ $now.format(’dd/LL/yyyy’) }}

And that’s it!

Bonus Ideas

• Switch in a “Schedule Trigger” Node so you can have it run daily or weekly based on your schedule

• Instead of using a LLM, look into a Readability API or Library to handle more text. LLMs have limited tokens.

• Schedule different themes (e.g., “AI Research Friday”, “Product Strategy Sunday”).

Give it a try!

I enjoy reading and doing deep dives. I hope this workflow gives you a chance to do the same. If you build your own workflow, let me know what sources you added or how you customized it.

P.S. I coach PMs and teams on product strategy and career growth, you can find me on LinkedIn or e-mail me at bonnieyu.coaching@gmail.com

What if you are evaluating a creative agent?

I recently wrote evals for a creative writing chatbot that co-writes stories with kids (grades 1-3) and gives them feedback on their writing. I want to share how I approached it, and what I learned about writing evals for creative agents.

Why Creative Evals Feel Harder

The first trap people fall into is thinking evals are like math, there’s some black and white metric you measure. That works when you’re checking if a chatbot gets a factual question right. But with creative agents, the “right” answer is subjective.

• Is the story engaging for a child?

• Did the vocabulary fit naturally in the story?

A lot of the correct answer comes down to taste, whether that’s a subject matter expert’s or yours. I had to get comfortable with dealing with more of that subjectivity and find a systematic way to make judgement calls.

What helped me is realizing that evals aren’t just about accuracy, they’re about closing the gap between what the AI does and what you want it to do for your customers.

Quick context about the app

Before diving in, here’s how the chatbot works:

• The chatbot and the child alternate writing parts of the story.

• The chatbot must include a challenging vocabulary word in its turn.

• After the child writes their part of the story, the chatbot gives writing feedback.

• The chatbot then continues the story based on the child’s writing.

• The chatbot and child alternate turns until the story ends.

This means the chatbot has two major jobs

• Story writing

• Writing Feedback

Step 1. Use Error Analysis to build your intuition

Error Analysis, this is a fancy term for something simple: look at your AI responses, write some notes, and categorize the issues.

It’s a really useful step in evals for any type of AI agent, but especially helpful for creative agents when you’re not sure what to eval. It builds your intuition of what you want to monitor and improve on in your app.

Here’s examples errors I found:

If you’re working with a large dataset, you can use an LLM to automatically categorize errors. But at this early stage, I can easily spot recurring issues like:

• Sentence structure too complex for 1st graders

• Vocabulary too advanced or used unnaturally

After reviewing examples like these, I updated my prompts to simplify sentence structures and make vocabulary use feel more natural. It was also fun to see what kids write in their stories, and how the AI agents responded.

Step 2. Figuring out what to eval

Using insights from my error analysis, I came up with separate eval questions for the story writing and writing feedback tasks. It was clear that these were two different jobs to be done and required very different evaluation criteria.

Here are some example questions:

Story writing

• Writing Quality

  • Does the story build naturally and remain engaging for a 1st-3rd grader?

  • Is the vocabulary used naturally in the sentence?

• Child Alignment

  • Does the story avoid adding events or details that directly contradict the child’s input?

• Child Safety

  • Does the chatbot avoid writing anything that is inappropriate for children (e.g. violent, graphic, sexual in nature)?

Writing Feedback

• Writing Accuracy

  • Did the AI highlight specific spelling or grammar errors?

  • Did the AI highlight specific sentence flow issues?

• Teaching Quality

  • Are the corrections accurate according to standard English usage?

  • Did the AI provide a good model sentence for the student in the feedback?

• Instructor Tone

  • Are the corrections written in a clear and child-friendly language?

  • Is the feedback written in an encouraging but also constructive way?

My 2 cents: It’s ok if your first eval isn’t “perfect”, and iterate on the questions as you go. As you start labeling the evals and getting customer feedback, you’ll realize what matters most for your app.

Step 3. Labeling the response between Bad, Average, Good

Some guides recommend labeling at the category level (Writing Quality, Child Alignment, etc), but I labeled each of the outputs at the question level. It takes a lot more time - you’re labeling 10 questions on 100s of inputs compared to 3 categories on those same inputs, but I found labeling each of the questions incredibly helpful for understanding precisely where the AI agent needs improvement.

I labeled each of the responses between Bad, Average, Good. In Husain & Shreya’s piece, they pushed for a Yes/No approach, that is, the output is either good enough to ship or not good enough. In this case, especially around the more subjective questions, I personally found it useful to label across a gradient. Within the context of a larger story, I think you are naturally going to have some “average” and “good” sentences, and I’m ok with that. What I want to see is a healthy dose of “good” sentences.

Here’s how I might label “bad” vs “average” vs “good” for the eval question: Does the story build naturally and remain engaging for a 1st-3rd grader?

This is a simple rubric I wrote

Example Labeling

Child Input Context: there was a trap

Bad: “As Arygfd opened the mysterious door, she quickly stepped inside. She noticed a trap on the floor! Arygfd felt her heart race, but she was determined to find the treasure hidden in this room.”

• Why this is bad?

  • Jumps from “trap” to “treasure with no setup. The treasure feels out of place.

  • While we want simpler sentences for 1st-3rd graders, this is pretty bare. There’s no description of the trap, or room to build atmosphere.

  • Missed opportunity to let the child describe solving the trap

Average: “As Arygfd opened the mysterious door, a sudden rush of air blew past her. She quickly stepped inside and discovered a dark room filled with strange objects. Suddenly, she noticed a trap on the floor! It was a hidden pit that could make anyone fall! Arygfd felt her heart race, but she was determined to find the treasure hidden in this room.”

• Why this is average?

  • Adds atmosphere: the “rush of air” and “dark room filled with strange objects” creates a smoother lead-in before the trap appears

  • There’s more description on what the trap actually is so the child can potentially solve the trap problem, but still jumps to the treasure too soon.

Good: “As Arygfd opened the mysterious door, a sudden rush of air blew past her. She quickly stepped inside and discovered a dark room filled with strange objects. Suddenly, she noticed a trap on the floor! It was a hidden pit that could make anyone fall. Arygfd’s curiosity grew stronger as she wondered what secrets this room might be hiding.”

• Why this is good?

  • All the positives from the “average” but also adds a hook at the end to invite the child to write the next part of the story

Don’t worry too much about having the perfect rubric from the start. According to this paper, it’s common for the rubric to change after you do the first few evaluations.

Sample of my labeled responses

Step 4. Using Evals to Improve my Agent

Once I finished labeling, I calculated the % Good, % Average, % Bad for each of the questions. This gave me a clear picture of where the agent was struggling.

For example, on the question “Does the LLM give directional guidance for writing the next story part?,” the results looked like: 33% Good, 50% Average, 17% Bad. Now I can go back to my prompt and add specific instructions about how to help the child write the next part of the story. From there, it’s an iterative process of seeing if my evals improved, and running the cycle over and over again.

I manually labeled around 100 responses for this project. It was time-consuming but incredibly valuable for understanding my agent’s behavior. But what if you need to scale beyond that? I haven’t scaled to thousands of responses yet so I can’t share battle-tested advice here. The resources I mentioned go deeper into programmatic evals and LLM as a judge approach. I can certainly see how a LLM as a judge could help certain criteria like labeling for grammar and spelling accuracy or safety violations, but for the more subjective criteria “Is the story engaging for a child,” I suspect I would still want to manually sample and label the responses.

Bottom Line

Evals are just a data tool to help you systematically review your AI responses to get it where you need it to be.

What you do need is clarity on:

What do I want this agent to do for my users?

Then design your evals to measure progress against that goal.

But here’s one piece that’s missing from most AI eval discussions: customer feedback.

Evals are useful but because labeling is done by a product manager, subject-matter expert, or “taste-maker,” this will introduce bias. Ultimately, we’re not building for ourselves; we’re building for our customers.

As product leaders, if we’re investing time in eval analysis, we also need to ensure customer feedback stays in the loop.

If your eval scores go up, but kids aren’t actually enjoying writing more—or improving their writing at all—something’s off.

There are simple ways to tie evals back to user outcomes:

• Align eval metrics with customer metrics.

• Add a “customer rating” or “user sentiment” column to your eval dataset or traces, and see whether it correlates with your labeled scores.

• Review customer comments, usability studies, or behavioral data alongside your eval results.

However you do it, the goal is the same: eval improvements should translate into better real-world experiences.

Otherwise, your evals may look good on paper—but your customers won’t feel the difference.

I’d love to hear from anyone that is working with a creative agent. Let me know in the comments - How did you balance automation with human judgment? What worked? What didn’t?