That’s the trick, isn’t it? You’re about to. The effort behind the work didn’t disappear when the sender delegated the writing. It moved. To you. You now get to work out what they actually meant (if anything), which parts have real value, which parts are filler, and which parts are subtly, confidently wrong.

We’ve all been on the receiving end of this. Generation got cheap; reading got expensive. Hold that thought, and I promise I’ll get to whole-organisation ROI.

The ceiling and the floor

You’ve heard the popular versions of what AI does to the quality of work. One camp says it levels everyone up: hand the tools to the whole team, and everyone performs like a senior. The other side says it drags everything down into banal mediocrity. Both are wrong, and they’re wrong in the same interesting way: the quality distribution is stretching.

The ceiling rises ( with some fine print). Bring solid skills and deep domain knowledge, and AI extends your reach: more ground covered in less time, without dropping your quality bar. The fine print is that you have to know what good looks like before* you ask for it, you have to be able to tell when you didn’t get it, and you have to actually care enough to put in the effort to get it out of the AI.

On the other hand, the floor drops. Before AI, lack of skill was self-limiting. You produced nothing, or you produced something visibly rough, which at least signalled “needs review”. Now the same skill gap produces fluent, formatted, plausible work. The floor dropped because bad work acquired camouflage. People didn’t get worse at their jobs, but that the heuristic of “polish equal value” is failing. (Side note: it’s the same cognitive bias that makes us rank confident speaker as more knowledgeable; confidence and polish no longer mean understanding).

calls this artificial competence, and his article is worth your time: AI expands your capability to do far faster than your capability to judge, and the gap between doing and judging quality is where the damage happens. Dunning-Kruger with a jetpack.

Side note on vocabulary: I’ve previously described that judging capability as “taste”. I’m walking back from that a little. “Taste” is a vibe word: it sounds profound while defining nothing. What the ceiling actually requires is skills, experience, and domain knowledge. Those are concrete. You can build them, test them, and hire for them. You don’t acquire taste magically, you do the reps for it.

So what does the research say?

We’re still learning the right way to get value out of AI, and the research is noisier than either camp admits.

The early studies (the 2023 wave of consultants and call-centre agents) found AI helps novices most. The floor rising, in other words, not falling. But look at what they measured: output that scored better. I suspect at least some of those studies mistook polished work for good work, similar to how most PMs preferred AI-written strategy docs which you can’t really evaluate without deeply understanding the org they were written for (see my talk with ). The novices’ output looked better, the AI’s just looks professional. Whether it was better is obscured by the polish, and needs to be evaluated in context.

Then there’s the METR study that made the rounds last year: experienced open-source developers were 19% slower with AI, while believing they were about 20% faster. That one measured a real effect, and it should give the hype crowd pause. But it’s also limited: it measured early-2025 tools, before the big jumps in coding capability, and it measured AI as a code writer. Using AI e.g. as a codebase explainer (building the very domain knowledge that raises your ceiling) is a different use entirely. There is more than one way to use AI, and not everything is equal – but those things seem to also change fast. The upshot is that AI is still very much in the chaotic domain and you need to constantly re-evaluate and retry in your own context.

So the evidence from research doesn’t settle anything, in either direction. What it shows consistently is that our measurements reward surface quality. Which is, conveniently, the point of this post.

What I’m proud to put my name to

Most of my attempts to have AI write anything consequential have failed. The output was recognisably AI: overly verbose with notorious verbal tics, structurally boring, and it often missed the point I was trying to make. Not something I want my name against.

Where it helps is when I tightly control the message. I own the understanding of the domain, the main point I’m communicating, and the war scars that anchor it. I use AI to strengthen my message, hunting for logical holes and counterpoints I haven’t addressed (maybe even convince me I was wrong in my analysis or assumptions). I.e. a thinking partner, not a ghost-writer.

I give you these blog posts as evidence. They aren’t “AI written”. The ideas, the opinions, and the anecdotes are mine. I lovingly craft the words (an author’s obsession with language). What I do is argue with an AI that has standing instructions to challenge my assumptions and tell me where I’m being sloppy. It pokes holes, but I decide which ones matter and how to fix them. If this post reads like me, that’s why 😉

Don’t go looking for a tidy rule like “human writes the first draft, AI polishes”, or the reverse “AI for the first and last drafts only”. There’s room for AI at every stage, from ideation, through structure, critique, tightening, and the inevitable spell-checking my fat fingers. It’s useful as long as it augments your message and your judgement rather than replacing them. Gilad makes a related point about AI as a judge: verification and critique are legitimate strong uses of these tools, but you still need enough domain knowledge to judge the judge.

Private gains, socialised costs

This is the bit I struggled to define clearly: The ceiling gains are private. Your task gets done faster, your boss is happy. The floor costs are socialised: the recipient decoding your document, the reviewer verifying your generated code, the team re-litigating a decision because the “alignment doc” was generated rather than thought through.

The producer saves an hour; three recipients each lose one. Individually rational, collectively negative. And no individual productivity metric captures the difference, which is why the dashboards show productivity up while delivery doesn’t budge and emotions grow frayed.

Time is the cheaper cost you pay; the dearer one is credibility, and it erodes in silence. I’ve stopped reading several blogs (and bloggers) once I could see the AI writing them. No angry comment, no unsubscribe survey, no feedback at all. I lose interest shortly after you lose interest in what you put out there. The senders will never know the price they paid by following all the influencers touting “content production agents”. (There’s more to say about AI writing, especially in light of Anthropic’s recent watermarking announcement, but that’s an upcoming post.)

Turns out this has a name now. Researchers from Stanford and BetterUp coined the term “workslop”: AI-generated work content that masquerades as good work, but lacks the substance to advance the task. Their findings, published in HBR, put numbers on the same mechanism: recipients spend real time and effort dealing with workslop, and afterwards rate the senders as less capable and less trustworthy. The cost of personal productivity is borne out by the team and the company, until people stop paying it.

The unit of production is a team

Now zoom out from you and your inbox, because this is where the organisational ROI question lives.

Despite what you hear, the smallest organisational unit that produces anything is a team. Software is built by a team, and teams run on communication and alignment: the shared context and mutual trust that keep everyone pulling in the same direction. That was true when the pyramids were built, it was true before AI, and no model (or any technology) release has changed it. I’ve argued before that the job boils down to organising your shit, communicating about your shit, and owning your shit - and two of those three are about other people.

The point of technology is to serve humans. For a team, that means serving communication and alignment. AI-generated artefacts mostly don’t. A generated PRD, status report, or summary is a 3-duck substitute wearing the trench coat of alignment. The artefact exists; the shared understanding doesn’t. Substituting for communication is worse than merely not helping with it, because everyone believes the conversation happened.

I watched this play out in a consulting engagement. The PRDs and tickets were long, polished, and plentiful, clearly generated to create volume rather than understanding. It held together right up until the developers started asking questions. The polish couldn’t compensate for the missing critical and creative thinking, for nobody having first understood the domain and then defined the problem. The team had tried to speed up the early stages, and ended up spending more time later, when the alignment gaps surfaced mid-build. Cheap artefacts now, expensive re-alignment later - with interest.

The same conditional applies here as at the personal level. AI can help team communications when it strengthens the human signal: critique your argument before the meeting, surface the objection you missed, translate your decision for a different audience after you’ve decided what you mean. It hurts when it generates volume in place of meaning. (In my own blog example, I’m not producing more posts, I’m aiming, and hopefully succeeding, at writing better ones).

So I wasn’t shocked when that widely-quoted stat did the rounds, the one about 95% of organisations seeing no measurable return on their AI investment. Take the number with salt (it was a small sample talking about early experiments), but the direction is confirmed by more published surveys and my own anecdotal experience. Organisations bought individual productivity, and the floor costs landed on the team, which is where output actually gets measured. The ROI was never going to come from typing faster, or even from shipping faster (people do, on occasion, ship faster; that’s a different post). It’s true for PRDs as it is for code, which creates a bottleneck in reviews or a degradation of product quality. Instead, the gains come from consistent, persistent pulling in the same direction. AI speeds up the pulling and muddles the unified direction.

So what can you do?

For yourself, it’s a progression, not a checklist:

1. Do the reps first. The ceiling lift is conditional on knowing what good looks like in your domain. There’s no shortcut through skills, experience, and domain knowledge. I’ve written about building that ladder before.

2. Own the message before you open the tool. Know your point, your audience, and your evidence. Then use AI to strengthen it: find the holes, tighten the argument, stress-test the logic. Maybe even change your mind when it surfaces something you’d missed. That’s using a tool, not delegating work.

3. Apply the name test. Never send anything you wouldn’t defend line by line. If the recipient can tell AI wrote it, you haven’t saved work. You’ve shifted your work onto them, and attached your name to the invoice.

For teams and leaders:

1. Make the recipient cost visible. Effort-shifting slop is a quality problem, not a productivity win. Treat it like any other defect that escapes downstream.

2. Norms, not bans. “You own what you send” covers most of it. AI use is fine; unreviewed AI output under your name is not.

3. Measure at the team level. If individual productivity is up and team delivery isn’t, the gap between those two numbers is the floor cost. Go find where it lies; it’s usually in someone else’s inbox.

The distinguishing question

Back to that document in your inbox. The question that separates ceiling from floor was never “did AI help write this?” in the same way it was never “was a chainsaw used to cut the tree?”. The right question is: would/could the sender defend every line? AI raises the ceiling for the people who can answer yes. It lowers the floor for everyone who can’t tell the difference.

This post was, as usual, argued out loud with an AI that kept telling me where my reasoning was thin and where I might be wrong or over-stating things. Every line above, I’ll defend (including the emdashes). That’s the whole trick, and it was never about the tools.

The standard discovery playbook of continuous experimentation, quantitative validation, and rapid iteration assumes a world where data is abundant, cycles are short, and you can test in production. That world exists. It’s just not your world.

This post is for the enterprise PM who’s been told they’re doing discovery wrong because they can’t run experiments on 10,000 users. You’re not doing it wrong. You’re navigating without GPS.

Navigation without GPS

B2C product discovery has GPS: real-time positioning from millions of data points. You always know roughly where you are. Take a wrong turn? Reroute in seconds. The data is abundant, the feedback loops are tight, and statistical significance is achievable before lunch.

Enterprise discovery has a compass, a hand-drawn chart, and a few locals who each know their stretch of coastline. You navigate by triangulation - taking bearings from multiple imperfect reference points, and finding your position where they converge. You never get GPS-level precision. But a local who knows the reef is worth more than a satellite overview, and the navigator who can read tides, landmarks, and currents will find the harbour when the GPS-dependent sailor is lost the moment the signal drops.

Here’s the reframe that matters: in enterprise, each signal is more valuable, not less. A single customer interview in enterprise can reveal more about the market than a thousand survey responses, because enterprise customers have been pondering their problem, can describe workflows in detail, and tell you things no analytics dashboard captures: the political dynamics, the regulatory constraints, the reason they bought your competitor and regretted it. Sometimes it’s hard to schedule interviews with the right people, but it’s never hard to get them to talk. You need to focus on finding those right people, the ones who know the reefs like a local.

Subscribe now

Every conversation is discovery

The books will tell you to schedule regular discovery interviews with customers. That’s good advice. It’s also an incomplete picture of how enterprise discovery actually works.

In enterprise, discovery happens every time you talk to a customer, not just in formal interviews. The trick is recognising it, and being ready to learn from it.

I’ve run professional services engagements where I helped customers install and configure the product. Those weren’t “discovery calls,” they were implementation sessions. But by asking about their environment, feeling the rough edges myself, and watching where the product met reality, I learned more about our customers’ actual workflows than any scheduled interview could have surfaced. You see the gap between what the product assumes and how the world actually works, and you feel it with your own hands, not through a screen share.

I’ve run workshops and webinars where I’d ask the audience about their context so I can “tailor this presentation to your situation.” That’s a facilitation technique. It’s also discovery in disguise. When 30 people in a room tell you what they care about and why, you’re collecting bearings. Different organisations, different roles, different priorities. The patterns that emerge across them are gold.

And I’ve jumped on customer support calls with angry customers. Those conversations, when you sit and listen to someone who is frustrated enough to escalate, teach you what people really care about and why. Spend time to go beyond the feature they’re requesting to the problem they’re trying to solve. In one case, I spent an hour on a call with a customer who was ready to walk. By the end of the call, I understood not just their issue but the organisational pressure driving it, something no NPS survey would have surfaced. We saved the account, and I won the adoration of CSM and Sales for months afterward, which, let me assure you, is a very useful form of social capital.

The principle: every customer conversation is a bearing. The formal discovery interview, the implementation session, the support escalation, the casual chat at a conference. You’re always triangulating. The question is whether you’re structured enough to capture what you learn and connect it to the broader picture.

Sales as your discovery network

Enterprise PMs have a discovery asset that B2C PMs can only dream of: a sales team that talks to prospects every single day.

Most product teams treat sales as a feature-request pipeline. “Customer X needs Y to close the deal.” That’s a data point, not a direction. But when five salespeople across different territories independently describe the same pain point (when you ask ‘why’ enough times), the same competitive loss pattern, the same objection they can’t overcome - that’s a strong bearing.

The investment is relational. Sit in on sales calls. Read deal post-mortems. When a deal is lost, pick up the phone. Talk to the prospect if you can, but also to the salesperson. Ask what happened. Ask what the prospect said. Ask what they said between the lines. Make it clear you value their intelligence, and they’ll bring you more of it. Ignore them, and they’ll route around you directly to engineering, to the CEO, to whoever will listen.

Win/loss analysis in enterprise is genuinely hard. Government officials often can’t speak beyond what’s been formally released. Regulated industry buyers are prohibited from accepting a dinner or a coffee. But it’s absolutely worth working the CRM, identifying the deals worth investigating, and picking up the phone. If you manage to speak to someone who chose your competitor, that single conversation can be worth more than a quarter’s worth of analytics. Treat it accordingly.

Design partners: the enterprise version of beta testing

Once you’ve triangulated the problem, enterprise solution discovery also works differently from the books. You can’t ship an MVP to millions and measure adoption. Instead, you work with design partners: a handful of customers who co-develop the solution with you, and give you feedback every step of the way.

I did this formally in one role, and it produced the most solid product foundation I’ve ever worked with. The process looked nothing like a B2C beta. We started with deliberately unstructured, open-ended discovery calls, to avoid seeding expectations. Then deeper dives into specific workflows. Then Figma prototypes, each accompanied by a list of questions we wanted answered (not “do you like it?” but “what would you expect to happen here?” and “where does this break your current process?”). Then demos of something still far from the prototype, but working enough to interact with real data and validate direction.

What made it work wasn’t the process. It was the relationship. We had a few people from their side, they had a few from ours, all sitting around trying to collaborate to solve a problem. Not vendor and customer. Partners. Those relationships lasted years and shaped not just that feature but the product’s entire direction.

The danger with design partners, and the navigator analogy holds here, is that a local pilot who guides you into harbour is invaluable for this harbour. Their knowledge of the next port is limited. Design partners give you deep, specific feedback, but their needs may not generalise. Not all user feedback is created equal: expert users push for power features, beginners want hand-holding, and the silent majority tells you nothing at all. You need design partners, AND you need your own chart.

Building conviction, not statistical significance

The hardest part of enterprise discovery isn’t gathering signal. It’s deciding when you have enough to act.

In B2C, you can define statistical significance and let the numbers decide. In enterprise, you’re building conviction from qualitative data and pattern recognition. One data point is better than none, but don’t get anchored on it. Collect customer anecdotes like you’re OCD; make it your obsession. You’ll be rolling them out again and again in stakeholder meetings, in engineering brainstorming, in board strategy reviews, and there is nothing like a real customer story to build empathy and alignment across an organisation.

Triangulation in enterprise isn’t as formal as actual navigation. It’s a constant process of connecting dots, of keeping those apparently unrelated anecdotes in mind until the pattern emerges. The professional services engagement where you felt the rough edge. The three salespeople in different regions hitting the same objection. The RFP that asked for something nobody offers yet. The support call where the real problem was three layers beneath the ticket. You hold these loosely, and one day they click together, and you know with the kind of conviction that no p-value provides where to point the product next.

This is where experience matters most, and where it connects back to building taste. The reps you’ve done, the patterns you’ve seen across multiple products and industries, the instinct that tells you “this is real” versus “this is one loud customer” - that’s product sense, and it’s built from exactly these kinds of qualitative signals, accumulated over years.

Practical navigation

For enterprise PMs doing discovery:

Structure your signal collection. Don’t leave insights scattered across call notes and Slack messages. Build a lightweight system for capturing and tagging discovery insights by source, theme, and confidence level. It doesn’t need to be complex; a shared document with consistent tags will outperform a fancy tool nobody uses. The point is making patterns visible across sources and over time.

Invest in sales relationships. Your sales team is your largest, most distributed discovery network. Build the relationship where they bring you insights, not just feature requests. Sit in on calls. Read the CRM. Make it clear their intelligence shapes the product, and they’ll give you more of it.

Triangulate before committing. One customer asking for something is a data point. Three customers and two salespeople independently describing the same pain is a bearing. The same pattern showing up in RFPs, support tickets, AND customer interviews? That’s a fix. Act on it.

Name your uncertainty. Enterprise discovery never reaches certainty. That’s fine. “We have high conviction from four independent sources” carries more weight than “a customer asked for this.” Being honest about confidence levels earns trust with stakeholders and makes prioritisation transparent. It also makes it easier to course-correct when new signal arrives, because you never pretended you had the final answer.

Uncharted territory

I should note: the principles behind continuous discovery are absolutely sound. You should be constantly talking to customers, collecting evidence, and validating assumptions. The enterprise version isn’t a rejection of those principles. It’s an adaptation to a world where the nature of conversations and the scale of evidence are fundamentally different. Once you sign on to high-end B2B, reality diverges from the textbooks. The principles hold. The practice looks nothing like the diagrams.

The enterprise PM navigates what the military calls VUCA: volatility, uncertainty, complexity, ambiguity. Uncharted territory. Not a known map, but a place you’ve never seen before. There’s fog, and it’s raining, and you can hear but can’t see the competitors over the other side of the hill.

But that’s still better than silence. Silence is the scariest data point in enterprise discovery. It means your customers aren’t talking to you - or worse, they’re talking to someone else. The navigator who’s out in the weather, taking imperfect bearings in rough conditions, is in a better position than the one sitting in port waiting for perfect visibility. It never comes. Set sail. Triangulate as you go. Adjust course when the signal sharpens. That’s the craft. After 20 years, I can tell you it never gets boring.

This isn’t cynicism, and I’m not having a go at the people who wrote it. Most organisations genuinely want to govern their AI responsibly. The problem is that they’re applying document-era thinking to a system-era problem. A policy tells people what they should do. Governance ensures they actually do it continuously, at scale, when nobody’s watching. The difference between a policy and a governance system is the difference between a speed limit sign and a speed camera.

The question isn’t “do you have AI governance?” Everyone has a PDF. The question is: does your AI governance actually govern anything?

You can’t have AI govern itself

Let’s start inside the stack.

You can’t have a bank regulate itself. Any entity that optimises for its own objectives cannot simultaneously enforce constraints on those objectives. People and organisations will find the path of least resistance, and the path of least resistance is rarely the path of most compliance. We know this. We’ve known it for centuries. It’s why regulators exist as separate institutions, with separate incentives, and the authority to enforce consequences.

The same principle applies to AI systems. An agent that optimises for task completion and token efficiency cannot be trusted to govern its own behaviour. Ask it to check whether its response was appropriate, and it will optimise for saying “yes.” It doesn’t have the consciousness to lie, it’s just been trained that “yes” is the completion of the task you just gave it. The incentive structure is baked in.

This is where the “death of programming” narrative gets it exactly backwards. The current story is that AI agents will handle everything, programming is obsolete, just give the agent some context and a set of tools and let it work. The reality is far messier. Composing AI into reliable systems is classic software engineering. You need decomposition (breaking processes into discrete, governable units), decoupling (each unit has defined inputs, outputs, and boundaries), and control logic (programmatic orchestration determines what runs when, what can access what, and what happens when something fails). This is the stuff they teach in second-year computer science, and it hasn’t stopped being true because the functions happen to contain a language model.

Each step in a workflow can call on AI, and should, where AI adds value at tolerable risk. But each step needs to be reliable by itself, governable by itself, before it can be composed into a larger workflow. That larger workflow can itself be wrapped as an AI “function” with its own governance layer above it. It’s turtles all the way up, but at every level there’s a programmatic shell around the AI core.

I’ve written about the Refrain/Restrict/Trap framework for governing a single AI interaction – one feature, one choreographed performance. That works for one component. When you have dozens of those components running simultaneously, feeding into each other, composing into a product, you need a governance layer above them: the orchestration. It can’t be another AI agent watching the first one (you can’t have one intern looking after another). This needs to be done through architecture, Control flow, and all the boring, unglamorous, absolutely essential software engineering that makes the whole thing work.

The practical implication for PMs: when you’re designing an AI-powered workflow, don’t design it as “the AI does X.” Design it as a sequence of governed steps, some of which use AI. Your architecture diagram should show control flow and governance checkpoints, not just a box labelled “AI magic.” If you can’t describe the governance at each step, you’re back to a PDF of well-meaning principles in SharePoint. You’re just crossing your fingers and hoping for the best.

Subscribe now

Humans in the loop, done right

Now rise above the stack into the human world.

“Human in the loop” is the most abused phrase in AI governance. It’s become a checkbox (”yes, we have human oversight, can we have our certification please?”) without anyone asking what that oversight actually looks like or whether it could possibly work at the scale the system operates.

Two examples from my consulting work that illustrate how different “human oversight” needs to be depending on context.

The first: an auto-filing system. Thousands of documents, automatically classified and filed. The founder insisted a human must review each filing decision. This is governance by wishful thinking. At that volume, a human cannot meaningfully review thousands of automated decisions per day. They’ll rubber-stamp, because rubber-stamping is the only option that doesn’t bring the operation to a halt. The oversight is illusory; it creates the appearance of governance while providing none of the substance.

The right mechanism here isn’t review-every-decision. It’s exception handling. The system files documents. When a user discovers a mis-filed document, they have a clear, easy path to correct it, and that correction feeds back into the system. Governance through designed recovery, not through bottleneck. In low stakes, humans don’t need to review every decision. Instead, they correct the ones that matter when it comes up, and ideally the system learns from those corrections.

The second: an online safety platform monitoring for behavioural signals among young and vulnerable populations. Here, the stakes are fundamentally different. There was no scenario – none – where the AI should take an action, or even recommend one.

What we designed instead was a governance cascade. At the first layer, AI filters low-value notifications (with its own control logic) to avoid the notification fatigue that would make human reviewers tune out everything. Significant signals surface as drillable summaries, showing the human not just what was detected but why, so they can form their own judgement. Critical signals escalate explicitly for review by internal staff, who can investigate further and, if needed, contact a person directly. At every layer of the cascade, the AI is helping the human understand signals and their meaning. At no layer is the AI deciding what those signals mean.

Only a human can interpret, judge, and act; AI findings make those more informed, but not replaced. That’s a fundamentally different design from “a human reviews the output.” And it only works because the governance was engineered into the product architecture from the start, not bolted on as an afterthought.

Don’t think of “human in the loop” as a design principle in the abstract. It’s a placeholder for a design you haven’t done yet. The real questions are: What is the human actually deciding? Can they meaningfully make that decision at the volume the system produces? What happens when the system is wrong and the human isn’t watching? What’s the recovery path? If you can’t answer those questions with specifics, your human-in-the-loop is an aspirational speed limit sign, not a speed camera.

Governance as a living system

Bring the internal and external layers together, and you get something that looks less like a policy and more like a product in its own right.

Governance is both proactive and reactive. On the proactive side you have eval loops built into the product from day one: outputs monitored, groundedness measured, failure modes tracked. The safety cascade I described above is proactive governance designed into the architecture. It doesn’t wait for something to go wrong. It continuously processes, filters, and surfaces what matters. This is someone actively looking at telemetry, not a quarterly audit.

On the reactive side come incident review, but not the kind you’re used to. AI failures have different root causes from software bugs. The model didn’t “break” – it behaved exactly as it was designed to, in a context nobody anticipated. The auto-filing system didn’t malfunction when it mis-filed a document. It classified based on the signals it had, and those signals were insufficient. The review needs to ask “what context did we fail to anticipate?” not “what code was wrong?” That’s a different conversation, requiring different people in the room and different questions on the table.

And governance must evolve with the product. I argued in the agentic AI post that you should constrain first and loosen later, as evidence accumulates. The governance system needs to adapt at the same pace. New capabilities mean new governance checkpoints. Loosened constraints mean updated monitoring. A steering committee that meets quarterly can’t keep up with a product that ships weekly. The governance needs to be as continuous as the development.

This is also where external regulation connects. The EU AI Act, Australia’s voluntary AI Safety Standard, industry-specific requirements. They all belong here, in the operational governance layer. But regulation is a topic for another post. The point for now: if your governance is operational, regulatory compliance becomes a reporting exercise. If your governance is a PDF in SharePoint, every new regulation is a crisis.

The architecture does the governing

At the feature level, you choreograph each AI interaction with constraints and guardrails. At the system level, you orchestrate those features with engineering discipline, with decomposition and control flow, and design human oversight appropriate to the stakes. At the regulatory level, the external world holds you accountable for both.

Contrary to the hype by those selling AI tools, programming isn’t dead. It’s never been more necessary. AI agents can do the work, but need the architecture to govern them. And the PDF in SharePoint? It can stay there. Just don’t confuse it with governance.

Every line of code will one day need to be maintained.

It’s usually attributed to one of the programming greats, but I’m afraid the exact origins are lost to time. But it still holds, decades later. Code is a liability. Not in the vague, hand-wavy sense – in the accounting sense. Every feature you ship is a commitment. You’ve made a choice, selected a path, and created something that now needs to be maintained. Not just technically, but in documentation, in support articles, in training materials, in customer expectations, in sales positioning. The direct cost of building is the down payment. The maintenance cost is the mortgage. And unlike a mortgage, this one never gets paid off. I’ve covered this before as product debt, the accumulated cost of every choice you’ve made, compounding quietly in the background. Code is just the most visible form of it.

Hand on heart: how often have you removed a feature from a product? I don’t mean deprecated it or hidden it behind a toggle for that one customer that still needs it. I mean actually ripped it out. If you have, you know the cost. If you haven’t, that tells you something too. Two-way doors are real in theory and rare in practice. Most shipped features are one-way doors wearing a two-way door costume. And every one of them is still drawing a maintenance salary.

I’ve written before about why killing features matters and how hard it is in practice. The theory is easy: “sunset underused features, focus on what matters.” The practice involves angry customers, contractual obligations, a key account that will churn if you take it away after years of use, and a support team that’s built an entire knowledge base around it. Sales have probably demoed it to someone interested last week. The gravity of shipped code is real. Once a feature exists, removing it is harder than building it was.

Building faster doesn’t help

There are two camps in the current AI discourse. Camp one: “AI lets everyone build and ship, ship, ship!” Camp two: “If you thought the speed of writing code was your problem, you have bigger problems.” Andrew Murphy makes the systems argument well. Using Goldratt’s Theory of Constraints, he shows that optimising a step that isn’t the bottleneck doesn’t speed up the system, it breaks it. More code means bigger PRs, longer review queues, higher work-in-progress, and more incidents. I won’t rehash it – go read his post.

But both camps are arguing about the wrong variable. The bottleneck isn’t in the delivery pipeline. It’s upstream, in the decision about what to build. AI tools that accelerate coding without improving that decision are giving you a faster car when you don’t know where you’re going. You arrive at the wrong destination sooner, and now you have more code to maintain when you inevitably change course.

Donald Knuth told us that “premature optimisation is the root of all evil.” He was talking about code. But apply it one level up: optimising the build step of the product process is itself a premature optimisation when the real constraint is knowing what’s worth building.

Subscribe now

Product isn’t what engineering produces

Here’s a reframe that trips up a lot of people, including a lot of PMs who should know better. Your product isn’t the code. It’s the whole go-to-market motion of how you exchange value with the customer: how you put the product in their hands, help them achieve their goals, and capture value in return. The code is the vehicle, not the destination.

In enterprise B2B, this is viscerally obvious. The “product” includes the sales process, the onboarding, the ongoing relationship, the support, the trust built over years of showing up when things go wrong. Every feature you ship adds weight to all of that. Engineering maintains the code. Product maintains the positioning. Sales maintains the pitch. Support maintains the knowledge base. Customers maintain the workflows that now depend on your feature existing exactly the way it does today. The liability is organisational, not just technical.

When your sales cycles are 18 months and your contracts are multi-year, every feature is a multi-year relationship commitment. “Move fast and break things” is a luxury for products where the customer can churn and come back next month. In enterprise, broken trust is a broken contract, and costs you the next three deals to boot. It’s the maths of a business where a single customer relationship can be worth millions over a decade.

Don’t commit too soon

I’ve always advocated a Just In Time approach to product decisions. Delay commitments until they’re genuinely needed, then make the best decision based on available data and move quickly. It’s disciplined restraint, not indecision; the difference between “we haven’t decided” and “we’ve decided not to decide yet, because we’re still learning.”

Code is a liability, so don’t take it on until you have to. The goal of discovery isn’t to generate a backlog of features to build. It’s to build understanding of the problem space so that when you do commit, you commit to the right thing with conviction. Every premature commitment, every feature built before the problem is fully understood, is a bet placed with incomplete information. Sometimes you have to bet early. But tools that make betting cheap don’t make betting smart.

Here’s the thing about AI-accelerated building: the overall number of features you ship shouldn’t increase. We’re not trying to throw more spaghetti at the wall to see what sticks. We’re trying to commit with more confidence. The speed goes into learning, not producing. You explore more options, kill more bad ideas, and arrive at the commitment point with a better understanding of what’s actually worth building. The end result? Fewer, better features, each one earning its justifying cost because you understood the problem before you started solving it.

Prototypes are for thinking, not shipping

This is where the speed of AI tools is genuinely useful. Not for building production features faster, but for building throwaway prototypes that help you think and communicate.

I’ve been using clickable prototypes as a communication tool for years; since a good designer introduced me to Figma. Before that, I used static screens. The difference was significant. Static mockups made sense in my head and in the designer’s mockups, but they were clunky when I used them in the field. The moment you could actually click through a flow, you felt where it broke. The conversation changed from “imagine if you could...” to “try this and tell me where it doesn’t work.”

AI coding tools take this further. I can now explore my own UX and flow ideas with rough, functional prototypes, before involving design. That’s not about cutting designers out, but about self-critique. I can build something, use it, realise it’s terrible, and kill it without wasting anyone’s time. When I do bring it to design, I’m not handing them a spec – I’m showing them what I’ve learned from trying it myself. “This flow felt wrong at step three, and I think it’s because...” is a better conversation starter than “here’s a wireframe I think works.”

A few practical notes, since we’re on the subject. Use whatever tools you’re comfortable with, the ones that handle the framework and visual scaffolding so you can focus on the flow, not the plumbing. Don’t get hung up on functionality. These are for illustration, not production. They are disposable. If you’re tempted to ship your prototype, you’ve missed the point.

And one deliberate tip: ask the tool to make it look rough. Like a sketch. The moment a prototype looks polished, stakeholders start treating it as a commitment rather than a conversation. A sketch says “let’s discuss.” A polished screen says “we’re building this.” You want the former. This avoids anchoring, the cognitive bias where the first thing people see becomes the reference point for everything that follows. Show them something that looks like a draft, and they’ll engage with the ideas. Show them something that looks finished, and they’ll critique the pixel spacing.

The broader principle: AI dropped the cost of exploration dramatically. Use that. Run three rough prototypes past a design partner instead of building one polished feature that might be wrong. The prototypes are conversations. The feature is a commitment. Know which one you’re having.

Commit when you’re ready

You may not regret every line of code (many will be genuinely useful), but it will one day need to be maintained. The question was never how fast you can write it. It’s whether it should exist at all.

AI didn’t change the fundamental equation of product development. It just made one side of it cheaper. The liabilities are still liabilities. The maintenance mortgage still never ends. The gravity of shipped features still makes them nearly impossible to remove.

The smart move isn’t to produce more code. It’s to produce less, better, and later – to use the speed for learning rather than committing. Explore more. Build throwaway things that teach you something. Kill your bad ideas in a prototype, not in production, because “learning by shipping” is expensive. And when you finally commit, commit with the confidence that comes from having done the thinking first.

The number of features you ship doesn’t go up. The quality and fitness of each one does. And that, for a product that needs to sustain multi-year relationships, is worth far more than velocity.

This isn’t one of those posts. Not because the tools don’t matter – they do, a bit – but because they’re the least interesting part. A Photoshop licence doesn’t make you a graphic designer. A Figma subscription doesn’t give you taste. And a Claude API key doesn’t make you good at using AI.

What makes you good at using AI is the same thing that makes you good at anything: understanding your domain deeply enough to know what “good” looks like, then using tools to get there faster.

The workshop

Walk into any craftsperson’s workshop and you’ll notice something immediately: it’s intensely personal. Tools arranged for their workflow, not alphabetically. Jigs and fixtures they’ve built themselves to solve specific problems. Materials organised by how they work, not how they’re sold. No two workshops look the same, because no two craftspeople think the same way.

That’s what a mature AI setup looks like. My daily workflow involves an AI coding assistant and a knowledge management tool running side by side (I’ll tell you the names later), with different project spaces for different work. I’ve built custom commands that automate my specific workflow: how I like to process meeting notes, track commitments, draft and revise written content. I use integrations to pull information in from various sources, and to push updates out carefully, with human review at every publishing step. Some of these tools I chose; some are whatever the team I’m working with uses. All of them are configured and customised for how I think, not how they shipped out of the box.

The point isn’t “use these tools.” The point is that this workshop was built over time, one jig at a time, each addition solving a specific friction point in how I work. It looks nothing like anyone else’s setup, because it shouldn’t. Most “How to AI” posts describe a factory floor with standardised workflows that could belong to anyone. That’s fine for getting started, but it’s not where the value lives. Conversely, most “How I AI” posts are interesting but not directly applicable to you. You’re watching someone else’s mise en place, and the useful exercise is figuring out what you might take away from it, not shoe-horning yourself into their workflow.

This is mine. Take what’s useful. Leave the rest.

Art, not science

AI is constantly evolving. Models from different vendors have subtle variabilities in what they handle well and what they fumble. Different flavours of the same model behave differently depending on context, prompting, and what you’re asking for. There are no stable recipes. There is no One True Way™️.

The only universal guidance I can offer: explore constantly. Use AI tools in domains you understand well, so you can evaluate the output. Push them to the edges of their capabilities. Develop an instinct for what researchers call the “jagged frontier” – the irregular boundary of what AI can do reliably, which zigs and zags unpredictably across tasks. You’ll find it’s brilliant at some things you expected it to struggle with, and terrible at things that seem simple. The only way to learn that boundary is to walk along it, repeatedly, with a critical eye.

This is more art than science. Like a sculptor who knows marble or a painter who knows how oils behave in humidity, the skill isn’t the chisel or the brush. It’s the thousands of hours of understanding the material that let you work with it rather than against it. You develop a feel for when the tool is helping and when it’s leading you astray. That feel can’t be taught in a course or captured in a prompt template. It comes from doing.

My personal quality test is simple: do I want my name against this? It’s personal taste built over years of doing the work without AI first, so I know what “good” looks like in my domain before I ask a machine to approximate it. Access to tools is near-universal now. You can (should) try them regularly, but that’s not the big deal. What’s important is whether you’d know a bad output if you saw one.

The maturity ladder

Here’s the progression that I think matters, both for individuals and for organisations:

First, learn the domain. There is no shortcut for this. Years of doing customer interviews, writing strategies, shipping products, making mistakes, and building the pattern library that lets you recognise when something is off. This is the apprenticeship.

Then, use AI where you already have expertise. Start with your own workflow, with tasks where you can immediately evaluate quality. When I first started playing with LLMs a few years ago, I used them for things I understood deeply: drafting communications I’d been writing for years, analysing data I’d collected myself, summarising research in areas where I could spot a hallucination at a glance. The AI was faster. I was the quality gate.

Then, extend to team and cross-functional processes. Once you’ve built intuition for what AI handles well and where it stumbles, you can start applying it to broader workflows such as internal tools, process automation, team efficiency. But now you’re responsible for other people’s reliance on the output, so the bar goes up.

Only then, put it in front of end users. This is where data quality, constraints, and careful design all come together. And it’s where the stakes are highest when you get it wrong.

My own career traced a version of this ladder, stretched over a decade. Eight to ten years ago, I was reading the foundational papers and talking to data scientists, trying to understand what machine learning could and couldn’t do. Three to five years ago, I was working deeply on metadata enrichment and content processing pipelines, the very plumbing that AI systems depend on. The last couple of years, I’ve been hands-on with LLMs and other models as they became viable, constantly probing what they can do reliably. Only in the past year have I been building the workshop, the custom tools and environments that fit my specific workflow, because that’s when the AI tools could support it to degree I was satisfied with.

You can’t skip rungs. And organisations can’t skip them either: audit your data, understand your domain, pilot internally, and only then face customers. The companies that rush straight to customer-facing AI without climbing the ladder are the ones writing apologetic blog posts six months later.

A note for early-career PMs

This note comes from empathy, born from watching what happens when the ladder gets skipped. I was young, rash, and brash too, once. Don’t read it as gatekeeping.

The concern isn’t that junior PMs use AI. Of course they should. The concern is that they might automate and abstract something they don’t yet understand. If you haven’t done enough customer interviews to recognise when an AI-generated summary misses the subtext, you’ll trust the summary and miss the insight: the frustration behind a polite feature request, the political tension in a stakeholder meeting, the thing the customer said between the lines.

Everyone’s seen the obvious failures. AI mis-transcribes a meeting. A search tool recommends, confidently and in detail, an API that doesn’t exist. An analysis of your data gets the basic relationships wrong – and you know it’s wrong because you collected the data. Those are easy to catch. The dangerous failures are the subtle ones you don’t catch, because you don’t yet know enough to notice. Without domain expertise, delegating to AI is just two interns looking at each other, hoping one of them is right. You can’t tell which output is good and which is wrong, because you haven’t done the reps.

The industry narrative that AI will “democratise” expertise assumes that expertise is just information. Sorry sunshine, it ain’t. It’s judgement. And judgement comes from doing the work, from years of getting it wrong, learning why, and building the instinct that tells you when something is off even when it looks fine on the surface. Do the reps. Build the taste. Then the tools become genuinely powerful, because you’re the senior in the room and not another intern.

One way to recognise where you need to invest more manual effort is to notice what you’re procrastinating on. The tasks you know you should do, but just can’t bring yourself to. Why? Is it boring because you’ve done it a thousand times, or because it’s pushing you out of your comfort zone? This is a visceral distinction (literally, you’ll feel it in your gut). When you’re avoiding getting out of your comfort zone, you’re avoiding growth, avoiding the opportunity to build the very product taste you need. I’ve had a good experience early in my career on pushing against the butterflies in my stomach, and learnt a lot about the skills of public speaking. I’ve learnt to recognise the symptoms, and force myself to do it and grow. And it’s at that point that you can make a judgement call about what to automate and delegate to AI, because you know when good looks like.

What’s in my workshop (today)

Since you’re reading a “How I AI” post, here’s the obligatory tool mention. These are what’s in my workshop right now. Ask me again in six months and half of them will have changed.

I use Claude Code for building custom tools and automations, and Obsidian as a knowledge management platform for deep thinking and organising info, keeping separate project spaces for each area of work. I have multiple Obsidian vaults open and a terminal with multiple tabs for Claude sessions, but I aim to work on one thing at a time for a stretch. The point is to have the tools open and ready to capture ideas, but more importantly to help me concentrate and do the deep thinking (distractions are free and abundant; keeping them at bay is a constant chore).

I’ve used others, these are my (current) favs, as they allow me to best bridge the gap between my thinking and the AI augmentation. Obsidian in particular, since I deal with documents rather than code. Markdown is great for both me and AI, Obsidian has a phenomenal plugin system, and when I can’t find something I can get Claude Code to write a plugin for me.

Meeting notes are captured and analysed (via MCP) in a process that makes sense to me, surfacing action items and tracking commitments in ways I actually pay attention to. Other tools for project and task management, prototyping, documentation are all the usual suspects from your preferred “top AI tools” list (like Lenny’s newsletter); some mine, some decided by the team I’m working with. I use integrations to pull information from various services for analysis, and to push updates out; reading freely, publishing carefully with human review at every step.

The specifics matter less than the principles: understand what you need before you choose the tool (please don’t make me bring up the hammer adage). Customise the tool to your workflow, not the other way around. Automate the tedious (note processing, status tracking, first drafts or copy edits). Preserve the valuable (deep thinking, critique, relationship building). And keep the human in the loop for anything that actually matters.

One thing that’s genuinely changed in the past six months: these tools have gotten good enough to teach you how to use them. You can ask the tool itself how to best leverage its capabilities, and it’ll give you a useful answer. Ask it to build a skill or a command, and work with it to customise your own environment to your needs. Don’t copy others’ generic implementation – spend a week to build and debug a skill that automates a task that used to take 5 minutes! 😜

But, and this is the taste paradox again, you still need to know what you’re trying to solve. The tool won’t volunteer that a capability exists unless you understand the limitation well enough to ask about it in the right terms. And you need to push back when its suggestions are wrong. This can range from the obvious (no, you can’t “clear the cache” with rm -rf /), to the subtle (actually understanding what makes sense to include in a PRD for your product, team, and organisation; it’s the same as copying Amazon’s 6-pager template and expecting to become Amazon).

Even self-teaching tools require a student who knows which questions to ask.

Own your shit

I’ve written before about the three core (and rather shitty) PM skills: organise your shit, communicate about your shit, and own your shit. AI maps neatly onto these, but not evenly.

It can genuinely help you organise. That’s what the workshop is for: automating the tedious parts of information management, keeping track of commitments, surfacing patterns in data. This is where AI earns its keep, doing the grunt work so you can focus on the thinking.

It can assist with communication. Drafting, summarising, critiquing, restructuring. But you still need to validate that the message actually carries across between humans: that the nuance is clear, that the tone is right, that you’re speaking in the other person’s language rather than a statistically plausible approximation of it.

But ownership – having real agency, putting your name against the work, deciding what matters and taking responsibility for the outcome – that is irreducibly human. No tool gives you that. And without it, you’re just two interns hoping one of them is right.

The workshop is never finished. There’s always a new tool to try, a new jig to build, an old process to rethink. That’s the art of it. The craftsperson who stops tinkering with their workshop has stopped learning.

But if I’m honest, the best thing AI has done for my PM practice isn’t any specific tool. While automating the boring bits helps, the constant exploration of AI’s boundaries has forced me to articulate what I actually value in my own work: which parts require human judgement, which parts are just friction, and where the line sits between them. That clarity (which is crystalised in writing these posts) is worth more than any tool I’ve added to the workshop.

As a fiction author, I can tell you: the best tools are the ones that disappear into the craft. You stop noticing the chisel and start seeing the sculpture.

In the AI industry, “agent” means a system that takes multiple steps to complete a task: search for something, summarise it, draft a response, maybe even send it. In philosophy, law, and common sense, “agent” means something quite different: an entity that makes autonomous decisions toward its own goals. An entity with agency.

These are not the same thing. And the confusion between them - sometimes innocent, sometimes very much not - is how you end up building products that hurt people.

I was working on a system to help prospective students explore career pathways at a university. It did a genuinely wonderful job: conversational, helpful, drawing on the institution’s course catalogue and career outcomes data. Right up until the point it helpfully recommended a competing institution’s program.

This comes under “Not Even Wrong” - it was trying to be helpful, can had no way to tell the difference. But it had been given too much latitude and too little constraint. The system was doing what it was built to do (find relevant information and present it helpfully), and in the absence of clear boundaries, “relevant” included the competition. We caught it well before public deployment, and it was a good illustration of why you need both guardrails and behavioural cross-checks. You can’t trust the system to infer what “helpful” means in your context. You have to define it.

The gap between “agentic” and “agency”

Let’s be precise about what we’re dealing with.

What the industry calls “agentic AI” is really multi-step task automation: an LLM connected to tools (search, APIs, databases) that can chain several operations together. Ask it to research a topic, and it searches, reads, synthesises, and presents. Ask it to process an inbox, and it reads, categorises, drafts responses, and flags exceptions. These chains can be impressive, and genuinely useful when designed well.

What “agency” means (the real kind) is autonomous decision-making with independent goals. The ability to decide what to pursue, not just how to pursue what you asked for. A system that decides on its own what to search for, who to email, and what to say. In regulated contexts such as government, legal, healthcare, financial services, that distinction isn’t academic. It’s the difference between a useful tool and an uncontrollable liability.

I noted in passing a while back that you can delegate tasks to “agents” where they don’t actually have any agency - that decision-making should be the responsibility of the users. That aside deserves unpacking, because the industry is moving fast in the opposite direction, and the product design implications are significant.

Choreography, not improv

Here’s a way to think about it. Picture a dance performance. A dancer leaps into the air at another who has their back turned - who then spins at precisely the right moment to catch them, lifts, and throws them into the next sequence. The audience gasps. It looks spontaneous, reactive, almost telepathic. Two performers apparently making independent decisions in real time, responding to each other with fluid intelligence.

It is, of course, nothing of the sort. Every step was designed. Every transition rehearsed. Every lift has a recovery move built in for when the timing is slightly off. The choreographer spent weeks and months mapping what the audience needs to see, what the dancers can do reliably, and what to do when something goes wrong. The audience sees magic. The choreographer sees engineering.

Good agentic product design is choreography. The user experiences what feels like an intelligent, responsive interaction. Behind the scenes: a system prompt (the script), guardrails (stage directions), the PM and product team (the choreographer), and the model can only access what the stage allows. When a performer stumbles, there are recovery moves, not an actor freelancing off-script and hoping for the best.

What much of the industry is currently selling is improv night at the comedy club. Hand the AI a microphone, point it at the audience, and see what happens. Sometimes brilliant. Often cringe. Occasionally offensive. And you never know which you’ll get on any given night.

Fine for entertainment. Not fine for a university advising students about their futures.

The spectrum of constraint

The amount of freedom you give an AI system isn’t a binary choice - it’s a spectrum, and where you set the dial should depend on the stakes.

On one end: a music recommendation engine. Low stakes. User can skip. A bad recommendation is mildly annoying. Give it latitude. Let it experiment.

On the other end: a government service answering questions about citizen entitlements. Or a law firm’s knowledge system. Or a healthcare triage tool. Wrong answers here can materially harm people’s lives. Constrain it heavily.

Most enterprise products sit toward the constrained end, which is where the real product design skill lives. AI companies promise you agentic systems that can do everything, and the differences between them are increasingly commoditised. But the real challenge is designing one that does the right things within the right boundaries and fails gracefully when it hits the edge of what it knows.

This connects directly to data quality and metadata. Remember the university example? Some courses listed in the catalogue are still being taught but are no longer available to new students: existing enrollments can finish, but the intake is closed. The content says “Bachelor of Arts in X.” The metadata says “closed to new enrolments as of 2026.” Without that metadata, your helpful AI agent enthusiastically recommends a course the student can’t actually take. The constraint - the metadata that tells the system what’s currently available - is what makes the agent useful rather than merely impressive.

Designing the choreography

I’ve written before about the principle of Refrain, Restrict, and Trap in the context of securing AI systems. But RRT isn’t just a security framework. It’s a product design framework for any system where you want the experience of agency without the risk of it.

Refrain is choosing the stage. Not every task should be choreographed on stage. The first design decision (and arguably the most important) is which tasks belong in the performance and which don’t. Our university system could help prospective students explore career pathways based on the institution’s offerings. It could not compare those offerings against competitors, advise on financial aid, or make promises about employment outcomes. Product Management 101: understand what the user actually needs, what the technology can reliably do, and where the boundary sits between them.

Restrict is setting the stage. Don’t treat Least-Privilege as an annoying security principle; instead think of it as a UX principle. The system only accesses what it needs for the specific task. Our answer engine doesn’t have access to backend administrative systems, only to curated, metadata-enriched content about available courses and career outcomes. Input length is bounded. Topics are filtered. The audience can feel that the stage has walls, and that’s what makes the performance feel safe.

Trap is the safety net. Input sanitisation and output evaluation are the choreographer’s recovery moves. The user’s query is checked and reframed. The system’s response is evaluated for groundedness and faithfulness to the source material before it reaches the user. When the performer stumbles, the stagehands catch it before the audience notices. This is why eval systems are so important, and why you need to make them an integral part of the performance, not an afterthought.

The industry push is in the opposite direction: more tools, more access, more autonomy. There are commercial reasons for that - more autonomy means more API calls means more revenue for model providers. But more autonomy also means more attack surface, more unpredictable behaviour, and more liability. “But if we constrain it, we lose the magic!” is the usual objection. I’d argue the opposite: choreography is the magic. The audience doesn’t want improv. They want a performance that reliably delivers what they came for. There are good reasons why tickets to the Bolshoi cost more than a basement comedy night. The constraints are what create the trust that makes the system valuable.

Practical choreography for PMs

If you’re building features that feel agentic, here’s where to start:

Start with the failure mode. Before designing what the agent does, design what happens when it goes wrong. What’s the blast radius? Can the user recover? Is a human notified? If you can’t answer these questions, you’re not ready to build the feature. You’re ready to build the safety net.

Choreograph, don’t improvise. Map every step the system takes. Design the transitions, the inputs, the outputs, the boundaries. Good old-fashioned business analysis and process mapping. Every interaction where the AI “decides” something should actually be the AI following a designed decision tree within defined constraints. If you can’t describe the choreography in detail, you don’t have a product. You have a demo.

Constrain first, loosen later. Launch with tight boundaries. Observe how the system behaves with real users and real data. Loosen where the evidence shows it’s safe. This is the opposite of the prevailing industry advice (”ship fast, add guardrails later”) and it’s the right approach for anything with real-world consequences. In enterprise and regulated contexts, a cautious launch that earns trust will outperform a flashy launch that erodes it.

Evaluate relentlessly. Build the eval loop from day one, not as a bolt-on after launch. Monitor outputs, measure groundedness, track failure modes and near-misses. Feed what you learn back into the constraints. The choreographer watches every rehearsal and every performance and adjusts the choreography based on what they see. Plan to do the same, before and during any interaction with users.

The best performances are always scripted

The most impressive performances are the ones where the audience forgets they’re watching something designed. That’s the goal for agentic AI products: users feel like they’re interacting with something intelligent and responsive, while the PM knows that every step was anticipated, every boundary was set, and every failure mode was accounted for.

The word “agent” comes from the Latin agere - to do, to act. The industry has the “do” part covered. The “act” part - in the theatrical sense - is what good product design adds. Not actual autonomy, but the compelling, trustworthy performance of it.

As a fiction author, I can tell you: the best performances are always scripted. The skill is in making the audience forget that.

This is why pricing work feels so similar to product work. It’s the other side of the same coin (pun intended). Understanding what your product is worth to the customer is understanding your product. And yet most PMs treat pricing as finance’s problem – something that happens after the product decisions are made. That’s like designing a bridge without checking how much weight it needs to carry.

The PMs who do engage with pricing usually land on “value-based pricing” – charge based on the value you deliver, not your costs. That’s the right starting point. But it tends to measure value in one dimension: what is this product worth to this customer in this transaction? The value that compounds – goodwill, trust, referrals, forgiveness when things go wrong, the renewal in three years – doesn’t fit neatly into that calculation. And that’s the value that matters most.

The whatchamacallit that costs $100

Let me sell you something. Say I have a gizmo or a service that delivers $100 of value to you – it saves you time, reduces risk, makes your life measurably better by a hundred dollars’ worth.

If I bill you $120, you’re not buying. Or you’re buying resentfully, under pressure, and you’ll leave the moment an alternative appears. This is the obvious case.

If I bill you $80, you’re happy. You feel you got a good deal. You like the product. You like me. When something goes wrong – and something always goes wrong – you give me the benefit of the doubt. You stay longer. You tell your colleagues about the good deal you got. You expand into other use cases. You forgive the rough edges.

But what happens when I bill you $99?

You buy. You’re not unhappy, exactly. But you’re neutral. The product had better deliver everything, in full, perfectly, because you’ve paid for nearly every penny of value it provides. Any friction, any gap, any moment where reality falls short of the promise – and you’re re-evaluating. You have no surplus goodwill to draw on, and neither do I.

The insight: the most profitable price isn’t the highest one you can charge. It’s the one that creates surplus on the customer’s side – because surplus is what turns a transaction into a relationship, and relationships compound.

Think about the last time you bought a car or got a haircut (same principle). If you walked out feeling you’d got a fair deal – maybe even a slightly good one, just happy with what you got for what you paid – you probably went back to the same dealership (or hairdresser) next time. If you walked out feeling squeezed, you didn’t. The car was the same. The surplus (or lack of it) determined whether there was a next time.

This isn’t charity. It’s strategy. The relationship over the next five years is worth more than the $19 “left on the table” this quarter. Optimising revenue per transaction is not the same as optimising revenue per relationship.

Subscribe now

Positioning is pricing is product

I worked on a product once that competed against a well-established, feature-rich incumbent. We had fewer capabilities – we knew that, the market knew that. The temptation was to price just below the competitor and hope the discount made up for the feature gap.

Instead, we positioned fundamentally differently. You weren’t buying a Mercedes. You were buying a ute (a “pickup truck,” for my American friends). It costs a quarter of the price, it does most of what you need, and you don’t cry when it gets scratched. The pricing wasn’t just “cheaper” – it changed the entire conversation. Customers weren’t comparing feature-for-feature against the incumbent. They were asking a different question: “do I need the Mercedes, or do I need the ute?” And for most of them, the honest answer was the ute.

That positioning created enormous value surplus. Customers felt they were getting a great deal – because they were. They weren’t paying for features they didn’t use. The relationship started from a position of goodwill, and that goodwill compounded into loyalty, referrals, and expansion. The customers who started with the ute often grew into needing more – and by then, they trusted us enough to buy it from us rather than switching to the incumbent.

Pricing signals who you are. It signals who you’re for. And it sets the tone for every interaction that follows.

This isn’t just about invoices

Here’s where it gets interesting for PMs who don’t touch pricing directly. The value exchange principle applies to every interaction you have – not just the ones that involve money.

Every time a stakeholder asks you for something, you’re in a value exchange. You provide something (time, expertise, effort). You receive something (trust, credibility, access, future cooperation). And the balance you strike determines the relationship.

The Sales VP needs you on a customer call. You can do the minimum: show up, answer the technical questions, leave. Or you can over-deliver. After the call, send a note: “the way that customer described their problem suggests they’re actually worried about X, not Y – here’s how you might frame it next time.” That costs you fifteen minutes. It changes how sales thinks about you.

Marketing asks you to review a blog post. You can mark up the typos and send it back. Or you can help them understand the core messaging – the language that resonates with the customers you’ve been talking to, the framing that makes the value proposition click. Now they’re writing better content, and next time you need something from marketing, you’re not starting from zero.

Sounds like more work. It is – at first. But each small over-delivery compounds. I got asked to join more sales calls. I got called by marketing more often. And then, one day, I was standing in front of a room delivering executive workshops – because the organisation had learned they could trust me in front of senior customers and partners. That didn’t happen because I applied for it. It happened because the surplus I’d created over months of small over-deliveries had accumulated into something I couldn’t have planned: a reputation that opened doors.

You can be utilitarian about this: invest in stakeholders and they’ll be there when you need them. That’s true, but it undersells the point. The real returns are the opportunities you can’t predict. The doors that open because someone mentioned your name in a meeting you weren’t in.

The quarterly target problem

I can hear the objection: “This is lovely, but I have a revenue number to hit this quarter.”

Fair. You can’t fight the tide. If the organisation optimises for quarterly extraction, you won’t single-handedly change that by being generous.

But you can still quietly over-deliver within the constraints you have. Give the customer a bit more than they expected. Be honest about what doesn’t work. Follow up after the deal closes, not just before. Build the relationship that outlasts the quarter.

The compound interest works even in hostile conditions. The customer who feels they got a fair deal will sing your praises to the CEO next time they share a drink at a conference. That warm introduction is worth more than whatever you could have squeezed out of the last negotiation. The PM who understands this – who sees willingness to pay as discovery data, who treats pricing as a product decision, who knows that the adoption gap is crossed with trust rather than features – is a PM who understands something fundamental about their product that the spreadsheet optimiser doesn’t.

The exchange that compounds

Every product, every interaction, every relationship is an exchange. The question isn’t “how much can I extract?” It’s “how much surplus can I create?”

Because surplus of goodwill is what makes people come back. Surplus is what turns a customer into an advocate, a stakeholder into a champion, a transaction into a relationship. And relationships – the ones built on genuine over-delivery, not on promises – are the asset that compounds long after the quarterly numbers are forgotten.

Leave a little on the table. It’s the best investment you’ll make.

Meanwhile, a policy from ten years ago - one that was still very much in effect - was buried in the results because the system assumed recent meant relevant.

The technology worked fine. The data wasn’t the problem either - it was all there, all accurate. What was missing was the understanding of the data. No one had tagged which policies were current, which were superseded, which had expiry dates tied to financial years rather than calendar dates. Without that context, even the best model in the world will confidently give you the wrong answer.

I’ve spent 20 years in information management, and I’ve seen this pattern more times than I can count. The specifics change - sometimes it’s policies, sometimes it’s emails, sometimes it’s case files - but the underlying problem is always the same. Everyone wants to talk about the clever technology. Nobody wants to talk about the data it’s consuming.

The Library of Babel

Jorge Luis Borges wrote a short story about a library that contains every possible book. Every truth humanity has ever known is in there. So is every lie, every piece of nonsense, and every possible arrangement of characters that could exist. Good fiction sits alongside trashy romance alongside the equivalent of the editing room floor - all the cuttings and drafts and half-formed ideas, shelved right next to the masterpieces.

It is simultaneously the most comprehensive and the most useless library ever conceived. Because without a card catalogue, without classification, without any way to distinguish a masterpiece from gibberish, having everything is functionally the same as having nothing.

Most enterprise data repositories are miniature Libraries of Babel. Decades of documents, policies, procedures, email threads, knowledge articles, and meeting notes, spread across SharePoint, Confluence, shared drives, and legacy systems. Organisations look at this and think “we have so much data - AI will love it!” And the AI does consume it all. Every last byte. And then it tells a citizen that the expired policy is the current one, because it has no way to know the difference.

The problem was never the reader. It was always the library.

“Just vector search it”

Every organisation goes through this phase, and that’s fine - it’s a necessary first step. You take your content, chunk it up, generate embeddings, and point a vector search at it. The first time you get a coherent, contextual answer drawn from your own data, it feels like magic. You show the demo to an executive, everyone gets excited, and for a brief, shining moment the AI hype feels justified.

But vector search on raw content is just step one. It matches on semantic similarity - essentially, “what chunk of text looks like it’s about the same topic as the question.” That’s genuinely useful. It’s also genuinely limited.

Semantic similarity has no concept of authority, currency, audience, or context. It doesn’t know that this policy superseded that one, that this version is a draft and that one is approved, that this document applies to Victoria and not Outer Mongolia. It can’t tell if the content is from an authoritative source or someone’s personal notes. It certainly can’t figure out that a January publication date means “expired” while a decade-old document is still the one you want.

This is where most organisations plateau. They got the dopamine hit, showed the demo, declared the AI project a success. Then the edge cases start creeping in. Users get a wrong answer, lose trust, and stop using the system. Someone suggests “maybe we need a better model” - when what they actually need is better-organised data.

I’ve written before about the maturity journey organisations go through with AI. Vector search on raw content is the “Tools and Tech” layer at the base of the pyramid. You have to go through it. But you have to go past it.

The card catalogue is the product

Here’s the thing that 20 years of information management has drilled into me: content tells you what something is. Metadata tells you what to do with it.

A document’s text will tell you it’s about parental leave policy. Metadata will tell you it was approved by HR in 2019, applies to permanent employees in Australia, supersedes the 2016 version, and is due for review in June. Without that context, your AI system is a very fast reader loose in the Library of Babel - it can find every book that mentions parental leave, but it can’t tell you which one to actually follow.

Metadata enrichment - adding structured context to content - is what turns the Library of Babel into a functioning library. And here’s what makes it interesting from a product and engineering perspective: metadata is orders of magnitude smaller than the content itself, but far richer in terms of what it enables. You’re not making the library bigger. You’re making the card catalogue smarter.

I’ve seen this pattern play out across very different products:

In one role, we worked on email correspondence where results needed to be aged as conversations progressed. The content of an email doesn’t change after it’s sent, but its relevance shifts as the thread moves on. A naïve timestamp sort would surface the latest email, but the critical context was often several messages back. We needed metadata about recency-in-context - the position and role of each message within a conversation - not just a date field.

In another, we built data processing pipelines, and a key design decision was building metadata enrichment into the pipeline itself. The pipeline wasn’t really the product. The enrichment was. Every step that added structured understanding to the raw data made everything downstream - search, analysis, reporting - meaningfully better. In fact, most steps were keyed to remove content based on enriched metadata. When you deal with incoming petabytes, pruning before searching is not an option unless you have enough coffee to last past the heat death of the universe.

In yet another role, we worked on records management with a rules engine. Classification alone wasn’t enough. The engine needed context to make decisions: is this CV from a job applicant or a current employee? How old is it? What’s the retention obligation for this category of document in this jurisdiction? The content says “CV.” The metadata tells you whether to keep it for seven years or delete it yesterday.

The pattern is consistent. Across email, forensic data, records management, and now AI-powered search: the organisations that invest in understanding and enriching their data get dramatically better results than those that throw raw content at the problem and hope for the best.

And here’s the kicker - this metadata enrichment often results in better outputs from smaller models. When you give a model structured context rather than asking it to infer everything from raw text, you need less brute-force capability. You don’t need a bigger reader. You need a better-organised library.

The skills nobody thought were sexy

Here’s the delicious irony. Information management, records management, taxonomy design, metadata schemas, retention policies, provenance tracking - the disciplines that made librarians’ and archivists’ eyes light up and everyone else’s glaze over - are exactly what AI systems need to function properly.

These aren’t new problems. Classification systems, authority controls, and content governance have been well-understood for decades. The AI industry is rediscovering them under shinier labels: “data curation,” “knowledge graphs,” “retrieval optimisation.” It’s the same principles in a new coat of paint. Every time I see a breathless LinkedIn post about the importance of “data pipelines for RAG,” I think of the records managers I’ve worked with who have been doing this - minus the acronyms - since before most AI engineers were born.

This also means the skills gap isn’t where people think it is. Everyone’s hiring prompt engineers and ML specialists. Almost nobody is hiring taxonomists, metadata architects, or content strategists for their AI teams. The people who know how to organise a library are the ones you need to make AI actually work - and they’ve been right there the whole time, misfiled under “not technical enough.” 😅

Maturing past the magic trick

So what do you actually do about this? Here’s a progression - not a rigid checklist, but a maturity journey. Every organisation will move through these at different speeds, and that’s fine.

Audit your library. What data do you actually have? Where does it live? Who owns it? How current is it? You can’t enrich what you haven’t mapped. This step is boring, essential, and almost always skipped in the rush to “do AI.” If you don’t know what’s on your shelves, no amount of clever technology will help you find the right book.

Establish authority. Not all content is created equal. Identify what’s authoritative versus draft, current versus expired, approved versus superseded. This is editorial judgement, not engineering work - and it’s one of the reasons you need humans who understand the domain, not just the technology.

Enrich with metadata. This is where the real value is created. Add the structured context that raw content lacks: effective and expiry dates, target audience, classification, provenance, and relationships to other content. Think of it as building the card catalogue that Borges’ library never had. It doesn’t need to be perfect on day one - even basic enrichment dramatically improves what your AI can do with the content.

Build governance. Who maintains the metadata? How often is it reviewed? What happens when content changes? Data quality isn’t a project with a completion date - it’s an ongoing practice. Just as a garden needs constant weeding and planting, your data needs constant pruning and tending. Skip this step, and your beautifully enriched library slowly degrades back into Babel.

Close the loop. Once AI is consuming your enriched data, monitor what it does with it. When answers go wrong, trace it back: is it a model problem or a metadata gap? More often than you’d think, it’s the latter. Build the feedback loop between AI output quality and data enrichment priorities, so that every bad answer makes the library a little better.

The librarians were right

Borges’ Library of Babel was infinite and therefore useless. Your data doesn’t have to be. The organisations that will get lasting value from AI aren’t the ones chasing the latest model release or the biggest context window. They’re the ones investing in their card catalogue - the metadata, classification, and governance that turn a pile of content into navigable, trustworthy knowledge.

I’ve spent 20 years telling organisations to care about their data. It only took the biggest technology hype cycle in history for anyone to start paying attention.

Better late than never, I suppose. The librarians were right all along. (My mother was one – I should know 😅)

These two books are slightly older (only a year or two!), but even though the industry moves fast — humans don’t. The problems identified have not materially changed as they deal with the creation of the models and the social impact of the hype.

These aren’t the first books I read and recommend on the subject (see AI Snake Oil), as I’m a firm believer in Responsible AI development. The hype, and its consequences, are pretty much as irresponsible as you can get. I find it’s useful to understand both what the technology can do, as well as what it can’t and the price it exacts. This helps me navigate my products to sustainability through the murk and upheavals.

The books mentioned below are highly recommended reading to anyone who understands the dark meaning behind the phrase “drinking the Kool-aid.”

The AI Con

By Emily Bender and Alex Hanna

Dr Bender is a professor of linguistics, working in computational linguistics an computer science. Dr Hanna is a director of research at the Distributed AI Research Institute (DAIR) and a lecturer in the School of Information at the University of California, Berkeley. They are well-positioned to observe the AI industry, and expose some of the shady (or downright dark) dealings behind it.

After introducing the (shoddy) history of Ai as a marketing term, the authors cover all sorts of hype around it — from thinking machines to superintelligence, from work automation to unsubstantiated claims around science. For each, they review who benefits from the hype, and what are the damages this causes. They show why “AI Doomers” are just as bad as “AI Boosters,” being largely the same — a distraction from present day evils.

Lastly, they do cover ‘life after hype,’ giving strategies to consumers, users, and organisational buyers to fight the hype and know what they are actually getting. While I find some of their recommendations naïve in this political climate, they are nonetheless something we should strive for as we pick up the pieces after the bubble collapses.

Blood in the Machine

Brian Merchant

I came across ’s excellent blog first, and through it learnt about his book. Merchant is a technology columnist, and has been reporting on Silicon Valley for a while, across reputable publications.

In this book, he covers the history of the Luddite movement in the early 19th century — and why the term ‘Luddite’ means something different than today’s vernacular of ‘progress hater.’ The Luddites didn’t oppose technology, but opposed the rapacious nature of the factory owners, who used technology to amass and concentrate power, and push skilled workers out by replacing their output with low quality products. His writing style is very approachable an engaging, bringing history to life and making it relevant to today.

In the last chapter, Merchant covers how similar things have been occurring over the past few years around knowledge work and other industries. Human nature (and mathematics of wealth) haven’t changes, and power accumulates. This time it’s Big Tech, at the expense of the rest of us.

Further Reading

Empire of AI, by Karen Hao

This one is specifically around OpenAI and Sam Altman, a company and CEO that give me the ick. Perhaps not as overtly evil as Musk’s MechaHitler or Zuck’s slave labour, but not far behind. Hao was a journalist with access to OpenAI’s insiders, and this book covers a lot that is rotten in that company’s culture.

I haven’t fully read this yet (hence not a qualified review), but I probably will — in the same way that one watches live reporting of disasters. You just can’t tear your gaze away from the train wreck.

Back in 2013, Altman posted about successful people build religions. When you view everything he’s doing in that light — as an evangelist who’s using religious fervor to scam followers — it makes a lot more sense. Everything he says (which is usually tied to who he’s talking to and how he’s trying to extract money from them), can be seen as building irrational fervor on unsubstantiated promised future, that’s always just around the corner.

Closing thoughts

Reader would know that I am always aiming for sound, common-sense product management, and try to apply the same to AI. That’s why I’m so allergic to hype, any hype, and why I try to bring back technology discussions to reality. It’s why I’m a proponent of responsible AI development, application, and integration into existing products.

Reading about some of the evils that are perpetrated on vulnerable populations world-wide is chilling. There are undeniable problems with how the technology has been built: I’m an author, and my books have been pirated and used to train LLMs — and even though I’ve been in the field for years I wasn’t aware of the extent of practical slave labour employed.

Tangent: I do so wish more technologists would learn humanities. It doesn’t take much to learn how history unfolds, or how rhetoric is about logic — not in the mathematical sense, but in making arguments — and spotting all the fallacies, fancies, and unsubstantiated claims hypers make. Some critical thinking would go a long way to see why Altman’s “solve physics!” is nonsense, see how and why Marc Andreessen calls on fascist arguments, and why Ethan Mollick should have watched Disney’s Fantasia as a child, and experienced the difference between art criticism and making art. You don’t have to read Cicero in the Latin to ask cui bono every time someone speaks.

And yet, I do think (hope) that once the bubble bursts and some of the worst offenders collapse, we’ll see a rise in ethically sourced and trained, small-and-efficient models, that are used for what they are good for. We could build knowledge (i.e. human understanding, not a collection of facts) and tools that serve us, and do so in ways we control and understand rather than implicitly trust their output. We could even start exploring other ‘AI’ avenues than just ‘throw more money at it.’

While the above books may come across as ‘AI is just fundamentally bad’ — and there is weight to their arguments when applied to current state — I do think that LLMs can have useful applications around NLP (Natural Language Processing) and coding (automating boilerplate). They are one tool in many, which can be useful if the cost is commensurate with the benefit (not burning the environment and economy), and we use it like other tools and not abdicate the responsibilities and expertise required to wield them. But it takes critical thinking, rigour, and resolve to limit these technologies to what they can do, and not fall into the traps of empty promises. Humanity, as it has demonstrated time and again, learns the hard way and doesn’t always draw conclusions. I guess we’ll just have to keep fighting hype and focus on appropriate use.

Note that out of the three current main use-cases for using LLMs — personal use, enterprise automation, and embedding in products — this post centres on the latter. This is about how to use the fuzzy and unpredictable output, and compensate for the times it occasionally goes wrong in the face of end-users.

What does cheese have to do with it? Read on…

When to Loop Humans

Human-centricity is important in any application, and knowing when an how to involve humans in the process is critical. I’ll give you two conflicting examples, to show the range and design thinking needed.

First product I’ve launched was around data leak prevention, attempting to stop people from accidentally disclosing sensitive information. The first version was heuristic and tended to warn users too often, so people got used to clicking OK. It was better to do some natural language processing, which wasn’t perfect but when it did pop up users paid more attention. That’s an example where imperfect prediction is still preferable overall.

Another product had to do with data retention: identifying sensitive info that should be disposed (y’know, before the hackers steal it). Problem was that the founder was a big believer in using AI for decision support without letting it make final calls. That left the information reviewers with lists of tens of thousands of entries… Which they had to manually review. It’s hard enough to explain to archivists that data past its retention schedule is a liability and needs to be disposed; giving them a tool that didn’t help was useless.

It should be painfully obvious that the key is knowing your users and understanding the context they operate with. Pretty much Product Management 101, regardless of the fact that we are dealing with non-deterministic, fuzzy systems.

So how can you build a product around this? What forces shape your decisions, and what options might you deploy?

• Consider the impact on human lives when it goes wrong

  • Always, always start with that. Some times a human must review and approve everything, other times it’s better to handle by exception.

• Consider the scale of interactions, and design a UI to support that

  • There’s a bias where people accept the ‘verdict’ of any algorithm as unbiased and correct, despite it not being the case. They also get used to clicking ‘OK,’ especially when it’s not their main task.

• Think about second-order effects

  • Unintended consequences of your design choices. It’s very hard to imagine, but tagging some QA and Security experts and doing observations on user interactions can help.

• Think about the tools

  • “AI” is a broad umbrella term, more marketing than any specific technology. As we know, a lot of it was built on plagiarism and slave labour, and it encodes so horrible Western-white-male biases. This about how can you use the tool in ways that do not cause harm to vulnerable people and the environment.

• Design escalation paths to a human

  • When does the human need to be involved? How do affected parties (including the direct user!) object to the output? Who monitors and handles those objections? Governance isn’t a policy document in SharePoint, it’s an ingrained practice.

Security by Design

I wrote before in general about securing generative AI, which — spoiler — comes down to putting your black hat on, thinking of all the possible second-order effects, and restricting inputs, outputs, and undesirable behaviour. A lot of the problems arise from this being new (so new attack vectors), the companies behind it prioritising addiction to win market share, and humans being humans (both gullible and misanthropic). That original article stressed exploring all the environmental, social, and governance factors you need to understand and build towards, and applying sound user-centric, human-centric really, product management principles.

In contrast, I now want to highlight how to practically embed an LLM in your product, using the principle of Refrain, Restrict, and Trap. This was first put forward like that in a research article about protecting from prompt injection. This comes down to choosing the use-case carefully, operating on least-privilege principles, and carefully controlling the inputs and outputs to models.

Keen observers would note that this is pretty much opposite what the industry is trying to push you to do. From OpenAI’s agents, to Perplexity’s agentic browser, to how MCP has been built by children. You can either let an unpredictable device run amok with your money, or you can have security — but not both.

Anyway, back to RRT. While the above article provided guidelines, here’s an example implementation from a PMs perspective. We built an answer engine as part of our search capability in the DXP (Digital Experience Platform; think websites on steroids). Our customers are governments, higher education, and highly-regulated industries — but they do like to keep up with the times. Offering a conversational search function, as natural language responses to searches, can help their audience, so long as it’s accurate and safe. It can be really helpful to use a technology that can match existing content to questions and provide direct answers (so users don’t need to click through a list of blue links).

Here’s one basic idea of how we ensure this safety and accuracy:

• Refrain
  The most basic thing is to understand when the technology is suitable and when it isn’t. A law firm can use it to help find partners who specialise in a particular area, but will never use it for legal advice. A university can use it to help prospective students navigate campus life, but not financial aid.

• Restrict
  The solution (which is much more than simple RAG), only has access to information about the selected topics. It doesn’t have access to backend systems. Input is limited (most jailbreaking techniques require a fairly long inputs) and filtered for allowed / disallowed topics.

• Trap
  The user’s query (input) is converted to a question, to help to trap anything untoward. The LLMs output is then checked for groundedness in and faithfulness to the source material, as well as compliance with other instructions. Only then is it passed back to the user.

This is just one example of a case where fuzzy output (being able to tailor the info to the specific question, especially in under-served non-critical areas) is useful, but how you still need to control for undesirable behaviour. We put a lot of thought into this, building the checks (Traps) into both input and output to ensure that the responses are accurate and do not cause harm.

In technical terms (see the marketing terms below 😉), these are user input sanitisation and an AI eval system which includes both deterministic checks and judge LLMs. Plenty of articles about it (eg this recent one from Lenny). I’d also suggest you visit OSWAP’s page on LLM risks, which covers many common pitfalls and mitigation techniques. I’ll do you, and your engineers, a world of good to keep them in mind when designing AI-backed solutions.

… and Cheese

I promised at the top that this has to do with cheese. So here’s a mental model you won’t forget in a hurry: think of LLMs as ripe Gorgonzola.

It’s fuzzy

Just like a good cheese has some fuzzy mould around (and inside) it, so do LLMs (and neural nets in general) have an in-built fuzziness of output. All these probabilistic models means the output isn’t sharply defined, but has a nice fuzz about it.

Some people love furry cheeses. Some people don’t. Some use-cases can thrive with this. Others can’t stand it.

It can be smelly

If you don’t pay attention to the fuzz, you can end up with something nasty and poisonous. Also, some people just don’t like it — don’t shove it down their throat. Keep tight controls and governance, and pull your users in rather than pushing it on them. Otherwise things will have the shape of that accompanying pear.

It’s a Marketer’s Dream

Our marketers named the approach above (using judging LLMs to cross-check inputs and and outputs) as “Guardian Agents.” And that’s just… cheesy 😜

But it does lend well with customers, and allows us to explain the methodology without getting too technical. True agency is years away, but using a combination of heuristic and probabilistic checks we can build a workflow that seems to be able to execute a specific task with fuzzy input and output that is nonetheless useful. As long as we can deliver on the promise of ‘guardian agents’ — and we do, by guiding our users and not over-promising — then there’s nothing wrong with a bit of cheese on occasion.

Too much isn’t good for you

Too much cheese and your doctor will start getting concerned about your cholesterol. Too much LLMs, and your audience and the environment won’t appreciate it. We make sure we use the smallest, most ethical models we could that can be controlled to generate the right result, in order to minimise the harm. Since ‘ethical models’ at this time is a bit of choosing the least-worse option, we’re poised to replace them when better options are available (i.e. no vendor lock-in). In the mean time, we minimise environmental and social harms as much as we can, and only use the functionality advisedly when the benefits outweigh the costs.

Hope you found this useful! Getting used to working with non-deterministic systems takes a bit of adjustment. Getting excited with a new hammer and trying to see what you can bash isn’t a good tactic, but using it when appropriate can be a wonderful boost.

Where and how have you seen good integrations of fuzzy logic like LLMs into products?

So if you find yourself in need of someone to talk to — whether it’s a specific problem in product management or general career advice, whether it’s breaking into Product or restructuring organisations — just reach out. Chances are I’ve seen this before, and a good conversation can get you unstuck and going.

Just to be clear, this is my way of paying it forward. I’m not building a coaching/mentoring business, I just like to help others. If we can build better practices together, and better products, everyone wins.

Best way to reach me is via a message on LinkedIn. We’ll find the time, and take it from there.

This isn’t financial advice, just a set of observations about lessons from history (and historians are only slightly better than economists in predicting the future). Neither is this anti-AI, just about the repercussions of the gap between the hype and underlying capabilities. I’ll hopefully leave you with a sober, practical approach to the nebulous future of ambiguous “AI”.

First, a definition

To talk about bubbles and their progression, it helps if we have a short definition — or at least description — of what asset bubbles are.

Bubbles happen when the valuation of an asset goes well above its intrinsic value. This sharp growth is fuelled by speculation and eventually collapses, leading to a ‘correction’ that may be localised or economy-wide, but either way wipes most speculators’ (especially late entrants’) investments.

There are broadly speaking two kinds, either a purely financial bubble or asset backed. Purely financial would be the stock market crash of 1929 or the GFC of 2008, and asset-backed relates to something physical. That something could have an intrinsic value (like railways) or not (like tulip bulbs or NFTs). Where the bubble deals with long-lived useful assets (railways in the 19th century or the fibre infrastructure around the dot-com bubble), they are sometimes called ‘useful bubbles’ as they enable future generations to make use of those ‘left over’ assets.

Some notable bubbles:

• The Dutch Tulip Mania (1634-1637)

• South Sea Bubble (1711-1720) — the first time the term ‘bubble’ was used

• UK Railroad Bubble (1840s)

• Wall Street Crash (1929)

• Black Monday (1987)

• Dot-com Bubble (2000)

• Global Financial (Credit) Crisis (2007-2008)

Bubbling up

Bubbles are characterised by a cycle of boom & bust, which sometimes is followed by a recession if the bubble affected the whole economy.

The fundamental cause for the boom is human psychology. We have a plethora of biases — greed, tendency for optimism, FOMO, deference for authority, recency and availability biases — that encourage speculation. Once assets valuations lift, there’s the Greater Fool Theory in action: I know it’s hyped, but it’s going up so I’ll be able to sell it at a profit to an even bigger fool. (This is basically the Venture Capital business model around exits 😜).

At some points, the asset price reaches such levels that the market (crowd) stops playing and everything collapses. Whether or not a recession follows depends on how localised the asset is vs how big of a spread it was in the community, and the size of it compared to the economy in general. (eg You probably haven’t heard of the Uranium Bubble of 2005-2007, because it didn’t have much effect outside of commodities trading).

Either way, it will be often followed by a regulatory response to prevent the same horse for bolting the stable, and a fiscal response in an attempt to restore the economy to balance. Those responses, especially if a recession occurs, almost invariably include the seeds of the next bubble: any stimulus now will encourage speculation later.

Bubble dynamics

In more detail, this is how economic markets and the socio-political balance around them behave around bubbles:

This slightly messier diagram shows the cyclical nature (feedback) of some aspects, as well as the other factors that may spark a boom:

• Technological innovation like the telegraph may increase speed of contagion and deepen information asymmetry. Other technological innovations (railways or internet) may cause a “new economy” thinking, the belief that this time it’s different.

• Financial innovation is linked to more sophisticated vehicles that speculators can employ, which usually relates to increasing leverage and spreading (and hiding) risks. From Joint Stock companies in 19th century to the CDO’s of the Global Financial Crisis, these tools allow new ways, often less regulated, to concentrate money.

The “rich get richer” mechanics are a mathematical certainty. Even if you start with a fully equal and balanced economy, slight variations cause concentration of wealth. This has been demonstrated by many models and research papers. The effect of wealth concentration is that those who already posses wealth are more prone for speculation — their assets are usually diversified, and they can afford to take the risk and survive if it doesn’t pay off. Since the risk isn’t spread evenly and the ‘heavy’ speculators are often early entrants, when the bubble collapses wealth tends to accumulate further, entrenching the divide. This eventually leads to political capture, which I’ll touch on below.

The AI Landscape

This is what I’m seeing in the 'AI ‘market’ these days (i.e. the economic, social, and political happenings around the bubble):

The stimulus of zero-interest following the GFC encouraged venture capital to take a high degree of risk and speculate wildly, and zero interest means high leverage. Add to that the ‘growth at all costs’ mentality of both established companies (miss your quarterly earning targets and get punished) and Silicon Valley startup culture, and you get intense risk taking.

“AI” — the umbrella terms that captures Generative AI, LLMs, and neural networks at large — is an interesting technology. As such there’s a lot of “this time it’s different” talk, a new economy that encourages people not to learn from history. Remember that societies absorb innovations at the speed of people, not of inventions.

The technology itself has a lot of uses, but there are fundamental limitations:

• Neural networks are great at predicting/generating what they’ve been trained on, but not great outside of that sample range (read ’ Substack and original papers, or — see my book review).

• They are fundamentally a probabilistic processes, the output of which isn’t predictable. The fact that we see some emergent properties is amazing, but as I mentioned when talking on the Twin Gaps, just because it can do something doesn’t mean it can do so reliably enough for many applications.

Compare those fundamental properties to the talk of ‘industry titans’ around “PhD level experts on any topic! Replacing all jobs! The Singularity!” and you can see several of the above-mentioned biases at work. The people who stand to win the most (and have accumulated enough wealth to afford wild speculation) encourage others (Greater Fools), who in turn rely on them (deference to authority), and fear missing out. Add the media buzz (they need catchy headlines and, as people, have own biases) for recency and availability of information acting as confirmation bias, and well, you get a hyper-hype narrative.

This narrative drives asset prices that are far above the underlying value of the assets. AI companies are losing billions of dollars, which has to stop at some point soon. The froth around the industry is impacting the whole US economy (side note: ’s column is great at exploring all the dark sides of the AI industry).

As a result, we have an over-valuation of AI companies. Whether it’s foundation model providers who burn billions without a sustainable business model, or the over-investment in data centres in anticipation that future usage will justify it, these are speculations. If model providers and hyper-scalers were to be profitable, presumably by charging the true costs of doing business, would the demand match up? Or would a technology that is “kinda good enough” at a low price point would suddenly be far less appealing?

Depressing thoughts

Things don’t necessarily end up with a recession… it can get worse.

If you read Paul Kedrosky’s Honey, AI Capex is Eating the Economy, you would have noted the alarming size of the bubble. The over-investment in data centres has surpassed the dot-com bubble in terms of GDP portion. That is actually masking some fundamental weakness in the US economy (which despite the tendency to shoot the messenger, is still public data in terms of jobs, growth, tariff debacles, etc.) We’ve also all seen weakened political institutions following DOGE and other gutting of federal agencies.

On the sidelines, notice also how the tech oligarchy is aligning with the political right, and reaching political capture. That is the point where wealth is concentrated to such a degree that the government serves the interest of the rich, rather than the populace at large. You can read about the Gini Coefficient if you really like math and economics, but as a simplification: when the 1% of populace holds 35% of the wealth, you have political capture to that degree. In the USA today, this is around 31%, with UK and Australia around 15-20%. Methodology matters in calculating this, but in broad terms, 35% is when the social tension starts reaching breaking point. What happens when people, stressed with job losses, climate change (aggravated by power-hungry data-centers), wars, suffering alienation due to social media suddenly face a collapsing economy with a clear ‘at fault’ group? Blaming Big Tech for cynically releasing addictive and extractive application and and the governments that enables them (as well as other social issues) could get conflated.

Recessions require government spending (something the current administration is averse to), and firm, thoughtful handling to contain them softly. Considering the size of the AI bubble, it won’t take much mismanagement to make the recession into a depression. More than the economy, a softer landing and recovery also require rewriting the social contract. How much do you trust the current world governments with thoughtful reforms and spending?

To be fair, this is an extreme scenario.

My estimate is that while a recession, probably a big one, is very likely, the factors that would lead to a depression are far more ambiguous, and the further collapse into civil unrest is anything but guaranteed. Things have to be a lot worse to mobilise the middle class, judging from historical precedents. There is too much socio-political complexity around this to make any prediction. Most notably, we are dealing with hyper-complex interconnected systems — it’s never the One Thing™️ that leads to collapse. We often describe revolutions as an inexorable force, something that was bound to happen. But when you look at the details, they are often a series of unlikely events that were anything but certain at the time. For those reasons, as I said above, historians, while able to sometime identify themes and forces at play, aren’t great at predicting the future.

Note that this is quite separate from further wealth accumulation, political capture, and possible transition to authoritarianism. While the first two are almost guaranteed in the wake of a recession, the latter is a just possibility given current tensions. Rewriting social contracts can take many forms, and authoritarianism is just one possibility.

The cyclical nature of forms of rule has been noticed by Polybius two millennia ago (best explained with an image). The basic idea is that each form of government rises in response to a problem, but contains within itself the seeds of its own collapse. While that's happening there are winners and losers, and we know from modelling that ‘Rich get richer’ is mathematically guaranteed, which leads to the rise of oligarchs and political capture. But the specifics of what happens and in what ways is determined by many factors, from the reaction of the oligarchy to change, to the strength of institutions, to external factors like climate and wars. Redistribution of wealth and power doesn’t have to be French/Russian Revolution-style violent. For example, the UK elites lost some power throughout the 19th and early 20th century without social collapse, which enabled them to keep wealth and power in the long run. Then again, the question of Ireland and Home Rule just before WW1 was on the brink of revolt (see the Easter Rising), and land ownership was a large issue. So one can say Britain avoided civil war only due to an external war, and that redistribution does require violence of some sort.

The outcomes of the coming recession will depend on the political power-plays within the USA, global contagion, and feedback loops. In some respects, the current isolationist movement of the US and socio-economic upheaval that will make the recession worse, make it less appealing to foreign investors; thus dismantling the global trade status quo may reduce or slow the contagion to the rest of the world.

So an economic depression and full social collapse is a worry, but perhaps not a stay-up-at-night worry. We’ll have to see how it plays out. Like the title says, Fun Times Ahead.

Counter arguments

On the subject of it may not be that bad, here are some counter arguments.

• There are some real productivity gains!
  True, but not conclusive. For some people in some cases, AI helps. As an industry, the jury is still out. There are big differences in using AI tools personally, as an enterprise helper integrated with other systems, or embedded in a product. Much of the evidence is anecdotal, and research (both ways) doesn’t always differentiate which use-cases were examined.

• AI Startups see measurable revenue growth!
  True, but that is not the same as profits. Most still lose money on top of the revenue. Unit economics will change drastically with underlying changes to costs of foundation models — just ask Cursor about Anthropic raising their prices.

• It’s a strategic necessity!
  Maybe. Depends on how useful it actually turns out to be, once the “be the first to AGI” race cools down. I deal with speculative fiction enough in my other life, here I deal with concrete evidence. So far, due to the underlying limitations mentioned above, I think LLMs/VLMs will be a component but not the center of future AI innovations.

• The investment now will build the future!
  Unlike railways or fiber networks, GPUs have a drastically shorter usable lifespan. The long-term utility of the large data centers is TBD.

• It’s only the tech sector!
  Correct, but to a degree that is currently propping up the US economy. The top 7 tech companies take up 35% of the stock market (not the same as the economy, but still), and the investment in data-centres is a noticeable percentage of GDP that’s larger than the dot-com. A bubble collapse will spill over to the rest of the economy (via job loss, consumer confidence, etc).

• It’s necessary to stay competitive!
  Maybe. That depends on the ROI of integrating AI, which covers both the costs of doing so and the market willingness to pay for it. There is a difference between robust process automation and possible backlash against public-facing features. Remember that automating tedious, repetitive tasks is great for the bottom line, but those tedious tasks were never the core business and their effect on the broader health of a company are usually smaller.

What I see is mostly a massive, speculative investment on something that is yet to prove itself. While a non-deterministic generator has some valuable use-cases (heck, I’ve built and launched several successful products with it!), the stats on enterprise successful adoptions are still emerging and it remains to be seen what the economies would be. All I know is that companies that burn billions more than they make can’t last long.

In a perverse possibility, the collapse of the big brand names may leave a vacuum for good-enough-but-cheap models. Something low-cost and efficient like DeepSeek, hopefully trained on ethically sourced data (to avoid those pesky legal copyright issues and public backlash), that can be run on discounted hardware (aging GPUs). That, I think, will be the Plateau of Productivity for the current Generative AI buzz, when the industry will stop being obsessed with burning money (and the planet) and start exploring other avenues for advances in the field.

What can we do?

Forget about changing the course of history. Also, again, don’t take this as financial advice. But as product managers we can subtly influence our products and companies to be better positioned to survive the coming recession.

First, you must understand the Good, Bad, and Ugly of AI. You need to understand what it’s good for and what it’s not, and how to work with non-deterministic technology. This requires exploration and development of new ways of doing things, but a lot of good product management & engineering practices hold true.

Once you understand the capabilities and limitations of the tool, you can find cases where fuzzy output is preferable or tolerable. That is, cases where the benefits in being able to generate language or predictions outweigh the associated in-built randomness and inaccuracies. That very much depends on whether you use it personally, embedding it in a product, or are building an in-house tool. Once you have that, you can consider the downstream impact on humans when (not if) the fuzzy output is wrong, and how to account for it in your workflows. You certainly can engineer around this, to make robust solutions both technically (like Refrain, Restrict, Trap), and socially (with human in the loop and escalation paths).

If it’s powering features in your product, you need to find ways of integrating it in ways that will be palatable to the consumer, and that will be so even if the costs suddenly go up 10x-100x. It’s harder to limit your costs in those cases, unlike the other two scenarios (personal use or in-house tool), because increased consumer usage means directly increased costs. Rather than betting the farm on it, consider your product and business fundamentals: what problems are you solving for whom, that are valuable enough to them? In what ways could this exciting-but-limited tech can you serve them better? Is it cost-cutting at the back-end, or a shiny-and-highly-useful thing that wasn’t possible before? How much will consumers be willing to pay for it?

However you use AI tools and under whichever scenario, just being prepared for non-deterministic output, having a plan B for costs blow-out, and considering both best and worst case scenarios in planning, would position you well to survive the recession any further fall-out.

Hope you didn’t find this post too depressing. This isn’t the first bubble I’ve seen play out in person, and I’ve studies more for the love of history. It will be rough times with inexorable progression, but that’s just life. With a bit of consideration, you can position yourself to survive, maybe even thrive, in the coming upheaval.

Below are my impressions of the conference, and some notes of the talk I gave together with Cam Ross — all about the different demands of product leadership, and how to tell if it’s for you — which I hope you’ll find enlightening.

Short version: worth coming to Melbourne for the conference, and product leadership is all about the people 😜

Product Camp is an unconference. This means that, apart from the keynote, attendees have 30 seconds to pitch their talks, votes are collected (physical dot voting), and then whomever get selected fills up the later sessions.

It is run completely by the community and for the community, with volunteers to organise and sponsors to pay for location, coffee, food, coffee, prizes, tea, and after-party. Pretty amazing achievement for a community-run event.

Actually, that’s part of the charm. A lot more human-centric than the slicker, corporate-run conferences. My deep thanks and admiration to Liz & Jen from Product Anonymous, to Steve Bauer the MC, and the hordes of cheerful volunteers. I’m grateful for the chance of connecting in person to people I only knew as talking heads on zoom (Steve, Lucy, Marco), and for all the new Product friends I made (Maxy, Jana, Nandhini, Jack, William, Yau, Pavi, Vijay… I’m never sure if it’s better to name people risking forgetting a few — please don’t be offended!)

Anyway, Cam and my talk about “Who’s insane enough to want to be a CPO?” got voted in. This was no-slides, just “telling it like it is” and sharing our scars from HoP / executive roles. We both had them, and went back to IC roles.

Some key-takeaways if you’re curious:

• It's all about the people.

  • "Politics" is the role..

  • Navigating people and egos at a whole different level than you’re used to, which simultaneously being more removed. You’ll get to understand why executives seem to have ADHD.

  • You need to be sure you can work and get along with the specific people you’ll be surrounded with. Don’t ignore red flags (including gut feelings!) about the people when interviewing.

  • You’ll need external support, you won’t have a ‘home team’ of peers as such. Find a mentor or executive support group.

• You will be operating at different scales

  • Building a product and team (PM) > building an organisation and portfolio (director) > building a market (CPO).

  • It sounds exciting, but…

    • The skill that got you here, won’t get you there (even if they sound the same). It’s actually much less “Product” work.

    • It’s nothing like the books. Being a mentor and being a line manager are very different, and despite your best intentions, you’ll be doing more line management than mentoring.

• It's not the only path

  • Consider ahead of time if this is the right path for you (don't chase title or pay). Consider why you want a leadership role: don’t assume you’ll automatically be able to do more (people won’t “just” listen), and think about what you’re giving up (team, harder to find next role).

  • Product skills are transferable to other fields, leadership isn’t the only path.

As for the obvious question: will we take another executive role? In both cases that answer was: yes — but it has to be absolutely the right thing. We won’t take it just because we feel it’s the thing to do, we’ll take it because we believe we can work with the people and make an impact.

Both Cam and I are open to chat with anyone and help them think through career choices. Just reach out to Cam or me on LinkedIn.

Top-left: Marco Tebaldi (a conference volunteer and fellow Squizzer), Cam, and I; Top-centre: with Jack Rust, being “interviewed”; Top-right: with the amazing Lucy Spence; Bottom: Cam and I talking about the realities of leadership/executive roles.

Speaking of Lucy, her talk — Defence and Stakeholders and other Vampire Hunters — was pretty amazing. It ended up being a very funny, useful, and included very candid talk about all the “hard won” lessons (read: career limiting mistakes) in stakeholder management. She also absolutely put everyone else to shame with her hand-drawn slides. I suggest you watch her recorded trial-run here.

The other talks I attended:

• Keynote my Dr Maria Camacho about design thinking. Nice to hear people talk about it, and even though I know the subject getting fresh perspectives and new anecdotes is always welcome.

• Lego Serious Play: an interesting technique to elicit thinking with tactile method from a group. It’s essentially going around conscious ‘deep’ analysis, and into story-telling. Great in person when you have big questions (and lots of Legos 😁).

• Exploratory User Interviews, with Jack Rust. If you ever worked with me you know I love doing them, and I write often about the topic (as well as how to be subversively helpful to get access to customers). So when Jack asked for volunteers I tentatively put my hand up… and found myself on stage again. It was fun being the interviewee for a change! (And I also got to talk a bit about AI). It was a live ‘demo’ that Jack used to highlight techniques and principles about exploratory interviews. If you weren’t there, I always recommend Talking to Humans by Giff Constable.

A talk I would have loved to hear but missed, was Product Architecture by . Luckily you can find the article behind it here on Substack.

In short, it was an amazing conference, and I hold deep gratitude and awe to Liz and Jen from Product Anonymous and all the volunteers who organised everything. I’m hoping to make it to Melbourne next year as well!

(And if you can’t wait to meet in person and need so career or product management advice — just reach out! I’m always up for a chat 😊)

We were particularly interested in the impacts AI has on the experience of people with accessibility issues across digital systems, but the discussion went wider than that. Below are some of my thoughts of the subject, in the hope that it helps raise awareness for this important subject.

Digital Accessibility

Both Neil and I have been around to see the rise of the PC and Internet, mobile phones, and now AI, so this isn’t the first technology revolution — or hype cycle — we’ve seen play out. While the pace is increasing, it’s interesting to compare how each wave has transformed the experiences of people with disabilities and accessibility issues.

I’ve asked Neil about the good and the bad in AI. Much of his interactions with AI are via a mobile device, and he is using dedicated apps as well as ones from all the major AI providers. Some of the wonderful developments is being able to point a phone’s camera and get a verbal description of a scene. He recounted how he can now “look” at pictures back from his childhood or his wedding, and have the verbal descriptions revive and relive those pleasant memories. It is something that we take for granted, so take a moment to appreciate being able to ‘see’ mementos this way. Other apps help with daily lives, like checking whether an outfit ‘goes together’ and isn’t a fashion disaster.

Of course, AI is prone to hallucinations. While I could certainly use that fashion advice myself (assuming the AI may be correct more often), I recall when we had a friend who was temporarily blind. She used a similar app which, when describing my wife, decided to very authoritatively add two years to her ago. Hilarity ensued. It can also be more problematic, as when it described to Neil an object being on the left, when in fact it was on the right. (And in a typical fashion, pointing the error to the AI made it argue back like a toddler). Luckily it was in an environment he was familiar with, and could catch the error without harm.

These kind of applications are a good case where even having something that isn’t super-accurate is an improvement. One still needs to verify the output, but it’s a marked improvement in the quality of life.

Digital Divide

Neil is technically savvy, understands how to use the tools, and can work with them. That is not always the case. Digital literacy, financial resources, and other similar aspects beyond physical disabilities, can greatly affect people’s access and derived value from technology. For those vulnerable populations, increased reliance on technology deepens the divide between those who can and those who struggle.

Just think about trying to contact a utility’s call centre, and being constantly redirected to do your business on the web. It’s not helpful when the website isn’t accessible, or when an automated agent gives you wrong advice and service. Or think of the subscription fees all those apps and services require. The people who need them the most, those with disabilities, are often in lower income brackets due to struggles with full time employment. Those new apps and associated fees may keep them out of reach of the people who would benefit the most, which again exacerbates the situation.

Automated Accessibility

So you want to build an accessible product or service. How do you do it?

A big problem with many automated accessibility checkers is that they only catch the 30-40% of errors that can be detected automatically. False positives and negatives also abound, for example an image with the alt-text of “alt text goes here” would pass the check, while other issues may be triggered which are fine. There is no replacement for having a human who understands the experience review it.

Adding AI into the mix has its own sets of issues. Asking an AI coding agent to build a site that is accessible is like asking it to build a site that is secure; it’ll happily insist it did so, regardless of the actual code. And AI-based interfaces themselves have issues, from the direct accessibility to screen-readers to the content itself that may be harder to verify and increase the digital divide.

That said, there are certainly cases where AI technologies can be used to enhance accessibility. This is a topic we at Squiz are passionate about, as part of our overall mission of improving digital experiences to enhance off-line lives. AI tools can find unfit alt-text and help correct it. They can help authors “translate” the text written by experts and approved by lawyers into something most humans can understand.

The trick, as always, is to understand that AI is just another tool. Just like the automated site-scanners aren’t sufficient to ensure accessibility, neither is AI a magic solution. You need to understand the capabilities of the technology (the jagged edge), it’s capacity to solve a particular issue, and keep a human expert in the loop to direct it.

The Future

Both Neil and I seem to have misplaced our crystal balls 😜

That said, even though these are times of great change and a dawn of a new type of general-purpose technology, there are some observations we can make from history that will apply to AI as well:

• Humans tend to over-estimate the short-term effects and under-estimate the long term effects.

  • The AI hype cycle will crash at some point, due to capability and adoption gaps. The long-term effects will likely be something surprising, not what we may predict now. Social media, for example, didn’t mean the rise of world-wide connected democracies, and we’re only now realising the psychological harms of automated content.

• Magic hammers and silver bullets are the stuff of fairy-tales, not reality.

  • AI is just another tool, which needs to be wielded by experts in their field who know what they’re doing, directing and verifying the output.

• Accessibility isn’t a policy in SharePoint; it’s the lived experience, and the difference between good and bad has huge impacts on whole communities.

  • You can — and should — absolutely start today, and build it into your workflow. Accessibility is not a perfect state, but constantly evolving thing you need to work on

If you’d like to watch the full webinar or read a more detailed report of key insights, you can do so here: https://www.squiz.net/blog/what-organizations-can-learn-from-real-experiences-with-ai-and-accessibility

That’s it for now. This is another view on Responsible AI, which I hope you find useful. It is something I always cared about, and why I love working for a company that built it into its ethos. It goes to show that while there is plenty of bad associated with AI (suggested reading: I Love Generative AI and Hate the Companies Building It), it can be used for good — it just takes some conscious effort.

Reach out if you’d like to talk about Responsible AI, about digital accessibility, or about building digital experiences that make a positive impact.

Some things don’t change, or at least change very slowly. Human nature is one of them, and this fundamentally affects technology and Product work. This should help you understand what you’re seeing in practice, and hopefully in talking about this with execs.

Invention, Innovation, and Diffusion

These are the three stages of technological progress. Problems usually arise when people underestimate the time to progress between them. Let’s start by defining them, and talk about why the gaps matter.

Invention

Invention is the development of something new. A new way to do things, a process that didn’t exist before. Think of steam engines or Bell’s transmission of audio signals over copper wires.

Innovation

Innovation is finding a new application of technology. This is a new way of doing things, but the focus is on a different application rather than inventing a completely new way. Thing of putting steam engines on wheels to move trains, or codifying visual images over audio signals to create fax machines that operate on telephone wires.

Diffusion

Diffusion is the adoption of the innovation across society. Building up a network of railways takes time; the initial effort is to move people and goods between two terminal points, and as that diffuses we see whole changes urban landscape and the creation of commuter societies. The value of faxes is often used as an example of network effects, where the more people have one, the more useful it is.

A key observation is that few things are “true” inventions. Nothing exists in vacuum, and we build on what came before. It also takes a lot of tinkering to make something useful, which leads us to the next section.

The Twin Gaps

When you move between the above stages of technical development and adoption, you’ll encounter two main gaps that you’ll need to cross.

The Capability Gap

Just because a machine can do something, doesn’t mean it can do it well. Obviously “doing well” is context dependent, and will mean different things to different people. The invention of movable type together with the printing press improved over the use of wood-block printing. There were still a lot of improvements along the centuries — small inventions and innovations — until we could see something like affordable newspapers and mass-market paperbacks.

The Adoption Gap

This is directly related to the definition of “well.” The first cars required maintenance relying on mechanical knowledge, and not useful to the average person, even those might have afforded the car itself. Decrease in direct price, increases in reliability, and the availability of support services (mechanics and gas stations) slowly tipped the balance. It still took decades to reconfigure cities for the changed patterns of transport.

Related concepts

In tech circles, “crossing the chasm” is an old concept that speaks about the difficulty most companies run into when they saturated the enthusiasts and early adopter crowd, and are now trying to reach the early majority. This is when companies used to progressing at the speed of technology, suddenly hit a wall when their customer base is progressing at the speed of people. I have yet to meet the executive that likes to hear “take a deep breath and slow down,” even if that is what’s needed.

Today’s AI Gaps

Much of the AI industry today is in these early stages. The various inventions and innovations which led to ChatGPT’s launch in late 2022 took a while while to build up. But the public at large is still facing those two gaps, and will continue to for a while due to the very nature of AI as probabilistic technology.

First, is how well do AI systems handle cases. I’ve written before about that sense of magic when you first ask a model something and get a coherent, cogent reply, or see it figure out an answer after ‘thinking’ about the inputs. But then you scratch the surface, and it’s often superficial (or outright wrong).

Which begs the question, of what use is a technology that’s only 80% correct? Or 95%? or even 99.9%?

The use is in the use-case. For some applications, like suggesting alternative marketing slogans, getting it right four out of five times is good enough. For others, like giving official tax advice, even one-in-a-thousand wrong answer is unacceptable. That’s why when I work with clients (often government agencies or regulated industries), I always start with the impact on someone’s life — what happens when, not if, the technology goes wrong?

Past the gap in capabilities, lies the adoption gap. Just because the technology might make less mistakes than a human — is it accountable? Does society accept this? When it’s your life on the line, how much will you trust the AI lawyer or doctor — or car mechanic, for that matter — as primary care giver?

Despite the efforts of billionaire CEOs to ramp up their stock prices by over-promising capabilities, society will move in a different pace, commensurate with the capabilities and humanity’s capacity to adapt. The intervening period will be… interesting.

Final note

This article is looking at AI from a product manager and an historian’s perspective about the forces of innovation and the diffusion of technology. There are, of course, many other considerations that need to be applied to AI systems, like the environmental costs, legal and moral issues, societal impact in its application, lack of governance, etc.

While I fervently wish these would be addressed sooner, the current political climate is mirroring other trends in history, both broadly and specifically around AI, and makes it unlikely. It seems like affairs will progress as previous such socio-technical revolutions did. But that is a subject for another time.

For now, there are steps you can take to better plan for closing those gaps with your products (whether AI or not). These revolve on selecting the right technology for the use-case, and accounting for adoption. No matter the speed of technical advancement (real or imagined), the speed of diffusion is order of magnitude slower.

As always, good Product Management is always about the long game.

This post presents a framework to selecting good use-cases for AI in your daily work. This should apply both to how you use AI as well as how you build AI-powered features into your product.

You’re (probably) treating AI wrong

With all the hype coming out of Silicone Valley and all the new vibe-coding tools that you’re supposed to be using to automate your everything, one would think that AI is a silver bullet, a magic hammer, and a business moat all rolled into one.

Sorry sunshine, it ain’t.

There are two things that are core to any knowledge work (as opposed to physical work), which are creative and critical thinking. AI, despite the hype, is far from it — it’s a regurgitation of the probable average. You still need to be in charge of those aspects, directing creativity and doing the critical thinking work.

AI can be sometimes be used for automating other tasks that take away from your time doing creative work and critical thinking, and, y’know, talking to customers, and that’s great — but why were you spending so much time on low-value tasks to start with? There might be something else going on which needs to be solved first (like building an enlightened team that doesn’t need so much hand-holding).

So being good with AI to do your PM job is good, but only useful if you use it for the primary activity of a PM — communications. It’s great to build rapid prototypes or improve your writing (whether stylistically or as a thinking partner). It absolutely sucks when you use it to replace communications, by doing the thinking and writing for you. Creating beautifully formatted pages on pages of PRDs is only useful in organisations that think the value of PRDs is in having them, not in the research that went in to building them and the discussions around them to build shared understanding.

But what about your product? There it’s even worse. All this rapid prototyping and speedy coding means that if you can build it that fast, so can your competitors. If you had a bright idea and put it out, it won’t take much to replicate. And that’s beside the fact that the code produced is often buggy and insecure; after all, coding is only a small portion of what a developer does.

I’ll leave this as a tongue-in-cheek illustration of trying too many things, too fast:

Remember:

Doing all the things faster isn’t always a good idea. Automating tedium is great, automating critical and creative thinking — the things that build product taste — is not.

The value of technology and cost moats will erode, and you’re left with the human-centric distinct advantages — like networks for B2C or large-scale B2B, and relationships for Enterprise.

How to think about integrating AI

There are two kinds of solutions you can integrate into your product (or workflows, for that matter).

Data-centric

Let’s start with using proprietary data and building your own models. This will be very domain specific, but you should think about the general principles:

• Life impact

  • All machine learning is probabilistic, so it will go wrong at some point. When it does, what’s the worst case scenario? It’s one thing to make a bad song recommendation, it’s another to deny someone financial benefits.

• Legal and Moral rights

  • You should have legal rights to the data (ref what is currently going on with all the copyright lawsuits). But you should also have moral rights. Just because you quietly changed your T&C so that you can now legally train models on user data, doesn’t make it right — and your users will show their displeasure with their wallets.

• Human-centricity

  • The model should aid humans, not rob them of agency. It should be build on inclusive data sets. It should minimise discrimination and allow for the inherent randomness of human lives, rather than codify existing societal biases.

• Explainability

  • Applications should present transparent results, and any decision-making should be explainable and the inner logic interpretable.

  • When things go wrong, there should be a clear path to escalation to humans, who could review and revert any automated decisions.

• The usual

  • All of the above is on top of data privacy, regulatory compliance, cybersecurity, environmental and social impact, etc. The application, including all the hosted data, should be secure and robust, reliable in its performance, and assured to remain so with strict governance.

Yes, I know you can make more money by doing the opposite. That’s how ‘Big Tech™️’ has been running for years now. But in your case, you’ll likely face ruin by rushing to market with ill-conceived, biased models that over-promise and under-deliver, where the only reason no one can tell it’s so is because there’s no way to raise complaints or understand why it happened.

I happen to believe being humanity comes before profit. Even if you don’t, you might be surprised that it’s actually also better for business, at least in the long run.

General purpose

When it comes to integrating general-purpose models into your product (LLMs, multimedia generators, or the recent crop of multi-modal models), most of the above principles still apply. Think how your app or feature might impact human lives, and consider all the biases that went into training generative models.

But faced with the current hype around LLMs and the over-promising that comes out of Silicone Valley, this is usually where the ‘slap a chatbot on it’ crowd jumps on the AI train. Not sure what to do? Why not let the public do anything and everything by asking in plain English? Well, because it doesn’t, for a start, and because it’s often a shitty UX, for another.

The first time you experience the emergent properties of LLMs in ‘figuring out’ how to write an answer based on input text (RAG) or list the steps it needs to perform to achieve a task is pretty magical. But getting an AI to follow a set process (algorithm) for any meaningful length is not reliable at best, ‘research’ tools rely on finding existing data, and ‘reasoning’ models actually perform worse than regular LLMs on anything outside their training data. Can you really afford to rely on a technology that goes wrong often enough to disappoint users? Or that has been known to cause significant backlash even when it does work?

Side note: remember that those telling you to “build now because by the time you do the technology will be there,” have a financial interest in hooking you on it. I’d like to see that even approach sensibility before I trust these claims, because as far as I can tell it will require materially different architectures that current models.

On the other hand, a key observation is that virtual assistants in the form of a blank prompt (“Ask me anything! I can do everything!”) is a poor user experience. First, it will take a long time for the technology to mature sufficiently to be trusted. Second, users aren’t experts in prompt engineering or thinking up solutions.

Both these observations indicate that you will need to think up a more structured UX and build the UI accordingly. You can then delegate tasks to “agents” where they don’t actually have any agency — that part should be the responsible of the users. In that respect they can assist and speed up, by helping the user think (that shallow level “have you thought of these 10 considerations that might apply”) and do (execute specific tasks, that can be constrained, validated, and controlled for blast radius when they go wrong).

In Summary

You can do a lot of good with AI, if you pay attention to all the dangers and pitfalls, avoid the hype, and plan accordingly. Chatbots aren’t the best user interface, or even the most interesting use case. Doing things ‘faster’ isn’t a replacement for doing things well.

You should be playing with AI, exploring capabilities and limitations, but don’t think putting in “AI” (of whatever flavour”) is a replacement of actual understanding of your customers, their needs, and the business model of giving them a solution.

The basic premise (or “TL;DR” if you spent too much on the internet), is that I always found the discussion about ‘empowerment’ of product teams a bit weird. Instead, what I see as the basic ‘elbow grease’ of product management (distributing context) also serves really well on the critical path of building high performing teams (and keeping you sane).

Though my quibble with empowerment is solidly based on the type of products I worked on, I think the principle of enlightenment for teams works across all types (B2C, B2B, and enterprise), as well as all levels (as an IC PM as well as as a leader).

My quibble with Empowerment

This goes beyond the Marty Cagan book on the subject (see my opinions on that here). Perhaps it’s because I’ve always worked in high-touch enterprise software, and usually in small to to mid-sized companies, well outside of Silicon Valley. There were never infinite resources, and allowing teams to just ‘go think up and pursue something meaningful for yourself’ just doesn’t make much business sense, even if constrained to a particular area.

And yes, I know I take the piss out of empowerment in that description, and that it’s never meant to be that way. It’s all about massive 6-page press-releases (with 300 pages appendices) and OKR trees that hide the forest… 😜

Fine, fine, enough jokes. But the intersection of Users, Technology, and Business is crowded, and more often than not teams in enterprise software can’t just explore an avenue for themselves in isolation. That’s because the complexity of the product implementation at customer sites just doesn’t lend itself to quick experimentation, nor do they have the volume of data to support that obsession.

Most information collected is qualitative because all you have is feedback; cohort analysis on a 9-months sales cycle and 6 month implementation project is a joke, and there are too many interconnected factors tipping affairs into the complex domain. And of course, throughout all of this you need to align marketing, sales, support, legal, cybersec, and random stakeholders in trench coats — just to see something go out the door and into the hands of a customer, who then have to go through the same stakeholder-hoops in their organisation.

But even if you aren’t subject to that Byzantine world, I find that the basic principles of product management, at least as I see them, can carry a dev team quite a long way into understanding the customer and the business.

The Path to Enlightenment

As a step between “we’re gonna code what the ticket says as per acceptance criteria” and “we’re going on an adventure”, there are enlightened teams. By this I mean product teams (dev, PM, design, and whatever else makes sense to build the product), who understand the customer and the company well enough to pull together in the same direction.

That means that you, as a PM, won’t have to be constantly involved in every minute decision or discussion, and you, as a leader, can trust that the right thing is done the right way even in your absence. The team is “enlightened”, in the sense of understanding the business and customer needs, and knowing how they can address them.

Question is, how to build such teams?

My CPO recently circulated this around:

It’s my belief that as a product manager you deal with context, collecting it and ensuring it circulates throughout the company. (This is true for data-driven environments as well). And so the path to enlightened teams lies directly in sharing this context — whether user, buyer, business, or other — with your developers, designers, and stakeholders.

For that purpose, the best tool is over-sharing.

In practice, this takes the form of:

• Sharing the vision

  • If you want everyone to pull together in the same direction, they need to know where you’re heading.

  • Same goes with any data you may have about the course.

• Inviting developers to join customer calls.

  • You’ll be surprised at the interest, and even if they don’t participate during you’ll see the effects of empathy after.

• Sharing lessons learnt

  • In regular team meetings and in dedicated sessions. Any insight that you gained from talking to customers.

  • Recording, transcribing, and summarising customer sessions is useful, but bringing up that one anecdote at the right moment is priceless.

• Take longer to answer questions

  • When a developer asks a “quick question” don’t give them the answer, but the whole why behind it. Share that customer context!

That’s it for now! This is something I truly believe in, and have employed several times to much success. It’s about product teams that understand the strategic context of the business and have empathy to customer challenges. It’s not hard, just take dedicated over-sharing of information to align people.

And when it works, it’s magic! Suddenly you don’t have to drown in Jira, and have enough time for the Product Manager’s One Job™️.

For a bit of background, I work for Squiz, a prime DPX (Digital Experience Platform) provider. Think about it like a website on steroids, where you can build smart solutions for visitors (occupying the niche between static sites like WordPress and a fully-featured web-app, for service-based organisations and government agencies).

I spend much of my time guiding customers on how to build AI-powered DX for the future, from how to start with AI through navigating the organisational change it requires, to envisaging a future that’s more than “slap a chatbot on it and replace employees.”

In that light, I’ve written two posts on the company’s blog, which I’ll summarise here and link to in case you want to delve deeper. These are around avoiding pitfalls and implementing AI in your online DX layer. Note that since they are company posts, they have been copy-edited by our marketing department 😉

Is your website AI-ready? Avoid these 3 common pitfalls

This article deals with the pressure to “do something AI” that many of our customers face, while at the same time they’re under the constraint of not damaging the brand. For most of our customers, the implications of AI going off-script are dire. This isn’t just a bad product recommendation, it could negatively impact people lives.

To that order, we show how implementing AI depends on more than just technology, and how you go through maturity levels: Automation, Adaptation, and Action. These are navigated as both the organisation and AI tools mature, and help you shift from ‘website information architecture’ to ‘intelligent experience architecture.’

At the same time, we cover the common pitfalls. These are, not surprisingly, more to do with attitude than technology.

If you want to read more about how to prepare your public-facing (or, indeed, internal facing — designing digital experiences for your employees can be just as, if not more, impactful!) read the full post here:

https://www.squiz.net/blog/is-your-website-ai-ready-avoid-these-3-common-pitfalls

The next article was, if you’ll excuse the marketing speak, about:

The ultimate 6-step roadmap to implementing AI on your website

The idea being that the 6-step roadmap is not just about launching a new feature. It walks through how to embed AI into your broader digital strategy in a way that's sustainable, user-centric, and aligned with your organization's goals.

You know, product management style.

The 6 steps are:

• Step 1: Define AI in your organizational context

• Step 2: Create shared understanding across leadership

• Step 3: Choose a high-impact pilot that proves the model

• Step 4: Appoint an AI-readiness lead (or task force)

• Step 5: Assess your current digital content for AI-compatibility

• Step 6: Embed continuous learning into digital operations

• Bringing it all together on your website experience

Yes, a lot of it is borrowed for organisational transformation, but adapted to AI and DX specifically. This is revolutionary technology, but revolutions still (and always will) happen at human speeds. So this focuses on making the best out of AI, and — hopefully — helping larger enterprises moves at something closer to start up speeds.

If you’d like to read the full article, find it on:

https://www.squiz.net/blog/the-ultimate-6-step-roadmap-to-implementing-ai-on-your-website

That’s it for now. I aim to get back to fortnightly blogging, covering both Product Management and AI (and product management of AI).

Experts in any field I’ve spoken with, have a basic curve of reactions to AI content. It’s hauntingly similar to Gartner’s hype cycle curve: “It’s so cool it can do that! Actually, when you look closely it’s utter crap. But then, perhaps I could use it just for some of the boring bits…”

So the question for today is: Why do we react the way we do to the various forms of AI content?

In large part, this is an expansion of my note in my review of Co-Intelligence by . This article is trying to delve deeper into the meaning of art, thinking, creating, and the limits of generative AI on the path to artificial general intelligence (AGI).

Subscribe now

How we react?

Let’s start with images.

Back in October, Scott Alexander on ran an AI Art Turing Test. he then analysed and published the results in November. It isn’t quite scientific, but with fifty images and 11,000 responses it’s pretty comprehensive.

I did find his columns after the results were published and read the analysis first, so when I went back and took the test myself I was forewarned and forearmed. I did reasonably well (better than the average) in classifying the pictures not mentioned in the analysis, but of course failed a few.

Alexander has specifically excluded ‘obvious’ AI pictures, thus not covering the ‘typical’ output of Midjourney / DALL-E you might encounter on social media, so some of the results might be skewed. Then again, he also chose works of experienced human artists so both sides can claim the only best were taken, not your ‘industry average.’ Still, in the wild, people will tend to recognise AI content better than the below results.

That said, I’d like to draw your attention to one particular conclusion:

The average participant scored 60%, but people who hated AI art scored 64%, professional artists scored 66%, and people who were both professional artists and hated AI art scored 68%.

Go read that whole article, and in particular that last point where an artist explains everything subtly wrong in a particular image. As Alexander concludes:

So maybe some people hate AI because they have an artist's eye for small inadequacies and it drives them crazy.

It may not be conscious, but I would interpret it as people being more sensitive to the uncanny valley. Beyond that, when you have an expertise in a field, then the small nuances and subtle mistakes are more prominent. In ‘factual’ fields like logic and math it’s easier to highlight, but in creative and artistic ones it’s harder define. That doesn’t make it any less visible or important to people, both novices and experts.

Of course, ‘creative efforts’ is a range — coding, for example, is not pure logic and requires creative problem solving, and is prone to subtle errors that aren’t immediately observable (all those bugs that leak through any testing). Many creative tasks are somewhere on the range between logic and art.

The point is, that expertise brings the ability to discern subtleties, and experience brings discernment and taste in selecting the right approach. Teaching someone a skill is one of the best ways you can improve yourself and achieve expertise, as it forces you to analyse works for faults and answer questions clearly.

So detecting faults (in human or AI content) can be conscious or subconscious; when we look at AI output and think it’s off, we may or may not be able to put our finger on what’s wrong with it, but the more attuned we are to the field the better we get.

What about writing?

I have uncovered the secret to effective use of AI in writing fiction! Let me share with you this nugget of wisdom. Ready?…

Every so often, I try one of the bots and get it to create and/or edit a text. I use a variety of prompts and techniques, and refine further and further, until I max out the capacity to produce the best possible text. And the results are, invariably, … shit.

Utter bollocks. Derivative drivel. Grammatically correct garbage. Soulless slop.

OK, enough exaggeration by alliteration. There are two good usages for the output of LLMs in purely creative writing. One, is as an exercise in editing, where I force myself to articulate all the problems in the AI’s output 😜 Cheekiness aside, it cab help me build up my editing and critiquing skills, though I certainly enjoy it more when I edit the works of fellow humans.

Even when skipping outright creation and using AI to edit works, whether a plain LLM or something like Grammarly, is fraught with its own issues. While it will do its best to make sure everything is grammatically correct and the language is clear and accessible, it will do so by ensuring it conforms to some version of “internet average” language and will strip and personality and voice from it. Developing your own voice is significant to authors, just as visual style it for artists. (For example, if you’ve read more than a couple of my posts, you should get a sense of my style, of me, and should be able to recognise it in other posts). While an improved AI could perhaps learn to imitate my voice, it won’t help my develop it. This is also the minor problem: the bigger one is that writing is thinking. One has to be very careful and consider any second-order effects (like lazy experts), and ensure that the crutch doesn’t become indispensable.

The second use I found for AI in writing fiction is pure motivation. Spending time with the bot to get such a mediocre output and knowing I could have written something much better quicker is a kick to stop procrastinating and get writing…

Getting back to a more serious discussion, there are cases where this kind of language is the right thing to use. For example, writing book blurbs is notoriously hard for authors. How to you condense a 100,000 words novel into a paragraph of marketing-speak? Using an LLM not to write your blurb but to quickly generate a few options, allows one to select the better features and get over the block, building something that fulfils the needs of describing the work while being generic enough to fit in a particular genre’s tropes, set expectations, and appeal to the right readers.

But creative writing (fiction, strategy docs, or creative copy) is my area of expertise, and why I find the base LLM output problematic in those cases. To become an expert myself, I must read and analyse and struggle with writing to express my thinking, and not outsource that to an AI. Using an LLM is like using crutches — no amount of relying on such will get me beyond the mediocre. There is a truth and a reason why ‘art’ is often described as a painful struggle, and why experts in any field are required to put in the hours of practice-feedback loops.

Subscribe now

Language is thinking

When we look at the output of LLMs, we anthropomorphise it as ‘thinking,’ as having put in mental effort to reach a conclusion. This is especially true when you play with a reasoning model and look at the stream it produces along the way.

We don’t have an explanation (yet) on how the low-level bio-chemistry of the brain produces abstract, linguistic, thinking — let alone our sense of self (the one ‘having’ the brain and doing the thinking). We say that this is an emergent property, one that rises out of the complexity of the underlying systems. We think, or hope, or fear, that LLMs will show that same level of emergence.

Maybe it’s already showing some — we can’t always pinpoint the way it learns and produces output, and there is some appearance of emergence of higher order understanding and capabilities. But is a reasoning model really thinking?

This is my current pet hypothesis:

We mainly think in words. Even visual artists that think in pictures when creating art think mostly in words at other times. We certainly mostly communicate with others using words. And we evaluate the intelligence of others using words. It’s why we find it hard to evaluate the intelligence of corvids and dolphins, and the bias against non-native speakers of our language. So when we encounter the stream of consciousness of a reasoning model, which looks coherent and sounds like a person might describe how they reason through a problem, we ascribe thinking to it.

But is it?

Specifically, how can we evaluate the apparent level of ‘thinking,’ and whether it’s a real emergent quality or just really good parroting? Is that stream budding consciousness, or an illusion based on our biases? That’s where I think turning to art can help assess some of the issues.

In part, language models engage in “pure” language generation. It may encode our biases (as they encoded in the body of words the LLM was trained on), but thinking is more than just language. For example, when the brain suffers damage in the capacity to generate emotions, people cannot make decisions — all decisions are emotional. (Side note: When we build decision support systems and market them as impartial, it often turns out that we just found a way to encode biases; often not the result we were aiming for — with the added bonus of extra opacity).

Our ‘thinking’ is dependent on many physical aspects of our brains and bodies, and cannot be separated. A model’s reasoning chain is, as anyone who tried meditation would tell you, far too coherent. Humans don’t quite think like that. So does mimicking language, learnt from humanity at large but still performed in isolation of any cohesive physical experience, still count as thinking? Would it support the emergence of all the other properties of thinking, such as feelings, awareness of time and space, morality? Back to the point about art above, right now, AI output hits that uncanny valley pretty bad. Though the example in this video is trivial, it’s exactly the point Alexander makes about driving people crazy.

Abstracting thinking only to language brings up the Chinese Room argument. This is a thought experiment that goes as follows. There is a room, inside of which is a person who does not know Chinese (or the language of your choice; my Greek is better than my Chinese, so I’ll stick to that). However, the man has a master codex on a fast computer, that species for each conceivable input of Chinese characters a corresponding output. By passing notes under the door, a different person outside can fluently communicate with the person inside in Chinese, even though the one inside does not speak the language.

There are many interesting philosophical responses, not the least of which is the system view that perhaps the room speaks Chinese.

For our discussion, though, note that the person inside is like a chatbot — they sit inside, waiting for input, for which they then provide an output. They lack agency and self-direction. They provide us with an output that makes us think they are thinking, but this is merely an output that is mimicry without discernment. It’s the diffusion model saying to itself ‘a hand has five fingers, I must remember to paint only five’ while still losing the plot on the cat merging into the curtain in the background. It’s the Deep Research (pick any of the multiple AI products with that name) who writes out ‘this is interesting, let me delve down this Reddit rabbit hole’ because it learned that this style of output pleases the humans who built it, without the discernment of a researcher knowing reputable sources from internet memes.

That is perhaps a bit flippant, but my experience with the Deep Research tools was a bit hit and miss. Even when the sources gathered were appropriate, the extraction of information accurate, and the report correct (a process we can assume will improve over time), it still leaves a few open issues.

First, just like with human interns, there will be subtle errors, necessitating a careful review of the output. That’s the above expert noticing and correcting the output, rather than accepting it.

Second, it only works if the information is already available on the internet, and does not cover original thinking. The written artefact is a representation of that understanding work. Writing is thinking, and without the effort to write, well, it’s like getting a report from an external consultancy on how to solve your problems. That is, not all that helpful.

Third, is that that value of these reports is rarely in the artefact, but in the thinking and effort that went into producing it. That’s part of growing the intern’s skill level, as well as understanding the material in depth — the struggle in the creative process, that brings insights now and later.

We’ll probably solve the training at inference time, making models that can learn from their own experience (and remember this learning when you close the chat). But that won’t be under current LLM architectures; we’ll need a different way to approach the problem. It also won’t touch embodied experience, which is integral to human thinking, and it won’t help us understand any topic outsourced this way to any depth. Critically, it will never solve ‘grokking’ as defined by Robert A. Heinlein: “to understand intuitively or by empathy, to establish rapport with, and to empathize or communicate sympathetically (with); also, to experience enjoyment.” It’s that innate understanding of a topic that you gained through struggling with it.

Perhaps one day, someone at some AI lab will issue the command “go out and live,” after which a future AI model will come alive and continue to pursue its own interests ad infinitum. It might also scheme against us when we try to put it back in the box. But this will be only the start — if it will want to produce art at that point, it will have to do it like the rest of us, observing, trying, getting feedback, and repeating — not merely emulating. Without this, all we have is mimicry without understanding, techniques without the taste to discern between right and wrong.

What can we do

Don’t get me wrong, I think LLMs can be very useful, including for creatives. But there’s still work required to build this up, guided by principles like ‘human in the loop’ and ‘trust but verify.’ And that doesn’t even touch on the ethical issues of the companies building those systems; I’m focusing here only on the technology, and how to approach implementation by considering second order effects.

When we implement AI systems, even accounting for human in the loop, even when building different tools for experts and novices, there are still many dangers.

Because this is a new field, the research is new as well and sometime conflicting. We can see instances where it improves capabilities, but at the same time (and occasionally due to second order effects) we see humans with good AI performing worse, or even a decline in cognitive capabilities because where AI helped with specific skills it reduced the capacity to problem-solve. (This was my main quibble with Mollick’s book — he claims that everyone benefits and AI is levelling the playing field, helping everyone who uses it).

In order to build expertise, there can be no replacement to human mentorship and guidance, and to shortcut to putting in the effort. Even if the exact type of effort morphs, it you go for the cheap option you get the cheap result.

‘Art’ is hard to define, but it’s my observation that art is half observation and half expression. It’s hard to separate the two, because we humans “train at inference time” to use the AI terminology. It takes sustained effort under competent guidance to build those skills (like a computer science major in their first job getting a lot of feedback from the team lead on what they naively thought was brilliant code 😉).

So while human-in-the-loop is absolutely the right way to go, there are still open problems in and around how they are marketed and implemented, how people interact with those systems, and how we account for varying levels of discernment (expertise) as well as allow for building future generations of experts. This doesn’t mean we should avoid AI — just like Photoshop changed graphic design but did not eliminate it, we need to find new ways of supporting students, practitioners, teachers, and experts.

When designing an AI-based solution, one (of many) questions you must ask yourself is whether an “internet average” is the right approach or not for the use-case, and how you should guide your user community to utilise it without harm.

In Summary

When it comes to art, the struggle is real.

When it comes to LLMs, we are far away from consciousness.

When designing AI solutions, having a “human in the loop” is still a must, but may not be enough.

We need to ensure that the tools support both novices and experts, do not reduce the expert’s capacity to exercise taste, and importantly allow novices to learn from human experts. The AI in the rest of the loop is a crutch. A very useful crutch, but one that should be be allowed to exclude other forms of movement, growth.

What to Expect

An 'outsider's' take on AI. is a business academic, not an AI researcher, and has been investigating, using, and adapting LLMs in both business and teaching contexts. That gives him a good perspective on the utility and implications of the technology.

The book is structured in two parts. First is around the promise, the problems, and sensible engagement rules for Generative AI. The second part explores Gen AI usages around work and teaching, and ends with exploring four possible futures about how AI might develop and the possible impacts it would have.

What I liked

I appreciated Mollick not shying away from the problems around Gen AI — both in terms of the technical frontier as well as the ethical implications. In my view he could have gone further exploring that, rather than settling for stating it, accepting and moving on. He notes that we, as a society, must come to terms with it and decide on acceptable usage, but while he offers advice on how to integrate GenAI into our working lives he doesn't offer the same in terms of broader ethical frameworks.

That said, much of the problems with Generative AI are not with the technology itself (it’s just a tool) but with the companies that build it. You know, those who think ‘stuff they found on the internet’ is free, just like ‘stuff they found in the neighbour’s shed.’ This is more solvable (well, theoretically, perhaps not in the current political climate), which still leaves the main tenets Mollick posits as foundational to working with AI as the right direction.

Subscribe now

What to be aware of

The book only focuses on LLMs, with little about other generative models or types of AI (such as predictive systems). The focus isn't on explaining the technical details, but on abstracting rules to maximise utility while the technology keeps evolving. This is a good position (as the AI landscape shifts rapidly).

Mollick is also very bullish about AI enabling everyone to perform at higher capacity, with lesser skilled individuals benefiting the most. The research I have seen (some published after the book was written and published), is more ambivalent. It is not yet clear if novices or experts benefit more, how this interplays across domains, and the long-term effects on overall skill levels.

A Creative’s Perspective

I am both a creative (fantasy author) and working with AI for my day job. Mollick does make a distinction between creative tasks versus creating art. He naturally focuses on the tasks at work that require creativity. In my view, many people in the AI world (and tech in general) miss the art aspect — which in turn makes them misunderstand some of the 'applied art' process, namely those creative tasks.

It is my view that this is a spectrum, a gamut, with fuzzy lines. You can't be 'just' a graphic designer without having an eye for visual art, you can't be 'just' a copywriter without understanding stroytelling. Conversely, the artistic process informs and enhances the more task-oriented creative process.

While defining what is art is complex, from a practical perspective I like to say that art is half observation and half expression. It's why painters learn to observe and draw shapes and light, and why authors must read voraciously. Art does not exist in vacuum, but is a response to what came before it and the world around the artist. The spark lies in the insights from observation and the journey of expression.

My fear, therefore, is that when applied to creative tasks Generative AI will force a regression to the mean — it may elevate the novice, but as a crutch that won’t lead to building expertise; and it may also hamper the expert, who stops struggling when the AI is ‘good enough.’

Summary

It's well written, practical advice, and will remain current for the next few years as AI improves. As we (humanity at large) will be forced to interface and interact with AI systems more and more, the guidelines presented by Mollick will be practical to almost anyone. Considering who reads this blog, I’d recommend it.