I am in charge of resetting the breakers. And slicing the potatoes.

The thing is, in my early years of living in Europe, this British dish got me very confused. Why do they call them chips? Why not french fries?

And why are potato chips called crisps in England? When dining in a restaurant, a burger and chips is potato chips, but fish and chips, is, well, french fries.

Context - and culture - is the difference between a correct meal and a mistake, whether a disappointed birthday girl or disgruntled patron.

The same is true with context for AI agents.

Humans are adept at answering ambiguous questions, AI agents are not.

Semantic layers are one way AI agents go from probabilistic results to deterministic, trusted insights. But this also depends on how the semantic layer is used and how much context is leveraged. Given the criticality of this in an agentic world, data teams are pursuing semantic layers as a foundation for AI.

They are right in treating semantic layers with the utmost importance. But they are wrong in thinking there should be one semantic layer to rule them all.

The Fallacy of One Semantic Layer

I understand why data teams want one semantic layer - it goes back to our industry’s wish for a single version of the truth. The answer to our quest for one truth, initially, was a data warehouse. Put all the data in one place and users could trust the data! I will spare you the history lesson in spread marts, data lakes, data swamps, and data mesh.

The bottom line is that each function has their own version of truth. Also, sometimes business definitions change.

When a CFO refers to revenue, the CFO is referring to GAAP revenue, based on invoices, net of returns. In a SaaS world, this is annual recurring revenue.

When a sales person refers to revenue, they most often are referring to order value, whether invoiced or not, and excluding professional services as there is no commission here..

Both are right. The difference is in context, here based on the person’s job function. With my food example, the definition of chips changes based on the main course (Fish or Burger) or country (British versus American).

In this regard, context must be part of your semantic layer strategy and is why organizations should be pursuing a mesh of layers, federated across the organization.

Where Does Knowledge Exist?

In a workshop and I led at the Gartner data and analytics summit, we asked people to write down where business definitions most often reside. Many wrote, “in someone’s head.” A few wrote, “in our data catalog.” Others, “in a spreadsheet.”

As and Tony Seale shared on The Data and AI Chief podcast, how your business operates is your unique IP. How your business operates may be partially documented in employee manuals or corporate intranets, but more often it is part of the tribal knowledge of a company.

The idea that we can get everyone first to document, then to agree, then to centralize, then to maintain business definitions seems laughable to me. I think of one customer I have advised who told me they have 33 different definitions of customer! I could easily understand three (prospective, active, former), but not 33.

And yet, if we do not have one single semantic layer, how else do we stop people from talking past one another or debating whose number is right?

An Operating Model for Semantic and Context Layers

I can envision a central information steward documenting these 33 different definitions, but unlikely that the same data steward would maintain such meta data as definitions are rationalized or refined. Data catalogs and information glossaries started in this vein but have met with mixed success. Business users and data experts recognize the need. The organizational muscle to share cross functionally and commit to maintaining is weak.

Hence, my recommendation for the agentic AI era is to pursue a layered, federated approach:

1. Centralize commonly used elements that require company-wide alignment

2. Decentralize and federate domain and business-specific terms

3. Leverage AI to capture existing context

4. Allow business users to dynamically update context

5. Periodically synchronize and merge centralized context with federated context

You will note data mesh principles here such as domain ownership. Vanguard shared on The Data and AI Chief podcast why they are federating their semantic layer approach (tune in here).

I suspect the biggest discomfort proponents of a single semantic layer will have are with points 4 and 5. Here, I am being a pragmatist rather than a purist. The control freak, single-version-of-the-truth seeker in me wants to tell a business user that they MUST tell the data steward when a business definition changes. The pragmatist in me knows this will not happen.

Just last week, as I was looking at podcast listener stats, I told Spotter (ThoughtSpot’s agentic analytics agent), whenever I ask for podcast listeners by time, use the download field. If I ask about particular episodes, use the listener field (a strange way the source system tracks data). Should I go back to our Snowflake DBA and ask him to annotate the data data model with more precise definitions? What about the owner of that ThoughtSpot Spotter semantic model?

I don’t do either. I am too busy (so is the data engineer and ThoughtSpot admin). And I am grateful that the context I gave Spotter while I asked a question is automatically shared with my podcast producers and preserved in Spotter Memory. I consider this a reasonable, federated approach in which I am largely the domain owner of podcast data.

Context can be provided during the analytics workflow but it also exists in a variety of systems, both structured and unstructured. Databricks recently previewed Genie Ontology that will crawl tables, queries, and dashboards for context. Glean differentiates itself on its context layer. Atlan has repositioned itself from data catalogue to open context layer. Such context is critical in ensuring trusted insights and efficient usage of tokens.

I want these definitions to be domain controlled and dynamically enhanced by AI.

On the other hand, for something as widely used and shared as “customer”, I want a different approach. Likewise for publicly reported KPIs and shared data in regulated industries, I want greater centralization but one that is flexible.

A Medallion Model for Semantics

With data modeling, we have gotten comfortable with multiple levels of data quality, with the gold layer being the highest quality in a lakehouse.

The same should happen with certain company-wide KPIs and dimensions. For a publicly traded company, GAAP revenue should be in that gold semantic layer. The most widely used dimensions such as customer and product, should also be in a gold semantic layer. Federated semantic models may be considered silver and user-specific definitions, bronze.

Where it should not be is in any layer that is closed and proprietary. That includes many legacy BI tools.

We also need a mechanism by which definitions enriched by domain users - while asking for insights - can be promoted back to this gold layer. Efforts such as Apache OSSIE (an evolution from OSI) are promising in allowing such interoperability. But not all database vendors are supporting OSSIE, nor are a range of AI and BI tools. So the onus is on each customer to define their semantic layer strategy - the ownership, the update process, the sources of truth.

But what about those three (or 33) different customer definitions? The definitions should be explicit to contain the context. Active Customer. Free Trial Customer. Inactive Customer. It’s okay if the generally accepted default for “customer” is “active customer.”

An intelligent AI agent will also use information such role, function, region, or frequently used data models as context to reduce ambiguity.

What’s The Best Place for a Gold Semantic Layer?

As I work with customers on developing their semantic and context layer strategy, I have noticed differences in reasoning here:

• Some want it closer to the database for re-usability.

• Others are deciding based on the ease of updating and whoever is closest to the business domain.

• DBT customers like the code-based approach for auditability and version control. The same is said of ThoughtSpot customers who export their models to TML.

What I have yet to hear is how organizations envision maintaining these models as context is enriched. Nobody wants to dumb down a context layer that is more mature than a particular database or further ahead than standards within OSSIE.

Back to my fish and chips analogy. Humans can deal with such subtleties and ambiguity. We automatically factor in context. AI agents are more rigid. Quality semantics with robust context is the answer to ensure trusted insights. If we hope to scale agents for trusted insights, the tech has to evolve and so does an organization’s semantic layer strategy and operating model.

I look forward to hearing how you are approaching this critical part of the AI stack.

A Word About My Biases (aka Experiences)

I am a business person at heart who is technically savvy. In deploying Business Objects and authoring three Complete Reference books, my universes were well designed, complete with descriptions for every field - imported from spreadsheets, the only source anyone remotely maintainted. In using and deploying Tableau, I did many one off analyses and never bothered to document any metric. At ThoughtSpot, I advise our top customers on data and AI strategy with a business-first mindset and right-sized governance approach. I am delighted to see that Semantic Layers are Suddenly Sexy Again.

In May, an AI consultant told Axios that one of their enterprise clients ran up a $500 million Claude bill in a single month.1 The company had put no limit on how many licenses employees could use. People ran long agentic workflows that compounded token spend quietly in the background, and the consultant noted expensive tooling getting pointed at things a person could do in seconds, like checking the weather. Uber burned its entire 2026 AI budget by April, four months in, after Claude Code adoption jumped from 32 percent of engineers in February to 84 percent by March. One Uber executive ran up $1,200 in a single two-hour coding session. Microsoft pulled Claude Code licenses stating engineers were costing $500 to $2,000 a month.2

That is the base rate. The individual stories underneath it are far worse. So I can understand why I’ve been getting questions about what specific actions should be taken to reduce AI costs. And what can make the costs more predictable? These are hard questions that can often end with “it depends.” So, I’m going to do my best to leave you a framework for controlling costs and some specific actions you can take right now.

Observe, Optimize, and Operate (O³)

In my mind, I think of these as visibility, optimization, and continuous improvement. I’ve cleverly named it O³ or “O Cubed”. In each of these areas, I’ll lay out some specifics that will help you right now. It’s worth noting that one-time reviews and adjustments to your agentic stack aren’t going to control your AI costs and that’s why you need a system or a framework. So let’s dive in.

O1: Observe

You can’t optimize what you can’t see, and most organizations can’t see very much. So, observation comes first because every recommendation later in the framework depends on having something or a number to compare against.

Before we begin observing, let’s make sure we understand what we are looking at. For our use case, let’s first understand tokens.

A token is four line items, not one

Input tokens are what you send into the model. It’s your prompt, the system instructions, all the documents and data you stuffed into the context. Note: I find that is the line item that gets inflated by design choices that nobody ever revisits. Recently, Anthropic’s own engineering team found they could delete repeated tool-use examples from a system prompt entirely. They just included the same instructions in the tool descriptions but only once, with no loss of capability. This removed redundant tokens from every single call.3

Cache read tokens are the input you have sent before. They are served back at a steep discount, as much as 90 percent. That discount compounds fast at volume for relevant questions but can be costly for unrelated questions. Cache is the single easiest win on this list but it’s not full-proof.

Thinking tokens represent the reasoning that a model does before it answers. We’ve all seen Claude ‘thinking’, ‘ruminating’, and ‘crunching’. Instead, Claude should say ‘generating more tokens and costs’. It’s important because those reasoning models are billed at output token rates which is the most expensive tier. It can also wreck your forecast because the amount is not fixed or bounded. Meaning, the same question you sent to the same model twice, can result in very different amounts of reasoning.

Output tokens cost the most per token. So your output costs are driven by verbosity, and by result sets that get echoed back through a context window instead of rendered somewhere cheaper.

To put real numbers on it: Claude Sonnet 5 runs $2 per million input tokens and $10 per million output, Opus 5 runs $5 in and $25 out.4 A model doing heavy reasoning can burn several times its visible output in thinking tokens you never see, all billed at the output rate. If your team runs agentic workflows with multi-step tool calls, every tool definition, every intermediate result, and every retry adds tokens you did not budget for.

There is one more level we need to observe; the tokenizer. While the rates for tokens may have held steady between model releases, the tokenizer may have changed. Claude 4.7 token generation jumped by 30% according to Anthropic. Call it ‘Shrinikflation’ if you like. Meaning, you can’t just watch the dollars billed, you have to look at the tokens count too.

The unit economics nobody models

In my experience, unit economics of AI and agentic workloads is one of the hardest capabilities to get right. Yes, it’s even more difficult than it was for cloud data platforms. Why, because gap in understanding between AI token usage and a measurable business outcome is huge. It was much easier to connect a compute-hour to a completed job or pipeline.

Let’s break this down. I see three layers between what is billed and the business value delivered. API pricing is easy; it’s dollars per million tokens. Application cost, in the middle, is harder. It’s how many tokens one real user interaction actually consumes including multi-turn interactions, retries, context growth, and tool calls. For the middle layer, consider the following:

1. Cost as a distribution. Because token usage for a single business question is unpredictable, the cost arrives as a distribution rather than a single point.

2. There is a language tax. Verbose prompts, bloated context windows, and multilingual requests ( yes, non-english tokenization just cost more), are all driving up cost without anyone deciding.

3. Shrinkflation is real. Your token cost may surge upward in new releases while value delivery remains flat.

Business outcomes are at the top. You can imagine a deferred support call, a closed ticket, or an hour saved by an analyst. Note: before you finalize the forecast, find every token meter in your AI stack.5

Report outcomes instead of activities

Let’s be clear, the frontier labs bill for activity not business value. They have no incentive to help you measure value, because consumption is the product. So now you know that it’s on you to show value. Here’s what to you shouldn’t report:

Activity - “We executed 10,000 prompts this quarter.”

Instead report the outcomes, “We deferred 500 support calls, recovered $50,000 in subscription revenue at risk, and saved 200 hours of analyst work.” Which would you want to hear?

Cloud data teams have taught us a lot here. I was on those teams when we had this same approach 5 or 6 years ago. A executive leader would ask what the data team had accomplished and we would proudly claim, “we ran 40K dbt models”. Not good. Today, we are seeing the same maturity gap with agents; only now the tools are different.

💡 Best practice: tag every agent

Categorize every agent by team, use case, and environment. Note whether it’s a background agent or an interactive agent before it ever gets deployed. We all know that teams are facing pressure to deliver quickly and that often leads skipping perceived unimportant details like tagging. But without we’ve lost visibility and puts limits on cost visibility, allocation, forecasting, and ultimately the unit economics.

Why it saves money: you can’t cut what you can’t attribute to a team, a product, or a workflow. This is a first principle in cost controls. Without proper tagging, your team is turning every cost metric into a guess.

💡 Best practice: audit business value on a cadence

Survey the business teams that are using running agents weekly, monthly, or quarterly. Ask whether the work still produces value, separately from whether the agent is still running. Do it need to run every hour, every day, on weekends? The there’s lots of quick wins here. Be ruthless! Retire every agent fails to show value.

Why it saves money: usage always outlives value. For example, a workflow built for a product launch keeps consuming tokens long after the launch is over, and nobody notices, because nothing is broken.

O2: Optimize

Visibility and surveys implemented, check. The next move is architecture. One instinct is to route everything to your most capable model, which is often the expensive model and usually not necessary. The other instinct is to tell people to prompt less, which may be even worse. It’s worse because it trades a cost problem for an adoption problem. In my opinion, the most expensive prompt is the one your domain experts are afraid ask due to ai rationing.

Task routing equals warehouse re-sizing

We’ve done this before with Snowflake compute. Right-sizing the model with the complexity of the task is what’s important here. For example, classification and extraction go to small, cheap models just like we did with simple data ingestions using Snowflake x-small compute. If you have genuinely hard multi-step reasoning tasks, those deserve the expensive models and should be governed appropriately. After all, not every user needs to solve that 100 year old unsolved math problem.

Cache aggressively wherever context repeats. Design orchestration so failures happen cheap and fast instead of retrying an expensive call three times. It’s easy to say and I recognize that it’s not always easy to implement. So, think big, start small, move fast, and iterate!

Note: Right-sizing alone won’t solve the unpredictable cost problems. Again, that’s why we need a framework.

⚠️ Caution: right-size to the task, then verify against realized cost

The same query doesn’t cost the same thing twice. Identical question, identical model, repeated runs: the most expensive run cost close to ten times the cheapest. Databricks reached the same conclusion independently, that per-token pricing is an unreliable indicator of total task cost. Cheaper-per-token models routinely cost more per completed task. 6

Why it saves money: capability, rather than price, determines how many turns a task takes to successfully. The rule here is to route based on the measured cost per completed task. Use the list price as just one input among many.

💡 Best practice: decompose into multi-step workflows and treat session length as a cost control

Break a task into steps, each handled by the cheapest model that is capable of completing it. Rather than sending full context to a single expensive call. Start new sessions for new topics instead of letting one session accumulate results across unrelated questions.

Why it saves money: a well-orchestrated pipeline can lower total spend even as it adds steps. This is because each step’s input context remains small. Conversely, session length compounds costs in the wrong direction, since results re-sent on every turn cost way more than results generated just once.

Vendors are shipping the tooling, and that covers half the job

FinOps for AI is becoming a software category of its own. And it’s filling in fast, which is a good sign that the problem is real.

• Ramp launched AI Token Spend Management in July. It provides one dashboard across AI providers, weekly briefings on usage trends and efficiency opportunities. It also has real-time controls and alerts to stop an overrun while it is happening.7

• Nvidia and Snowflake moved into the model router market in August. Routers productize the task-to-model matching this section describes, sending each request to the cheapest model capable of handling it.

• Linux Foundation has launched a Tokenomics Foundation.

My recommendation is to evaluate what’s best for your organization and buy them. They are genuinely useful, and building your own meter from scratch is a poor use of an engineering budget. If coding agents have made us all builders, the build vs buy question now is “Could We” vs “Should We”.

You have now reached the area where most cost control efforts stall. Observe and Optimize only pay off when you apply them to your own business processes. No vendor can answer the hard questions for you, like which department’s work justifies more spend, what counts as an accepted output in your context, which agent deserves to still be running next month, and what a deflected support call is actually worth on your P&L. You and your domain experts rely on experience and judgement to answer those questions and then determine where to apply your resources. Don’t delegate that part.

O3: Operate

You can think of Observe and Optimize stages as constrained projects with end dates and deliverables. But, Operate is different. It’s the part that runs forever. It’s also the part that needs continual care and feeding before it takes root. Once you set the cadence and maturity take hold, you can sit back and harvest the value. Your process will have become a FinOps practice.

⚠️ Callout: set a time-to-live on every background agent

Every background or scheduled agent gets an expiration date or a review date which should be set on deployment of the agent. When it expires, someone reviews the ongoing value and renews it deliberately. Otherwise, it stops running and accumulating costs.

Why it saves money: background agents are easy to forget especially when the creator doesn’t pay the bill. Nobody sees them, nobody complains about them, yet they bill every night. Setting TTL keeps the business user engaged, increases accountability, and makes value decisions explicit.

Cost per accepted task

If you aren’t familiar with cost per accepted task, it’s the metric that matters most. The main problems are that few organizations track it and the frontier labs can’t track it. It’s not cost per query or cost per prompt. It’s the cost per output your team actually kept and used.

I break acceptance rate into three categories and track them. The categories are as-is, edited and used, and discarded. Now track those. You’ll soon see examples of workflows that cost three times more per call and get accepted nine times more often.

Let’s compare Workflow A, that premium workflow, with Workflow B, the cheaper workflow. Workflow B has a 50 percent discarded rate. So, the more expensive workflow wins on cost every time. Why? Because you pay twice for the rejected half, once for the bad output and again for the human hour spent redoing it.

Lastly, don’t forget to measure the failure rate. Track all three numbers together. How often you get a correct answer, what an average correct answer costs, and what the worst runs cost. Think of these as best case, average case, and worst case scenarios. The failure rate or worst case is usually ignored but it is the one that wrecks your forecast.

Demand determinism for anything governed

One last operating principle, and it is the one I feel most strongly about. Trusted answers are non-negotiable! Using probabilistic SQL against any governed metric is a problem for your budget and your compliance audit. Wherever a number has to be defensible, push the resolution into a deterministic layer and accept that the layer will sometimes fail closed and tell you it cannot answer. Some people will find that infuriating but I think it’s the entire point.

Wrap up

Frontier labs will keep billing based on tokens, because tokens are what they can meter. Your job is to build the layer above the meter. That’s the layer that translates consumption into value, so own it. You’ll be using the same concepts that we used to build FinOps on top of cloud consumption. And your team probably has the skills already, because you have been doing it for the cloud data warehouse for years.

Observe what a token actually costs and who is spending it. Optimize the routing and the orchestration, with realized cost as your evidence rather than a pricing page. Operate with TTLs, acceptance rates, and one outcome metric per workflow; ensure somebody owns that metric.

So, the bill arrives in tokens, full stop. But, you control whether leadership sees the number of tokens or a story about the business value driven by those tokens.

Leave a comment

As anxiety around token costs becomes real for any data and AI leader with budgets scrutinised and inference fin-ops becoming real, I think about the power tool my dad would never allow me to use… the mortising machine - the ultimate way to offload cost.

If offloading AI cost via MCP, or, cutting mortises into timber are of interest to you - here are three concrete (see what I did there?) ways to offload AI cost with MCP.

When Refactoring becomes a Factory

A headache I had as a Chief Data Officer was my convoluted mess of dbt models, SQL scripts, and legacy stored procedures, all so dependent on each other that a single column definition change could break multiple dashboards downstream.

Even a simple metric change was a drama... if leadership wanted to change for example, the loss ratio formula from claims paid / gross written premium to claims paid / earned premium, (I worked at an insurance company) you had to find everywhere that loss ratio was calculated; dbt staging, intermediate SQL, the semantic layer etc, make the change then retest downstream dashboards, models and reports.

LLMs didn’t really help... sure you could give an LLM an instruction such as: ‘refactor loss ratio to use earned premium not gross written, find every definition across the repo and list all reports affected.’ But to understand that instruction the LLM would need the full repo inputted into the context, easily ~500k tokens. Also take into account that agents, as they search, reason, fail, retry, tend to ‘loop’ as they work. So a 20 person team changing metrics regularly can run to five figures a month just reading code. So much context stuffing also raises hallucination risks, not great in any organisation but especially not in something as regulated as insurance!

MCP handles this more efficiently. The agent calls a dbt/Git MCP server for the metric and gets back only the 3 models where loss ratio is defined. A lineage call returns the 4 affected downstream models as JSON. The LLM inspects the 3 files, updates the logic and opens a PR for human review costing ~1.5k tokens, not ~500k! Deterministic tools handle the indexing and the LLM handles reasoning.

Structured Data - Hard and Expensive for LLMs

Here’s where we get to the mortises! If you’re wondering what mortises are, they’re slots in timber that joins fit into. You can certainly pay a joiner to hand chisel the 80 or so mortises you’ll need for a wooden staircase, or, you canhave the mortise machine do it for you.

When you ask Claude (or any LLM running on your structured data) a question like; ‘what is net revenue by region?’ it must read your entire semantic layer metadata to understand what ‘net revenue’ and ‘region’ mean. It then runs the query on the warehouse, formats the raw JSON output into text and charts. A single query can burn ~13k tokens. Multiplied across teams, a prototype built in an afternoon can become a $75,000 a month beast Yes, you can build in prompt caching and that will cut what you’re billed for repeat context if subsequent questions are asked. It will require engineers to build and oversee it though so whilst token cost will go down, engineering cost will not.

A tool that is NOT text to SQL but tokenised search will do this very differently. I’ll use ThoughtSpot’s Spotter here (yes, thank you for asking, I do work for ThoughtSpot!) Rather than context stuffing raw schemas into the LLM, a lightweight text to token knowledge graph maps queries directly to database schemas, using a small fraction of the tokens a text to LLM tool would use. The SQL compilation and chart rendering then happen deterministically on the warehouse at no token cost at all.

The tools can easily be used together so that when the question is asked to the LLM, it makes an MCP to Spotter which will then do the heavy lifting of the compute (via it’s deterministic mode of action already explained). Spotter sends a response back to Claude to format. I haven’t formally benchmarked the saving, but as a principle you’re replacing a large, unpredictable token spend with a small, mostly fixed one.

Tokens are the Tip of the Iceberg: Custom Code for Integrations

When you have to write code for each new integration to oversee authentication, error handling etc. it quickly becomes a large engineering burden. I know because I’ve been there and it’s very…. multiplicative… e.g. 3 x AI assistance for 4 x enterprise systems becomes 12 custom integrations.

And this is really only the start. Every time lets say, SAP updates their API, every time you upgrade your agents, the connections break requiring engineering support. Last time I looked at a P&L I did not find engineers to be cheap. Worse that that, this approach keeps engineers in this zone of ‘invisible work’ where they’re busy on work items neither seen nor understood by execs. ‘But what do your team do all day?’ is a question I was asked as a CDO and having to justify your team’s existence is not a good position to be in. However if you build a single MCP server that any MCP compatible agent can plug into and this problem largely goes away.

If we take a scenario where a company wants its internal AI agent to perform weekly sales health checks by calling data from ThoughtSpot (revenue metrics), Salesforce (pipeline deals), and Jira (to see delivery blockers). Without MCP engineers need to write 3 python wrappers, security layers and converters taking about 6 weeks to build. MCP would require a few days only to configure permissions.

This is essentially Total Cost of Ownership. Execs currently are edgy about token costs but will often not see nor understand ~£10,000 of engineering costs. Be mindful of what’s visible to them and what isn’t… and educate them.

Just as TCO is important, so is security with MCP don’t miss the opportunity to build trust by engaging your security team early

MCP should be a no-brainer but I still see a lot of point connections. Organisational silos, prototypes that become full production solutions etc. it’s essentially a new type of tech debt.

Conclusion: Offload the Load

If the timber was cash in my dad’s eyes, the tokens certainly are cash in any data & AI leader's eyes… and things get expensive when you’re not using the right power tools. MCP is your mortise machine, so use it to offload cost and token anxiety.

As my dad would say… the right tools for the right job.

I had an angry mom recently blast me on why she hates AI, fearing for her son’s job prospects. Is college even worth it if students are just regurgitating what LLMs tell them?

Teens are using ChatGPT and Claude as study guides, essay writers, vacation planners, poster-makers and more. AI is now a life skill. Just as high schoolers learn essential skills such as driving, algebra, biology, and world history, AI has joined the ranks of foundational skills. There are AI skills to acquire, but there also are skills that we should consider NOT outsourcing to AI.

This applies to teens as well as adults.

We do not want to outsource the skills that make us uniquely human. When we outsource too much to AI, those skills atrophy and risk disappearing altogether. Let’s take map reading as an example. Without a map on a student’s smart phone, could they find their way while visiting a new country? Paper maps and the ability to read them seems to be a lost skill for GenZ. Most teens and young adults are fine with this dying skill as long as they have cell service, wifi, a well charged phone, and there is no mass power outage. Drop them on a safari … or hiking in remote parts … and it’s a challenge!

Here are the three skills I would not concede to AI:

1. Critical thinking. Newer AI models have advanced reasoning modes. But critical thinking is a uniquely human skill that goes back to how we are coded for survival. Critical thinking involves solving complex problems, reflecting on what are the most important problems to solve, and interpreting assumptions and alternatives. This includes asking better questions of AI. When AI starts spewing answers that are confidently inaccurate, critical thinking skills allow you to challenge the results. That vacation you planned using AI in which the proposed flight connection does not exist? – critical thinking skills come into play here. AI has been trained on the knowns, not on the new, novel, and possible.

2. Writing. Phew, Substack lit up last week with the introduction of AI checker Pangram. If you have used Claude or ChatGPT all year long to draft your papers, without first thinking about the points you want to make, your writing skills have already atrophied. Anecdotally, people have told me the decline is steep in a matter of weeks. Further, the ability to organize your thoughts into a persuasive argument or problem solving strategy is weakened. Brain research confirms this decline. So, then, imagine you are now in your first job or hoping to land the most desirable internship. You are in a meeting and your boss poses a problem, asks you to assess the situation and recommend what to do. Your ability to articulate and think on the spot will be weaker than the co-workers who have only been using AI as a thought partner. In this regard, you should use AI primarily as a thought partner and a super spell checker, but not as the writer of your first drafts. Draft it yourself first, think critically about the points that are most important, then use AI help you make it better such as for fact checking or critiquing your arguments. Further, research shows that handwriting allows for further memory retention and creative sparks.

Now, that weekly summary of a status report that requires little thought – that is a good task to outsource to AI. Students in which English is a second language have also shared how letting AI write for them is helpful. For sure, my rusty German writing I absolutely butcher the language. But I would apply the same principles. How much do you want to hone your English or foreign language skills? Draft first,then use AI to correct.

3. Emotional Intelligence and Empathy. Being smart is often associated with high IQ. EQ is a high emotional intelligence, defined as the ability to recognize and regulate your emotions, while also understanding and empathizing with others. Empathy is a uniquely human skill. Empathy is the ability to put yourself in someone else’s shoes – to truly feel what another person is feeling emotionally. The idea that AI has feelings or conscience has repeatedly been refuted. Sycophancy – AI telling you what you want to hear – is a known problem with AI. Will I Am framed this so eloquently at the Davos World Economic Forum, Imagination in Action break out, earlier this year, saying “AI will f*** up relationships.” AI girlfriends and boyfriends will tell you exactly what you want to hear, setting up unrealistic expectations for healthy relationships. As humans, we are wired for connection. Deeper connections are enhanced by empathy. As the amount of AI generated content explodes, humans are looking for more authentic content and relationships. The most successful leaders have high EQ and use EQ to motivate others.

Now that you know what skills not to outsource to AI, here are the AI skills to acquire and hone.

1. AI privacy and safety. Every AI company has their own privacy settings that includes how they use your prompts, the degree they retain information about you, and what can be done with that information. Deepfakes are increasingly used by scammers to trick someone into sending money. The emotional damage of a photo of a real person manipulated into a nude photo is profound. Schools also may suspend students for such behavior. Certain AI platforms take AI safety seriously and ban such manipulation, while others allow it.

2. Prompt engineering aka asking good questions. Prompt engineering is really about asking good questions and giving AI the most context that will yield optimal output at the least cost. Also use the right LLM for the right task, leveraging cheaper models for simple tasks (see this blog on tokenomics).

3. Training data. Large language models are trained on vast sources of public data. This may be news articles, social media posts, and the like. It becomes a game of numbers and skews heavily to western cultures. As you apply your critical thinking, consider gaps in data that may skew answers. For example, one Mark Cuban Foundation AI bootcamp student asked AI to create an image of a successful business woman in Singapore. It defaulted to a male business person, even though Singapore has the highest percentage of female C level executives. Models also may not be trained on the most recent data, so rapidly changing news and sports results may cause AI to give you wrong results.

4. Hallucinations. AI hallucinates, which basically means it will make up answers and give false answers. Think of AI as one big Mad Libs game where it is largely predicting the next word in a sentence. Sometimes it gets that predicted word wrong, and even more so, early models were terrible at basic math. In other words, you cannot blindly trust AI. There are some things you must fact check.

5. Semantic layers and context. Semantic layers are a layer that sits above the data sources to tell AI where to find data, what particular terms mean, and how that definition varies by role or region. It ensures accurate AI. Humans are good at ambiguity, AI is not. So when you are at a restaurant and you order “Fisn and Chips,” the waiter knows you want french fries with that fish (Chips is the British phrase for what Americans call french fries). Get the context wrong and AI

The teens that will come out ahead in the next decade are not the ones who let AI think for them, or those that refuse to use it at all. They are the students who know what is still important: the questions to wrestle with before you begin to format a prompt, the first draft that you struggled through by hand, and the human relationships that develop your experience. AI can be a thought partner, but only for a teen who still thinks for themselves. That distinction, more than any particular skill, is what separates agency from dependence and success from mediocrity.

Thank you to , Director of Education at MCF, for posing this question to me and thank you to both Charlotte and for feedback as I wrote this, sometimes on post-it-notes, but mostly by computer!

In Part 1, I left off wondering why our optimized Claude to Cortex Analyst text-to-sql solution only yielded us 15% improvement in token usage. Recall what I called it Pillar one, Claude calling MCP with Cortex Analyst using semantic views to get deterministic SQL, governed joins, and metric definitions that don’t drift between sessions.

I said that number was a ceiling, not an average. You still don’t get is predictable costs and a budget you can forecast. This is the post where I show you why. Let’s start with asking, what happens when your analyst asks the 2nd and 3rd questions and how much that really costs.

Why state is the expensive part

In an architecture where the model does the orchestration, there are two conversation histories running in parallel, and they are vastly different sizes.

Snowflake’s history carries your questions and the generated SQL (see the Cortex Analyst line on the chart above). Cortex Analyst does support multi-turn Q&A but you pass the prior prompts and response back in a messages array. So, Snowflake built a dedicated agent whose only job is to take that history plus your new question and reframe it into one fully-specified standalone question.1 Elegant design. But look at what is in that array: questions and SQL. Ten turns is maybe two thousand tokens.

The model’s history carries your questions, the generated SQL, and every result set you have looked at. That’s the Claude line from above. Ten turns at a few thousand tokens per result set is roughly fifty thousand tokens or more, re-sent on every turn.

Same conversation. Two histories. One of them is an order of magnitude larger than the other. And, it’s the one being metered per token.

This is the single most useful reframe I can offer: the expensive context is not the reasoning. It is the data. And in architectures 1 and 2, only one of the two systems is holding the data.2

That explains the limitation we opened with. Cortex Analyst cannot answer “what’s the revenue of the second product” because result rows were never in its history to begin with. It can refine a question so that “now just the last 7 days” works fine, because that only needs the prior question. It can never reference the result or the answer.

So who does resolve it, when the composite system gets it right? The model does. The rows are sitting in the model’s context window. It reads them, works out that “the second product” means Widget Pro. Next the model rewrites the follow-up question as a self-contained question, and sends that to the semantic service.

Read that paragraph again, because there is something important in it. The frontier model has become your state layer. Not by design. By default, because it was the only component holding the results. You didn’t explicitly make this decision, its a consequence of the architecture.

The optimization that isn’t

Once a team sees the token bill, someone usually proposes an obvious fix: stop sending all those rows to the model. Let’s aggregate the answer and just send the top 10.

Wait what? That breaks everything. The moment the aggregated rows leave the context, every follow-up question breaks, because the model no longer holds the facts it needs to resolve a reference. Yes you have optimized but at what cost. You have taken away the very thing that makes drill-down work at all.

Unfortunately that means that the most expensive component in your text-to-sql architecture is also load-bearing. You cannot remove it, and you cannot shrink it without losing the conversational behavior you built the system for. WTH!

Tokens are not outcomes

Let’s step back from the mechanics of our text-to-sql for a second and look at what you are actually getting for you money.

You are not paying for insights. You are paying for tokens processed. Insights and tokens align reasonably well if you can get them on turn one. But… they come apart badly after that.

By turn ten, most of what you are paying for is re-reading data you already have. The model is not learning anything new about your business when it re-ingests the same six category totals for the ninth time. It’s being given the same rows again and again because it has no memory. At this point, it’s just overhead with a metered rate.

Worse, is what I call Value Inversion, the relationship between the business value and the tokens used actually inverts. Let me explain, the questions that generate the most tokens are often the questions that generated the least value. Meaning, that failed query that got retried, the ambiguous join that took three attempts, the long exploratory session that ended in “never mind”, those cost the most and provided the least. You end up paying the most per unit of insight for your worst outcomes.

Compare that to how the rest of your data stack is priced. A warehouse query costs what it costs based on the work it does (bytes scanned, compute consumed, etc). Run the same query twice and you pay roughly twice, which is annoying but honest.3 Nothing about that model charges you more for re-reading a result you already have.

What happens to the cost curve

Because each turn re-sends everything before it, cost grows faster than the number of turns. Turn one is cheap. Turn ten is expensive. Turn thirty is punishing.

Recent research auditing frontier model costs across multi-turn agentic benchmarks found exactly that. The replayed message history is the largest single cost component in multi-turn agent workloads, not the model’s reasoning.

Prompt caching helps, and you should absolutely use it. It cuts the cost of your system prompts and tool schemas to roughly a tenth. But caching protects the scaffolding of your system, not the data itself. Every new result set is new content. Cache reads soften the curve. They do not flatten it.

This is why “start a new session” turns out to be real cost advice and not a joke. Yes, it good for cost management and it’s real accuracy advice too. Both model providers and the broader research note that long or topic-shifting conversations degrade model performance. Cool! The same hygiene helps both problems, which should tell you something about how tightly coupled cost and correctness are here.

The thing underneath all of this

Every architecture pays for semantic resolution. There is no configuration where that work disappears. The only question is when you pay.

You pay once, at design time, by modeling your business logic into a governed layer that generates deterministic SQL. Or you pay every turn, at inference time, by having a probabilistic system re-derive that logic from scratch. That means pay the expensive output-token cost; coupled with a variance you cannot forecast and an answer that might differ from the last one.

That’s the actual build-versus-buy decision. It was never really about a chat interface or natural language input those are the easy parts now. The chat interface has been commoditized. What has not been commoditized is self-service analytics, knowing what your data means, holding onto that meaning between questions, and deterministically answering a question.

This is why I have come around to thinking of the semantic layer as a FinOps concern and not only a governance one. I have spent most of my career arguing for semantic layers on correctness and trust grounds, and those arguments are good, but they have not always won the budget fight. This one just might, because it shows up on an invoice.

For disclosure: I work at ThoughtSpot, and our architecture sits in that third category from Part 1. ThoughtSpot utilizes reasoning models for semantic resolution and determining intent but doesn’t meter AI tokens. It’s deterministic SQL generation from a governed semantic layer, orchestration that selects best fit models rather than exposing you to any single model’s cost profile, and result sets that never enter a context window. You should evaluate that claim as skeptically as any other vendor claim, and you should hold anyone else in this category to exactly the same standard.

But test the architecture, not the demo. Ask the second question. Then ask the third. Then the forth. Lastly measure cost per insight not cost per million tokens.

Leave a comment

I know what the cost per million tokens is for a frontier model like Claude Sonnet. That number is on Anthropic’s pricing page and it tells you almost nothing. I’m talking about a business question like “What are sales by category for the last 30 days.” What did that cost the organization, and what will thousands of them cost next quarter?

It’s a trick question, I can’t give you a number because there isn’t just one. The answer is it’s a distribution, and it’s rather wide. And the shape is determined by two architectural decisions that most teams never think about.

Here is the conversation that sent me looking.

“What are my products?”

“What’s the revenue of the second one?”

Every analyst I have ever worked with does some version of this a hundred times a week. They ask, evaluate, narrow, pivot, and drill. It’s not the nature of the question, heck it’s not even a clever question or an edge case. But it is the job.

As a data leader and Snowflake Data Superhero I know a little bit about this. It’s a two step process and you need access to the result set from the original question. Yet, Snowflake states in their own product docs that Cortex Analyst can’t access results from previous SQL queries.1 The example above is the identical scenario from their documentation: ask for a list of products, then ask about the second one to demonstrate that the service has no way to know which product you mean.

I want to be careful here, because this is not a knock on Snowflake. It is the opposite. Snowflake is doing the right thing that most of this industry is not doing: publishing the architectural limits of their own service in plain language. I wish more vendors did it. So, I’m more concerned about why the limitation exists. The answer falls out from an architectural decision, and it’s this same decision that determines what your bill looks like at the end of the month.

Most teams evaluating natural-language analytics are asking about accuracy, I do it all the time. Can it get the SQL right? And the answer right? Those are important questions, but also incomplete. The way I see it, there are two structural questions underneath it, and you probably answer both before you write a line of code, usually without realizing you have answered them at all.

Who resolves the semantics? That is: who understands the user intent and turns “revenue by category” into a join path, a grain, a filter, and an aggregate.

Who holds the state? That is: who remembers the last question, the last query, and critically, the last result set.

The first question determines how predictable your cost is. The second determines how fast your cost grows. Your cost is those two factors multiplied together. Nearly everything expensive and surprising about DIY text-to-SQL is downstream of these two answers.

Three architectures

Let me put some structure around this. I have been mapping the natural-language analytics landscape against those two axes, and there are essentially three configurations. I built this out concretely against a Snowflake MCP setup using a typical star schema, fact_sales with date, product, category, customer, order, city, and state dimensions, and one deliberately easy question: What are sales by category, by day, for the last 30 days?

Architecture 1: The LLM writes the SQL

You connect a frontier model to your warehouse over MCP, expose a SQL execution tool, and let it rip. No semantic model. The model, let’s say Claude Sonnet 4.x, discovers your schema, infers the join paths, guesses the grain, writes the SQL, and interprets the results.

The frontier model, let’s say Claude Sonnet, determines every piece of business meaning. Which table is “category”? Which key ties the fact to the dimension? What “revenue” actually means in your organization? All this is re-derived by a probabilistic system, from scratch, on every single turn. You pay for that derivation in output tokens, and you get a slightly different derivation every time.

Architecture 2: A service writes the SQL from a semantic model

Turn on something like Cortex Analyst. Now Claude uses skills and tools to send your prompt to Snowflake Cortex Analyst. The text-to-SQL happens inside Snowflake, against a declared semantic view with real join paths and metric definitions. The SQL that comes back is deterministic and governed. This is a genuine improvement in correctness, and I do not want to undersell it.

But watch what happens to the orchestration. Cortex Analyst returns SQL text, not result rows.2 So the model has to take that SQL and hand it back to a separate execution tool. It pays output tokens to relay a query it did not author and cannot improve. I have started calling this the Echo Tax, and it is significant. It basically cancels out all of the savings you get from skipping schema discovery.

And the results still land in the model's context window. All the results, every row. Driving up context window size and token costs.

Architecture 3: The semantic layer resolves it, and holds the state

The third configuration is one where semantic resolution happens deterministically in a governed layer, and result sets never enter a context window at all. The model resolves intent. The layer generates the SQL. Results go to a rendering layer instead of a context window or thinking tokens.

This is where ThoughtSpot’s deterministic semantic layer and API resides. Yes, ThoughtSpot is where I work. I will come back to that at the end rather than pretending I have no stake in this.

_{Table 1 - Side by Side Comparison}3

The thing I want you to notice is that architectures 1 and 2 differ on pillar one (SQL Generation) and are identical on pillar two (Who holds results). Both put your result rows in a context window. That is why the token savings from adding a semantic service are far smaller than anyone expects. In my modeling of that single easy question, roughly 15 percent on the happy path, and effectively nothing on output tokens.4

15 percent, what? For adding a semantic layer. That number surprised me enough that I went back and re-derived it twice.

The reason is simple once you see it: the biggest line item in both architectures is the result payload, and it is byte-identical in both. A semantic layer that only fixes who writes the SQL has fixed the correctness problem and left the cost problem completely untouched.

And that 15 percent is a best case. It’s just one question, on a fresh session, with nothing going wrong. It’s not a real business conversation, and the reason why is the second pillar: who holds the state.

Why probabilistic answers are unbudgetable

Everything else up to this point is structural. You can model it, draw it, and forecast it. And pillar one is much more problematic, because it introduces variance you cannot forecast at all.

I have been going through a recent study, “The Price Reversal Phenomenon”, auditing what frontier models actually cost in practice, versus their listed per-token price.5 And here three findings should change how anyone in a FinOps role thinks about model selection and costs..

The same query doesn’t cost the same thing twice. Send an identical question to an identical model over and over, and the cost swings wildly. In some cases, the most expensive run cost almost ten times as much as the cheapest run. Internal research from Databricks says the same thing, that per-token pricing is an unreliable indicator of overall task cost.6

Think of it this way; each new instance samples different reasoning paths, produces different levels of “thinking”, and uses fewer or more tokens. So even when the answer is correct, the reasoning paths may be different, implying that changing the prompt does not remove that randomness.

Key Takeaway : It demonstrates how these models work, and you can’t just re-prompt your way into efficient token usage.

So there is no such thing as a cost per query. It’s a range. Remember, when you hand finance a number and you’ve handed them just one roll of the dice.

Cheaper-per-token models routinely cost more per task. Lower-priced models tend to need more turns. They hit more dead ends, more backtracking, more looping. And each extra turn costs more than the last. When a run fails, a “cheaper” model can end up costing 3–10× more than what a stronger model would have spent to succeed. Meaning, you pay the biggest premium for the answers you don’t actually get. I can’t imagine a worse cost profile for a production system.

Averages obscure the actual risk to your budget. When comparing models on average cost, the cheaper model was secretly more expensive about 32% of the time.

The Prototype Trap

A common scenario that I see is what I call the Prototype Trap; I spoke about this at the Data + AI Summit last month. Teams are doing a prototype or a POC but they’re looking at only the easy questions.

On easy questions, almost every model performs well, uses very few turns, and costs about the same. Price reversals largely vanish. It’s on hard questions that have many turns, real ambiguity, and genuine reasoning that the differences. Those are driving the cost explosion.

The typical prototype, someone asks for revenue by region, got a right answer in four seconds. The room was sold. But real production use cases run the hard ones. The ambiguous ones, the ones with three plausible join paths, the ones where “customer” means something different in billing than in CRM.

The prototype didn’t actually test the use cases that determines whether it really works or not.

So, Here’s what I would actually do

I’d question whether or not I should you built it myself or buy it? But, that’s probably a topic for another post. So, Let’s get back on track.

I’ll be frank about the third architecture’s failure modes, because they are real. Semantic layers are intentional by design and they demand an upfront modeling investment in order to see the greatest value. All semantic layers constrain the question space to the data product that has been modeled. When you ask something outside the model, a good semantic layer fails closed. It tells you it cannot answer rather than guessing. Some people find that infuriating. I think it is the entire point, but I understand the frustration. So, if your organization cannot commit to the modeling discipline then you will not see the benefit.

To achieve a predictable analytics budget alongside reliable metrics and genuine drill-down capabilities, focus your architectural requirements on these key demands derived from the two pillars.

Aggregate in the warehouse, not in the model. Six category totals instead of a hundred and eighty daily rows is not a one-turn saving. Those rows are re-ingested on every remaining turn of the session. This is the single highest-leverage change available in a DIY architecture.

Keep results out of the context window. If your architecture requires result sets to live in a model’s context for follow-up questions to work, you have accepted superlinear cost growth as a permanent condition.

Treat session length as a cost control. New topic, new session. It bounds context growth and improves accuracy simultaneously.

Demand determinism for anything governed. Probabilistic SQL is always wrong but you cannot budget a probability, and you cannot audit one either.

Measure what a successful answer costs. Per-token pricing tells you nothing here, and neither does a clean run on an easy question. Track three numbers instead: how often you get a right answer, what an average right answer costs, and what the worst runs cost. That last one is often skipped, and it’s the one that wrecks a forecast. An evaluation that throws out the failures is measuring a system you don’t have.

What this leaves you with

Pillar one, where the semantics live, is where everyone does the optimization. The most logical solution is to move the semantic resolution out of the frontier model and into a semantic model or to a lesser extent semantic views. Great start, you get deterministic SQL, governed joins, and metric definitions that don’t drift between sessions. Take the win.

What you don’t get is predictable costs and a budget you can forecast. You narrowed the distribution with fewer retries and unambiguous joins but it’s still a distribution So, you’ll get about 15% improvement on the happy path which is only the easiest questions in a single turn.

That’s because we have not discussed the second pillar yet. Recall, in architectures 1 and 2, every row you look at lands in a context window and it gets re-sent on every turn. You are still paying the Echo tax and making the data the most expensive part of the cost.

In Part 2, “Nobody Asks One Questions”, we get to what happens when your analyst asks the 2nd and 3rd questions and how much that really costs.

Thanks for reading Brains & Bots!
Share this essay with a friend or coworker. Your support is appreciated 💙

Share Brains & Bots

Leave a comment