Introduction

The finance industry — my industry! — does not use AI enough. Outside of a few pockets of early adopters, and despite the proliferation of AI tools, we lag software, law, sales and marketing, and even education and healthcare.

This essay argues that the missing ingredient for finance is data quality. It shows how a large number of “AI failures” can be traced to bad data quality, either when sending data to AI, or evaluating data that comes from it. And it provides solutions: best practices and mental models to ensure that AI works for you, and works well.

The essay does all of this through a finance lens, centering the very specific requirements and failure modes of our industry — not just Wall Street finance, but Main Street finance as well.

Data quality, AI and finance — good looks, brains and money — what more does one need to succeed? Read on!

Who This Essay Is For

This essay is written for non-technical finance professionals who use AI and/or want to use AI more, especially those who work outside of Wall Street and institutional capital markets.

More specifically:

• If you’re a finance manager or director, a CFO or VP of finance, a wealth advisor or RIA, an analyst or underwriter, an investor, a finance ops lead, an FP&A specialist, a bookkeeper or controller, a family office allocator, or any role along those lines;

• If your work involves complex, detail-oriented, knowledge-intensive tasks where accuracy is paramount;

• If you’ve ever prompted an LLM, used an app that uses AI, tried automating tasks or stitching together workflows, or are just curious and excited about AI;

then this essay is for you.

This essay is also for anyone, in any field, who uses, builds with, or builds on top of AI for professional tasks. Founders and operators, data people, executives and investors, function leaders, informed laypersons, strangers on the internet: the principles in this essay generalize perfectly to your work.

This essay is not written for specialist AI researchers and builders. Training and improving foundation models is a massive and insanely rapidly-evolving field with its own nuanced and distinctive body of work on data quality.

Let’s jump in!

Data, Three Ways

It’s helpful to distinguish three ways in which LLMs interact with data quality. There’s data quality for input to AI; data quality for output from AI, and AI as an evaluator of data quality. Let’s look at each of these in turn, starting with input.

Quality is Everywhere …

“Input” is broader than you think. Your prompts are input. Any files or resources you share are input. Connectors are input. Any app you use, that uses AI, is input to AI. Your history is input. Your skills are input. It’s all input, and it’s all data.

This point is often overlooked. We’re used to thinking of data as fields in a database, or documents in a folder. LLMs have a wider view, and treat any and every type of content as data input – facts, figures, codes, writing, images, audio, websites, traces, logs. If it’s bits on a disk, it’s data.

And if it’s data, it’s subject to quality. Any one of these inputs can be high-quality or low-quality, and the difference matters a great deal.

… And Must Be Protected

Almost all historical “quality engineering” has focused on the narrow view of data, as structured fields or documents1. This approach hasn’t kept pace with the way LLMs use data. We need to widen our view.

Garbage in, garbage out remains as true as ever. The problem is that LLM outputs are horribly plausible2. Detecting garbage is easy; detecting plausible untruths is much harder. Which means we have to be even more careful about what we send in.

It’s not just garbage (semantic junk) you have to watch out for, it’s lazy or shoddy or noisy inputs. LLMs excel at taking those, and polishing them, and making assertions with unfounded confidence. Careless in, convincing out3.

The Careful Feeding of Models

Here are a few things you should do to avoid the CICO pattern. Almost all of these can be reduced to a simple philosophy: maintain closer control of fewer but better data inputs.

Curate ruthlessly. Resist the temptation to throw everything into context, just because you can. More is almost always worse.

Provide constant grounding. Define model-independent sources of truth, and reference them across your entire workflow, not just at the beginning or end. Do NOT let your LLM write to your source of truth.

Watch for hidden handoffs. When chaining tools, or building workflows, or using apps, watch for seams where data moves from one AI-powered application to another, unknown and unseen by you. These are where quality degrades.

Stay current. Models are easily confused by version conflicts and stale data. This is especially noticeable in RAG chunks or file uploads, but it shows up everywhere.

Inspect the raw material. Never ever lose the habit of looking at the rawest possible versions of your inputs – files, transcripts, tables.

Beware sycophancy. Prompt quality is part of input data quality, and it’s all too easy for a model to “lead” the user into low-quality prompts that merely maximize engagement and rewards.

Avoid context rot. Even with careful curation, long sessions and complicated projects can lead to confusion, distraction, attention dilution, repetition and poisoning (persistent errors). You can mitigate this with fresh context windows, concise summaries, and single-task focus.

Interestingly, these are all quality actions that don’t really have a great analogue in the pre-LLM world. Quality keeps evolving!

Data Quality for Outputs from AI

Now let’s look at LLM outputs. Just like every input to an LLM can be considered data, so can every output4. We’ve seen how to evaluate (and improve) data quality on the input side; how do we do the same on the output side?

Ask for grounding. The mirror image of providing grounding. Ask for grounding (facts, references, citations, logic) at every stage, not just the final output.

Show your work. LLMs tend to skip steps; over-extrapolate; generalize; hand-wave; make up plausible data. (This is as true for numerical output as it is for written output). Asking models to show their work is a different and equally valuable form of grounding.

Promote dissent. Pay special attention to outliers, contrary opinions, and revised priors. Explicitly ask for steelman and devil’s advocate cases.

Recognize signatures. Train yourself to spot places where AI tends to err. Many people can recognize bad AI writing (“The honest answer — it’s not writing; it’s expression”); similar patterns exist in finance, analysis, and data.

Watch out for these AI tells in particular:

• Spurious precision

• Confusion between inputs, assumptions and outputs

• Internal inconsistency and calculation errors

• Rationalization of suspiciously “clean” results

• Scale and unit errors

• No sense of materiality or proportion

Climbing the Quality Ladder

Notice that most of the above practices operate at the granular (unit) and aggregate (corpus) levels of data quality. But there are some things you should do at the fitness-for-purpose and business-outcome levels as well:

Beware fluency. LLMs are hyper-fluent generalists. They cosplay competence. Don’t mistake their fluency for correctness or expertise. Stay skeptical.

Watch for reward-hacking. Humans are notoriously poor at knowing what’s good for them. It’s easier for an LLM to fool a human into thinking something is good, than it is to actually produce something good.

Know what you don’t know. Estimate confidence and uncertainty, and explicitly match these to your goals and budget. Require your models to do the same.

Beware flattening. LLM outputs, especially at the end of a long process, tend to converge to a mushy middle: homogenized, averaged-out, general-purpose analysis or artefacts. Don’t accept this; constantly steer them away from generalizations and towards specifics.

Don’t anchor. If the output of a model, workflow, app or tool just repeats what you already know, what good is it?

Loop in the humans. Candidly, humans aren’t great at detecting errors. But at least their weaknesses are imperfectly correlated with AI weaknesses, so you get a diversification benefit.

Notice that many of these practices are about overruling intuitions (some would say biases) drawn from working with other human beings — fluency-expertise, agreeableness, self-confidence, alignment, generalization and confirmation.

An Optimistic Interlude

You might read all these examples and conclude that LLMs are hopeless at tasks involving complexity, nuance, or quality. Not so!

LLMs are smart, enthusiastic, energetic generalists. They’re amazing at managing large amounts of information, at building complex artefacts, at breadth across a dizzying range of competences.

For tasks on which they’ve been tuned — like software engineering — using LLMs feels like magic. Not the marketing-copy version of magic, or the stage-trick version of magic: the real thing. Take it from me, LLMs are incredible.

But LLMs are just tools. And like all tools, they’re not perfect; they can be used incorrectly; they can get things wrong.

The point of this essay is to unlock their use. Learn how to use these spectacular tools to their fullest potential. Steer, don’t fear!

Marking Your Own Homework: AI as Evaluator

AI is such a great, general-purpose tool. Why not use it to evaluate its own output? This is the third way that AI interacts with data quality.

The pros are obvious: scale, speed, cost, consistency, stamina, coverage. LLMs can handle more material, faster and cheaper than any human; they know so much, never get tired, never get bored.

The cons are equally obvious: self-certification and untethering. Self-certification is when models audit themselves. LLMs are not very good at this; as mentioned before, they don’t know what they don’t know, and when confronted with contrary evidence, they tend to either double down or oscillate wildly. This not exactly a behaviour that inspires confidence5.

Untethering is a different failure mode, where a closed cognitive loop (input → output → evaluate → more input → more output) leads to models drifting away from reality, detached from their grounding. We’ll talk more about this below.

And of course AI judges have their own biases, blind spots, dependencies, confident errors, and so on. These are bad in themselves; they are also gameable by the evaluatee. So you have to ask, who evaluates the evaluators, and that just leads to infinite regress...

Contamination

There’s a theme running through all the above: contamination.

We’ve talked a lot about the importance of grounding: of connecting your model — both inputs and outputs — to an authoritative external source of truth. But what if that external authority is itself the product of an LLM?6

LLMs are great at producing large quantities of output. LLMs are great at processing large quantities of input. The temptation is to connect the two: to pipe one AI app’s output to another AI app’s input. From there, it’s a small step to having LLMs write to your internal sources of truth. Contamination.

If your LLM — or an app built on AI, which means most apps these days — writes directly to your general ledger, your CRM, your customer metrics, your operating handbook, you should be worried.

Errors, once created, persist and propagate. Downstream AI apps read bad data, assume it’s correct, and write even more bad data back into your files. Nuance and detail are lost; falsehoods and inventions become enshrined as truth; and your models build ever-more elaborate superstructures on ever-more shaky foundations. Soon, your organization is subtly detached from reality; your context has been poisoned7.

The way to avoid this is informational discipline. Isolate your primary sources. If you have genuinely proprietary data, quarantine it now! Flag LLM outputs, keep them separate, be explicit and conscious of when and how you join them. Supervise and inspect. Chain rarely. None of this is new; it’s the same set of practices we’ve been talking about all along. But contamination makes these practices a must-have, not just a nice-to-have.

In Conclusion

This brings us to the end of another Pivotal essay.

Articles about AI are often too vague to be useful, or too specific to last beyond the next model update. In this essay, I’ve tried to find a middle ground, with concrete examples and suggestions but also general principles that stand the test of time. I hope you found it helpful!

Toronto, August 2026

Never miss a post!

Pivotal is a free newsletter of long-form essays on finance, AI, data, startups and tech. I write infrequently and in depth; I like to think my pieces are worth writing, and worth reading.

Subscribe to Pivotal and share it with your friends and colleagues: especially those who work in Wall Street and Main Street finance. Encourage what you want to see more of in the world.

Subscribe now

Introduction

Data quality. We love it, we want it, we praise it, we aspire to it. Even in these benighted and degenerate times, if there’s one belief that unites all sensible individuals, it is the belief that data quality is a Good Thing.

It’s a pity, then, that nobody seems to know what data quality is.

Ask six practitioners to define data quality and you’ll get six different answers. In fact it’s worse than that: give the same data to six practitioners, and you’ll get six different evaluations of its quality. Data is the elephant and we are the blind men of Hindustan.

Fortunately, Pivotal is here to save the day. Today we shall learn all about data quality. Read on!

Standards Are Poor

Let’s start with the “standard” definitions of data quality. They are, unfortunately, not very helpful.

ISO 8000 defines quality data as data that meets its stated requirements. This is one of those tautological statements that is perfectly accurate and completely useless.

ISO 25012 defines data quality using 15 attributes, including all the usual suspects: accuracy, completeness, consistency and so on. This too is correct, but incomplete.

I take a somewhat different approach.

A Modest Assertion

I begin with an assertion: data has no innate quality. Quality is a purely emergent phenomenon, conditional entirely on use case.

Readers of How to Price a Data Asset will recognize this line of thinking. In that essay, I argued that data has no intrinsic value; instead, the value of data is the value of what can be done with it.

Data quality is that which increases data value.

Since data value is a function of usage, so too is data quality. Data quality can only be assessed with reference to what can be done with the data.

We care about data quality precisely because it allows us to do more; do better, faster, cheaper; or just do differently with our data.

This is still a bit abstract and hand-wavy. We’re going to make it more concrete.

Levels of the Game

Our first insight is this: data quality comes in levels. These levels are not separate or mutually exclusive; they exist simultaneously; and much of the noise around data quality stems from level confusion.

These levels are ordered and dependent. Ordered: data quality can pertain to individual record, to data corpus, to application, or to business outcome. And dependent: each level requires the ones below and above, for coherence and usability.

I’ll explain all these terms in a bit, but first, let’s examine the levels and what they cover.

Granular Quality

The first level of data quality is granular or unit-level quality.

Think of an individual “unit” of data – a single database record, or sentence, or question-answer pair, or labeled example. You can test this granular unit for accuracy, precision, recency, well-formed-ness, internal consistency, plausibility, provenance, interpretability, confidence, and more1 . This is what many data quality evaluators do, and where they stop; it’s the realm of ISO 25012, of observability and monitoring.

Two facts jump out. First, all these quality attributes exist at the level of individual units of data. You don’t need to inspect other records to know if a given record is accurate, precise, recent and so on. This is why we call this granular quality. Each unit stands alone.

Second, all these attributes are downstream of clear usage/value questions: is the data true, is it usable, is it current, and is it coherent? And the questions themselves are conditional. True, in what context? Current, relative to what? Usable, how?

Aggregate Quality

The second level of data quality is aggregate or corpus-level quality.

All your individual units or records might be high-quality, but that doesn’t mean your data corpus is high-quality. At corpus level, you care about attributes like coverage, deduplication, granularity, representativeness and balance, cross-record and label consistency, distributions and aggregate statistics, volume and sufficiency, continuity, joinability, and drift.

These are attributes that emerge from your data taken in aggregate; no individual piece suffices to establish these attributes. The questions being addressed here are: is the data all there, is it clean, does it mirror the world, and is it stable over time and space?

These questions too are context and use-case specific: what does “all” mean, how clean is clean enough, what’s the world being mirrored, what are the time and space constraints. Again, the reason we ask these questions is because without knowing the answers, we can’t use and get value from the data.

Fitness for Purpose

The third level of data quality is fitness-for-purpose quality.

Quoting Pivotal4 :

It’s meaningless to talk about data value [and hence data quality – ed.] without specifying how the data will be used. Financial statements aren’t useful for an advertising campaign. Audience profiles aren’t useful for equity analysis. But flip those around, and the datasets are not just useful; they’re essential. The use case is everything.

We’ve already talked about how granular quality and aggregate quality are questions you ask of the data, conditioned by use case. Fitness-for-purpose is where the questions shift to the interaction between data and application.

This takes a couple of different forms. There’s “informational fit”, which includes data relevance, adequacy, sufficiency and necessity – in short, does the data answer the questions you want answered? And there’s “operational fit”, which includes data availability, licensing/compliance, interoperability, and risk/reward calibration – in short, can you use the data effectively?

Business Value

A dataset might have great unit-level quality, excellent corpus-level attributes, and perfect fitness-for-purpose. That’s still no guarantee that it will add business value. You can do everything right, and still fail.

This brings us to our final facet: business-outcome quality. Does the data actually deliver value to the business? Does it lead to higher eval scores, or stickier enterprise revenue, or superior risk-adjusted returns, or better customer conversion? This, ultimately, is what we care about: the value of data, and the measure of its quality, is the value of what we can do with it.

As before, you can break this down into a few questions: was the data used, did it change anything, and was the change worth it?

“Was the data used?” means measuring data adoption, influence on decisions, and delta in actions. “Did it change anything?” means measuring delta in outcomes, attributing it correctly, and judging materiality. And “was the change worth it?” encompasses ROI, timeliness, durability and risk.

And so the answer is not about the data itself. (That’s what the lower levels are for!). The answer is forming better hypotheses about the value the data will deliver5, instrumenting the data-usage-result pathway, and scaling back or doubling down as the results indicate. This is the secret: at the highest level, data quality is not about the data. You have to zoom out.

Quality is a Ladder …

The levels I just described are ordered and dependent. You can’t get to the higher levels of data quality (fitness for purpose, business outcomes) without first traversing the lower levels (granular and aggregate quality). But the lower levels generate no value in themselves. You need both.

This resolves the definitional problem we started with. The failure mode of ISO 25012 is endless checklists, aka getting stuck at the lower levels – “we measured the data against 127 quality dimensions, yet our business remains unimproved; now what?”. The failure mode of ISO 8000 is non-actionable tautologies, aka getting stuck at the higher levels – “this data is good because it does good things; now what?”.

Quality as a ladder is the organizing principle that subsumes and transcends both of these definitions. At lower rungs, ask yourself: am I tunnel-visioned on attributes and neglecting my business use case? At higher rungs, ask yourself: am I tunnel-visioned on results and neglecting foundational hygiene? Everything else follows.

Many disputes on data quality are the result of people operating at and talking about different levels of the ladder. Hence the elephant. It’s hard to tell a meticulous data ops engineer that their perfectly labelled records have no business value; it’s equally hard to tell a visionary CEO that their perfect operating model is built on sketchy input data. The former’s instinctive response to problems is to look for granular fixes; the latter’s is to look for a strategy that works. Neither is a panacea.

… And You Shouldn’t Skip Steps

Good data hygiene means doing all the things: confirming unit-level, corpus-level, fit-for-purpose, and business-outcome quality.

This is hard. And so the temptation is to skip steps. There are two bad ways, and one maybe-okay way, to do this.

First, the two bad ways:

• Failure to launch. Focus too much on the lower rungs of the ladder; build immaculate quality at granular, aggregate and purpose levels; deliver zero business value. This is astonishingly common, probably because it’s easy. The lower rungs are tangible, measurable, easy to impact - in a word, “legible” - and so that’s where people tend to focus.

• Failure to ground. The opposite problem: ignore the lower rungs, and jump straight to solving for business value. If your target is well-defined and your feedback cycle is fast enough, this might work. The rationale here is that the (business) end justifies the (data) means – who cares about correctness, provenance, timeliness et al, as long as the results are good. But this is usually not sustainable; foundations matter.

The maybe-kinda-sorta-okay way is:

• Provenance as proof. Borrow quality from elsewhere; let somebody else do the work. If your source is unimpeachable – if you trust their data implicitly – then you can invest materially less in checking unit-level and corpus-level quality. Meanwhile, fitness-for-purpose can be solved by sticking to vertical-specific providers. (Of course, you still have to generate business-value yourself.)

Note that trust in data sources doesn’t happen by accident; it’s built up over time, with resources, and through results. Above all, it’s endogenously determined. If and as long as the data works, you trust the source; if and when it doesn’t, your trust dissipates.

Taking a Breather

This concludes the first part of this essay:

• why data quality doesn’t really exist on its own;

• how to think about it in layers;

• the quality ladder; and

• how to avoid getting stuck on any one level.

The second part is all about AI. How does AI change our intuitions about data quality? Spoiler: in a bunch of cool, non-obvious, and interesting ways. I explain the traps of data quality for AI, and offer a number of concrete suggestions for avoiding those traps. Read part two here!

Toronto, June 2026.

Never miss a post!

Pivotal is a free newsletter of long-form essays on finance, AI, data, startups and tech. I write infrequently and in depth; I like to think my pieces are worth writing, and worth reading.

Subscribe to Pivotal and share it with your friends and colleagues: especially those who work in Wall Street and Main Street finance. Encourage what you want to see more of in the world.

INTRODUCTION

Data Is The New Moat

There is a growing consensus that data is not the new oil or the new gold; it’s something better. Data is the new moat.

We’re in the middle of a remarkable land-grab in software. LLMs are changing the way software is built, opening up vast new markets previously untouched by tech (especially in services), and making many incumbents look decidedly long in the tooth. This is a generational opportunity for companies to capture market share, and many startups are doing precisely that.

Startups riding the AI wave are reporting growth rates that have never been seen before. Bolt grew to $20M ARR in 2 months; Cursor went from $1M to $100M ARR in 21 months; OpenAI has billions in revenue (and remember, GPT-3 was released less than 5 years ago). Stories of hyper-growth abound.

But so do cautionary tales. Competition is fierce: if you can build a crazy powerful app in a weekend, so can others. Any capabilities you create might be eaten by the next generation of foundation models. Churn is high, sometimes driven by disillusionment when your product doesn’t live up to the hype, sometimes driven by excitement when a competitor releases something that blows your solution away.

Massive opportunities; killer environment. What’s a company to do? Moats.

In warfare, moats are what prevent a castle from being stormed. In business, they’re what prevent a company from being overrun by competition, engulfment, or slow decay. Moats help you acquire and retain customers; they help you outperform and undercut rivals; they help you buy low and sell high, move fast and play bigger; they help you win, and keep winning.

Some moats are known and loved: network effects, user lock-in and switching costs, brand and positioning, process power, unique IP, economies of scale. Others are obscure or questionable.2

But with the advent of AI, a new kind of moat has come to prominence: the data moat. And this is not a coincidence.

AI companies have a special resonance with data moats, because data and AI are two sides of the same coin. LLMs require vast amounts of data, for training, fine-tuning, learning, reasoning. And LLMs unlock the value of data like almost no technology before. It’s a match made in business-model heaven.3

All the old moats are still valid — brand, for example, or network effects. You can (and should) build them. But they’re orthogonal to and independent of AI, in way that data moats are not. Data moats reinforce AI advantages, and AI advantages reinforce data moats.

What Even Is A Data Moat?

Everybody’s talking about how to build a data moat. Moats are in the air.4

Unfortunately, much of this conversation is ill-posed. There is an incredible amount of incomplete, inconsistent, or simply outdated thinking about data moats and how they work.

Classic mistakes in this vein include thinking data is a moat when it isn’t; relying too much on weak data moats; confusing other moats (like scale) for data moats; misunderstanding which attributes of data contribute to its “moatiness”; failing to distinguish between software moats and data moats; and not realizing when a data moat has lost its effectiveness. The Underpants Gnomes remain undefeated.5

This essay aims to bring some structure, rigour, and current best practices to the discussion. I’ll define a few different categories of data moats, explain when and how (and if!) they work, and explore some tactics to maximize their potency.

There will be case studies! Rampant speculation! Counter-intuitive conclusions! Clever turns of phrase which I hope go viral! And snarky asides!6 Read on.

Control And Loops

Let’s begin with some taxonomy.

I posit that there are two and exactly two categories of data moat. I call them data control and data loops; every type of data advantage can be put into one or both of these categories.

• Data Control. If you have sole control of a critical asset, you have a moat. In the world of data, this control comes in various forms: uniqueness, aggregation, movement, usage, records, action, catalysis and more. We’ll learn about data control in Part One of this essay.

• Data Loops. Many well-known business moats rely on positive feedback loops7 that accelerate some core business dynamic — for example, marketplace gravity, user network effects, protocol adoption. A number of data moats exhibit the same pattern. We’ll learn about data loops in Part Two of this essay,

PART ONE: DATA CONTROL

Controlling data really means controlling the value of data; moats arise when nobody else can access this value.

There are a few ways to do this: you can control the production and ownership of (unique) data; you can control the movement of data, either internally or externally; or you can control the usage of data, through technological or other means. Create value; transport value; extract value: each of these approaches can lead to a data moat.

A pre-condition here is that the data being controlled has to be meaningful. Otherwise, “control” is pointless: you can’t make money from tolls on a road going nowhere.

Unique And Proprietary Data

Okay, so you produce, capture or own unique data. Maybe the data’s about product usage, or customer behaviour, or industry dynamics, or something else. Do you have a moat? Probably not.

Unique data is neither necessary nor sufficient to establish a data moat. It’s not necessary because, as we shall see, there are other (often better) methods: controlling movement or usage of data, and building data loops. And it’s not sufficient, thanks to the “meaningfulness” criterion mentioned above. What does this criterion entail?

• The data must offer substantial value to either you or your customers. A small delta in value means you can be overtaken by someone who outperforms you on other fronts, even without this data; a large delta means you cannot.

• The data must be genuinely rivalrous. Your using it should prevent others from using it, or at least, from getting the same value from it.

• The data must have no functional substitutes. Competitors should not be able to achieve similar outcomes no matter what data they use, similar or not.

Most datasets don’t satisfy even one of these conditions, let alone all three. But if all three conditions are met — if you have unique, high-value, irreplaceable data that only you can use — then you may have a moat.8

Historically, there have been a few ways to get such data:

• As a by-product of your core business. (This is sometimes called exhaust data, by analogy with exhaust from a combustion engine). A good example here is stock market data, captured by NYSE and NASDAQ as a by-product of their core exchange business. But this is not a data moat. A larger core business might result in more (or better) exhaust data, but the converse is not true: NYSE’s data sales don’t give their exchange business any “extra” defensibility.9

• Via process power. Many well-established data businesses follow this template. Think of Factset’s financial statement data, or Moody’s ratings data, or Nielsen’s media consumption data: they all depend on decades of expertise working with specific datasets and internalizing all their nuances. This is a moat; whether you want to call it a data moat or a process moat is a matter of semantics.

• Through brute force investment of time and resources. Some examples of this pattern include search engines crawling the web, logistics and delivery firms mapping roads, and robotaxis recording driver-environment interactions. In each case, companies acquire data that becomes the foundation of their technology and hence their business model, that others cannot easily replicate. These are genuine data moats. But …

Brute Force Is Dead ...

The business theory behind brute force is “my capex is your barrier to entry”. Firms spend time and resources acquiring data before their rivals do, and use that data to gain market dominance.

Unfortunately, this works less well these days:

• LLMs make data acquisition easier. Not just a little easier, orders of magnitude easier. You don’t need 100s of curators working 1000s of hours, just tell an AI agent to go get the data for you. Companies that spent years building complex human-mediated data pipelines must now contend with upstarts who can replicate 99% of their work for 1% of the cost. Synthetic data is another end-run around brute-force methods.10

• Capital for data acquisition is cheap. Funding markets have fully internalized “the bitter lesson” and “the unreasonable effectiveness of data”. As a result, the capital to finance brute force data acquisition is cheaper than ever. Brute force is ultimately a bet on market timing and the cost of capital; change those, and the tactic vaporizes.11

• Knowledge diffuses, capabilities increase. The planet-scale infrastructure that Google built in the 2000s, that allowed them to crawl and index the entire internet fast, was a major moat for them (though PageRank gets all the credit). Today, there are dozens of firms who can do the same, and much cheaper. Knowledge diffuses, tools improve, hyperscalers service-ify anything computable, and Moore’s Law marches inexorably onwards. Yesterday’s edge is today’s commodity.

Using an expansive definition of “data” puts this effect in stark relief. Studio Ghibli spent decades painstakingly perfecting a gorgeous visual style that no other animation studio could replicate: the very essence of brute force to generate unique content.

And then last month, ChatGPT blew the doors open for anybody to create their own Ghibli-fied art.

What this implies for creativity, intellectual property, democratization and artist economics is beyond the scope of this essay, but the underlying theme — that LLMs make mere “effort” less of a moat — is clear.

... Long Live Brute Force!

So are moats from brute force data acquisition a thing of the past? Not quite. There are still a few cases where brute force has its place:

• In industries where that last 1% of accuracy or coverage or quality makes a meaningful difference — for example, finance. (And maybe art! TBD.)

• Upstream of LLMs — for example, labelling data for LLMs to train on (Scale, Mercor, and friends), synthetic data pipelines, proprietary evals and so on.

• In domains where LLMs are currently disadvantaged — for example, ‘real world’ data acquisition — audio, video, physics, bio. (This won’t last, btw) (which is also why there’s a land-grab happening here).

The other interesting play here is to recognize that sure, brute force isn’t a long-term moat, but if you have a funding advantage, you can use it to establish first-mover advantage, and then find your moat elsewhere — perhaps via workflow lock-in, or non-data network effects, or platform status, or brand. We’ll come back to this point.

Fragmented Data

A second powerful way to control data is to be the clearinghouse; the central repository that unifies fragmented data assets or data value.

This is a well-known pattern for data businesses; indeed, it’s probably the default mode for them. Think of Bloomberg, LexisNexis and CoStar: clearinghouses for financial, legal and real-estate data, respectively.

The very best clearinghouses add so much value to their fragmented source data — aggregating, harmonizing, licensing, transforming — that they essentially create new proprietary data value of their own. In other words, datasets that are low-value and commoditized in isolation, become high-value and unique in aggregate.

Clearinghouse / unification is a genuine data moat, combining elements of uniqueness, process power and brute force. And it’s not just for specialist data businesses! An interesting recent phenomenon is the emergence of a similar pattern for non-data businesses.

The idea is to aggregate / unify fragmented data, but then monetize it through a software offering. Here are some examples:

• Rippling, Gusto and Remote.com aggregate regulatory data on payroll, contractors, benefits, tax etc. from dozens of countries. This allows them to offer “unified global payroll” as a service.

• Stripe and Adyen do the same thing for “global payment processing”, again incorporating local rules and regulations on tax, authentication, KYC and AML, data privacy, reporting, dispute resolution etc., not to mention local customer habits and preferences.

• Aggregators like Kayak and Skyscanner, and OTAs like Expedia and Booking unify flight information, prices, seats and other travel data from multiple airlines, from GDS providers, and from each other, but they don’t sell the data. Instead, OTAs sell tickets, while aggregators charge referral fees.

• Zapier and Plaid aggregate information on “interfaces”, for SaaS tools and bank accounts respectively, which they can then offer unified access to. Plaid wraps multiple bank websites into a single API; Zapier wraps multiple SaaS APIs into a single no-code tool.

• Numeral and Quandri, both early stage startups, focus on state-by-state sales tax and personal insurance lines, respectively. These are highly fragmented regimes with lots of interaction effects, making data unification super valuable.

In each case, the data collected is both essential to the business offering, and difficult for others to collect: a data moat.

There’s a subtlety here. Firms like Airbnb, Shopify and Uber also collect data from around the world, but their customers don’t really care: any individual transaction is localized, and doesn’t benefit from aggregation. For clearinghouse effects to kick in, there needs to be a one-to-many relationship between the customer use case and the fragmented data.

Information Produces Action

Zooming out, what are we unifying here? Data, yes, but which data, and why?

Essentially, this is data that is a substrate for action. Rules and regulations have to be complied with, and they’re mostly deterministic: they lead directly to action. Flight info is both necessary and sufficient for the action of booking tickets; sales intel likewise for conducting outreach. Information produces action.12

Data unification is genuinely hard, which makes it a good moat. And data unification with an action layer on top often requires domain specialization, which makes it even moatier.

Sales enrichment shows how the frontier changes with time. The progression from D&B to Clearbit to ZoomInfo to Apollo to Clay is a story of value being captured from unifying fragmented data, making it API-accessible, exploiting network effects, adding workflows, and layering in AI actions, respectively.

Controlling Data Movement

This is a blurry category. Partly because of boundaries: where does data movement end, and data usage begin? And partly because of overlaps: companies with this type of control often achieve it via a combination of tactics, both data and non-data. Let’s look at some examples!

Visa has perhaps the single most famous network effect in all of business13. An intriguing (and perhaps provocative) claim is that their network effect is largely based on the control of data movement.

Consider: when I buy something with my credit card, money doesn’t actually flow. Instead, data flows: customer name and verification, transaction details, credit limits, outstanding balances, merchant and bank IDs, payment schedules. The money flows much later, and not 1-for-1 either.

The Visa network orchestrates all of this — across consumer, merchant, payment gateway, payment processor, acquiring bank, issuing bank, and more. The nodes and edges are the network effect; the knowledge (and control) of these nodes and edges is the data moat.

Any middleman business is vulnerable to disintermediation. In Visa’s case, merchants could communicate directly with banks to inquire about customer credit scores. But they don’t! Visa controls that interaction. Visa controls almost all data movement across the network. Abstracting away the (immense) complexity of the network is precisely what brings participants on board and keeps them from defecting; in a very real way, data control underlies the network effect.

Amadeus and Sabre are like specialized Visas for the travel industry, controlling the flow of data (inventory, bookings, identities) across airlines, hotels, car rentals, travel agents and aggregators, and of course travellers. Change Healthcare is a Visa for the healthcare industry, controlling the flow of data (and payments) between patients, healthcare providers, insurance companies, and government payers programs.

Businesses built on controlling external data movement are lucrative, but rare. More common are businesses that manage the flow of data internally for their clients. Do they have moats?

Usually, no. Managing the movement of data is not the same as controlling it. Moving data around is what 99% of software tools do, and most such tools self-evidently do not have moats.

The exception is data flow in highly regulated industries. In healthcare, for example, patient data is highly sensitive and you can’t just access it or move it around willy-nilly. So you have firms like Epic, that specialize in managing internal data access (this is also a system-of-record effect; see below), and firms like Datavant, that specialize in transporting data between organizations while remaining secure and privacy-compliant (via a data-standards effect; see below again). It’s not so easy to rip these out: there’s limited upside and large downside, so most clients stick instead of twisting.14

Data Usage

The final type of data control is the most powerful: when you control the usage of data. This is a large sub-category, encompassing systems of record and action, catalyst data, and exogenous data moats.

System Of Record

System of Record (“SoR”) is one of the oldest, best-known and most effective data moats.

In any large organization, information is scattered. It’s scattered across excel files and databases, emails and slack channels, PDFs and decks, user manuals and policy handbooks, contracts and filings.

There is substantial defensibility in owning the platform that collates all this scattered information. Such a platform is called a “system of record”, and the goal is for it to become the “single source of truth” for the organization. Data is piped into the SoR; queries are addressed to the SoR; answers come from the SoR. If it’s in the SoR, you can assume it’s true; if it isn’t, you cannot.15

The canonical example of a SoR is Salesforce, which collates everything businesses need to know about their customer and sales pipelines: dates, contact information, interaction history, pipeline stage, expected value, progress from opportunity to close, marketing campaigns, customer service and case management, and more (much much more). Entire sales orgs run on Salesforce; they cannot run without it.

Salesforce has analogies in other domains. Indeed, this is typical for SoRs. It’s impractical and inefficient to unify all an organization’s data in a single SoR; instead, there are separate SoRs for each function. Salesforce is aimed at the sales function; Oracle does financial management, Workday does human resources, Quickbooks does accounting, Ariba does supply chain and so on. Each of these SoRs acts as the single source of truth for their specific function.161718

It’s instructive to look at the founding years of some of these SoR companies:

• SAP: 1972

• Oracle: 1977

• Epic: 1979

• Quickbooks: 1983

• Ariba: 1996

• Zoho: 1996

• Salesforce: 1999

That’s pretty staggering. These are incredibly long-lived businesses — especially given the pace of change in software; they must have incredible moats.

Why Are SoRs So Sticky?

Salesforce is not, shall we say, a widely loved product. Nobody is passionate about their Salesforce instance. But they can’t live without it; Salesforce is as sticky as hell.

Why so? Because Salesforce controls data usage. If you need “accurate” data about your prospects and customers you have to get it from Salesforce19. Without Salesforce, you can’t do anything: you can’t email prospects, update their status, understand their needs, close a contract, support them post-close, model your pipeline, run a campaign, anything. Salesforce has a monopoly on your internal sales data.

What’s more, this privileged position means that most Salesforce instances have years of “workflow barnacles” adhering to them. These are both procedural: rules that sales and marketing and customer success staff have to follow — and technological: Salesforce has a whole app store’s worth of third-party tools to read, write, modify, visualize, present, and analyze its data.

Salesforce isn’t great, but the thought of how much work it would take to rip out an instance, export all its data, load it elsewhere, replicate all that app functionality, transition all the users, and get back to full productive flow, intimidates almost all switchers. And this goes for most at-scale SoRs, which is why they’re so moated.20

Sticky No More?

And then came LLMs. It turns out that exporting data from a SoR is just the kind of tedious task that AI agents excel at.

One of my favourite go-to-market motions from recent years is where would-be SoR disruptors offer to do all the migration work themselves: we’ll export the data, put it into the new system, add in new app hooks, you name it. The key insight here is that this is not a “risk”, it’s just a “cost”, and vendors are happy to pay that cost in return for years of LTV.

Ironically, it’s probably better for this GTM if the cost doesn’t go down too much. The difficulty of migrating SoRs leads to low churn rates and high LTVs, which justifies the investment in doing the migration work as a vendor — but if migration costs drop too far, others can do the same to you; churn will tick back up, meaning LTVs drop, and soon you’re in a race to the bottom. Data viscosity is your friend, until it isn’t.

Systems Of Action

Systems of Record have a monopoly on internal data. This control makes them both valuable and defensible. But what if they could do more?

After all, the value of data lies, solely and entirely, in the value of what can be done with it. You don’t want your SoR to be the place where data goes to die21 ; you want to act on the data.

This thought leads to the next, perhaps even more powerful type of data moat: the System of Action (“SoA”).

Systems of Action don’t just store data passively; they enable actions on top of it. The key, and what sets SoAs apart from SoRs, is how specific these actions are: how well the action layer coheres with the data layer and the user function.

An example will make this clear. Consider “version control” — the process of maintaining a software code base. This is a classic System of Record: you want a centralized repository of all your source code with no ambiguity about what’s in production, what’s under development, and what’s deprecated. The earliest version control systems did this, and not much more; they were essentially just file management systems.

Over time, it became clear that version control required specialized hooks corresponding to the needs of code management: “locking” (so two programmers don’t work on the same file at the same time), “deltas” (store only the differences between successive versions, for efficiency), “repos” (manage groups of files instead of piecemeal), and “checkin/checkout” (another way to solve the conflicts problem). These were still SoR-ish; still fundamentally based on managing records.

But as codebases (and programming teams) expanded, even more functions became necessary. Modern version control systems offer atomic commits, complex branching logic, pull requests, “blame” and other collaboration features, hosting, CI/CD integrations, social profiles, metadata and asset management, co-pilots, and much more. Calling Github or Gitlab a mere SoR vastly undersells how tightly embedded — and finely tuned these systems are with every aspect of a programmer’s productive life. Systems of Action.

Agentic Systems

The obvious next step is for the system to take action itself. Wait ... whose music is that I hear?

Enter LLMs.

Systems-of-record store data; systems-of-action empower humans to act on data; systems-of-agents act on data themselves.

Let’s continue with the example above. What’s the primary action that humans take, on top of a software SoR / code repository? Why, programming, of course!

It turns out that LLMs are very very good at programming.

Possibly the hottest area within the supernova that is AI, is LLM code agents. Multiple firms are staking out this space, each with different lines of attack: “virtual junior devs” like Devin, in-IDE tools like Replit’s Ghostwriter, workflow solutions like Cursor, Bolt and Lovable, and of course SoR-level agents like Github Co-pilot.

Which approach will win? That’s the ten-billion-dollar question22 . I like this summary of the disruptor thesis:

The counter-thesis is that SoR owners will block these apps, build them on their own, and fast-follow any user-facing improvements; and their head-starts in data and distribution will suffice to win the market.23

Exogenous Control

The next set of data moats is a catch-all category that I call exogenous control. In this pattern, you control the usage of data, not through any attribute of the data itself, nor even of the software that manages the data, but instead through external carrots or sticks. Here are some examples:

• IP Rights: it doesn’t matter if the data is unique, or fragmented, in a SoR, protected by process power, or none of the above: if you have exclusive IP rights to its usage, you control it. Consider the S&P 500 Index: the underlying data is public, and the index itself is trivially easy to replicate, yet S&P Global makes ~$1B a year from licensing it — to asset managers (for benchmarking and ETFs), exchanges (for index-linked derivatives) and banks (for structured products).24

• Contractual Monopolies: where you corner a primary data source via a favourable contract. Impossible in an efficient market, but data markets are not efficient; datasets are often (massively) mispriced. Unfortunately, this moat is temporary: if the data turns out to hold value, the contract will almost certainly get renegotiated on renewal. The best strategy here is to use your contractual moat to buy time to establish other sources of defensibility; IQVIA did precisely this for pharmacy data, while Neustar failed to do this for telecom data.

• Regulatory and Compliance Moats: If the government mandates that people use a particular dataset, that’s a moat for the company owning, controlling, or implementing that data. A good case study here is the ENERGY STAR program: companies like ICFI, Leidos, DNV and Guidehouse make massive annual revenue providing these certifications to both government and private sector customers. Similar patterns exist for other programs like CAFE, air and water quality, FDA labeling ...

You’ll notice that exogenous control is often linked to government action. And government action is strongly inertial: hard to get started, but even harder to stop once started. State-sponsored data moats! 25

Catalyst Data

A specific flavour of unique data that’s worth calling out is catalyst data: data whose value comes from enabling or activating the usage of other data. This category is interesting because it’s a type of “indirect control” — you don’t need to control the enabled data directly, it may not be unique or proprietary to you, but you do control the ability to extract value from it — which means you can capture disproportionate economics.

Here are a few examples; in each case, “activates” is a synonym for “materially increases the value of”.

• Google: user intent data activates search results data

• Amazon: purchase history data activates product listings data

• Acxiom: customer profile data activates basic marketing lists

• Any social media company: viewer history activates new content

• CUSIP, DUNS, LiveRamp, Datavant: unique identifiers activate siloed intel

• FICO, Nielsen, ratings agencies, IQVIA: consensus benchmarks activate unanchored performance data

In each example, the second dataset has some baseline value in itself, but is made very much more valuable by the addition of the first. Indeed, you could argue that the above companies grew to dominate their respective industries precisely because they were the first to figure out how to unlock the value of the “enabled” datasets.26

An interesting aspect of catalyst data is that, empirically, it seems to result in winner-take-all or at least winner-take-most markets. This is partly survivorship bias: after all, you never hear of the catalyst datasets that don’t result in huge outcomes.

But it also reflect two patterns. First, catalyst data, when it works, tends to work really well — it adds substantial value to (untapped but often very lucrative) data assets. Second, catalyst data often works in sync with various data loops: industry standards, consensus benchmarks, user network effects and so on. We’ll explore this in more detail later in this essay.

Data Control, Summarized

Here’s a handy chart summarizing what we’ve learned so far:

Interlude

We’re about halfway through this essay. If you haven’t subscribed to my newsletter yet, now is a great time to do so.

Subscribe now

I write infrequent, in-depth, original explorations of topics that I have substantial professional expertise in: data, investing, and startups. Learn more here.

PART TWO: DATA LOOPS

The second major category of data moat is the data loop: a positive feedback process that links data and business value in a virtuous cycle. Data improves the business, and the business improves the data (for some value of the word “improve”), and the flywheel spins sufficiently fast that no competitor can catch up.

This, for many, is the most familiar form of data moat27. It’s also the most misunderstood. Some data loops make strong and certain moats. Others are weak, have limited scale, or hidden vulnerabilities. Still others are effective, but they’re not data moats at all; they rely on scale or network effects, and you could take the data part out with no loss.

There are three main families of data loop: the quantity loop, the learning loop, and the usage/value loop. Let’s dig deeper into each of these.

Quantity Loops

The quantity loop is the simplest family of data loop: data attracts data. This can occur through quite a few different mechanisms:

User-Generated-Content (UGC) Loop

This is the loop that drives Facebook, Youtube, Instagram, TikTok, X, and even LinkedIn. All these platforms host user-generated content for free: photos, videos, posts, resumes. This content attracts other users, who post even more content. The more content, the more users; also, the more content, the better the recommendations, and hence again, the more users. The presence of all these users (and their attention!) attracts advertisers, who subsidize all of this.

Content, of course, is just another word for data. This is a perfect data quantity loop, and once it reaches maturity, both lucrative and very hard to displace.

Equally fascinating are the firms that fizzled after reaching non-trivial scale using this loop. Consider MySpace, Tumblr, Quora, Vine, Digg, and perhaps Stack Overflow28. What happened to their (supposed) data moats?

The reasons are varied — failure is overdetermined! — disastrous M&A (MySpace by News Corp, Tumblr by Yahoo), failure to monetize effectively (Quora’s paywall), self-inflicted wounds (Tumblr’s NSFW ban), product missteps and technical debt (most of them), and competing with the apex predator that is Facebook (all of them). 2930

But fundamentally, these are just triggers. The problem with the UGC data loop is that it can reverse just as fast as it builds. Everybody goes where the lights are brightest; and conversely, everybody flees the ghost town. Missteps, if not quickly reversed, become death sentences; momentum is a fickle friend. So this moat is deceptive: more vulnerable than it appears.

Search-Engine-Optimization (SEO) Loop

Once scaled, the classic UGC loop tends to lead to “walled gardens” of content, that users never leave or indeed want to leave. But there’s another, very similar loop where users are constantly re-acquired, and SEO is the engine of this re-acquisition.

This is the SEO data loop. Users create content, or the platform itself generates content programmatically; other users, looking for that specific content, find their way to the platform via Google or another search engine.31

The phrase “looking for that specific content” is important. Unlike the UGC loop, the SEO loop is task-oriented, not feed-oriented. The content has to be useful, and actually answer the user’s search query.

What are some examples of useful content? It’s a wide spectrum:

• Reddit and Quora answer specific questions

• Expedia, Booking, Kayak et al provide travel information and actions

• Yelp and TripAdvisor provide service reviews

• Zillow does house prices

• Glassdoor and LinkedIn cover various aspects of professional life

These companies monetize in different ways. The broad horizontal platforms tend to monetize via ads, while the vertical ones mostly do affiliate or lead-gen. And then there are a few that monetize via subscriptions or services. Lead-gen, in particular, is sufficiently lucrative — think insurance, financial products, legal services, healthcare, education, travel, home maintenance — that there’s a whole cottage industry of service directories that exist solely to aggregate provider data, capture Google traffic, and take an introducer’s fee.

In each case, adding more data gives these sites more search equity, leading to more traffic, and hence (either directly or indirectly) more data. Flywheel unlocked, moat established.

How moaty is this moat? In the Google Era — roughly speaking, 2005 to 2020 — it was pretty damn moaty. Multiple billion-dollar businesses were built, and defended, using this loop.

But this era might be coming to an end, along with this moat. One reason is over-saturation: there’s so much AI slop out there, Google search is simply less useful32 33. Another is dis-intermediation: LLMs are already replacing search for purely informational queries, and agents may soon do the same for navigational and transactional queries. This would bypass the entire search-learn-select-purchase funnel on which the SEO loop is built.34 A final headwind for the SEO loop is the migration of content behind paywalls and logins. People are locking down their data assets; farewell, fully-open web.

Stack Overflow offers a cautionary tale. This graph of monthly questions posted on the popular programming site, created by Gergely Orosz based on data from Theodore Smith, says it all:

There’s an initial steep rise fuelled by search traffic. Then there’s an almost ten-year period of stability, despite minimal changes to the product. Surely competitors should have emerged during that window? The fact that they didn’t is testament to the power of this moat.

And indeed, the final decline isn’t due to direct competition; it’s because the world suddenly bypassed Google + Stack Overflow as the best way to answer coding questions. The dates line up almost perfectly: Stack Overflow’s traffic peaks in May 2020, and then in June, OpenAI releases GPT-3 ...

SaaS Data Gravity

Stars form by accretion. Amidst vast clouds of diffuse cosmic dust, local concentrations of matter attract more matter, in a gravitational spiral that ends with temperatures and pressures high enough to initiate nuclear fusion.

Empirically, software exhibits a similar pattern: diffusion is replaced by concentration. Here’s how it works.

A lot of software is initially sold via a “wedge” — a tightly-scoped, low-cost, low-risk insertion into the target customer’s stack. Step two is to “land-and-expand” into more users, applications, and revenue; often, this entails “going multi-product”.

But which software tools are able to expand, and which ones fizzle? The analogy with star formation provides the answer: the winners are those that already have local concentrations.

Sometimes, it’s a local concentration of “workflows” — customers like the productivity boost from combining multiple tools into one, but they hate switching their processes, so whichever platform owns the most frequent or most important workflows will tend to swallow the others.

And sometimes, it’s a local concentration of “data”. This is data gravity.

Tools that control the most important or valuable data tend to swallow tools whose data is peripheral or less productive or worse integrated.

Toast is a great case study here. Toast started out as a point-of-sale (POS) system for restaurants. Owning front-of-house order data gave them pole position to expand into kitchen display tickets, online and mobile ordering, consumer-facing apps, delivery integrations, gift cards, payments — and then, eventually, into restaurant financing, payroll, HR and more. All those other functions had competitor apps with their own data, but Toast’s POS data was both the most important and the most central; its gravity enabled Toast to swallow the rest.35

Data gravity moats are similar to, but not quite the same as system-of-record moats. Both moats benefit from workflow lockin, data viscosity, and controlling usage. Where they differ is the dynamics.

There’s no inherent requirement or expectation of growth in a SoR: they are perfectly sticky even when static. But growth is everything for a data gravity loop: growth in data coverage, in product use cases, in audience, in surface area.36

Data gravity in vertical saas is one of the best data moats out there, not least because it feeds into many other (non-data) moats: workflow, trust, revenue control and network effects.

Give-To-Get (G2G) Loop

This is a well-known pattern for data businesses. In a G2G loop, customers of the business receive data if (and sometimes only if) they also contribute data to the business. The more data the business has, the more attractive it is to customers — and hence the more likely they will sign up and contribute data! Positive feedback thus kicks in above a certain initial critical mass.

A pure give-to-get loop is when the data contributed is the same as the data received; and value is created by the act of aggregation. Waze is a great example: users contribute (individual) and receive (aggregated) traffic data. Geographic aggregation in fact seems to be a common theme for G2G models — think of OpenStreetMap, Weather Underground and GasBuddy.

Data consortia (aka data co-ops) are a special case of G2G where a group of competing businesses decide to pool their data resources to generate insight unavailable to them individually — and thus outperform rivals outside the group. The canonical example here is Verisk. Verisk’s ISO (Insurance Sales Office) served as “neutral territory” for a consortium of 7 large insurers; they aggregated and standardized data from these 7 firms, leading to better underwriting for all of them.

Marketplaces with 2-way review systems (like Airbnb — hosts review guests, guests review hosts) are also give-to-get, but here the value isn’t in aggregation, it’s in eliminating selection bias in the reviews, and hence fostering trust.

The theme of eliminating bias is why benchmarking is a very common application area for give-to-get data models. Dun & Bradstreet does this for B2B credit; PayScale does it for salaries; Glassdoor does it for employer reputation; Cambridge Associates does it for private fund performance; and so on.

Of course, the very best examples of benchmarking using a give-to-get data model are the big consulting firms: McKinsey, BCG, Bain et al. The G2G part isn’t explicit, and these certainly aren’t software or data firms, but the loop (and the moat) is very real. (The data in question is sometimes called “best practices”.)

Give-to-get is pretty moaty once you reach critical mass. The hard part is getting there: it’s a classic “cold start” problem, with challenges and solutions that are very reminiscent of the 0-to-1 step for marketplace businesses.37

Anonymization

Aggregation is not the only benefit of clearinghouse and give-to-get models. Anonymization also matters. This is for competitive reasons — there will always be specific business details that companies don’t want to share with competitors, but sometimes it’s hard to “give” data without also giving away those details. So you need a neutral party who can navigate those nuances. And it’s for regulatory reasons — privacy laws dictate what information companies can and cannot share, and adding an anonymity layer makes the process much easier, safer, and more compliant. This isn’t really a moat in itself, but it makes the actual moat (clearinghouse or quantity loop) more resilient.

Learning Loops

The learning loop is the next major family of data loop. Businesses use (“learn from”) data to run better; and running better helps them get more and better data. I wrote about this flywheel in Data in the Age of AI; here’s a stylized view:

This loop works, and at scale, it works extraordinarily well. But it is not a moat.

In fact, “data learning loops” are tied with “unique data” for first place in the category of “moats that aren’t really moats”. I already talked about unique data; but why is learning so un-moaty?38

It’s a question of limits. Business efficiency doesn’t scale indefinitely with data inputs; instead, the value of learning plateaus above a certain level. Meanwhile, costs show the opposite pattern: long-tail and edge-case effects kick in at scale, making “move-the-needle” data collection more expensive. Acquisition costs go up and marginal data value goes down; in other words, advantages diminish with scale instead of accelerating. This is no moat.

There are two and a half exceptions.

The first is what I call the business model unlock: when your learning loop reaches a magic threshold that enables a business model that simply isn’t possible without that learning. Threshold effects are important here: you need a discontinuity in the customer value function for this moat to work.

Amazon Prime is a good example of the business model unlock. Once Amazon acquired enough data (“learned”) about customer behaviour, purchase patterns, order frequency, warehouse location, inventory management, delivery scheduling, route optimization and so on, it was able to offer “free” 2-day delivery. This unlocked multiple virtuous cycles (in both time — order frequency and packed routes — and space — warehouse and driver density), which demolished every other horizontal marketplace. The data didn’t merely drive an iterative, incremental, quantitative improvement in Amazon’s operations (the classic, un-moaty learning loop); it enabled a whole new service that competitors simply could not match. An incredible moat.

The second exception is data businesses. It’s not that (increasing) cost and (diminishing) value effects don’t apply to data businesses; it’s just that the data gives them access to a bunch of other levers (unique data products, easier GTM, price discrimination, ecosystem tactics) that counter these effects. I wrote about these levers in The Economics of Data Businesses.

The half-exception is AI. In AI, outcomes do appear to scale indefinitely with data inputs, taking away that particular limit. On the other hand, there’s no actual loop here; it all happens during pre-training. Recent advances in test-time inference (and especially the tantalizing prospect of cross-user / cross-session learning) could change this, but we’re not quite there — yet.39

Secondary Learning Loops

There also exist “non-core” or “secondary” data learning loops, which are even weaker and less moat-y than core learning loops. I include them here for the sake of completeness:

• Data quality loop: somewhat valuable, but quality is not a moat.

• Product recommendation loop: eigenvalues and e-commerce are a cool combination, but also not a moat.

• Product optimization loop: lol, no. A/B testing is not even a shallow ditch.40

I’ve heard each of these being referred to as a material source of defensibility. Nope.

If all these data learning loops are so weak, why do they have such a hold on the popular imagination? I suspect the reason is largely political. “We win because we use data and technology to provide a better service” is a much better sell than “we win because we charge monopolistic tolls on attention, commerce, and devices”.

Aside: The Barton And Von Ahn Loops

I want to highlight two data-loop entrepreneurs: Rich Barton and Luis von Ahn. They’ve each founded multiple decacorns or decacorn equivalents, using clear and distinctive approaches:

• The Rich Barton Playbook: find an industry with knowledge that is valuable, fragmented, and opaque; make this hidden knowledge public; dominate search traffic; own “customer demand” in the industry; repeat. He’s done this for travel (Expedia), employment (Glassdoor), and homebuying (Zillow). Kevin Kwok wrote an excellent essay about this.

• The Luis von Ahn Playbook: find a labelling problem where users get utility from the act of labelling — aka the two-sided learning loop. In Duolingo, users learn a language but also translate untranslated books. In Recaptcha, users authenticate themselves, but also label ambiguous images. (Mercor, not a von Ahn company, appears to be working along similar lines.)

What other repeatable data-moat playbooks exist? In The Economics of Data Businesses, I hypothesized that we’d see the emergence of lots of niche, vertical-specific, B2B data businesses. A year later, Travis May launched Shaper Capital to incubate precisely that style of business — “solving data fragmentation in various industries”, as he previously did at unicorns LiveRamp (identity) and Datavant (health). I’ll be watching the Travis May Playbook with great interest.

The Bootstrap And Switch

While learning loops may not be a sustainable moat in and of themselves, they are still valuable as the first leg of an extremely effective pattern that I call the bootstrap and switch.

In this pattern, you begin with a data learning loop to improve your product and grow your customer base; but as you reach scale, you build out other network effects and defensibilities that become your long-term moat. This is seen most often in content businesses (Facebook, Netflix, Youtube), and in marketplaces (Doordash, Uber, Airbnb).

Whenever recommendation or matching plays an important role in the value added by a platform — as in the examples listed above — learning becomes super valuable, and drives rapid growth. But the long term moats for those businesses come from network density, user trust, economies of scale, and attention-aggregation. Not from learning.

Usage/Value Loops

The next type of data loop, and possibly my favourite, is the usage/value loop: the more widely a particular data asset is used, the more valuable it becomes to its users. Whoever owns or controls this data asset mints money.

This loop comes in a few different flavours:

Data Exchange Standards

You provide an industry-wide “primary key” that can link records held by different companies. Factset’s CUSIP identifier is a good example: it allows capital markets participants to agree on precisely which securities they’re trading, clearly and unambiguously.

In a financialized world, with multiple debt and equity classes, listing venues, and security types, this is critical: firms simply cannot function in capital markets without CUSIP. And the more people use it, the more valuable / universal / lucrative / essential it becomes. It’s a toll on the industry, with increasing returns to adoption!

(Similar products to CUSIP are DUNS, VIN and ISBN; this category seems to have a fondness for acronyms.)

Business Evaluation Standards

You provide an industry-wide “benchmark” that everybody uses to evaluate or price transactions. S&P’s eponymous S&P 500 Index is a good example: every investment manager (and every LP in an investment manager) compares their performance to this index, and it’s also used to price derivatives, structured products, ETFs, you name it. Again, the more it’s used, the more dominant it becomes. Another tax!

(Similar products: Nielsen’s viewership benchmarks, corporate and sovereign risk ratings, FICO scores.)

Pass-Through Loop

I talked earlier about why most data learning loops aren’t moats: because there are limits to the returns from data. But what if you could explicitly pass through some of those gains back to your customers? For a select few companies, in a select few markets, this means the limit does not exist. The bigger you get, the greater your gains.

For example, Stripe uses its massive transaction volume to flag and prevent fraud more effectively, and they pass that through to merchants in the form of lower take rates and fewer chargebacks. This is better than mere “product improvement” because it is both unbounded and dynamic41, and the more people participate, the better it is for everyone.

“Better underwriting through data” is an interesting and perennially tempting target for a pass-through loop. It all hinges on the word “better”. Most challengers take it to mean “broader”, which is problematic — it means you’re going after the marginal creditor, with all sorts of adverse selection, moral hazard, and short convexity effects. But “better” can also mean “faster”, in which case you’re broadening the market in time not in space: more turns for each dollar lent, hence higher IRR, which you can then pass through. The general (counter-intuitive) lesson here is that if “learning” unlocks “speed”, that’s usually more moaty than unlocking mere “quality”.

Trust Loop

In many industries, “ground truth” isn’t fully objective, it’s subjective and endogenous — it’s defined by what people believe. In this scenario, being the “trusted source of truth” is a usage/value loop: the more people trust you, the more your data or content offering is accepted as ground truth, and the more people should trust you.

Examples include peer-review (Elsevier, Clarivate), security (Verisign), market research (IRI, Kantar), insurance (Verisk), and financial ratings (Fitch, Moody’s, Morningstar). You can bootstrap this trust with features like transparency or auditability, but beyond a certain point, these become self-fulfilling prophecies, hence moats. (And yes, these are all data or content assets.) 42

Implicit Knowledge Capture

This is an emerging flow that combines proprietary data, learning, trust, and usage/value. Many application-layer AI companies follow a certain template:

• begin with a foundation model;

• fine-tune it on vertical-specific knowledge;

• automate basic workflows in that vertical;

• discover edge cases and interaction effects and other wrinkles;

• incorporate human feedback to get better at these messy “real world” tasks;

• gain the trust of their human sponsors;

• widen the aperture to replace or assist on ever more workflows;

• repeat the above steps;

• become irreplaceable.

The usage/value step here is capturing implicit (undocumented) human knowledge, which leads to better outcomes, which leads to more trust, which leads to more surface area. And the limiting factor here is thus the complexity of the real world (which is much higher than the point at which diminishing returns set in for a vanilla data learning loop).

Network Effects And Scale Effects

A business has a network effect when adding a new node (typically, a customer) to the network benefits all the other nodes (other customers), either directly or indirectly. This is a well-known and well-studied pattern, with examples from Amazon, Alipay and Airbnb to Zoom, Zomato and Zelle.

A data network effect is when the node of interest is data; hence, adding new data A makes data B more valuable, and vice versa.

True data network effects are rarer than once thought.

Sure, there are plenty of companies that exhibit weak data network effects. Several of the loops above exhibit this: they’re tenuous, or they plateau, or they decay with time. Meanwhile, there exist companies that exhibit strong data network effects, but they are often one-of-ones.

But there also exist data scale effects, and some of these can be quite powerful:

• fixed cost amortization across a wider base of users and data

• advantages in GTM: price discrimination, market coverage, segmentation

• brand and trust effects: dominant positioning, category creation, inbound acquisition, efficient conversion, frictionless retention

• market power: acquisitions, setting standards, commoditizing the complement

Scale effects aren’t quite as moaty as network effects, but they’re nothing to sneeze at either.

The catch of course is that there are also diseconomies of scale: long-tail effects, diminishing marginal performance, disruption from below, quantity-quality tradeoffs and so on. So it’s a question of whether the lift outweighs the costs, and the answer is usually quite domain and business-model-specific.

Data Loops, Summarized

That brings us to the end of our survey of data loops. Here’s handy chart the second:

Overlaps And Evolution In Data Moats

Two things should be clear by now.

First, moats overlap. Many of the companies listed above have multiple data moats, and they’re often mutually reinforcing. Hybrid moats exist, combining elements of both data control and data loops. Catalyst data can arise from an acquisition loop; implicit knowledge capture can be built on top of a system of action; process power can lead to customer data aggregation; and so on. Looking for a “single” data moat is analytically reductionist, and probably not the best business strategy either.

Second, moats have lifecycles. Many moats only kick in above a certain scale or critical mass: brute force, network knowledge or complexity, clearinghouse, UGC and SEO, data gravity, give-to-get, biz model unlock, exchange and evaluation standards, trust and brand effects. And many moats have upper limits to their potency, thanks to LLM replacement, diminishing marginal value (of data quantity or quality), long-tail effects, and diseconomies of scale. This suggests that in addition to defining your moat, you need to understand where you are in its effectiveness curve.

CONCLUSION

How To Use This Essay

“Philosophers have only interpreted the world. The point, however, is to change it.”

This essay is for founders, investors, and operators who care about data moats: who want to build them for their own companies, or compete with rivals who have moats of their own. But as we’ve seen, “data moats” are a wide and varied concept.

Step 1, then, is to understand what type of moat you have (or can build). Do you have unique, valuable data, acquired in a future-proof manner? Do you unify fragmented data and build products or services on top of it? Do you control a system of record, or control data movement in a complex or regulated network? Do you have exclusive IP or contractual or compliance rights? Do you own data that gets more valuable with adoption, like an exchange or evaluation standard? Do you benefit from data brand effects? Do you have a quantity loop like UGC, SEO, G2G or data gravity? Can you unlock new business models with data? Is your moat future-proof and specifically LLM-proof? Do you have a moat that’s not in the list above?43

And so on. The set of questions is lengthy, but finite; the detailed exploration above should help you identify exactly what moats you have access to.

Step 2 is to understand how strong, and how durable, those moats are, and where you are in its lifecycle. Here’s handy chart the third:

As you can see, not all data moats are created equal44. Few understand this.

And therein lies alpha. Knowing the strengths and weaknesses of your own moats, of your rivals’ moats, and knowing what you (and they) can potentially build towards, is a major strategic advantage. Data moats are real, and they’re powerful, and AI makes them even more so; harness this power!

Have fun and good luck!

Toronto, April 2025.

END NOTES, REFERENCES, BONUS THOUGHTS

Essay Meta

“This tale grew in the telling” — J. R. R. Tolkien, foreword to The Lord of the Rings.

I feel an odd kinship with the good professor. At the start of every essay, I tell myself I’m going to keep it short and sweet this time. But the process of writing uncovers new ideas and insights, new underlying structures and themes, new data points and anecdotes, new illustrations and examples — all of which I want to include, because I think they make my essays stronger.

Today’s essay is no exception. It’s my longest yet, but I also think it’s one of my meatiest. I hope you agree!

My philosophy of essay-writing is “fewer but better”. I publish 2-3 essays a year. Each essay is 5-7k words long, takes 2-8 weeks of active writing, and at least 5-6 months of thinking before that. I only write when I have something meaningful, original, and material to say. I use Claude as a thought partner and research assistant, but every single word (and every em-dash) is written by me.

The algorithms hate this, but it works for me, and I think it selects for a high-quality group of readers who appreciate what I do and the work I put in. Thank you for being one of them!

… And A Request

I hope you found this essay interesting, informative, and useful.

If you did, I would greatly appreciate it if you could help spread it far and wide::

• Send it to your colleagues and friends.

• Share it on social media — X, LinkedIn, or the platform of your choice.

• And of course, subscribe!

Subscribe now

Your likes, comments, shares, and subscribes are the best incentive for me to keep writing these pieces, and giving them away for free.45

Tapping A Tuning Fork

After selling Quandl, the company I co-founded, to Nasdaq a few years ago, I’ve been on sabbatical — writing this newsletter, angel investing, and advising / serving on boards. But now I’m feeling the itch to jump back into a more active role, and so I’m tapping a tuning fork to see who resonates.

Specifically, I’m actively exploring startup ideas in the data + defensibility space covered by this essay. If you’re interested in working with me, and especially if you’re technical, please do reach out.

I’m also very open to and welcome interesting third-party opportunities. If you have an interesting opportunity for me to explore, especially if it touches one or more of data, investing, and startups (the Pivotal trifecta), I’d love to hear from you.

Tapping A Tuning Fork, Part Two

I’m an active angel investor in technology startups, and I put my money where my mouth is: almost all my portfolio companies have (or are building) defensibility through data. Here are just a few examples:

• Daloopa is building a foundational data layer for equity investors, involving elements of process power, clearinghouse, knowledge capture, and user trust.

• Quandri is building an agentic system for insurance, incorporating a system of action, unique data, knowledge capture, and surface/area trust.

• Arima has a unique data asset that is partly synthetic, partly clearinghouse; and is also a catalyst for customers to unlock their own data value.

• Setyl is a system of record for IT assets, with a strong data-gravity effect.

• Canopy is a system of vision / engagement for multi-family managers, with aspects of give-to-get and benchmarking baked in.

• Cascade is a system of record / action for private credit, with data gravity, pass-through value capture, and network / complexity moats.

• Citylitics has unique data, industry-specific labelling, implicit knowledge capture in a domain-tuned LLM, workflow embed, and G2G customer data aggregation.

These companies had negligible revenue when I invested; now they’re collectively at around $25M ARR. Data moats work!

If you’re the founder of a company with interesting data defensibility, or if you’re a VC who’d like to jam on these ideas or collaborate, please do reach out.

Companies Mentioned In This Essay

Acxiom, Adyen, Airbnb, Alipay, Amadeus, Amazon, Apollo, Ariba, Arima, Bain, BCG, BenefitFocus, Bolt, Booking.com, Cambridge Associates, Canopy, Cascade, Change Healthcare, Citylitics, Clay, Clearbit, Clarivate, Cursor, Daloopa, Datavant, Devin, Digg, DNV, Doordash, Dun & Bradstreet, Elsevier, Epic, Expedia, Facebook, Factset, FICO, Fitch, GasBuddy, Github, Gitlab, Glassdoor, Google, Greenhouse, Guidehouse, Gusto, ICFI, Instagram, IRI, IQVIA, Kantar, Kayak, Lattice, Leidos, LinkedIn, LiveRamp, Lovable, Mastercard, McKinsey, Mercor, Microsoft, Moody’s, Morningstar, MySpace, NASDAQ, Netflix, Neustar, News Corp, Nielsen, Numeral, NYSE, OpenAI, OpenStreetMap, Oracle, Palantir, PayScale, Plaid, Quandl, Quandri, Quora, Reddit, Remote.com, Replit, Rippling, S&P Global, Sabre, Salesforce, SAP, Scale, Setyl, Shopify, Stack Overflow, Stripe, Studio Ghibli, TikTok, Toast, TripAdvisor, Tumblr, Twitter, Uber, Verisign, Verisk, Vine, Visa, Waze, Weather Underground, Workday, X, Yahoo, Yelp, YouTube, Zapier, Zelle, Zillow, Zomato, Zoho, Zoom, ZoomInfo.

Further Reading

• The Power of Data Network Effects — Matt Turck

• The Empty Promise of Data Moats — Martin Casado

• The Six Moats of Data Businesses — Travis May

• Unfair Data Moats and Regulatory Capture — Alex Izydorczyk

• Making Uncommon Knowledge Common — Kevin Kwok

• Capabilities and Moats - Rick Zullo

• DaaS Bible and Daas Bible 2.0 — Auren Hoffman

• The Vertical SaaS Knowledge Project — Dave Yuan and Tidemark

Acknowledgements

As a general principle, I do not share drafts of my essays or ask for feedback before publishing them; I don’t want to dilute my voice or opinions. I haven’t done so for this essay either. However, I do want to acknowledge the many conversations on this topic that I’ve had over the years with people whose thinking I respect:

Tripp Jones, Dave Marquardt, Dave Yuan, Kevin Kwok, Matt Ober, Sheetal Bahl, Bill Dague, Tammer Kamel, Carrie Shaw, Beth Adams, Sergei Ryshkevich, Auren Hoffman, Matt Turck, Greg Neufeld, Jeremy Giffon, Alex Izydorczyk, Evan Reich, Thomas Li, Ahmed Badruddin, Jackson and Jamieson Fregeau, Sunny Juneja, Winston Li, Christopher Batts, Joe Schmidt, Philippe Kwiatkowski, Nick Achkarian, Jeremy Diamond, Ruben Schreurs, Jared Bochner, Alex Niehenke, Leo Polovets, Ben Rollert, Susan Liu, “The Terminalist”, Abhishek Sharma, and the late, much missed Naren Gupta.

Thank you! (All errors are, of course, my own).

— AT.

Footnotes

One of the themes of that essay was the ‘compression of timelines’ in venture during the boom. Startups were growing faster, rounds were closing faster, funds were being deployed faster. In a footnote, I speculated why time, in particular, should be the variable of interest:

If Wall Street is in the business of spatial arbitrage, Silicon Valley is in the business of temporal arbitrage.

Today I want to unpack that statement, because I think it holds some genuine, and non-obvious, insight. It’s a useful framework to explain what’s happened in venture over the last decade or two, and what we can expect to happen next.

Making Dough from Bread

I bought a sourdough baguette from my neighbourhood bakery this morning. I paid $4 for the privilege.

Why $4? Why not $40, or $0.40? What determines the price of bread?

The list is long. There’s the cost of the ingredients; the value of the baker’s time; regional wages and rents; demand for artisanal baked goods from local essayists; the distance to the nearest supermarket; and so on. These immediate factors in turn are determined by further factors upstream: for example, the cost of flour is a function of the price of wheat, which in turn is a function of sunshine and drought, crop yields and acreage, imports and inventories and substitutes.

It quickly gets complex. But it’s not random. Each part of the bread supply chain is coupled. If wheat prices rise, baguette prices inevitably follow. If consumer demand craters, prices drop. If the proverbial butterfly flaps its wings in Brazil and it leads to a tornado in Texas, guess what: wheat prices react.

But how do these relationships work, exactly? What causal mechanisms link inventory levels at a grain elevator in Saskatchewan to prices at a bakery in Toronto?

It’s commonplace to invoke ‘the magic of markets’ to explain this. But even magic requires magicians. The reality is that these relationships are enforced, by profit-seeking intermediaries. The stage is set for spatial arbitrage.

To Wheat, To Who

Here’s a highly stylized sketch of the wheat/bread supply chain. Wheat is planted, harvested, stored, milled, distributed, baked, and sold. Every stage depends on the stages before and after it. But crucially, no single participant needs to know the full picture in order to be effective.

Imagine I’m Bobby, a baker. I don’t need to know about elevator inventory levels, or fertilizer shortages, or any of that stuff. I set my price based on the price of flour and yeast and salt; my rent and operating costs; my hours worked. If I set it too high, customers will go to my rival Pat’s patisserie down the street; if I set it too low, I’ll go out of business.

Now imagine I’m Dale, a distributor. I don’t need to know about Bobby’s customers or their behaviour; but I do need to know the price of flour from various mills, the demand from various industrial customers, the cost of storage and of trucking. And I’ll preferentially shift my buying and selling and shipping to whatever combination makes the most economic sense for me.

This is spatial arbitrage. Shoppers go where bread is cheapest. Bakers buy from whichever wholesaler is cheapest. Wholesalers and distributors source from where flour is cheapest, and sell where it’s most expensive. If you’ve ever driven a mile out of your way to buy cheaper groceries, then you too, gentle reader, have participated in spatial arbitrage.

The fascinating thing about spatial arbitrage is how it propagates. Participants take actions to maximize their local profits, and these actions transmit ‘fundamentals’ across the economy. If, say, persistent droughts impact the wheat crop in a particular region, its harvest may fall and prices rise; but other producers increase their acreage; distributors reroute their trucks; storage elevators adjust; and eventually a new equilibrium establishes itself, taking into account the new dynamics of supply and demand1.

Futures, Perfected

So this works, and it has worked for centuries; for wheat, and coal, and textiles, and silk, and spices. Local actions lead to global supply chain equilibria.

It works, but it’s not terribly efficient. Rerouting trucks, switching distributors, managing storage levels, planning acreage — there’s friction and effort and uncertainty at each stage, and reaction times are slow.

Enter Wall Street. About 150 years ago, the first wheat futures contracts began to trade. Instead of trading physical wheat, traders could trade pieces of paper that represented wheat, to be delivered on some future date.

These contracts specified two things: what and where.

• What: the quantity and quality of grain that is acceptable for delivery.

• Where: the location and timing of this delivery.

It may not be obvious, but this simple specification unlocked massive efficiencies in the wheat market.

Let’s think about spatial arbitrage again. Let’s say the prices of wheat at Kansas City and Chicago are out of whack: farm prices in KC are $5 per bushel lower than mill prices in Chicago, and it costs only $1 to transport a bushel of wheat from KC to Chicago.

In the bad old world without futures contracts, an arbitrageur would have to buy wheat from a farmer in Kansas City, arrange for it to be transported to Chicago by train or barge or truck, and then sell it to a miller in Chicago. Their actions push KC prices up, and C prices down, until the arbitrage disappears.

This is doable, but it’s complex, and risky, and takes both capital and effort: the arbitrage rewards would have to be quite high to justify such a plan.

Now let’s introduce futures. You can break down the spread between KC wheat and Chicago wheat like this:

And this means you can decompose the trade into sub-trades:

Here’s what happens now:

• Kim the Kansas City trader looks only at the spread between physical wheat and wheat futures in Kansas City. If physical wheat gets too cheap relative to wheat futures, they buy physical and deliver it into the futures contract. They may have to hold some inventory, but they don’t care about the price in any other city, or the cost of trucking.

• Charlie the Chicago trader does the same thing, in (wait for it) Chicago.

• Sam the spread trader doesn’t care about physical prices at all. Instead, they look at the spread between the two cities’ futures contracts. If it rises above the cost of transport between the two cities, they sell the spread and move wheat from one futures delivery hub to the other until it normalizes. They don’t need to interact with farmers or millers or care about initial supply or final demand or the absolute level of wheat prices or anything else; the spread is all that matters.

The beauty of this setup is that you get cross-city spatial arbitrage for free. If farm prices in KC are cheaper than mill prices in Chicago, you no longer need to do an end-to-end trade with all its attendant complexities (and costs). Instead, if each of the intermediary traders does their thing, physical prices at either end of the chain will adjust automatically.

Why is this better?

• Each intermediary arb is lower risk, and requires less capital to support

• Each arbitrageur can specialize and focus on just one spread

• This makes spatial arbitrage easier to execute

• And as a result we get tighter markets, faster price discovery, lower volatility, less uncertainty

There’s a shorthand for this: more efficiency. Efficiency is a good thing: it means the economy is allocating resources properly

Efficient markets mean it’s easier for people to participate: farmers, millers, bakers, and ultimately, your humble essayist, clutching his morning baguette.

Divide, Define, and Conquer

Let’s generalize. What’s the structural innovation here?

It’s two-fold: dividing and defining.

Dividing means slicing up the value chain into smaller stages: wheat and flour and baguettes and croutons. And defining means specifying exactly when and where and how the object (product, service) is handed over from one stage to the next: it’s not just wheat, it’s this particular variety of wheat, in this quantity, to be delivered at this place on this date.

And then you financialize all the things: you map these finely divided and carefully defined physical entities to abstract representations (aka securities), and allow people to trade them against each other and against the underlyings.

THIS ONE NEAT TRICK is all you need to supercharge spatial arbitrage.

The Canny Valley

But enough about wheat. Let’s talk about Silicon Valley!

It’s a long road from inception to IPO. As an investor, how do you finance ideas, not knowing if they’ll work? As a founder, how do you get your ideas funded, not knowing how much they’re worth?

There’s a great quote from the classic reference book Numerical Recipes in C:

If the desired X is in between the largest and smallest of the known Xi’s, the problem is called interpolation; if X is outside that range, it is called extrapolation, which is considerably more hazardous (as many former stock-market analysts can attest).

The genius of modern venture capital is that it has figured out a way to convert the extrapolation problem (how do you predict the future of technological innovations?) to an interpolation problem (how do you bridge the gap between idea stage startups and mature public companies?).

And what is this secret, you ask? It’s the same trick that we saw for wheat: divide and define. But instead of slicing and specifying features across space, Silicon Valley does the same for stages across time. Temporal arbitrage!

These stages are called ‘rounds’. These rounds are labelled: angel, seed, Series A, Series B, Series C-D-E-F, and finally IPO or ‘exit’. This is the divide step.

Each round corresponds to some idea of progress in the underlying business. Roughly speaking, Seed is for a team or hypothesis; Series A is for product-market-fit and initial traction, Series B and beyond are for growth and scaleable economics. This is the define step.

What’s more, specialist investors exist to finance each step: to bridge the gap between each stage and the next. Angel investors invest in angel rounds; seed funds invest in seed rounds; and so on2.

And with those simple steps, the full end-to-end financing journey is unlocked.

A seed investor doesn’t have to have a complete analysis of the path to IPO — that’s way too complex, and risky, and uncertain, and unpredictable. All they need to care about is whether the company can plausibly get to the next round. After that it’s the responsibility / decision of the next round’s investor to get to the round after that, and so on.

The comparison to the wheat supply chain is obvious. Kim the Kansas trader doesn’t have to think about trucking costs; that’s Sam the spread-trader’s department. Sam doesn’t have to think about miller demand; that’s Charlie the Chicago trader’s lookout. Ultimately, wheat goes from farm to granary to mill to bakery to supermarket to customer, transforming from grain to flour to bread along the way, but each leg can focus on what it knows (and does best). The analogy is complete.

Efficiency Unlocks Volume

It’s kind of easy to take this for granted, but this is a genuine innovation. It derisks what would otherwise be an impossible sector to underwrite.

More precisely, it identifies, decomposes and then allocates the risks of the sector to people who want to take those exact risks — early stage investors seeking 100x returns on 1 out of every 20 investments and okay with seeing the rest fail, pre-IPO investors seeking 2x returns on all their deals with strong downside protection on the misses, and everyone in between.

This dramatically increases the number of investors who can invest. Less obviously, it also dramatically increases the number who want to invest. In financial markets, diversification is everything; investors flock to whoever can offer them new, uncorrelated, finely-grained sources of risk and return. Supply creates demand, and the invention of stage-based venture financing unlocked a juicy new source of investment supply. Demand — that is, massive capital inflows — followed.

The entire blossoming in venture over the last 15 years has been driven by the maturing of temporal arbitrage. Efficiency unlocks volume. Just as spatial arbitrage ultimately leads to more bread production (better economics, less wastage, more choice, higher quality, specialization and scale effects), temporal arbitrage ultimately leads to more startup creation (for exactly the same list of reasons). To use the terminology of Seeing Like a State, temporal arbitrage made venture legible.

Herd Animal Spirits

Writing about the collapse of Silicon Valley Bank in 2023, Matt Levine had an all-time great burn:

I am sorry to be rude, but there is another reason that it is maybe not great to be the Bank of Startups, which is that nobody on Earth is more of a herd animal than Silicon Valley venture capitalists.

But here’s the thing. Given the world that VCs live in — the world of temporal arbitrage — this is a feature, not a bug. VCs must establish strong shared consensus in order for their ecosystem to work.

Why so? Recall that a futures contract specifies two things: the object being transacted, and the location of said transaction. Without these specifications, participants wouldn’t know what to deliver or receive, or where; spatial arbitrage wouldn’t be possible.

Now in the commodities market, these specifications are set by futures exchanges like the CME or the CBOT. But in venture, no such centralized exchanges exist. Instead, the industry as a whole works to create implicit shared knowledge of what and where — or rather, when.

The ‘when’ question is why venture participants are so obsessed with round-specific benchmarks. Product-market-fit! Scalable unit economics! 1M revenue for a Series A and 100M for an IPO3 ! These rules exist to reify the consensus trajectory from inception to exit: they’re not just a roadmap for founders, they’re also a price guide for investors. Inflection points and milestones are more legible than steady compounding.

The ‘what’ question is also why participants talk so much about fundability. It’s not enough to build a good business per se; you need to build a business that is fundable by downstream investors: in other words, that hits the (arbitrary, but consensus) milestones that downstream investors care about.

People poking fun at VCs for being consensus are missing the point. You need alignment on stages and on milestones in order to make the transitions work; and you need support from downstream investors to keep the journey going. Without it, the whole ecosystem comes crashing down, to nobody’s benefit.

Consensus is the Winning Strategy

Common investing wisdom says that to make money, you must be contrarian and right. Venture subverts this. In venture, contrarian/consensus and right/wrong are not orthogonal axes; being right in multi-stage venture is defined by follow-on rounds and markups — in other words, by becoming part of the consensus. Front-running consensus is the winning strategy! Venture is the Keynesian beauty contest in its purest form.

In this framing, a VC’s core competence is knowing when a company is fundable, not just by themselves, but also by downstream investors. Consequently, 80% of the value-add offered by 80% of VCs is helping companies with downstream funding; it’s a goal in itself, not merely a means to an end.

A lot of things now make sense:

• the VC preference for in-network and pedigreed founders

• the emphasis on self-fulfilling prophecies, in business models and funding rounds

• the sense of ‘pre-anointed’ winners (and losers) in the market

• the way the entire industry pivots, en masse, into today’s hot sectors, and memory-holes yesterday’s (NFTs, anyone?)

• the coded signals and elaborate dance of Bay Area conversations: everyone is trading narratives, all the time

• the career risk that a VC takes when investing in an ‘unfundable’ startup (as opposed to a merely unsuccessful one)

• the inordinate amount of time that VCs spend talking about round benchmarks, valuations and ‘market’

• the backward-induction approach that founders are advised to use when planning budgets and strategy

• the difficulty of raising ‘tweener’ rounds, especially at early stage

• the disconnect between price, value, and valuation4

• the disconnect between startup financial metrics and the financial metrics used by everyone else in the world5

None of this seems obviously a part of ‘find the best companies and give them money’, but here we are.

Movers, Shakers, and Venture Market Makers

Don’t consensus decisions lead to mediocre returns?

Yes, absolutely — and that’s okay! The seemingly illogical and perverse behaviour I just described is merely the price paid for making venture legible, which in turn is what has led to massive capital inflows into the asset class, helping investors and founders alike6. A rising tide lifts all boats; 50th percentile in a booming sector is preferable to 95th percentile in a struggling one7. For the last decade, it’s been totally fine (and extremely lucrative) to be a mediocre VC.

But there’s an even better play here.

Public market investors like mutual funds and hedge funds tend to be price takers: Mister Market sets a price, and the investor chooses whether or not to buy (sell) at that price, depending on whether or not they think the price is cheap (rich) relative to some notion of target or fair or expected value.

VC firms have never been that; startup financing markets are too illiquid, and startups themselves too unique, for there ever to be a ‘market price’. Instead, classic VC firms had to be price makers — setting the price and terms for every deal they printed.

Modern multi-stage VC firms are different again. Modern VC firms are market makers. Yes, they still set terms, and yes, they still want the price to go up. But the consideration of intrinsic or fair value is almost strictly secondary to the question of whether a downstream investor will follow on.

This is market-maker behaviour. Market-makers don’t care (much) about intrinsic value; they care about matching supply and demand. Supply here is founders; demand is (ultimately) public market investors; temporal arbitrage aka multi-stage venture financing enables this large gap to be bridged; and the market-makers take their cut8.

Modern venture exhibits all the classic attributes of market-making:

• Consensus, obviously. A market-maker lives and dies by the ability to front-run the crowd; if you’re contrarian, you’re toast.

• Price-agnosticism. “You have to play the game on the field” is the purest statement of this attitude; can you imagine Warren Buffett, or any price-taker, saying this?

• Coverage. Market-makers care about surface area: they want to see every trade. VCs are exactly the same: dealflow is everything. Top firms religiously measure the number of deals in their mandate that they see (and don’t see) — a metric of zero interest to price takers.

• Inventory. You can’t be a market-maker without holding inventory. Hence the proliferation of exploratory and scout cheques from the big firms. The expected returns from these cheques are minimal (if not negative, once you account for non-dollar costs), but collectively they offer optionality on supply.

• Correlated books. The big firms are in all the same deals, and they trade with each other incessantly to lay off (but really, merely transfer) risk.

• Size-pilling. VCs prefer A- businesses in massive markets to A+ businesses in small ones. Why? Power law math is part of the reason; another part is that large, competitive TAMs mean more dollars deployed. Prop traders hate this; flow traders love it.

• Brand. Ever wonder why hedge funds are so secretive about the markets they’re active in, while sell-side banks shout from the rooftops about their position in the league tables? Now observe VC behaviour: do they look more like the former (prop), or the latter (flow)?

• Selection. I’m just going to quote @pmarca here: “The core dynamic [that creates scale economies in venture] is that a few firms have positive selection on their side; the other firms have adverse selection working against them”. Every market-maker knows this dynamic in their bones.

• Winner’s curse. The higher you bid on a deal, the more likely you are to win it — but there’s a line above which the deal becomes less attractive, no matter how price-agnostic you are.

• Franchise protection. Despite adverse selection and winner’s curse, it’s desirable to occasionally print trades that you know you’ll lose money on, because it keeps you in the game. Irrelevancy is the greatest sin.

• Winner-takes-all. Market-making is lucrative, but only for the top firms. It’s no good being the 19th largest market-maker — you don’t see enough flow, and you have to pay worse prices; it’s a vicious cycle. Conversely, the top firms benefit from a virtuous cycle of brand, access, capital, pricing power, and returns.

None of this, btw, will come as a surprise to anyone who has spent time on a trading floor9.

The Rise of the Venture Majors

Market-making is simultaneously more lucrative, less risky, and more scaleable than price-taking. As a result, the handful of firms that succeed in capturing market-maker network effects quickly achieve escape velocity. This has led to a new class of player in the industry: the venture majors.

These firms are vertically integrated — they play at every stage, from pre-seed to Series Z. They are horizontally expansive — they play in every sector, from software to robotics to biotech to defence to crypto. And increasingly, they play across geographies, and up and down the capital structure stack. No tech financing event is out of scope for them10.

At the same time, they’re not really hunting alpha. Outperformance is not the point; rather, because of their increasing economies of scale, they just want to be part of every (material) deal. The result is, essentially, beta on the private tech market — it may not be a true ‘index’, but it offers highly-correlated directional exposure to the market as a whole, which is almost the same thing.

LPs love this. Most institutional investors want sector exposure first, and in-sector outperformance second11. The venture majors provide precisely this.

And the incentives don’t stop there. Allocator capital scales better than allocator diligence — it’s so much easier to deploy $20M into one fund than $1M into each of 20 funds. And of course GPs are happy to make 2% of the biggest number possible.

Everybody points to the 2024 statistic that 50% of new LP dollars went to just 9 firms; nobody seems to have a theory of why rational LPs would do this. Market-maker concentration provides the answer!

There’s a clear parallel with history here. Many of today’s largest investment banks — Goldman, JP, Citi — got their start as prop lenders during a previous technological buildout: canals, railroads and coal in the 19th century. They then broadened their franchise into more beta-like businesses like brokerage, depository and market-making — less glamorous, but far more lindy. Prop traders come and go, but banks go on forever.

The goal of the venture majors is to become the investment banks of the technology world. And it certainly looks like they’re succeeding! They say venture doesn’t scale, but this is not venture; it’s a completely different product, with completely different risk-return and investor profiles12.

Ironically, this transition might not be enough! The AI-industrial complex (energy, data centres, compute hardware, training runs, data acq) seems to be full steam ahead on the largest tech buildout ever, and even the venture majors aren’t big enough to finance it; instead the dollars are coming from defense budgets, sovereign wealth funds, and megacap public tech cos (FAANG and friends). The New Deal and Manhattan Project might be better historical analogies — we shall see.

Truly, we live in the most interesting of times!

Toronto, Jan 2025

Notes, Questions, and Further Reading

• I use the terms arbitrage, market-making, and spread-trading somewhat interchangeably in this essay. This is not sloppiness or imprecision; rather, it reflects the fact that the boundaries between these actions are blurry, especially when you’re in an illiquid, opaque, discrete market. Normally the pedant (and ex hedge fundie) in me would be up in arms about this; not this time.

• It’s a little cringe to cite your own work, but I think my 2022 essay Minsky Moments in Venture Capital remains the best distillation of the dynamics of venture at the height of the bubble. And my predictions around lengthening timescales proved to be very accurate.

• One danger with focusing solely on getting to the next round of funding is that it can be gamed. Venture funding can be used to ‘buy’ artifical growth, which in turn can be used to justify the next round of funding, which buys the next round of growth, and so on. You can keep this cycle going for quite a long time, even if the underlying business is a lemon. Paper markups, liquidity from secondaries, and high-profile headlines keep everyone happy along the way. The cynical view is that this enriches founders, VCs, Bay Area landlords, and Google-Facebook-Amazon, all at the expense of LPs. There's probably some truth to that, but note that many LPs are willing accomplices in this game. There are principal-agent problems in every industry.

• Two questions I’m currently thinking about. First, what happens if the right anchor of the financing chain becomes permanently untethered, because of a lack of IPOs — can companies stay private forever, and does this increase or decrease the attractiveness of the venture market-making model?

• Second, we all saw how ZIRP distorted funding markets. How might future changes in the time value of money — specifically, from AI productivity, or from tariffs and money-printing — affect recurring revenue models and the temporal arbitrage that funds them?

And Finally

A quick word about me: I’m Abraham Thomas, a former portfolio manager at a quant hedge fund, turned successful startup founder (Quandl, acquired by Nasdaq), turned private investor.

Pivotal is my deep-dive newsletter on data, investing, and startups. I write occasional long-form essays on topics where I have meaningful professional expertise: data and data businesses, AI and data, venture capital and angel investing, quant investing in public markets, software startups and business models, and being a tech founder.

If you enjoyed this essay, please take a minute to like, comment, subscribe and share. Thank you for reading!

Subscribe now

If you’re reading this, you probably know me as a writer on data, investing and startups. Some of you may also know me as the co-founder of a tech startup myself — Quandl, a venture-backed company that was acquired by Nasdaq a few years ago — or perhaps as an angel investor. But I suspect very few know of my previous life, as a quant, trader and portfolio manager at a Japanese hedge fund. So I thought I’d write an essay about one particular adventure from those days. Read on!

“Better Lucky Than Smart”

My first job out of university was as a programmer-analyst at a Japanese hedge fund, Simplex Asset Management. I started work in August 1998. Exactly one month later, Long Term Capital Management, the world's largest and most celebrated hedge fund, blew up — spectacularly.

This was problematic, to say the least.

In the short term, it was actually good news for us. Our fund, newly launched at the time, proposed to trade many of the same quantitative strategies that led to LTCM's demise1. With LTCM out of the way, we were often the only capital chasing those opportunities. If we had launched 3 years earlier, we would have blown up alongside LTCM; 3 years later, and those opportunities would have been much diminished. As my boss liked to say, “better lucky than smart”.

But over a longer horizon, we were worried. LTCM’s collapse raised doubts about the fundamental viability of the class of strategies they, and we, traded. Luck wouldn't suffice; we needed a Plan B.

This is the story of that Plan B. I’d like to say it was completely intentional and strategic, but the reality was much more exploratory and emergent. At the end of it, we discovered we had built something for which there wasn’t really a name at that time: a system that tracked market prices, ran models, identified opportunities, designed trades, processed tickets, managed hedges, and exited positions — all in a matter of seconds.

Seconds? Yes, seconds. Laughably slow compared to today’s HFT systems; miles ahead of the market back then. Our proto-HFT system drove most of Simplex's trading activity (and profits) for the best part of a decade. I was one of the lead builders and main traders on the system; here's how it all happened.

PART ONE: SETTING THE STAGE

Why Genius Failed

LTCM and Simplex both specialized in ‘convergence trading’ — building quantitative models of relationships between different securities; placing bets to exploit inconsistencies (‘mispricings’) in those relationships; and profiting when those inconsistencies resolved (‘converged’).

But what happens if the mispricings don't converge? Or even worse, if they diverge? You can double down on your trades, but as Keynes memorably (if apocryphally) said, markets can remain irrational longer than you can remain solvent.

This was what happened to LTCM. In the summer of 1998, their portfolio was buffeted by a perfect storm of adverse market moves, seemingly across every single position they held2. This was no coincidence; instead, it reflected the pattern — not captured in LTCM’s historical data — that in a crisis, “all correlations go to one”. Many other investors had very similar positions to LTCM, and were heavily levered to boot; when they stopped out of those positions, it caused all of LTCM’s trades to diverge simultaneously. LTCM didn’t stand a chance3.

We observed all of this at Simplex, and were determined not to suffer the same fate. But it seemed an inevitable part of convergence trading. The risk profile of this strategy is often described as “picking up nickels in front of a steamroller”: you can be smart, and agile, and nimble, but eventually you’ll get squashed.

We needed something new. Ideally, something that:

• didn’t rely on market-to-model convergence

• didn’t use excessive leverage

• didn’t correlate to other investors’ positions

It seemed impossible. It wasn’t.

The Big Idea: Monetizing Noise

Let’s say you have a model that identifies or predicts a relationship between two or more securities. Deviations from this relationship can then be modelled as a ‘spread’ — the difference between two bond yields, say, or a bond and a bond future. As long as the relationship holds, the spread should ‘mean-revert’, i.e. it should return to zero, like this:

And this means you can build a trading strategy around it. You buy (sell) every time the spread goes below (above) a certain threshold value, and exit when it converges to zero.

This is the classic Salomon-LTCM style of trading: you make money for a while, but eventually the market diverges instead of converging, and you blow up. (Even if it converges again later.)

But let’s zoom in on the middle section of this graph, highlighted below:

Notice that although the spread never converges during this window, there are nonetheless a number of peaks and troughs. What if we could trade this noise — sell all the interim peaks, buy all the interim troughs? We’d make money despite the lack of convergence!

This might seem blindingly obvious today, but believe me, it wasn’t so at the time. Extremely smart, mathematically skilled, financially sophisticated quant researchers and traders would get hung up on two questions: “How can it possibly make sense to buy the spread when it’s above zero — that trade has negative expected value!” and “How can you possibly make money with a convergence model if the spread never converges?”.

And that was just our colleagues on the desk! Getting our investment committee, risk management, institutional LPs, prime broker and others on board took material time and effort. We had to write multiple research notes, run simulations and back-tests, show actual live trading results, and people still didn’t understand how this could work. Old habits die hard.

The End of Bid-Ask

Of course, saying you’re going to trade the noise is one thing. Doing it effectively is another.

The problem, as always, is bid-ask. The more granular the noise you want to target, the harder it is to overcome transaction costs. If you’re paying 1bp to enter or exit a trade, you need 2bps of opportunity just to break even, and at least 3-4 times that to make it worth the risk. (This is why there’s an entry threshold for your trades). In the late 1990s, the market was inefficient, but it wasn't that inefficient; the oscillations we were trying to capture were much smaller.

But we found some techniques that worked4:

• Instead of trading spread constituents directly, trade liquid proxies. What you give up in precise targeting, you make up in round-trip trade count.

• Layer strategies on top of each other, and trade only the ‘net change’ across multiple strategies.

• Leg into multi-sided trades opportunistically. Randomize this to minimize directional risk.

• Track block trades to anticipate short-term hedging flows by market-makers.

• React faster than the competition to price discrepancies across venues. (This was still possible back then!)

• Trade a lot during volatile markets; enter or exit convergence positions while the rest of the market is still reacting to macro events.

• Maximize trade count in both time (frequency) and space (strategies).

• Realize that a rough hedge done instantly is superior to a perfect hedge that takes time or costs money.

All of these pointed to a single conclusion: we needed rapid, automated execution across a portfolio that traded half a dozen different (but overlapping) convergence signals in a liquid market.

Now we had to build it.

PART TWO: THE DEVIL IN THE DETAILS

There’s Treasure in Treasuries

We decided to start with US Treasury bonds, a market that was efficient enough for quant modeling to work; volatile enough for noise-trading to be feasible; and liquid enough for zero bid-ask to be achievable.

Liquid and efficient it may have been, but technologically, the bond market was still stuck in the 1980s. The vast majority of trades were done on the phone. Electronic venues were fragmented and opaque; electronic execution was less than 10% of the market. Only a handful of participants knew how to price the Treasury yield curve correctly; those that did, typically relied on overnight model runs, with Excel spreadsheets for intraday updates. (And this included traders at the world's biggest banks and asset managers). A perfect market for us!

Data Rules Everything Around Me

The first step was getting the data. The first step is always getting the data5.

In those days there was no single convenient API we could use to get bond market data6, so we brute-forced it:

• We asked 5 different investment banks (the ones with the biggest Treasury franchises) for their morning, noon and end-of-day pricing runs, that they used to mark their own books.

• (We eventually asked each bank to do this for each of their main desks — TKY, LON and NYC — for more coverage).

• Sometimes these were in plaintext, sometimes these were images or PDFs or other document formats. OCR tech wasn't great in those days, so we had two separate back offices, in Tokyo and Hong Kong, manually entering prices from these runs.

• We did automated intraday screen-grabs of all the major (executable) trading venues.

• We did the same for our Bloomberg and Reuters terminals, focusing especially on prices around the market close.

• Our parent firm had multiple prime brokerage relationships (for different funds, trading strategies, geographics), and we got prices from them as well.

Of course we then had to error-correct, remove outliers, fix systematic biases and so on7. Since prices for most bonds were quoted as a spread to a handful of ‘benchmark’ issues, we also had to normalize all our quotes so that the benchmarks were aligned. And then we had to distill all this raw material into a single ‘golden’ time-stamped price for each bond.

We're not done yet! Treasuries, unlike equities, have a fixed maturity. In modelling terms, this means that bonds are not the same from one day to the next. Today you have a 10-year bond; tomorrow it's a 9-year, 364-day bond. For perfect day-to-day model consistency — which will become important in the next section — we needed to construct a universe of ‘virtual’ bonds with constant maturities, that were linear combinations of actual bonds.

This turned out to be non-trivial, since the virtual bonds had to obey a set of conditions that were hard to satisfy simultaneously (well-behavedness, smooth weights, yield and coupon matching, sparse data and asymmetry handling). I spent quite a lot of time figuring out the math on this8.

Good isn't Fast Enough; but Fast is Good Enough

Our workhorse was a yield curve model we called N3, short for ‘Normal 3-Factor’. N3 was a viciously non-linear set of coupled differential equations that, given 12 input parameters, would spit out the yield at any maturity. 8 of the parameters were constants, denoting unchanging structural aspects of the economy; we'd calibrate these about once a year, running an optimization (the EM algorithm) that took many hours to run, over a decade-plus of historical data. The remaining 4 parameters changed from day to day, reflecting the market's implicit values for 4 numbers that defined current economic conditions: the overnight funding rate, the expected (real) growth rate, the expected inflation rate, and the risk premium9.

The classic way to use N3 was, we'd pick 4 ‘anchor’ points on the yield curve — interest rates at constant 1-week, 2-year, 10-year and 30-year maturities — and use them to solve for the 4 daily-changing parameters (4 equations, 4 unknowns). The model would then be able to predict yields at every other point; we could then buy (sell) bonds that appeared cheap (rich) relative to their predicted yields. (Taking into account coupon effects, cashflow timing, financing costs, liquidity and other niggly details of course.)

But solving was slow. Not the 6-8 hours that a full EM calibration would take, but 10s of seconds up to several minutes. That wasn't good enough for the opportunities we wanted to capture.

We spent a lot of time looking for ways to do this faster, both hardware and software. Nothing worked10.

And then, a breakthrough!

We realized we didn't need to re-solve N3 for each tick of market data. N3 was impossibly complex, but for small-ish moves, it could be approximated linearly. So we did a full solve every 60 minutes or so; during that full solve, we also calculated all the partial derivatives of each parameter with respect to small moves in each of the anchor bond prices. Then, for any market move in between our 60-minute resets, we’d just do a linear approximation — a simple matrix multiplication sufficed to generate the new parameter values, and another one sufficed to generate the predicted yields elsewhere on the curve11.

We applied this conceptual breakthrough throughout the workflow. Every intraday calculation (actual bonds to virtual bonds, price space to parameter space, factor sensitivities, hedge ratios) was converted to a linear approximation. Matrices all the way down!

Structure and Infrastructure

So we had a system that could identify market opportunities fast. The next step was to hook it up to trade execution infrastructure.

There were (are) 3 broad categories of participant in the Treasury market:

• Clients: asset managers who wish to buy and sell bonds (including hedge funds like Simplex)

• Dealers: the big banks, who facilitate those trades by making markets and holding inventory

• Brokers: specialist firms who provide an execution venue for inter-dealer trades, but don’t trade or hold inventory themselves

Back then, client-dealer trades were entirely done by voice. Dealer-dealer trades were sometimes voice, sometimes electronic. Some of those were direct, others went through the brokers. And finally, client-client and client-broker trades didn’t exist: dealers guarded their intermediary position jealously.

We did an end-run around them. We convinced a few brokers to allow us to trade directly on their electronic platforms, giving us access to liquidity at a speed the dealers couldn’t match. A couple of the electronic platforms also had automated ticketing systems, which was great.

Seven Seconds Or Less

But we couldn’t do without the dealers. They still controlled the largest pools of liquidity, and for many bond issues they were the only game in town. And their flow desks transacted with clients exclusively on voice.

We needed a way to execute voice trades faster than anybody else.

This ended up being classic workflow engineering. We figured out the essential actions that went into a voice trade, and automated as much of it as we could:

• identify an opportunity that required dealer (not broker) execution

• design the trade: legs, directions, bond IDs and notionals

• auto-generate a request-for-quote message

• paste the message into Bloomberg and send it to a dealer salesperson

• receive the quote and parse the prices

• pick up the phone and say ‘done’ or ‘nope’

• get a trade ticket from the dealer and parse that

• feed the trade into our portfolio

Most of these steps were automatable; the only one that required a human was sending the BBG message, and saying yea or nay to the reply. The hardest part was training dealer salespeople to quote prices in a consistent format so that we could parse them automatically. (Oh what would I have given for an LLM to do that for us). We ended up building a proto trade capture tool — a widget where you could paste the dealer’s message and the app would extract all the details and flow it through the system. We had one for live quotes, and another for trade confirms, and they worked astonishingly well.

We didn’t quite get it down to seven seconds or less, but we were pretty damn fast.

Happenings Under the Hood

There were lots of smaller hacks that accompanied these broad brush strokes:

• Our status monitoring machine had no sound card. I hacked its bootup beep (using clock cycles) so that it could play a tune, if the tune was written down as frequencies. From then on, whenever the system went down, the Imperial March would beep through the office.

• Because of database speed and field and size limitations, at one point we encoded the entire swap market into a single ‘binary large object’ (BLOB) which we could load into memory and query. (I cannot tell you how much database technology sucked in those days.)

• We needed a UX to display positions, actions, hedges, live P&L and risk. Excel had a stream function that in theory could do this, but it was slow, prone to hang or crash, and unauditable12. We found an Excel clone called MarketView which was optimized for quote streaming; it completely sucked at being a spreadsheet, but we didn’t care — we did all the calculations server-side, and used MV as a dumb display.

• We initially thought we'd build and trade the system from our head office in Tokyo, but I eventually moved to NJ and set up a satellite trading office there. An early example of co-location!

Sidetracks and Surprises

There were also some misadventures along the way:

• That time our super-automated trade reconciliation system emailed all our positions to all our counterparties instead of to our back office. Amazingly, this bug was offset by another bug, introduced by the same code update, that bricked our email server. So we lived to trade another day13.

• That time I messed up the zeros in a trade notional — to be fair, it was a yen trade, there were lots of zeros — and ended up executing a super-complicated multi-legged trade for a minuscule position. I caught the error the next day; for once, I was glad that the market moved against me, because I was able to add the full (intended) size at a much better level14.

• That time the whole trading system had to be paused for weeks because of a devilish bug: a low-level optimization routine kept flipping between two equally valid solutions (it was a ‘slightly’ under-determined system) — we eventually figured out it was due to an unstable interaction between our annealing algorithm, our random number generator, and the way eigenvalues work.

• That time — more than once, actually — that our prime broker called us in a panic, because their trading desk told them we had bought (or sold) many times our financing limit. They didn’t know that we had already offset those trades elsewhere — the idea that a fund could round-trip that amount of volume without facing ruinous transaction costs seems to have never occurred to them, and we weren’t about to tell them.

• Not quite a misadventure, but an amusing sidenote: Each of our dealer counterparties, seeing the volume of business we were printing with them (but not knowing about the offsets), were convinced that we were financing our positions with their competitors15. So they began to offer us better and better deals to finance with them. This too became information we could trade against.

• That time we discovered that it didn't matter where we set ‘fair value’ for any convergence spread: as long as the spread was noisy, our P&L was insensitive to its actual level. This obvious-in-retrospect discovery drove our academic research team slightly insane.

How Does It Feel To Win?

Switching to this new HFT approach was a slow process. We probably spent a year thinking about and around the problem, doing research, building prototypes, back-testing, solving various mathematical details etc. Another year building the actual core system. And then another year gradually ramping up our trading volume, figuring out areas to automate, ironing out bugs, getting ready for prime-time.

But it worked. And it worked better than we had any right to expect. In the US market, high-frequency curve trading quickly went from 10% to 50% (and in some months, 80%) of our P&L.

Even better than the P&L was the trading footprint. We discovered a major, major loophole in the system. Portfolio risk, credit limits, correlations — industry practice was to calculate all of these based on end-of-day positions. But our end-of-day positions were typically ‘flat’! The noise we were trading would mean-revert intraday, so all our buys and sell would cancel out. A counterparty looking only at our end-of-day book would conclude that we were taking very little risk, and therefore didn't require us to post much margin16.

It also meant we didn't need much financing, which meant very little overnight leverage: our return on balance sheet was excellent. And finally, our P&L was rarely correlated with close-to-close market moves, which meant it was rarely correlated with other market participants, even if they were trading similar models and strategies.

No requirement of model convergence; limited leverage or financing needs; minuscule margins to post; and low correlation with the rest of the market: this was the holy grail all right.

What was happening was that our proprietary, research-intensive, model-driven prop trading strategy had begun to take on the behavioural profile of a successful flow desk. This made a sort of intuitive sense: after all, we were consistently buying low and selling high and never holding trades for very long; the underlying logic may have been determined by a convergence model, but the trading pattern was that of a market-maker17. Indeed, we even started thinking of ourselves as ‘multi-security market-makers’, where our core models allowed us to span a wider range of hedges than any single-instrument or sector specialist.

It was, and I say this with all modesty, beautiful.

Markets Are Complex, But We Are Simplex

Yes, this was our unofficial motto. Yes, it’s terrible.

With the core infrastructure in place and proven to work, we wanted more. We added layers upon layers to our stack:

• N3 was a great model, but it was just one model. We realized that if ‘fair value’ doesn’t matter and if ‘market-to-model convergence’ doesn't matter — if the only thing that matters is intraday noise — then any model that consistently identifies intraday noise should work in our system. So we added multiple models and ran them in parallel; some of these were economically-founded, others were just good old fashioned correlation hunting; all data, no math.

• We also started trading more instruments: swaps, eurodollars, futures, options, cross-currency basis — anything and everything that exhibited any sort of intraday correlation or mean-reversion versus our core Treasury book, as long as it could be traded fast. (But we never ran a truly global book, even though the math made sense; we felt that there were too many ‘unknown unknowns’ and asymmetric macro risks to get comfy with.)

• We became a lot more sophisticated about market impact: when to trade and how much; red flags and green flags in market structure; half-life of opportunities and residual flows; seasonal (really, diurnal) effects; how to trade around macro events and more.

Small But Mighty

Given how ambitious this project was (in technical scope, in dollars traded, and in novelty), we accomplished all of this with an astonishingly small team:

• TB was the senior portfolio manager and driving force behind the project; he's now a senior exec at BlackRock.

• HL was the lead programmer and architect building the system; he subsequently became CTO at a couple of other hedge funds.

• RO and GG were junior analysts who did a lot of the block-and-tackle work on coding and data; they eventually started trading on the system we built, and they’re both successful porfolio managers today.

• As for my role in all of this: I bridged trading and tech. I was a trader and portfolio manager who was also technical, and so I did a lot of the model / data / strategy R&D and built many of the prototypes; I was also the most consistent day-to-day trader on the system once it went live.

Five people! Small teams punch above their weight, always.

Looking back, one other thing that leaps out at me is how impossibly young we all were. TB and HL were the senior citizens on the team; I don't think either of them had reached 30. I was 24, and clad in the invincibility of youth. RO and GG were even younger, fresh out of undergrad.

All Good Things

But alpha decay is inexorable, and all good things come to an end.

I mentioned how our HFT book went from 0% to as much as 80% of our total P&L in its heyday. It remained at those high levels for a while. But then it then went back to around 20%, and stayed there.

What happened? Competition. I don’t think anything we built was truly unique or irreproducible, and I’m sure many others were experimenting with similar approaches; there was something in the air18. I know of at least one bank whose curve-trading infra was pretty much identical to ours, and no doubt there were multiple funds playing the same game. Round-trip opportunities of 1-2bp, which we’d reliably see at least once or twice a day, became rarer and rarer, before disappearing completely. Any mispricing was quickly counter-traded.

It was a good run while it lasted. I'm sure there were firms that invested resources only to go live right when the opportunity set declined; we were fortunate (both lucky and smart) to be ahead of the curve and thus get a few years of strong performance. We were also reasonably quick to recognize when the alpha began to decay, and pivoted into a different set of strategies, rather than trying to compensate with ever larger positions.

I continued to work at Simplex for a while longer, but eventually grew bored and quit. A couple of years later, I co-founded Quandl. But that’s a story for another day!

Toronto, Dec 2024

Introduction

Data is the new oil, they say; data is the new gold. Very well, then: oil costs eighty dollars a barrel, and gold is twenty three hundred dollars an ounce. How much does data cost?

It’s a meaningless question.

The factors driving oil prices may be complex, but there’s a well-established consensus on transaction criteria: volume, location, grade, and date. There are exchanges which specify delivery rules for benchmark contracts like WTI, Brent and Dubai. When you buy a barrel of crude, you know what you’re getting.

Data ... is not like that. Data is inherently heterogeneous. Dataset A and Dataset B may both be bits on a drive somewhere, but often have absolutely nothing in common beyond that. Different fields, schemas, specs; different themes, coverages, informational content; different consumers, use cases, and value. Every barrel of WTI crude is identical; no two datasets are identical.

Does this mean that data pricing is all art, no science? Not quite. Data’s innate heterogeneity means that no criteria can be absolute; there’s no single formula you can apply. But there are definite principles that generalize across a wide range of data assets.

I was the co-founder and chief data officer of Quandl, a successful data marketplace (now owned by Nasdaq). In that role, I evaluated thousands of data assets and priced hundreds of data products. I can say, with some confidence, that I’ve priced more — and more varied — data products than almost anyone in the world.

In this essay, I’ll share a few of the things I’ve learned. I’ll start with some basic axioms of data value; then I’ll lay out the implications of those axioms. But first, an important aside on why this matters now.

There’s a new buyer in town

Historically, two industries have dominated transactions in data: finance, and adtech1. These are the only two industries with multiple buyers, multiple valuable data assets, multiple use cases; also the ability to pay consistent material recurring revenue for data2.

This has changed. There’s a new buyer in town: AI. AI models have an insatiable appetite for training data — it's almost certainly the current limiting factor for growth in their abilities — and so their sponsors (MSFT, OpenAI, Anthropic and friends) go to great lengths to acquire such data. But the data that they need isn’t necessarily like the data that finance and adtech need, and their utility/value curves are also different. This has implications for the pricing of training data, rendering much past intuition irrelevant. I’ll try to highlight some of these new effects as we go through our list of pricing principles. Let’s go!

PART ONE: AXIOMS

Data has no innate value

Starting with the obvious (but often misunderstood): data has no innate value. The value of data comes from the value of what can be done with it3. Therefore, every discussion of price has to begin with understanding that value: how the data will be used, and by whom.

Data value depends on the use case

It’s meaningless to talk about data value without specifying how the data will be used. Financial statements aren’t useful for an advertising campaign. Audience profiles aren’t useful for equity analysis. But flip those around, and the datasets are not just useful; they’re essential. The use case is everything.

Data value depends on the user

The identical dataset, with the identical use case, may nonetheless offer different value to different users. The value of training data to OpenAI is very different from its value to a solo hacker. The value of capital markets data to Citadel is very different from its value to a retail investor. Every data user is unique.

Some of this is just a scale effect; the same data has more impact when deployed against millions of customers or billions in capital or trillions of parameters. But it's also a capability effect: Citadel and OpenAI can get a lot more out of the same data than a smaller, less sophisticated user. A large part of data pricing is finding useful proxies for these two effects. We'll come back to this idea.

Data is fundamentally additive

Data is additive in a way that software is not. If you have one CRM, you don't need a second; adding HubSpot and Pipedrive to your Salesforce doesn't improve your sales performance. The same goes for your ticket tracking system, your HRIS, your payroll and expense systems, indeed all your software. Duplication is anathema; don’t repeat yourself.

This is not the case for data! Adding more names to a list of prospects makes the list more valuable. And in fact you can add data along more than one dimension: more names, but also more fields for each name, and more details for each field; you can even combine three perfectly overlapping datasets to generate a higher-quality merge. Additive!

Data is actually rivalrous ...

An almost universal misconception is that data is a ‘non-rival good’: person A using a dataset does not prevent person B using the same data. This is incorrect.

It’s incorrect because it focuses purely on the mechanics of data consumption. It’s trivial to duplicate most datasets, and in that sense, yes, A and B can both ‘use’ the same data. But that does not mean they derive the same value from it.

Financial markets provide the clearest example of this. If a particular dataset (satellite images, say, or credit card transactions) holds unpriced information content, then hedge fund A can trade on that data to make money. But once that's done, hedge fund B cannot! The opportunity goes away. The data is effectively rival: only one party can act on it.

The non-rival misconception stems from treating data as having innate value. If that were the case, merely having the data would suffice. But as we know, the value of data lies in the value of what you can do with it. And that’s often contingent on nobody else doing the same thing. In actual data business practice, the more valuable a dataset is, the more effort its owners expend on keeping it exclusive, proprietary and protected. This wouldn’t happen if data were a non-rival good.

Note that with the planet-scale datasets used for AI model training, even the starting assumption — that it’s trivial to duplicate and consume data — is incorrect; these datasets are huge.

... until it isn't

Advantages don’t last forever — not in capital markets, not in AI training, not anywhere. Datasets become commoditized, and substitutes arise. At this point, the data becomes non-rival.

This hurts most data vendors, but is a massive boost for a select few. The dream of every data owner is for their data to become table stakes — commoditized, but essential.

Data assets have well-defined lifecycles

When pricing a data asset for sale, it’s critical to understand where it is in its specific lifecycle.

Early on, neither the dataset nor the market are mature enough to drive value. The dataset is incomplete, inaccurate, slow, irrelevant. The market lacks the tools or sophistication to use the dataset effectively. There may be exploratory activity, but transactions are rare.

The second stage is when early adopters realize there is alpha to be found in the data — better targeted ads, or better model evals, or excess market returns. (I'm using the word alpha here in its broadest sense — an edge over the rest of the industry). At this point the data is super-valuable (and priced accordingly) but the audience is still narrow. Most data assets, by the way, never make it this far.

The third stage is for a given dataset to become widespread in its industry. Substitutes proliferate, and prices decline. With more suppliers and also more users of this data, its alpha decays; there’s still some there, but not much4.

The final frontier is when the dataset becomes table stakes: when people use it, not because it confers an advantage, but because not using it would put them at a disadvantage. At this point, prices go back up — not as high as during the alpha phase, but higher than during the decay phase — and usage expands dramatically. This is the best position to be in as a data owner; companies that get to this stage can tap years if not decades of revenue from a single asset5.

It's the marginal lift that matters

Combining point 1 (the value of data lies in what can be done with it) and point 3 (data is fundamentally additive) leads us to a rigorous definition of data value:

The value of data is the value of the marginal change in actions taken after adding the data to your business process. 

where that business process could be your model training step, your quant trading strategy, your ad auction, or anything else that’s data-driven.

Finding ways to estimate this marginal value is the key to effective data pricing6 7.

PART TWO: IMPLICATIONS

Unique data is uniquely valuable

We now begin to understand why truly unique (aka proprietary) data is so valuable.

First, it's universally additive — you can combine it with almost any existing corpus, and increase its utility, and thus its value8.

Second, the data owner can (attempt to) control its lifecycle. By controlling the pace of commodification / alpha decay / transition to table stakes, you maximize the area under the [price x transactions] curve.

Third, if and when a unique data product becomes table stakes, its owner has monopoly power; this is tantamount to collecting a tax on an entire industry. Paying for the data is just ‘the cost of doing business’.

Beware of functional substitutes

The value of data is the value of what can be done with it. Therefore, completely different datasets may be competitors, vying to offer the same value! This is ‘functional substitution’, and it throws off many who think their data assets are unique.

Here are two examples:

• Foot traffic, email receipts, and credit card transaction logs are all very different datasets, but they offer similar insight: what are people buying at the mall. Your foot traffic data may be proprietary and unique, but credit card transactions are a functional substitute, and so your data isn’t really unique in the value it offers.

• Demographic profiles, social network behaviour, and search histories are all very different datasets, but they offer similar insight: what are people interested in and hence what are they likely to buy.

Note two things here. First, functional substitutes can be (and usually are) additive. You can combine them to paint a richer picture of reality, especially if the underlying sources / mechanisms are sufficiently uncorrelated.

Second, there’s usually a hierarchy of value among functional substitutes, and it has to do with which dataset is ‘closest to the sun’ — i.e., most tightly linked to the underlying event of interest. Yes, demographics can sometimes predict buyer intent, but it's not as powerful (or valuable) a signal as e-commerce search activity, because the latter is much closer to the act of purchase.

Standard software pricing fails for data ...

The vast majority of enterprise software today is sold via tiered subscription plans. Tiers — with labels like bronze, silver, gold; or maybe individual, professional, enterprise — are distinguished by features, seat count, usage limits and other proxies for value delivered9.

Unfortunately, most of these proxies don’t work for data10. For example:

• Pricing by seat doesn't work, because data value doesn’t scale linearly with user count. Adding one more data user is not like adding one more software user or login account. Data is used by teams; you can sometimes charge per team, but teams are heterogeneous in size, use case, and value generated.

• Pricing by feature (in the software sense, not the ML sense) doesn't work. Indeed it’s not clear what a ‘data feature’ would even be. There’s rarely a slider that makes sense here.

• Pricing by raw volume (i.e., how many terabytes are you buying) doesn't work unless the data is perfectly fungible, somewhat commoditized, and probably quite unstructured. But note that this actually is the case for some AI training datasets.

• Pricing by API call doesn't work unless the data changes / decays extremely rapidly. (But if the API call triggers an action, not just a download, then this can work; see the discussion of wrappers below).

• Pricing by download —the equivalent of paying for an on-prem software license — doesn’t work because data is trivial to copy, and auditing is hard.

But there are other angles you can take that do work:

• Pricing by structured volume works: you pay more for more records, or for more fields (if the data is tabular-ish), or for more coverage, or for more granularity. For example, paying for more profiles (in a marketing database) or longer history (in a financial database).

• Pricing by quality kind of works. Note that quality exists across multiple dimensions — accuracy, completeness, annotations, structure. This deserves an entire section to itself; see below.

• Pricing by access works. You can charge more for speed, recency, update cadence, exclusivity, and custom usage rights. These are genuine sliders, and many data vendors will explicitly tier their pricing based on what buyers want and need11.

• Pricing by use case is something you can do with data that you cannot do with software. Software use cases are largely fixed: DocuSign won’t do your accounts, and Quickbooks won’t manage your signatures12. But the same dataset may help different users perform completely different tasks. (This of course is why data pricing is hard!)

• Pricing by customer scale works. This is obvious, but worth being explicit about: large customers get more value from a given data asset than small ones, hence they should be willing to pay more. And this is not just a deep-pockets effect; it's rational behaviour on all sides.

• Pricing by business unit is a special (and useful) case of pricing by customer scale; the business unit here could be a team, or a geography, or a product line, or even a specific model generation. Customers are usually more amenable to this slider than paying straight up for scale.

AI changes one thing in all the above, which is that pricing by raw volume now works. Data quantity is unreasonably effective in improving model outcomes, and so it makes sense to not quibble over structure, quality and access, and just pay by the petabyte.

... unless you wrap the data in software …

A common pattern is to identify the most valuable use case for your data asset yourself; wrap it into a software app; and then sell the app. Google is the canonical example here: one of their data assets is ‘knowledge of customer intent’, which they wrap into their Ads business. Advertisers covet the ability to get their product in front of the ‘right’ customers; they’d do this themselves if they knew who the right customers were; Google has that info, and so does it for them, at a price.

Experian does the same for consumer credit data; Bloomberg built a whole terminal to manage access to its financial data assets; and for that matter ChatGPT is just a user interface, built on GPT-4’s model weights, which in turn rely on oceans of training data. (One could argue that given the pace of progress in both closed- and open-source LLM architectures, it’s the training data — and resulting model weights — that's the true secret sauce here.)

Wrapping your data into software has two major advantages: first, it’s now much easier to link data value to value delivered; and second, you can now sell using the traditional software pricing axes. (BBG has per seat, Experian has per inquiry, Google has the incredible keyword auction.)

Note, by the way, that monetizing your data asset by wrapping it in software is not at all the same as using data to improve the performance of your software asset. (This is an extremely common category error.) The latter is the classic ‘data learning loop’ used by businesses from Amazon to Zendesk; but data is not the core offering for those companies; data merely helps them optimize their software and operations.

… or you use the data to wrap a service

An increasingly common business model is to deliver what is essentially a service, in the form of a data asset.

Consider Scale.ai or Clearbit or Datavant. They each offer what is fundamentally a service — data labelling, profile enrichment, and medical record anonymization, respectively — but they offer this service in the form of a data product, that you access via download or API call or common key.

The beauty of this is that unlike traditional service models, this approach scales up: perform the service once, but sell it many times. And just as with software wrappers, you can monetize service wrappers along the traditional software pricing axes — by action, or by API call, or even by seat.

Data quality is multi-dimensional

High-quality data costs more. But what exactly do we mean by high quality? It turns out that — as with all things data — the definition of data quality depends on the use to which the data is put.

Quant and systematic hedge funds — perhaps the largest purchasers of raw data in the world — care about accuracy and precision. Their business is predicated on identifying data points that represent violations of efficient markets — outlier prices, predictable patterns, unexpected correlations, internal inconsistencies, misunderstood risk. Bad data points are dangerous because they suggest violations where none exist.

Adtech platforms — the other major buyer/user of data at scale — care about coverage and depth. Outliers and individual bad/missing data points don't matter that much; their business is all about capturing as much of the heart of the distribution as possible, so that when an advertiser defines a target profile, keyword or behaviour, they have a rich audience to offer them.

AI models in need of training data — the new buyer in town — care about structure and internals. As of right now (and this field evolves incredibly rapidly, so who knows how long this will remain true), the quality attributes of training data that appear to have the most marginal impact on model performance are structural (as opposed to informational) cleanliness (i.e. deduped, denoised, debiased data); annotations; data diversity; and perhaps some amount of domain specificity.

Note that these quality factors aren't mutually exclusive; ideally you want all of them (and more), no matter what your use case. But their relative importance varies.

How does this affect data pricing? Price discrimination is the obvious answer: the same data asset may have different value to different users, based on its distribution of quality factors. (This is just a restatement of axiom 3 above.)

A subtler point is that many of these quality attributes are improvable. You can annotate raw data; you can combine datasets to increase coverage; you can cross-reference datasets to boost accuracy. Taking these actions is an easy way to boost data value, either for external sale or for internal use.

(Who is best placed to take these actions? Data producers have the advantage that they can amortize the costs of these actions across multiple buyers; data consumers have the advantage that they know their own use cases and can therefore prioritize the most effective actions.)

Value also derives from non-quality factors

Depending on the user profile and specific use case, there may be other drivers of dataset value:

• Provable compliance becomes more valuable as a data asset traverses its maturity curve. In the early days, when the utility of the data is unknown, participants are less inclined to pay for compliance. But as the ecosystem matures, this becomes more of a priority13.

• Data provenance is valuable because it makes compliance easier and it reduces the scope for error introduction. Primary sources are the gold standard here.

• Uncontaminated data is incredibly valuable in any use case involving prediction (gen AI, or quant investing). This is data that has never been mined or fitted against, and that therefore has not biased your evals. Uncontaminated data is a non-renewable asset; the minute you use it to either train or test, it loses its virgin status, and all future uses must reckon with the risk of overfitting.

• Data fungibility — the idea that every record in your dataset is like every other — is good because it makes the data more tractable, more additive, and more repeatable in its value generation.

These are all internal drivers of value: attributes, not necessarily what you would think of as “quality”, that are nonetheless inherent to a dataset. But there are also external drivers of value, and many of them have to do with table stakes status.

Table stakes data comes in different flavours

Datasets can attain table stakes status in different ways:

• Some datasets become standards for data exchange. S&P’s CUSIP for publicly traded securities, D&B’s DUNS for private companies, Datavant’s universal patient key for medical records, LiveRamp’s RampID for advertiser audiences are all examples here, albeit for varying degrees of “table-stakes-ness / industry adoption”.

• Some datasets become evaluation benchmarks. The dream here is to offer “ground truth” that every industry participant has to measure themselves against. S&P, Nasdaq and MSCI do this for investor performance via their market indices. Nielsen does this for TV advertising: both advertiser and network rely on Nielsen ratings to mark their contracts.

• Some data owners have quasi-monopoly power, either driven by a data flywheel, or from a non-data network effect of some sort. Meta and Google are examples of the former: their knowledge graphs, constantly updated by user content and intent, dominate online advertising. Bloomberg is an example of the latter, thanks to the ubiquity of its terminal.

• Some datasets rely on bundled usage to become table stakes. To work in a given industry, you have to (effectively) purchase a dataset; and conversely, to use the data, you have to be licensed to work in that industry. The relationship between individual Realtors, various Realtor organizations / brokerages / local certification bodies, and the MLS real estate database(s) is a good example of this.

Usage rights are monetizable

The value of data is the value of what you can do with it. Therefore, the more you’re allowed to do — the more usage rights the seller grants you — the more you should be willing to pay! This is a pricing effect that’s completely independent of quality, quantity, dataset internals, or table stakes status.

Common usage rights include:

• Scope of use rights: in-house, in-product, customer-facing, full re-distribution

• Ownership rights: transfer versus license, explicit versus implicit permissions, approved and forbidden use cases

• Audit rights: usage tracking, observation, compliance, post-contract deletion

• Derived data rights: modifications, contribution and attribution (e.g. if the data is combined with other datasets), ownership of downstream products

• Compliance rights: liability, reps and warranties on the data, legal shields

Sophisticated data contracts delineate very carefully what you can and cannot do with the data you license, and these allowances often have a dollar value attached to them.

Payment-in-kind is an emergent pattern

An interesting recent development is the emergence of payment-in-kind as a compensation pattern for data assets. When (say) OpenAI licenses content from a news media org, part of what they offer is to highlight that organization’s brand in chat conversations and link placements. This has multiple benefits: the media org gets ‘AI-organic’ traffic; the LLM gets fresh data; both sides get to use click-through and engagement data to measure the actual lift and value of the content; and there’s a recurring component to it all. Squint and you can even see analogies to Google’s one-two punch of Adwords and SEO for content creators.

For AI, data quantity matters — a lot

Does a small amount of high-quality (accurate, factual) data outperform a large amount of ‘good enough’ data? For AI use cases, the answer is increasingly in the negative.

We see this in ideas like ‘the unreasonable effectiveness of data’, and ‘scale is all you need’. There seems to be no upper limit to how much better models become, the more training data you throw at them. Fine-tuning and domain-specificity and especially human heuristics tend to plateau; this is ‘the bitter lesson’ that many researchers learned the hard way.

(A telling recent example is the way GPT-4, a one-shot model trained on vastly more tokens, outperforms BloombergGPT on almost all financial analysis tasks.)

One consequence here is that Sturgeon’s Law — “90% of everything is junk” — no longer holds so strongly for data. In the past, the vast majority of datasets in the world held no insight, catalyzed no actions, and had no value. AI’s voracious appetite for all sorts of training data means this is no longer the case: even the junky stuff helps model evals (perhaps not by a lot, but it’s not zero).

So does quantity unequivocally beat quality for AI? It's not quite that simple, because we also care about marginal impact and hence ROI. Sprinkling just a little bit of quality on top of your massive corpus — for example, via simple de-duping — has dramatic effects on model performance. As training sets grow ever larger, it’s often more efficient to do this than to acquire the next token; beyond a certain point, data quality scales better than data quantity14.

The implications for data pricing are straightforward. Quantity matters; high-quality quantity is even better; and there's no upper limit.

Recurring revenue is always the key

The big challenge with pricing data for AI use cases is that most of the training value comes from the historical corpus, meaning there's limited opportunity for recurring revenue from ongoing updates. Reddit’s decade-plus of archived content is far more interesting to OpenAI than an incremental day, week, or month of new content15.

Recurring revenue is, of course, the secret to effective long-term data pricing and economics. It works best when the data has some combination of short shelf-life, regular update cadence and repeatable usage. AI training datasets typically don’t have these attributes16.

There are ways to mitigate this. You can charge higher prices, of course, to make up for future payments; or you can try to charge per model or per training run.

The long-term solution here is to build a data flywheel — a ‘perpetual data machine’ that generates or captures a steady stream of new data, or constantly improves your existing data. These flywheels can be based on user behaviour (Reddit’s post data), or business model (Nasdaq’s exchange data), or tech (Google’s content-intent loop), or many other archetypes; the key is that they constantly offer new data to train against.

Synthetic data offers new economics

Synthetic data pipelines offer a way to generate unlimited quantities of high-quality, always-new training data, at far lower costs than acquiring comparable data ‘in the wild’. (Both the quantity and cost effects span orders of magnitude). And early indications are that well-constructed synthetic data is almost as effective as natural data in training frontier models. This offers the tantalizing prospect of bootstrapping the data curve indefinitely: use each generation of LLMs to generate synthetic data to train the next generation17.

The viability of synthetic data and the importance of quantity over quality are both negatives for the price of existing data assets. Proprietary data ain’t what it used to be!

(That being said, purely synthetic data tends to degrade over time, in a sort of entropic drift. A parallel phenomenon may occur in the wild, as LLM output proliferates across the web. The worst-case here is a Gresham effect, whereby bad content drives out good. All of these possibilities argue that provably-human proprietary data has its place after all.)

Sales cycle determines ACV

Normally, ACV determines sales cycle. Your price point decides your go-to-market motion (enterprise sales, inside sales, self-serve etc.), and hence the time it takes to close a deal. This is true for data sales as well, but there’s an additional effect where the causation runs the opposite way: sales cycle determines ACV.

Here’s why. In enterprise software, much of the time taken to close a sale is spent arming your champion, doing feature bakeoffs, getting buy-in from various stakeholders, choosing tiers and negotiating prices, onboarding users and so on. Very little of that process translates to data sales (no tiers, no features, no users).

Instead, much of the time in the data sales cycle is spent onboarding, linking, sampling, testing and using the actual data. And this is expensive; MAD-stack engineers don’t come cheap. The more time and resources a buyer spends on a dataset, the stronger a signal it is that the buyer thinks the dataset is valuable.

Given how opaque data value often is, this signal is meaningful — especially if the buyer is large and sophisticated. Sellers recognize this, and bump up their prices18.

Legibility determines market size

Much of this essay has been about pricing individual data assets or transactions. But to data owners, it’s also important to know the size of the market; after all, revenue is not price alone, it’s price times transactions. The most lucrative data assets are often those with a (relatively) low price and a large transaction base.

What determines market size? Legibility.

The more legible a dataset is — the easier it is to objectively compute its ROI — the larger its market. This manifests as more customers, more types of customer, more usage within a given customer, higher prices and lower acquisition costs19. It’s no coincidence that adtech and finance are the most lucrative data verticals: those are also the industries where it’s easiest to put a dollar value on databases, datasets, and sometimes individual data records. Will AI training data reach the same level of maturity? We shall see!

Conclusion

So there you have it: 5000+ words on data pricing. We’ve covered use cases and users; quality and quantity; internal and external value factors; pricing axes and maturity curves; table stakes and usage rights; and much more. If there’s anything I’ve missed, I’d love to hear from you in the comments. Happy data pricing!

If you liked this essay:

• Share it with others who might like it.

• Subscribe to my newsletter, and get your friends to subscribe too!

• Read my previous essays on related topics:

  • The Economics of Data Businesses

  • Data in the Age of AI

Subscribe now

I’m an active early-stage investor in companies with interesting data assets; if this is you, please reach out.

I write infrequently, but I like to think my essays are worth the wait. Your subscriptions, shares, comments and likes are the strongest incentive for me to write more!

Toronto, May 2024

For many years, Marc Andreessen’s Twitter bio read: ‘Strong Opinions, Weakly Held’. What do these words mean? I’ve heard quite a few explanations, and they all seem to miss the point.

Imagine you’re lost in the countryside, and you come to an unmarked fork in the road. You have no idea which direction leads to the nearest village, but night is falling, and you must choose. What do you do?

The absolute worst thing you can do is stay put. Without the data to make an informed choice, you make no choice at all, and are paralyzed. This is a bad idea, and everybody agrees that it’s a bad idea, from Aristotle onwards.

So you pick a direction at random, and off you go. But here’s the subtle trap. Because you worry that your choice is incorrect, you go slowly.

This is the second worst thing you can do. Your lack of certainty has led to a lack of commitment. A slow pace may be psychologically comforting, but as a method of getting un-lost, it’s terrible.

Part one of Andreessen’s aphorism addresses this: strong opinions. If you’re going to do something, do it at full throttle. Night is falling; get to the village as quickly as you can.

But if you find that the road you’re on dwindles into nothingness, don’t be stubborn about it; turn around! This is the second part of the aphorism: weakly held.

The beauty of this combination is that it maximizes rate of learning. The faster you go down a path (strong opinions), the sooner you’ll know whether you need to reverse direction (weakly held). SOWH is a phenomenal learning tool, and not just for lost travellers.

Most people utterly fail at this combination. They don’t truly commit to the path they’re on — they don’t give it all they’ve got — but at the same time, they’re unwilling to change their direction. Having made a choice — even if the choice was made on limited or outdated information — they feel they have to stick to it.

Let's dig a bit deeper. When does SOWH work, and when does it fail? One way to answer this is with a classic consultant's 2x2.

There are 4 possible combinations of commitment and certainty:

• Strong opinions strongly held is the realm of determinism. You know what you need to do, now go and do it. If you’re in a well-defined and well-understood problem space, this can be a great strategy.

• Weak opinions weakly held is the path of least resistance. You don’t commit deeply, and you don’t mind changing directions. This strategy doesn’t necessarily lead to forward progress, but it’s low-cost and inoffensive.

• Weak opinions strongly held is surprising. You’d think this quadrant would be empty: after all, who in their right mind would be stubborn and didactic about something they have no information about? The answer is, quite a lot of people. This is fundamentally a static and safe choice: never change your mind, and never push hard enough that your mind needs changing.

• Strong opinions weakly held is the true empty quadrant. People don’t like to admit ignorance, and they don’t like to appear changeable. (Understandably so; many organizational cultures penalize them for those attributes.) And yet it’s the optimal strategy if what you care about is rate of learning1.

SOWH is a technique for decision-making under imperfect information.

This is not quite the same as decision-making under uncertainty. The premise of SOWH is that information exists — information that can be used to mitigate the uncertainty — but you don’t have it yet. SOWH is a way of acquiring that information as efficiently as possible.

I’ve found myself using variants of SOWH all through my career: as a trader at a hedge fund, then as a startup founder, and now as an angel investor. All three professions require(d) me to place bets without knowing all the facts; in all three professions, I was rewarded for rethinking those bets when new facts came to light. But I didn’t have a formal way of defining this approach until I came across Andreessen’s aphorism.

Strong Opinions, Weakly Held is now my mantra. May it soon be yours as well!

Toronto, Oct 2023

End Notes

• I’ve been angel investing in tech startups for over a decade, with a strong track record. I’m now launching my own syndicate on AngelList, and opening up my first deal. If you’re an accredited investor interested in early-stage opportunities, I’d be happy to hear from you.

• Last week’s short note on ‘Mediocre Success’ turned out to be quite popular, so I thought I’d write another short note, on another mental framework that maximizes rate of learning for startups. Let me know if you like it! (But the next piece will probably be more of a traditional Pivotal deep dive).

• As always, if you enjoyed this piece, please subscribe, and share it with others who you think might like it.

Subscribe now

Footnotes

Now, this is not a simplistic exhortation to go big or go home, hit a home run or strike out, be blindly ambitious, any of those things. It's subtler than that. Let me explain.

Startups are defined by uncertainty. As a founder, you have to discover almost everything about your business: What is the product? Who are your customers? How will you reach them? How much will they pay? Who are your competitors? How is the industry evolving? The list goes on.

One common way to answer these questions is, essentially, the scientific method, applied to tech startups. Frame a hypothesis; run an experiment to test the hypothesis; confirm or disprove the hypothesis; learn and iterate and learn and iterate.

For example, your hypothesis could be that ‘Cold-calling customers will lead to sales’. So you hire a couple of sales reps and tell them to cold-call 100 customers. If you get 30 new sales out of that (a terrific hit rate) — great, the hypothesis is true! You can hire more sales reps and double down on this tactic. And if you get 0 new sales — that’s also great, the hypothesis is false! You can move on to other acquisition tactics like FB ads or SEO or events or whatever. Either way, your experiment worked in confirming or disproving the hypothesis.

The worst outcome, the very worst outcome, is to get a small but non-zero number of sales — say 1 or 2. Because now you're in a bind. Do you double down or pull the plug? Does cold-calling work or not? Could it be that the method works, but the sales reps aren’t hustling enough, or they’re not following the right script, or they’re not calling the right people? Or maybe the method is flawed and your reps just got lucky? You just don't know!

That’s the danger of the mediocre success. The point of startup experimentation isn't the success itself; it’s the learning that comes with clear-cut success or failure. You don't really care about the sales revenue generated by your first two reps; you care about whether this is a strategy you can scale to dozens and then hundreds of reps, or whether you need to use a completely different strategy. It’s all about the learning. And mediocre successes don’t give you any learning.

Unfortunately, there’s a natural human tendency to hedge our bets — to make design choices in our experiments such that a mediocre success is the most likely outcome. There’s also a tendency to only do experiments that you know in advance will work — but such experiments are not useful: the delta in information is close to zero. For example, and continuing the sales experiment: as a founder, you could do the calls yourself; reach out only to the very best, most qualified prospects; create custom collateral; offer sweetheart pricing. All those actions will increase the chances of closing any one deal, and as a result they’re very tempting, but do they tell you if cold-calling is a viable sales strategy at scale? Nope.

Making it worse is that we’re all heavily socialized to aim for mediocre success. Schools, universities, large organizations — they don’t want big swings and big misses; they want safety and consistency. A steady 7 is better than 10s interspersed with 0s. This might work well in structured, predictable environments, but in startup-land it’s anathema.

So when a startup comes to me with an idea for an experiment, the one thing I tell them is: make sure that there’s a well-defined distinction between success and failure. Don't fall in the messy middle. If the hypothesis fails, make sure it fails clearly and unambiguously; if it succeeds, make sure it succeeds equally clearly and unambiguously. And remember that a hypothesis failing means the experiment succeeded; you learned something. That’s what it's all about.

The worst outcome is a mediocre success!

Toronto, Sep 2023.

End Notes

• This essay is about tactics, and specifically about how mediocre successes hamper your learning. There’s a whole separate essay to be written about mediocre success in the realm of strategy, which is largely about opportunity cost.

• I’m experimenting with the content and form factor of this newsletter. In addition to the long-form, deep-dive pieces that I’ve written so far, I’m going to write some shorter single-topic pieces like this one, hopefully at a higher cadence. I’ll also widen the scope to include more personal anecdotes, tactical nuggets and random riffs. Let’s see how it goes!

• If you enjoyed this article, please subscribe, and share it with 2-3 other folks who you think might enjoy it as well. I’d like to get more subscribers, and Twitter/X is no longer the reader-content discovery engine it once was.

Subscribe now

How does the sudden explosion in AI affect data and data businesses?

In this essay I present two answers and explore their implications:

• A material change in the relative values of information and compute, with implications for both software and data business models.

• An even more dramatic change in the absolute quantities of data and compute available in the world, with implications for trust, identity, quality and curation.

But I’ll begin with some history. Data and data loops are the key driver of the best performing business models of the last decade, and through them, of the AI revolution itself. Let’s find out how!

Memory is Destiny

I taught myself to code on a IBM PC clone running MS-DOS on an 8086. This was in the late 80s; the computer was cheap, functional and remarkably ugly, but it worked. And it had what was — for its time — a princely amount of storage: 30 whole megabytes of hard disk space.

A decade later, working my first job as a programmer, storage was easier to come by, but it was never out of my mind. I cared about memory leaks and access times and efficiency. Creating a clever data structure to hold yield curves for rapid derivatives pricing was one of my prouder achievements at the time. We didn't store everything, only what we needed to.

Another decade later, I just didn't care. At Quandl, the startup I founded, we saved everything. Not just the data assets that formed the core of our business, with all their updates and vintages and versions, but also all our usage logs, and API records, and customer reports, and website patterns. Everything.

What happened? Well, this:

The Data Explosion …

Quantitative changes that result in qualitative changes are always worth watching. Because it wasn't just Quandl that stored all its data; it was everyone. Everyone everywhere, all at once.

The 2010s were the decade of the data explosion. Driven by falling hardware costs, and — equally as important — by business models that made said hardware easy to access (shoutout to Amazon S3!), the world began to create, log, save, and use more data than ever before.

… and Business Models

A huge number of the business models that ‘won’ the last decade are downstream of the data explosion:

• the entire content-adtech-social ecosystem

• the entire ecommerce-delivery-logistics ecosystem

• the infrastructure required to support these ecosystems

• the devtools to build that infrastructure

How so?

Consider advertising, still the largest economic engine of the internet. Facebook, Reddit, Youtube, Instagram, Tiktok and Twitter all rely on the exact same loop, linking users, content and advertisers:

Companies are able to offer users unlimited free storage for photos, videos, blog posts, updates and music; the content attracts more users, who attract advertisers, who subsidize the platform, and the cycle continues. None of this is possible without cheap, plentiful storage.

Or consider delivery and logistics. It's easy to take for granted today, but the operational chops required to power same-day e-commerce, or ride-sharing, or global supply chains, are simply staggering. The famous Amazon and Uber flywheels are just special cases of a data learning loop:

These flywheels require up-to-date knowledge of customer locations, purchase and travel habits, store inventories and route geographies, driver and car availability, and a whole lot more. Again, none of this is possible without cheap, plentiful data1.

Note, incidentally, that all these flywheels are not just powered by data; they generate new data in turn. The data explosion is not just an explosion; it's a genuine chain reaction. Data begets data.

Data Is Software’s Best Friend

All these business models are, essentially, software. And that's no surprise! Data and software are two sides of the same coin.

The software used to optimize businesses is useless without data to apply itself to. And data is worthless without software to interpret and act on it.

More generally, tools are useless without materials; materials don't have value unless worked on with tools.

Complementary Inputs

Economists call these ‘perfectly complementary inputs’: you need both to generate the desired output, and you can't substitute one for the other. An immediate consequence is that if the price of one input falls by a lot — perhaps due to a positive productivity shock — then the price of other is almost certain to rise.

Imagine you're a tailor. To sew clothes, you need needles and fabric, and you're limited only by the quantity of each of those you can afford. (Needles and fabric are complementary inputs to the process of sewing).

Now imagine that the price of needles plummets, due to a new needle-manufacturing technology. Your response? Use the savings to buy more fabric, and thus sew more clothes! But if every tailor does this, then the price of fabric will rise. Tailors and consumers are both better off, the total quantity of clothing produced goes up, but the relative value of needles and fabric has changed.

For the last decade plus, the quantity of data in the world has been exploding: its price, therefore, implicitly declining. Software has been the relatively scarce input, and its price has increased: you can see this in everything from the salaries of software engineers to the market cap of top software companies. Software ate the world, with a huge assist from cheap, plentiful data.

And then came GPT, and everything changed.

The Peace Dividend of the Content Wars

GPT is a child of the data explosion. The flood of new data, generated by users but also by content farms and click factories and link bots and overzealous SEO agencies, necessitated the invention of new techniques to handle all that data. And it was a team of researchers at Google who wrote Attention is All You Need, the paper that introduced the Transformers architecture underlying pretty much every modern generative AI model.

The original peace dividend was that of the Cold War. Transistors, satellites and the internet were all offshoots of that conflict; today, they're used for more — far more — than just launching and tracking missiles. Similarly, although LLMs were invented to manage the data spun off by the Content Wars, they’re going to be used for a lot more than just Search.

The Compute Explosion

GPT is prima facie a massive productivity boost for software. Technologists talk about the 10x programmer: the genius who can write high-quality code 10 times faster than anybody else. But thanks to GPT, every programmer has the potential to be 10x more productive than the baseline from just 2 years ago.

We are about to see the effects.

Move over data explosion; say hello to the compute explosion!

Data Rules Everything Around Me

The first and perhaps most obvious consequence of the compute revolution is that data just got a whole lot more valuable.

This naturally benefits companies who already own data. But what’s valuable in an AI world is subtly different from what was valuable in the past.

Some companies with unique data assets will be able to monetize those assets more effectively. BloombergGPT is my favourite example: it’s trained on decades of high-quality financial data that few others have. To quote a (regrettably but understandably anonymous) senior exec in the fin-data industry: “Bloomberg just bought themselves a twenty year lease of life with this”.

Other companies will realize that they are sitting on latent data assets — data whose value was unrecognized, or at any rate unmonetized. Not any more! Reddit is a good example: it's a treasure trove of high-quality human-generated content, surfaced by a hugely effective moderation and upvoting system. But now you have to pay for it.

You don't need huge content archives or expensive training to get meaningful results. Techniques like LoRA let you supplement large base models with your own prop data at relatively low cost. As a result, small custom data can hold a lot of value.

Quantity has a quality that’s all its own, but when it comes to training data, the converse is also true. ‘Data quality scales better than data size’: above a certain corpus size, the ROI from improving quality almost always outweighs that from increasing coverage. This suggests that golden data — data of exceptional quality for a given use case — is, well, golden.

Picks and Shovels, Deals and Steals

The increasing value of data has some downstream implications. In a previous essay, I wrote about the economics of data assets:

The gold-rush metaphor may be over-used, but it’s still valid. Prospecting is a lottery; picks-and-shovels has the best risk-reward; jewellers make a decent living; and a handful of gold-mine owners become fabulously rich.

The very best data assets, reshaped for AI use cases, are the new gold mines. But there are terrific opportunities for picks-and-shovels specifically designed around the increased salience of data in an AI-first world:

• tools to build new data assets for AI;

• tools to connect existing data assets to AI infra;

• tools to extract latent data using AI;

• tools to monetize data assets of every sort.

More generally, the entire data stack needs to be refactored, such that generative models become first-class consumers as well as producers of data. Dozens of companies are emerging to do precisely this, from low-level infra providers like Pinecone and Chroma, to high-level content engines like Jasper and Regie, to glue layers like LangChain, and everything in between.

Quite apart from tooling, there's an entire commercial ecosystem waiting to be built around data in the age of AI. Pricing and usage models, compliance and data rights, a new generation of data marketplaces: everything needs to be updated. No more ‘content without consent’; even gold-rush towns need their sheriffs.

High-value information assets; a new generation of picks-and-shovels; a reimagined ecosystem for data: the world of data business just got a lot more interesting!

An Accelerating Flywheel

The second major consequence of AI is that the quantity of both data and compute in the world is going to increase dramatically. There’s flywheel acceleration: data feeds the compute explosion and compute feeds the data explosion. And there’s also a direct effect: after all, generative models don't just consume data; they produce it.

Right now the output is mostly ephemeral. But that's already changing, as ever more business processes begin to incorporate generative components.

What does this imply for data?

The Confidence Chain

We’re entering a world of unlimited content. Some of it is legit, but much of it isn’t — spam bots and engagement farmers, deep fakes and psyops, hallucinations and artefacts. Confronted with this infinite buffet, how do you maintain a healthy information diet?

The answer is the confidence chain — a series of proofs, only as strong as its weakest link. Who created this data or content; can you prove that they created it; can you prove they are who they say they are; is what they created ‘good’; and does it match what I need or want? Signatures, provenance, identity, quality, curation.

The first three are closely linked. Signatures, provenance and identity: where does a something really come from, and can you prove it? After all, “on the internet, nobody knows you’re an AI”. Technologically, this space remains largely undefined, and therefore interesting. (The irony is that it’s an almost perfect use case for zero-knowledge crypto — and crypto has been utterly supplanted in the public imagination by AI.)

The last two are also closely linked. Curation is how quality gets surfaced, and we’re already seeing the emergence of trust hierarchies that achieve this. Right now, my best guess at an order is something like this:

filter bubbles > friends > domain experts ~= influencers > second-degree connections > institutions > anonymous experts ~= AI ~= random strangers > obvious trolls

But it's not at all clear where the final order will shake out, and I wouldn’t be surprised to see “curated AI” move up the list.

A nuance that often gets lost here is that curation is not about ranking; it's about matching. If generative AI increases the total volume of content in the world 100x, that does not imply that your quality filter needs to be 1/100 as restrictive. There's no ‘law of conservation of quality’; no upper limit to the number of high-quality creations possible. The limiting factor is your bandwidth as a consumer of content.

Hence the goal of curation is not to rank the ‘best’ X of any category, but rather to find the X that (conditional on a minimum quality cutoff) best matches your profile. (And your profile may not be neutral or objective; this is why filter bubbles are at the very top of the trust hierarchy.)

(Note that the entirety of this section is applicable to B2B use cases and a wide range of data types, not just consumer use cases and social content.)

Compute All The Things

Data becomes more valuable; ecosystems need to be retooled; the data explosion will accelerate; and trust chains will emerge. What about compute itself?

Just like the default behaviour for data flipped from ‘conserve memory’ to ‘save everything’, the default behaviour for software is going to flip to ‘compute everything’.

What does it mean to compute all the things? Agents, agents everywhere. We used to talk about human-in-the-loop to improve software processes; increasingly, we're going to see software-in-the-loop to streamline human processes. This manifests as AI pilots and co-pilots, AI research and logistics assistants, AI interlocutors and tutors, and a plethora of AI productivity apps.

Some of these help on the data/content generation side; they’re productivity tools. Others help on the data/content consumption side; they’re custom curators, tuned to your personal matching preferences.

More pithily, if you’re a Neal Stephenson fan:

An open question is whether these agents will compress or magnify current differentials in power, wealth and access. Will the rich and famous have better AI agents, and get even richer and more famous compared to the masses? History suggests not; that productivity is a democratizer; but you never know.

New Abundance, New Scarcity

The 19th century British economist William Jevons observed a paradox in the coal industry. Even though individual coal plants became more efficient over time — using less coal per unit of energy produced — the total amount of coal used by the industry did not decline; it increased. Efficiency lowered the price of coal energy, leading to more demand for that energy from society at large.

Something very similar is happening with the data-software complex. It’s not just that data and software reinforce each other in a productivity flywheel. It’s not just that generative models produce and consume data, produce and consume code. It’s that the price of ‘informed computation’ has fallen, and the consequence is that there will be a lot more informed computation in the world.

A possibly lucrative question to ask is, where are the new areas of scarcity? The hardware that powers compute and data is an obvious candidate: as the latter exponentiate, the former cannot keep up. Persistent and recurring chip shortages are a symptom of this; my hypothesis is that this is not a problem of insufficient supply, it’s a problem of literally insatiable demand, driven by the Jevons effect.

Another candidate for scarcity is energy. Large model training consumes a huge amount of energy, but at least it’s confined to a few firms. But once you add in accelerating flywheels, the compute explosion and agents everywhere, the quantities become vast. We haven’t felt the pinch yet because of recent improvements in energy infra — solar efficiency, battery storage and fracking — and there’s hope that informed compute will help maintain those learning curves.

Watch out for artificial scarcity. Society may benefit from abundance, but individuals and corporations have different incentives; they make seek to constrain or capture the gains from ubiquitous, cheap, powerful data and computation.

Finally and most provocatively, what happens to human beings in this brave new world? Are we a scarce and valuable resource, and if so why — for that nebulous entity we call ‘creativity’, or for our ability to accomplish physical tasks? Will AI augment human capacity, or automate it away? I believe in abundance and I’m optimistic; the only way to find out is to go exploring. We live in interesting times!

Toronto, 16 May 2023.

End Notes and Requests

• If you found this essay thought-provoking, please share it — on social media, with friends, enemies, or anyone else you think might enjoy it.

• Please subscribe to my newsletter, Pivotal. I write essays on data, investing, and startups. My essays are infrequent, usually in-depth, and hopefully insightful. Also, they’re free.

  Subscribe now

• Writing in public is an exercise in ‘tapping a tuning fork and seeing who resonates’. If this essay resonated with you, come and say hi!

• I’m an active angel investor in tech startups, many of whom fit the templates of companies described in today’s essay:

  • Arima, Citylitics and Daloopa are building new data assets

  • Setyl, Ubico and Getware are extracting latent data

  • Quandri, Mero and Canopy are data capture companies

  • Syro is working on identity and secret

• If you’re building a company along similar lines, I’d love to hear from you.

• [Off-topic] Is high-quality preference-matched content really that valuable? To test this hypothesis, I’ve written a curated guide to visiting Japan. If you like my essay, you might like this guide: it’s selective, deep and informed. Check it out!

Here are two things I believe:

1. Speed is a startup's key advantage.

2. Startups are a marathon, not a sprint.

Unfortunately, they're mutually contradictory!

If startups are a marathon, then staying power should count for more than speed. Conversely, if speed is the key, then why worry about stamina and resilience and the long haul?

One way to resolve this contradiction is to simply say, this is why startups are hard. You have to do both: go as fast as you can for as long as you can. Sprint the marathon.

But I think there's a deeper resolution, and I found it in events from over a hundred years ago. Read on!

1909: The Worst Journey In The World

Apsley Cherry-Garrard was an unlikely explorer. Physically frail, painfully shy, nearsighted to the point of blindness — it was a surprise to all involved when he was selected to join Robert Falcon Scott's ill-fated 1909 expedition to the South Pole.

But Cherry quickly proved his worth. He was a willing worker, cheerful and kind, grittily determined and impossibly brave. Nowhere were these qualities more evident than during ‘The Winter Journey’, a harrowing trip across the Antarctic glacier in the depths of the long polar night to retrieve eggs from a colony of Emperor penguins. Along with two others, Cherry man-hauled 350kg of food and supplies hundreds of miles by sledge across rock and crag and ice, in temperatures reaching -75°C and amid almost total darkness.

This was Cherry's finest hour, a feat of superhuman endurance and staggering courage. Yet it was also this journey that broke him; it rendered him unfit for the more glamorous dash to the Pole itself, and left him sickly and pensive for the rest of his days.

Worse was to come. The next Antarctic summer, Scott's polar expedition party were starving, exhausted, stricken by injury and frostbite and scurvy. Cherry led one of the relief teams searching for them. But he was forced, by bad weather and depleted rations, to turn back — just a few miles miles away from the tent where (unknown to him) Scott and his companions lay helpless and dying. It was a decision that would haunt Cherry for the rest of his life, and it colours every page of his memoir of the expedition, The Worst Journey In The World.

The Worst Journey is a minor masterpiece, one of the classics of travel and exploration. And yes, it's a tragedy — but it's also a book about heroism. If courage is grace under pressure, Scott's company embodied it: from the unquenchable optimism of Birdie Bowers, to the calm competence of Crean and Lashly, to the laconic dignity of Titus Oates. In the face of impossible conditions, they kept on. It's one of my favourite books.

A Pleasant Jaunt

But Scott lost. Scott lost.

He lost not only his race to be first at the pole, but also his life, and the lives of the four men he reached the pole with. Glorious failure makes for inspiring memoirs, but it is failure nonetheless.

Scott's great rival, the Norwegian explorer Roald Amundsen, succeeded. And he did so with comparative ease. While Scott's 12-man summit party were laboriously dragging 1000 kg of food and supplies up a 10,000 ft glacier on sticky wooden sledges, often relaying so that they had to cover each piece of ground three times, Amundsen's party used skis and dogsleds and didn't have to man-haul even once.

Scott's party suffered terribly from deficiency disease, injuries (Wilson's leg and Edgar Evans' hand, rendering them unable to work or even walk sometimes), snow blindness, frostbite, gangrene and more. Lashly's diary, detailing his return journey, is graphic about the progressive effects of scurvy on his companion Teddy Evans:

Meanwhile, the worst affliction to befall Amundsen's party was a toothache.

Scott's team had to contend with constant gnawing hunger, cold and thirst, going on short rations and running out of fuel even while performing prodigious amounts of work. Meanwhile, Amundsen and his companions actually gained weight on their journey. In his journal, Amundsen describes feasting on seal meat and biscuits: “we are living in the fleshpots”. And while Scott cut his margins very fine — too fine, as the final tragedy revealed — Amundsen did everything with time and to spare.

Amundsen had a third of the funds that Scott had, and a fifth of the men. Facing the same conditions, with far fewer resources, he outdid Scott on every measure. Amundsen's polar party covered 3440 km in 99 days — with effort, yes, but no real risk. Scott's party covered 3200 km in 148 days with injury, travail and death.

Scott writes of the Pole “Good God! this is an awful place”. Amundsen's companions write, variously: “the dogs are enjoying life” ... “it'll be fun, the finish to this race” ... “the most wonderful conditions” .... “splendid”.

What happened?

Knowing One Big Secret

Why did Amundsen succeed where Scott failed? Ultimately, Amundsen knew one big secret that Scott didn't.

In polar exploration, moving fast reduces your risk.

In most fields of human experience, this is emphatically not true. Driving fast is riskier than driving slowly. Sprinting up (or down) a slope is riskier than walking steadily. Cycling downhill without brakes is riskier than with.

This generalizes. Structures built quickly tend to be less stable. Work performed rapidly tends to be sloppy. Crash diets aren't as effective as lifestyle changes. Slow and steady wins the race; short cuts lead to long delays; haste makes waste; and so on. This is the calculus that most of us have internalized, and it usually works.

But not in Antarctica. Why not?

Humans cannot survive unassisted on the Polar Plateau. It’s too cold; the air is too thin; there's no source of food or fuel or drinkable water. When your supplies run out, you die.

That's bad enough if you're staying at a base like Scott’s Cape Evans or Amundsen’s Framheim, but at least you can hunker down and wait for ships from New Zealand or Argentina or Australia to re-supply or rescue you. The situation is much worse if you plan to leave your base and journey into the unknown.

You have to carry all your resources with you, for as long as you’ll be away from base. You’re pioneering a route into the unknown, and you don't know what obstacles will arise in your path: unclimbable mountains, treacherous crevasses, impassable ‘pressure ice’. You don’t know what weather to expect: blizzards that last for days, or snow-glare that renders you blind. You don't know how long your journey will take, or how your team will deal with conditions nobody has ever encountered before.

And those are just the known unknowns! The longer you spend, the more vulnerable you become to unknown unknowns. Accidents become death sentences. But so do seemingly trivial incidents: a leaky oil stopper, for instance, or a misplaced glove. Or simple ignorance — most notably, of the role of fresh food in preventing scurvy1.

This is what Amundsen realized, and Scott did not. The risks of failure — from exhaustion, starvation, injury, illness, accident, mishap — increase super-linearly with time; so the correct, risk-averse thing to do is to spend as little time as possible on the journey.

Digression: Trickle Down Speedonomics

Every single thing about Amundsen's journey was optimized for speed: equipment, transport, personnel, and perhaps most importantly of all, strategy.

Equipment: over the winter before the polar dash, Amundsen's team laboriously reviewed every single piece of gear they would carry: shaving down the skis to save a few ounces of weight; unpacking their bulk rations and repacking it into daily allowances; sewing two tents together to halve their setup time; designing storage canisters that could be opened without unlashing them from the sledges - anything and everything to save time.

Transport: Amundsen famously used dogs while Scott used dogs, ponies, motorized sledges and old-fashioned human effort. All of these technologies had been used, with some success, on previous polar expeditions — but dogs were the fastest by far. 

Personnel: Amundsen put together a small team of specialists. Small teams consume less, hence can travel lighter, hence move faster than large ones. And specialists work faster than generalists. In fact Amundsen did even better: he insisted that his team have multiple specialties, like Olav Bjaaland, a national ski champion who was *also* a highly skilled carpenter; Helmer Hanssen, a superb navigator and experienced dog handler; and Sverre Hassel, a qualified ship's mate, navigator, sail-maker and leather-worker. Having assembled this team, Amundsen gave them wide discretion to act independently: again, in the service of speed.

Strategy: Amundsen had a single strategic goal: to get to the Pole and back as quickly as possible. He was happy to take tactical risks if they served this overall strategic goal. For instance, both his base and his route were on uncharted (and hence dangerous) territory, chosen because they shortened his journey by 100s of miles; he correctly judged that this was the lesser risk. Similarly, Amundsen didn't mind skiing in fog or snow or even moderate blizzard; it was less risky than staying put. At the same time, he firmly resisted the temptation to ski for more than 6 hours on any given day; he didn't want to take the risk of tiredness slowing down his team. 

Above all, he was flexible: unafraid to change his plans, but fully committed to whatever plan was currently in place.


Slow And Steady is Starved And Scurvied

The contrast with Scott's last expedition is jarring.

Scott believed there was safety in numbers: that a large, slow-moving convoy faced fewer risks than a small and agile team. He believed that the success of such a convoy depended on perfect adherence to complex plans, often written down weeks or months in advance. He believed that strict Naval discipline and a rigid hierarchy were the best way to ensure there were no deviations from these plans. He believed that having his team tackle multiple goals mitigated failure in any one of them. He believed in redundancy over flexibility: that most problems could be solved by throwing resources at them. And he believed that bloody-minded persistence could make up for any amount of shoddy execution.

All in all, Scott believed too much. 

And when conditions on the ground belied his assumptions, he refused to change. This might have worked in the Royal Navy (a large but nonetheless mostly high-functioning organization where discipline and structure and detailed planning ruled the day), but in the Antarctic, it was a disaster. Scott's polar party paid the price. 

A Day Late, A Dollar Short, A Startup Gone

By now, gentle reader, you will have realized that this essay is not about polar exploration at all; it’s about technology startups.

Tech startups, like Scott and Amundsen, operate in conditions that are unknown, and unforgiving.

Unknown: Early stage founders have to discover almost everything about their business. What is the product? Who is the customer? How do we reach them? What will they pay? How do we hire, and scale, and compete, and disrupt, and defend? It's like putting together a jigsaw puzzle where you have to craft each piece from scratch and you don't know what picture you're assembling.

Unforgiving: Early stage founders need to hire, build, sell and grow, before they run out of cash. It's a race against the clock, and most startups are ‘default dead’ — their runway will end before they reach profitability. This is like venturing on the Polar Plateau: you have to reach safety or perish; time is not your friend. (The jigsaw puzzle will explode if you don't finish it in 6 months).

What's the best way to mitigate this deadly duo? Amundsen knew the answer. Speed.

Speed is the single best way to de-risk a startup. The way to survive the unforgiving marathon that is startup life is, paradoxically, to sprint as fast as you can. Speed is a startup's best friend.

They Often Call Me Speedo …

… ‘cos I don’t believe in wastin’ time.

Through what specific mechanisms does speed reduce startup risk?

Speed is a competitive advantage in itself. In fact, it’s the only sustainable advantage that startups have. Startups are famously under-resourced compared to big companies; they lack money, brand, people, scale, reach, robustness and pretty much everything else. The one thing that startups can do better than incumbents is move fast. In dynamic environments (shifting macro trends, rapidly evolving technology, demanding customers, ruthless competition), this alone might suffice to win the market!

Speed gives you more shots at goal. This is the big one. When you have to discover everything, it helps to have lots of chances at discovery. Each shot doesn’t have to be perfect; in fact it’s best if they’re not: perfect is the enemy of good enough. But you can learn a lot from ‘good enough’.

Speed hedges against burnout. Why do startups fail? Superficially, they fail when they run out of money. But fundamentally, startups fail when their founders lose faith. And the single most common reason for this loss of faith is burnout: pushing that boulder endlessly up the hill. Slow and steady magnifies this feeling; it’s hard to feel energized when every day is a slog.

This is deeply, deeply counterintuitive. It goes against the physical intuition we've all acquired from youthful scraped knees and twisted ankles. It goes against the conventional wisdom drummed into us at school. It goes against the operating principles beloved of large organizations — plan for every eventuality, diversify your efforts, hedge all risks, operate redundantly, work to a cadence.

But this intuition and this wisdom and these principles are all anti-patterns for early-stage startups. Startup best practices are diametrically opposite, and they're all downstream of a single goal: speed.

Let's go through a list of startup cliches, shall we?

Move fast and break things: because slowing down is riskier than making mistakes.

Do things that don't scale: because building for scale wastes time that you don’t have.

Strong opinions, weakly held: because hedging your bets slows you down2.

Two-pizza teams: because beyond a certain minimal size, additional hires also slow you down.

Hire for slope not intercept: because the best predictor of future velocity is past velocity.

Make the main thing the main thing: because focus creates alignment and alignment fosters speed3.

Find your North Star: because measuring what matters is the best way to not get hung up on irrelevancies4.

Digression: The Airspeed Velocity Of An Unladen ... Godwit?

One of the most interesting scientific papers I've read in recent years is about the bar-tailed godwit. 

Researchers discovered that this seemingly unremarkable bird shatters previous known records for migration. Every September, godwits fly non-stop directly across the vast central Pacific Ocean: a journey of between 8000 and 12000km. They don't stop to rest or feed or sleep; they just fly. They maintain an energy output that is 8 to 10 times their basal metabolic rate, for 8 to 10 days, contending with extremes of dehydration and sleep deprivation along the way. Long migrations are not unknown in the animal kingdom, but the godwit is something else.

Why? The paper lays out some fascinating (and convincing) hypotheses. And they all come down to one thing: speed as a risk mitigation strategy.  

Speed minimizes total energy cost:

> Mortality during migration may be much higher than during other parts of the annual cycle, so selection should favour behaviours that minimize risks of mortality during the migration period, including time spent accumulating the fuel resources needed for migration. A single transoceanic flight probably minimizes overall time and total energy cost of migration ...

Speed minimizes risk of predation:

> The flight corridor across the Pacific is essentially devoid of avian predators capable of taking godwits. The peregrine falcon, though recorded from most archipelagos in Oceania and a known predator of godwits elsewhere, would be of little concern to godwits during their non-stop, largely open ocean flight since peregrines could neither consume a godwit while in flight nor land on the ocean to consume its prey.

Speed minimizes risk of disease:

> The trans-Pacific route also provides a corridor probably free of pathogens and parasites. Long-distance migratory birds that use numerous stopovers are exposed to a diverse pathogen fauna. Birds engaged in endurance flights presumably become energetically stressed; under such conditions, immune function could become suppressed. If godwits flew along a continental route, the greater distance would mandate at least one stopover to refuel, probably requiring several weeks' duration. Any immunosuppression associated with their long-distance flights could render them more susceptible to infection at stopover sites. By flying non-stop, godwits minimize their risks to novel pathogens and parasites, and the costs of activating and maintaining their immune system may be reallocated into flight costs.

Less total energy, fewer predators, less exposure and better resistance to pathogens and parasites: it appears that flying fast and direct -- across the entire Pacific Ocean! -- is less risky than going slow and steady. Sound familiar?

It's Not Just For The Birds

Once you recognize this pattern, you begin to see it everywhere.

The best alpinists are known for their preternaturally fast climbing. HFT, the most lucrative investment style of the last two decades, is based not on being ‘smarter’ but on being ‘faster’ than other investors. Military theory from Hannibal to Alexander to Genghis Khan to Guderian to Boyd emphasizes speed over almost every other attribute. Magic: The Gathering regularly bans certain cards from tournament play because they're too fast; they break the game via turn-1 kills.

Whenever there's a complex, dynamic, unpredictable and unknowable environment — high-altitude mountaineering, modern financial markets, the theatre of war, zero-sum strategy games — speed is the key to success5.

I Believe You, Truly I Do. (No I Don’t.)

Luke: I don’t believe it.     
Yoda: That is why you fail.

Almost everybody in the startup ecosystem agrees that speed is a good thing. Almost nobody acts like it.

We’ve been socialized to believe that going too fast is risky. We fear launching too soon; we abhor looking sloppy; we shy away from criticism; we look for side-quests; we think ‘one more feature’ will save us; we bike-shed. All these tendencies slow us down, but we think they reduce our chance of failure, and more importantly, the appearance of failure.

We forget that for startups, not going fast is riskier still.

This essay aims to change that. This essay was written to tell you, gentle founder, that it’s okay to go faster than comfort dictates; faster than feels prudent; faster than you think you can. In fact, it’s not just okay; it’s optimal. And now you know why.

Toronto, 21 Jan 2023.

End Notes

Yes, I am exquisitely aware of the irony inherent in taking twelve months to publish an essay about the virtues of speed.

Also, subscribe!

Markets rise, and markets fall. This much, at least, is well-known. But why do market cycles occur? What causes the pendulum to swing from euphoria to crisis and back?

Hyman Minsky was a 20th-century economist whose ‘financial instability hypothesis’ is probably the best-known explanation for the boom and bust cycles that characterize public financial markets. But there’s far less examination — in fact, there's almost none — of how Minsky dynamics apply to private markets.

We’re currently in the midst of an unprecedented boom in private market activity. Tech entrepreneurship, angel investing, and venture capital have never been so widespread. Can Minsky cycles happen in this realm as well?

Let’s find out.

The Inevitable Briefness of Alpha

When I started my career as a bond trader at a quant hedge fund, arbitrage opportunities were relatively plentiful. Not many investors had the knowledge or infrastructure to effectively exploit these opportunities, so the early pioneers in the market made good money.

Of course, it didn’t last. Knowledge spreads, technology diffuses, and arbitrages disappear. Markets asymptote towards ever-greater efficiency. This process is extremely well known in capital markets, and even has a name: ‘alpha decay’.

Spreads representing 10s or even 100s of basis points of opportunity, common when I started trading, dwindled to mere 1s of basis points less than a decade later.

… Wait, You Did What?

Now, most individuals, when faced with declining opportunities, will reduce their exposure to those opportunities. If a particular strategy made 10% last year but is only expected to make 1% this year, common sense suggests allocating less capital to it.

But institutions … don’t work like that. Trading desks have quarterly P&L targets, and traders have annual bonuses they want to make. There's a widespread culture of “what have you done for me lately?” — you can't coast on past success. The implicit call option embedded in most traders’ compensation profiles exacerbates this.

As a result, faced with a 1/10 reduction in expected value for an opportunity, many institutional portfolio managers will actually 10x their exposure to the opportunity, so as to maintain their dollar P&L. I saw this multiple times in the early 2000s, and it's completely rational, given their incentives.

Mathematics Made Me Do It

But that's not the interesting part. The interesting part is that many risk and compliance models actively encourage investors to do this. Here's how it works.

Most risk models (whether implicit or explicit, quantitative or qualitative) measure the riskiness of a particular investment based on how it and similar investments have behaved in the past. Sounds reasonable, right?

Now, suppose that a particular class of investments, that used to be quite risky in the past, has grown less risky in recent years. A model trained on both historical and recent data would say it’s okay to put more capital to work against these investments than in the past; they’re just not that risky any more. Again, sounds eminently reasonable, right?

“The market is maturing”, is usually how people describe this. Or “The asset class has become more efficient”.

The Illusion Of Safety

Ah, but here's the catch. What if it's precisely the deployment of all this capital that causes the decrease in volatility and hence in perceived risk?

In my own little world of bond arbitrage, spreads were far less volatile in 2006 than in 1999 — because any tiny deviation from ‘fair value’ was quickly met by a flood of arbitrageur dollars pushing the other way. Those dollars simply didn't exist in 1999.

As a result, bond trading in 2006 ‘appeared’ a lot less risky than in 1999. Reward (expected value) declined but risk (realized volatility) declined even more; as a result, Sharpe ratios — roughly speaking, reward divided by risk: a widely used metric of investment performance — went through the roof.

This is the Minsky boom. Money entering a market boosts returns and reduces volatility, leading to very strong (realized) performance. This attracts more money, which improves performance even more. A positive feedback loop ensues.

And this is perfectly legit! Economies can and do reallocate resources all the time. This is how it works; this is how it’s expected to work.

What Goes Up …

The problem with feedback loops is that they tend to overshoot. Minsky booms in an asset class attract a constant influx of new money, but they also need that influx to continue marking up the price and marking down the risk of the asset class. And as the wise man said, “If something cannot go on forever, it won’t”.

Eventually — and fortunes have been made and lost, trying to predict just when that ‘eventually’ comes — something happens. It could be an exogenous shock like COVID, or a tightening Fed, or an election, or a war; it could be an industry-internal event like a particular firm blowing up or winding down. The new money stops, or maybe just slows down a touch, and prices soften. And that triggers all sorts of nasty consequences.

… Must Come Down

The first thing that happens is that risk-management dashboards start flashing red. Even a slight selloff causes returns to drop and risk to rise; portfolios that seemed well-balanced now appear just a little too aggressive.

Over-extended investors begin to trim their positions. Unfortunately, this causes more price declines, and more volatility, triggering another round of position-trimming. Even conservative investors suddenly realize that their portfolios are riskier than they thought. They sell as well. The ensuing vicious cycle is sometimes called a risk spiral.

Risk spirals are often accompanied by margin spirals. Banks and brokers require leveraged investors to post margins that are proportional to their portfolio risk. As their portfolios become riskier — and remember, the portfolios themselves are often unchanged; all that has changed is the level of volatility in the market — leveraged investors face margin calls. They have to sell assets to service these calls, creating further volatility and downward price pressure; the margin call becomes a self-fulfilling prophecy.

(That’s why it’s always best to be the first firm to unwind positions or call margin. Goldman — disclosure: Matt Levine used to work there — was very good at this, Lehman less so.)

The final domino is a redemption spiral. Seeing declining performance and increasing volatility, LPs in a fund request their money back. To service these redemption calls, the fund has to sell even more of its positions, triggering yet another feedback loop. Boom quickly turns to bust.

Hello Lehman My Old Friend

This is exactly what happened in credit markets in 2007-08. An influx of cash on the way up, accompanied by the sense that it was a can’t-lose trade; everyone was making money on housing and mortgages and credit. And then an equally unstoppable wave on the way down, as value-at-risk and margin calls and redemption spirals all worked to pull capital out of the credit market.

And Round And Round We Go

In good times, people are encouraged by past success to place larger and larger bets, thinking they're less risky than they actually are, when often it's the very existence of these large bets that drives present success and depresses perceived risk.

Investors believe the trend will continue indefinitely, and become complacent. They invest in lower quality instances of the asset, while increasing their leverage.

And then the music stops. Markdowns lead to deleveraging which lead to more markdowns; the positive feedback loop now operates in the other direction. Eventually, the asset class overshoots as investors become overly risk averse, setting the stage for the next bull market. Stability breeds instability, and vice versa.

Now, all of this is well known. Hyman Minsky fell out of fashion in the 80s and 90s, but his work was rediscovered and widely shared just in time for the GFC. Today it's part of the toolkit for most macro (and many micro) investors; you can also see it in regulatory ideas like market circuit-breakers and systemic backstops.

Is Venture Immune?

But credit is a foreign country; they do things differently there. Let’s talk about early stage tech and venture investing.

At first glance, venture capital seems an unlikely candidate for Minsky dynamics to take hold. Consider:

• VCs funds don’t use leverage

• They don’t offer redemptions or early liquidity to investors

• There are no counter-parties and no margin calls

• Volatility is actually good for most VC portfolios (long basket of options)

Without mark-to-market, there’s no chance of a risk-reduction spiral. Without leverage and counter-parties, there’s no chance of a margin spiral. And without investor liquidity, there’s no chance of a redemption spiral. What mechanism could force the liquidation of a venture portfolio, or incept a Minsky bust? For that matter, what’s the mechanism for a Minsky boom in venture?

I Have Confidence … In Confidence Alone

To answer that question, ask this one: where does confidence come from?

The key idea of Minsky cycles isn't that rising prices attract capital; that's just standard trend dynamics. The key Minsky idea is that increasing capital inflows reduce perceived risk.

In the run-up to the GFC, home prices (and much else) went up, but the underlying confidence of the market was rooted in a belief that advances in securitization (everything from Gaussian copulas to CDO-squareds) had delivered genuine structural innovation to the mortgage market, unlocking a swathe of value. And there was prima facie evidence for this belief in the fact that credit spreads were tighter than ever.

Only afterwards did it become clear that those tighter spreads were driven by capital flows into subprime securities, not by an actual reduction in economic risk1.

What's the analogy for venture capital? I suspect that the variable of interest is time.

Fast Is In Fashion

If you talk to almost anyone in venture today, you’ll hear a few themes again and again.

Startups are marked up faster than ever:

Rounds are closed faster than ever:

Funds are deployed faster than ever:

Across every aspect of venture, timelines keep compressing.

In Search of Shortened Time

Timelines in venture have compressed dramatically across the board. This is good news for founders. It’s even better news for investors.

Why so? The superficial answer is IRR. It typically takes 5-10 years for venture investments to generate cash returns. It’s impractical to wait that long, so LPs judge venture firms on their interim IRR. And fast markups dramatically boost IRR.

Firms use these boosted IRRs to aggressively raise new funds, and so they should; if you make 2 and 20 on every dollar you deploy, why deploy slowly? Maximize your lifetime-dollars-deployed.

But there’s a deeper answer. Remember the key Minsky idea: it’s not about returns, it’s about risk.

The ‘classic’ model of venture assumes that startup outcomes follow a power-law distribution: most startups fail, while a small number of outlier successes generate all the upside. Furthermore, “lemons ripen early”: failed startups fail fast — they don’t show the progress required to raise follow-on financing, and quickly go to zero. Meanwhile the outlier successes take time to grow into their full potential. Venture portfolios therefore exhibit a J-curve.

But this is no longer true.

Accelerated markups mean the venture J-curve no longer exists!

Let’s do the math. In the bad old days, if you invested in 10 startups, then 18 months later maybe 3 or 4 would have raised 1 round each of further financing at say a 2x markup, while the remainder would be dead or doomed. Your portfolio as a whole would be worth 0.6-0.8x what you invested: the negative stage of the J-curve. (Note that this is a portfolio that’s doing well!)

Today, if you invest in 10 startups, then 18 months later your 3-4 surviving firms might easily have raised 2-3 more rounds of financing at a 2x markup each time. Thanks to the velocity of financing, your portfolio as a whole could be worth 1.2-2.4x what you invested. Yes, the upper bound is higher, but crucially, so is the lower bound. The fast markups have completely compensated for your lemons! And as a result your risk appears minimal. This is terrific if you’re an investor, and funds know it:

Higher returns and lower risk means new money floods into the sector, accelerating the feedback loop. This is the classic template for a Minsky boom, and it’s all driven by compressed time2.

Many Things Can Be True At The Same Time

Like all Minsky booms, there are some genuine truths underlying the dynamics of the venture market today.

Startups are marked up faster than ever — but startups are also growing faster than ever. 3x year-over-year used to be considered strong; today it’s a bare minimum. The best companies grow at 5x, 10x or even more.

Rounds are closed faster than ever — but it’s easier than ever to evaluate the economics of software companies. SaaS diligence is a solved problem.

Funds are deployed faster than ever — but that’s what maximizes dollars returned, not some arbitrary investment schedule.

Or, in handy graphical form:

These arguments are obviously true. But it was also obviously true (and I’m not being sarcastic here) that there were some genuine structural advances in credit markets in the 2000s, broadening access to loans for borrowers while reducing risk for lenders. This did not stop the credit markets from imploding in 2008.

So are these arguments strong enough for venture to be immune to Minsky dynamics? Is it different this time?

Detour: True Risk and Measured Risk

One way to understand Minsky cycles is that they’re driven by the gap between ‘measured risk’ and ‘true risk’.

When you lend money, the ‘true risk’ you take is that the borrower defaults3. But you can’t know this directly; instead you measure it by proxy, using credit spreads. Credit spreads reflect default probabilities, but they also reflect investor demand for credit products. A subprime credit trading at a tight spread doesn’t necessarily imply that subprime loans have become less risky (though that could be true); the tight spread may also be driven by demand for subprime loans. Measured risk has deviated from true risk.

Similarly, when you invest in a startup, the ‘true risk’ that you take is that the startup fails. But you can’t know this directly; instead you measure it by proxy, using markups. Markups reflect inverse failure probabilities (the higher and faster the markup, the more successful the company, and hence the less likely it is to fail — at least, so one hopes). But markups also reflect investor demand for startup equity. Once again, measured risk has deviated from true risk.

During Minsky booms, measured risks decline. During Minsky busts, measured risks increase. The flip from boom to bust occurs when the market realizes that true risks haven’t gone away.

The Destination, Not The Journey

So now let’s rephrase the question. Has the true risk of venture investments changed? More rigorously:

Does the compression of timelines in venture change the distribution of terminal outcomes for venture-backed companies?

On that question, the jury is still out. It’s not obvious to me that accelerated markups change the power-law dynamics of venture portfolios. Markups change the journey of a business, but do they change the destination?

If the answer is yes, then there’s no Minsky dynamic at play; what we’re seeing is a rational evolution of the venture industry. Maybe startups are truly less risky now; maybe the market truly has matured. More capital, lower returns, safer investments4.

If the answer is no, then venture is very possibly in a Minsky boom, and we’re just waiting for the moment when it turns into a Minsky bust.

What could trigger such a moment?

Reasons For Momentary Lapses

This section is necessarily speculative, but I’ll begin with an observation. There is in fact one well-known death spiral in startup land, and it’s the dreaded down round.

In a down round, a startup running out of cash is forced to raise capital at a lower valuation than its previous financing. This is bad news. Anti-dilution provisions mean that early investors and common shareholders are wiped out. Recent hires whose options are now underwater begin to leave. The startup is perceived as damaged goods, and has to pay above market comp to replace them, attracting mercenaries instead of missionaries. Customers, not knowing if the startup will survive, churn. Finances worsen, predatory investors circle, and further down rounds loom.

A valuation spiral is bad enough. It’s usually accompanied by a talent spiral, which is worse. In a tight labour market, good operators have their choice of where to work. The best startups are able to attract the best talent5. Meanwhile, flailing startups tend to fill up with mediocre employees — the ones who can’t find work elsewhere. This makes it even harder to recruit excellent people 6. The spiral continues.

Down rounds are widely considered the harbinger of doom for venture-backed startups. Understandably, founders and investors go to great lengths to avoid them7.

How do they do this? The most common approach is to wait it out. Cut costs, squeeze out short-term revenue, raise bridge loans from less-known investors at the best terms you can, and hope to eventually ‘grow into your valuation’. Sometimes it even works.

This is — not coincidentally — a perfect mirror image of the logic used by many investors today. “I don’t mind paying up; on the current trajectory, even a doubling in price is easily recouped via just a few months of growth.”

But if my hypothesis about time is true, this could be dangerous. If compressed timelines are the driver of Minsky inflows into venture, then anything that delays funding cycles could precipitate a painful reversal. First some startups delay fund-raising because they need to grow into their valuations; then the VCs who invested in those startups have to delay their own fund-raising with LPs because they don’t have the requisite markups; then the LPs reconsider their (hitherto ever-increasing) allocations to venture because the latest returns are uninspiring; and before you know it, there’s an exodus from the asset class. Minsky giveth, and Minsky taketh away.

Toronto, 12 Feb 2022

Further reading

Everett Randle describes how Tiger built “the first structural, non-brand driven competitive advantage and flywheel at scale in venture” — and it’s all based on the effects of accelerated deployment on venture timelines and risk. Possibly my favourite essay of 2021.

I wrote that tech financing doesn’t use debt or leverage, but that’s changing fast. Startups like Pipe have already had great success securitizing software cashflows. Alex Danco predicted this in one of my favourite essays from 2020: Debt is coming.

Byrne Hobart applies the Minsky framework to supply chains, another topic of some current interest. His focus is more on liquidity and leverage than on risk, but the core ideas remain the same.

Housekeeping

I’d like to thank Ruben Schreurs and Venkatesh Rao for their suggestions on style and structure, which helped improve this post substantially.

Thank you for reading! If you liked this post, please do 3 things straight away:

• Email it to a friend

• Follow me on Twitter: @athomasq

• Subscribe!

A handful of business models dominate tech today: SaaS, marketplaces, e-commerce, on-demand, social networks and so on. Most of these business models have been studied widely, both their execution and their underlying dynamics.

But there’s one notable exception: data businesses. Despite the fact that many of the largest and most dominant tech firms in the world are data businesses, there are not many resources on the what, how and why of this business model.

This essay is an attempt to change that. Read on!

Definitional aside: What is a data business?

Every company uses data, but not every company is a data business. A company is a data business if, and only if, data is its core product. Data is central to the activity of the company; without the data, there is no company 1.

Google, Bloomberg, Yelp, and ZoomInfo are all data businesses. They acquire their data in different ways, and they generate revenue from that data in different ways. But for all these companies, data is the fundamental unit of value creation.

ONE: It's all about the data

THE FIRST fundamental truth of data business models is this: it’s all about the data.

Successful data businesses are all built around a unique or proprietary data asset. There are a few ways to build such an asset:

• Brute force: You throw resources at the task of primary data collection. Examples: Google crawling every website in the world; Planet launching 100s of micro-satellites; ZoomInfo cold-calling company switchboards to verify contact info. This is the most common and, despite the expense, often the best way to create a new data asset.

• Aggregate and harmonize: You take data that others have collected and published (often for free), and you aggregate, link, and harmonize the data. Example: Reuters standardizing printed financial statements in the 1970s.

• License and transform: You license commoditized data and transform it into a value-added form. Example: Scale.AI taking raw images and labelling them at, well, scale.

• Affiliate collection: You farm out data acquisition to partners with the right incentives. Example: Advertisers install the Facebook pixel to collect data on customers, which they send to Facebook to optimize their Facebook ads.

• Core business output: You create the data as part of your core business process. Example: Every transaction on the New York Stock Exchange generates data (price, volume, orders), which NYSE monetizes.

• Payment in kind: You offer a free service or tool, in exchange for data or data tracking. Example: Foursquare’s free SDKs for mobile app developers, which enable Foursquare to track mobile user location.

• Inbound network effects: You create a compounding advantage in getting data sources to come to you. Example: Google search is made continually better by site owners submitting data to Google (via SEO and other channels), leading to more Google searches and even stronger incentives for site owners.

• Give to get: Partners send you individual pieces of data in order to access the corpus as a whole. Example: Businesses send their counter-party data to Dun & Bradstreet, in order to access D&B’s B2B credit database — which is based on aggregating all these counter-party reports.

• (Data consortia are related to give-to-get, but with a peer-to-peer topology instead of hub-and-spoke.)

• Data exhaust: You collect or generate data as a by-product of your core business. Example: comparison shopping apps, email managers and personal finance tools all have visibility into consumer transactions; some of them use this to build data products.

• Data creation: You generate synthetic data, for applications where ‘real’ data is unnecessary, undesirable, or unachievable. Example: Tonic creates fake data that companies can test their systems on, before deploying to production.

Note that these methods are not mutually exclusive. If anything, they tend to be mutually reinforcing.

TWO: Control unique data to capture unique value

THE SECOND fundamental truth of data business models is this: whoever controls the data, captures the value. Intermediaries get squeezed.

A common failure mode is to build a business on top of somebody else’s data. If you depend on a single upstream source for your data inputs, they can simply raise prices until they capture all of the economics of your product. That’s a losing proposition.

So you should try to build your own primary data asset, or work with multiple upstream providers such that you’re not at the mercy of any single one.

You should also try to add proprietary value of your own, lest either your suppliers or your customers encroach and disintermediate you. A sufficiently large transformation of your source data is tantamount to creating a new data product of your own.

These tactics interact. Sometimes the very act of merging multiple datasets adds substantial value 2. Joining data correctly is hard! Other non-glamorous ways to add value include quality control, labelling and mapping, deduping, provenancing, and imposing data hygiene 3 4.

Some companies, discovering that they can neither control their data assets nor add intermediary value, pivot to picks-and-shovels instead. Tools to support data businesses — everything from monitoring to pipelines to governance — can be lucrative in their own right.

The gold-rush metaphor may be over-used, but it’s still valid. Prospecting is a lottery; picks-and-shovels has the best risk-reward; jewellers make a decent living; and a handful of gold-mine owners become fabulously rich.

THREE: Data business have slow beginnings

THE THIRD fundamental truth of data businesses is this: they start slow.

You’ll that none of the above data acquisition methods are ‘easy’. They need upfront investment or a certain amount of scale to work. Absent either of those, building a data asset is a process of slow bootstrapping.

Adding to the problem is the fact that almost all data products have a ‘minimum viable corpus’ — a size below which the data simply isn't useful. This parallels the concept of a minimum viable product in software, but an MVC is usually much harder to build than an MVP.

The analogy with software doesn’t end there. Almost every apart of the software business stack has an equivalent in the data business stack. Where software firms invest in devops, QA and product, data firms have to invest in data ops, data QA, and data product. These tend to be just as complex, with the extra hurdle that third-party providers are rare, hence you often have to build this infra in-house. All of this is expensive.

As a result, delivering one's first data product requires significant time and resources.

And this is a good thing! Remember the classic wisdom: my capex is your barrier to entry. The effort required to go from zero-to-one in data businesses is one reason they are so formidably defensible. It's also why ‘brute force’ remains one of the most popular strategies used by players in this game. A data product that can be built easily is a data product that can be replicated easily.

But even after you build your data asset, you’re not home free. For reasons we’ll get into later, most data products require category creation. You have to educate your ecosystem, evangelize your product, nurture your customers over time. Early sales cycles are long, and win rates are low. But it gets better — a lot better.

Aside: Nothing ventured, nothing gained

One would think that a business model that requires substantial upfront investment but pays off in buckets later on would be a perfect fit for venture financing. That may have been true in earlier eras, but not today. Tech investing in recent years has indexed heavily on growth rates; as a result, data businesses — with their slow early growth — often find it difficult to raise venture capital.

(It’s also the case that compelling opportunities in data have historically been rarer than opportunities in software, even if they're more lucrative. VCs are familiar with outlier math, but their lack of reps evaluating data businesses tells against them.)

FOUR: Growth accelerates over time

THE FOURTH fundamental truth of data businesses is this: they accelerate.

Everything starts slower on the data side. Building a valuable data asset takes time. Building the supporting infrastructure to actually deliver that data takes time. Sales cycles take time.

The classic mistake people make is to see this and jump to the conclusion that early-stage data businesses don't work and will never work.

But that's a category error, and it’s due to a fundamental difference in dynamics. Software business economics tend to degrade; data business economics tend to improve.

Why so? Here’s how it works:

• The marginal cost of acquiring data begins to decline. You begin to see economies of scale on the infrastructure side.

• Data sales get easier as your corpus is no longer minimal. Sales cycles shorten, sometimes dramatically.

• An expanding corpus also expands your audience: for example, there are many more buyers for data covering 50 US states or 10,000 public stocks than for data covering 10 states or 200 stocks.

• You can slice and dice your data for more effective targeting and price discrimination — shortening your sales cycle even more.

• As your data becomes widely used, it goes from optional to essential. Customers use it because other customers are using it. The dream of every data asset owner is to become an industry standard. (This doesn’t happen with most other business models).

• You can charge more for data. This is partly a corpus-size effect, and partly a table-stakes/must-have effect. Data maturity opens up new axes for pricing — per record, per API call, per data update — in addition to the usual SaaS axes of per use case and per seat.

• With more customers, you can amortize your fixed costs of data acquisition and delivery across a wider base — and they’re almost all fixed costs 5.

• You can charge recurring revenue; after all, nobody wants to work with obsolete data. (This is harder in the early days, not because of lack of buyer appetite, but because your update cadence probably isn’t good enough.)

• The combination of recurring revenue, avenues for upsell, and must-have status means your NRR and LTV are terrific.

• You can unlock new channels of data acquisition, most notably customer contribution loops. Models like give-to-get, payment in kind, and affiliate partnerships are now accessible to you. (They weren’t previously, because you were too small to be sufficiently attractive.)

• You can also create data quality loops: get customers to not just contribute, but also verify their own as well as third-party data for you, either explicitly or through various behavioural and software hooks.

• You can use your data to power your customer acquisition, most notably with a data content loop. Data content loops can be simultaneously cheaper, more scalable and more defensible than most other go-to-market channels, as shown by Expedia, GlassDoor and Zillow.

• You can build services on top of your data, creating a data learning loop. As your data improves, these services improve in parallel, growing and contributing even more data to your platform.

Now, many of these effects taper off eventually. Marginal data costs go back up once you start hitting the long tail; price flattens out once the marginal data point no longer adds insight; most real-world entities have finite (even if large) cardinality. These curves are sigmoid, not unbounded.

But you can get a very long way before that happens. At-scale data businesses are huge, and many of them are still growing fast.

The holy grail is when all these scale and network effects combine such that you can be both the lowest-cost acquirer and the highest-paying buyer of your data inputs —while still offering your data outputs to customers at the lowest price in the market. If you get this far, you’re unstoppable.

Case Study: ZoomInfo — from linear to loops

ZoomInfo (NASDAQ:ZI) is a data business whose core asset is a directory — names, titles, contact information — of corporate employees. If you're a salesperson and you want to sell product X to company Y, ZoomInfo will help you identify and contact the exact person Z who you should be talking to.

Here are two fascinating interviews with Henry Schuck, the founder and CEO, conducted a decade apart.

The first one, from 2012, is all about the brute force early stage:

Sramana: Would you talk about the product development process? What were your data sources? How did you put everything together?

Henry Schuck: We were gathering data directly from the companies we were profiling. That is what made our company different back then as well as today. We called into those companies to gather the data and collect the phone numbers, and we updated the information on those people. That is what made all the difference. We did not source data through a crawler.

Sramana: Would you just call the company switchboard and ask for names and numbers?

Henry Schuck: That is basically it. We would start with some online research and identify some top-level people. We would start there and work on building the organization out.

Sramana: It sounds like it was very labor intensive.

Henry Schuck: It was very labor intensive. At first it was just Kirk and me. We were spending 75% of our time on the phone and 25% of our time selling the product. The data was always of paramount importance for what we were doing. As we hired new people, we would split their time to 80% research and 20% sales and marketing.

The second one, from 2021, is all about scale effects:

Jesse: How are you actually getting the data?

Henry: We buy data. We gather data through public sources. We also have two contributory data models. One is we have a freemium model where people can get limited, free access to ZoomInfo in exchange for their email contacts. And so if you ever Google somebody and you see ZoomInfo come up as one of the top results, if you want free access to ZoomInfo, you could [share] your email contacts for that free access. And then we have a customer contributory network. And so a portion of our customers share data with us that we cleanse, validate and send back to them. And we kind of take the exhaust data off of that to help cleanse and manage the datasets. So for example, if you use a marketing automation system, you're going to share bounced data with us and confirmation email data with us. And we're going to use that to cleanse the hundred million records in our system.

If they bounced three times for our marketing automation system, we're going to take those people out and then cleanse the database. And then we have literally a million other unique sources that come in to ZoomInfo. And the thing that sits in the middle of all of that is this evidence-based machine learning algorithm that makes sense of all of the information. You see one person in seven people’s CRMs with different information. You see another person come through eight people's email contacts with different information. And so that machine learning algorithm is consolidating them, connecting them to an individual person and then publishing the most accurate version of that person into the platform.

Every additional customer who contributes, every additional freemium member who contributes, the data just gets better and better and better.

I want to break down all the different loops and motions in play here:

• Give-to-get / payment-in-kind: Users get free access to ZoomInfo's ‘Community Edition’ by giving ZI access to their own email contacts.

• Affiliate contribution network: Customers send raw data to ZoomInfo; ZI returns cleansed, validated data, and also keeps some of it for themselves.

• Data quality loop: ZoomInfo collects duplicate profiles from dozens of sources, and runs ML to infer the ‘most accurate’ version of each profile. The more sources, the more accuracy; and accuracy is a major selling point for ZI.

• Data exhaust loop: ZoomInfo uses customer exhaust data — for instance, email confirmations and bounces from marketing campaigns — to update and improve their own database.

• Data content loop: A big source of ZI's leads is SEO: when you search for a person, their ZoomInfo profile is often on page 1 of the results. The more profiles, the more leads; the more leads, the more profiles (thanks to the acquisition loops above).

And all of this pays off:

• Accelerating go-to-market: As the data gets bigger and better, it gets ever easier to sell. From the transcript:

  Henry: It’s just hard not to see how much value you get out of that the first day you turn it on.

  Jesse: Right. I’m sure their head explodes.

  Henry: Yeah. Their head explodes.

  Jesse: They become very skeptical. Then you go, “Go drive to the address and you can verify my data.”

  Henry: You can validate the data. Tell us the company you’ve sold to over the last couple of months, we’ll just show you them inside of ZoomInfo.

  ...

  Henry: Our average sales cycles are sub-30 days ... There are dozens of deals every month that we sell same-day.

• Multi-axis pricing: ZI can offer per-seat, per-record, and per-use-case (sales, marketing, recruiting) pricing, depending on the client.

• Brute force at scale: ZI has a data verification team of 100s of people, and crawls ~40M websites daily for profile updates.

• Exceptional economics: ZI’s margins grew from 50% in 2014 to 90% at the time of this interview. NRR is above 100% and accelerating. The go-to-market motion has a 6-8 month payback period and 15:1 LTV/CAC. That's not a typo.

Note that ZoomInfo did not raise venture financing; they were bootstrapped until a private equity round in 2014.

FIVE: Data businesses are super sticky

THE FIFTH fundamental truth of data businesses is this: they are virtually impossible to displace.

I’ve talked about this before 6:

The conventional wisdom in tech is that customers will switch to a new product if it’s 10x better than what they're currently using.

But what does ‘better’ mean for data?

If your data product is substantially similar to existing products, then one way to be ‘better’ is to offer a dramatically lower price. But this doesn’t work for data! The concept of minimum viable corpus means you need to be above a certain size and quality, else your data is almost worthless. And the very particular economics of data acquisition (brute force, economies of scale, data collection loops) mean that it's hard to build an MVC that’s dramatically cheaper than the incumbent offering. So ‘disruption from below’ rarely works.

Another option for being ‘better’ is to go the other way — offer dramatically higher quality. Typically, you need to be an order of magnitude faster, or more accurate, or more comprehensive — sometimes all three! — before you start poaching customers from the incumbents.

And even that might not suffice; the value of data lies in what can be done with it, and often, increasing speed or accuracy or comprehensiveness doesn’t really increase utility by enough. If you're competing with a table-stakes data product, heck, even increasing the utility by a huge amount won't work.

One way to think about this is in a ‘jobs-to-be-done’ framework. If a particular piece of software does a particular job, it's relatively easy to imagine a next-gen product doing that same job 10x faster or better or cheaper. But it's actually quite hard to envisage what an improved version of an existing data product can do that is 10x better than the existing data product.

In fact, typically, a successful new data product is not a variation on the existing data; it’s a brand new data asset from a completely different source, exploiting a completely different set of loops.

Meanwhile the original data asset continues to sell. Churn rates for mature data products are minuscule. So just as a matter of usage, it’s quite difficult to disrupt established data businesses.

Then there are all the ‘classic’ sources of defensibility mentioned previously: high barriers to entry; economies of scale in data acquisition and infrastructure; positive feedback loops / network effects in data and customer acquisition.

And finally, incumbents can use certain tactics to bolster their structural advantages:

‘Commoditize the complement’ is the best known — either with software, or with third-party data 7.

‘Creating data standards’ is another common one — every large data business has done this, from PCI to CUSIP.

‘If you can’t beat them, buy them’ — mature data businesses are extremely acquisitive. If they can buy data that is either additive or adjacent to their core product, they almost always will.

Put it all together, and mature data business have a set of flywheels and compounding structural advantages that are almost impossible to displace:

But data businesses aren’t quite winner-takes-all. Instead, the biggest ones tend to form duopolies — think Visa and Mastercard, Nasdaq and NYSE, Google and Facebook, Moody’s and S&P, Experian and Equifax, Bloomberg and Refinitiv. I suspect this has more to do with de facto government policy than with actual business dynamics.

Aside: 180 years, 4 presidents

Abraham Lincoln, Ulysses S. Grant, Grover Cleveland and William McKinley all worked for the same data company: Dun & Bradstreet.

Dun & Bradstreet (NYSE:DNB) is 180 years old. No business survives that long (two world wars and a civil war, recessions and depressions, market booms and busts, multiple technological revolutions, good and bad management, you name it) without an amazing structural moat. That’s the power of data businesses.

And DNB is far from the only centenarian data business in the world! Equifax was founded in 1899. Reuters was founded in 1851. Various Lloyd’s entities (the List, the Register, and the insurance mutual) were founded between 1686 and 1760. Standard & Poor’s dates back to 1860. All data businesses, all chugging along today, and none of them particularly close to being disrupted.

In SaaS a particular generation of software can become obsolete in a few years; products built in the early 2010s look decidedly old in the tooth today 8. But data businesses can last for decades, and as the above examples show, even centuries. Invest in building a data asset once, keep customers forever: the ratio of LTV to product creation costs is off the charts.

SIX: Successful data businesses are rare

THE SIXTH fundamental truth of data businesses is this: they are rare.

You might think that starting, buying or investing in data businesses is a slam dunk. Unfortunately, it’s not — and that’s because successful data businesses are rare.

This shouldn’t come as a surprise, because valuable data is also rare. There's a lot of data in the world, but most of it is junk. (Sturgeon’s law: 99% of everything is junk). The list of criteria that a data asset must meet in order to be the core of a successful data business is long and extremely restrictive.

And of course, many ‘obviously valuable’ data assets (information about companies, say, or people) have already been captured. That leaves non-obvious data assets9 — which in turn implies category creation. Category creation is hard!

But (speculation alert!) I wonder if we might see a ‘deployment era’ for data just like we’re currently seeing in software. Instead of massive horizontal platforms covering entire information categories like people, companies, or events, I wonder if more targeted, bespoke, vertical- or application-focused data businesses will emerge — data for very specific niches. We shall see!

Toronto, 29 Jan 2022.

Further reading

Kevin Kwok has a great essay on data content loops. This was a strategy we used to great effect at Quandl, the data business I co-founded. Kevin describes how Rich Barton used it thrice, at Expedia, Glassdoor and Zillow.

Auren Hoffman has a terrific and comprehensive Data-as-a-service Bible. I also recommend his World of DaaS podcast, especially his interviews with other data business founders.

I wish there were an essay with a truly in-depth treatment of Google as a data company (which it so clearly is!), not a search or advertising or content or AI company. If you come across such an essay, I'd love to read it.

Housekeeping

This is my very first Substack post; thank you sincerely for reading this far! If you have any comments or suggestions, please send them in.

If you liked this post, please do 3 things straight away:

• Email it to a friend.

• Follow me on Twitter: @athomasq

• Subscribe!

Subscribe now