Here is what puzzled me. We have never had more individual data and naive intuition says all that granular personal data should fragment us, splinter the world into a billion niches. And yet the opposite is happening. The more the machines learn about us as individuals, the more alike our worlds look. That paradox, more personalisation producing more sameness, is what this piece is about, and by the end I want to convince you this is a political economy story that needs and has an easy intervention.

The theoretical mechanism

Imagine forty options, cuisines, genres, shop types, and a standard algorithm: predict what people want, show them more of it, watch what they pick, update. Then let the loop run.

The mode gets shown more, so it gets picked more, so the model grows more confident the mode is what people want, and the tails starve. Within a few dozen rounds, a catalogue that began almost perfectly even collapses onto one dominant option. But why does it collapse toward the mode rather than fan out across all that individual data?

Almost every recommender is trained to minimise a prediction error, get the rating wrong by as little as possible, or maximise the chance you click. But the problem is that under squared-error loss, the prediction that minimises your expected error is the conditional mean. So an algorithm that is uncertain about you, and it is always at least a little uncertain, hedges toward the average. The more uncertain it is, the harder it pulls you toward the crowd. This is why more data doesn’t save us. Personalisation under a standard loss function is regression to the collective mean with extra steps. Variance, the technical word for the stuff that makes you you, is expensive to predict. Basically, diversity is variance, and optimisers are built to minimise variance.

What makes this political economy (and hence why I decided to write about it) is that the system doesn’t have to be right, it only has to be listened to. A recommender that’s a mediocre predictor of what you’d love in a rich, diverse world becomes an excellent predictor of what you’ll click in the impoverished one it creates.

This is also where I should say plainly that I didn’t invent any of this. Economists would call it performativity, the way a model, once acted on, bends the world into agreement (MacKenzie’s markets that reshaped themselves to fit their own equations). Machine-learning researchers have started calling a version of it model collapse: train a system on its own outputs for long enough and the variance drains away until everything converges on the mode. Urbanists have their own word, placelessness, for what’s left when everywhere optimises toward the same template. What I’m going to show here though is that these are one mechanism seen at four different scales. Think of it as temperature zero for culture: always take the most likely next token, and watch the surprise drain out of the sequence. Let’s make it concrete.

Test 1: physical environment

As a first test, I took the Greater London Authority's map of 640 high streets, mapped 18,000 London food places onto it, tagged each by cuisine, and flagged the chains, meaning any name recurring three or more times across the city (Subway, Pret, PizzaExpress, the M&S Food-to-Gos). Then I asked a question you can only ask with the whole city at once: how similar is any one high street to any other? For every pair I compared their cuisine mix, scored 0 for nothing shared, 1 for identical for all sixty thousand pairings.

Pick two London high streets at random and, on average, they already share almost half their food profile. For a city like London that sells itself on the variety of its neighbourhoods, that's a lot of similarity. But the average is just a snapshot, what drives it is even more interesting.

So, I sorted streets into quartiles by chain share. The streets with the fewest chains are the most distinctive; the other three quartiles all look noticeably more like the rest of London. Chains seem to act as a convergence layer: you don’t need a high street to be dominated by them. A relatively small number of repeated names is enough to make otherwise different streets resemble one another more.

And that’s not surprising since the chains are running exactly the loss-minimising loop from the last section. Gail's (my favourite UK middle class bakery/coffee shop) uses an AI site-selection system that learns from its existing bakeries, combining trading data with demographics, competition and location to forecast where the next one should work. Pret has used hundreds of data points, from historic sales to footfall, to decide where it opens. Waitrose has long run its own version with geodemographics, GIS and regression (raising my favourite question of what came first: the Waitrose or the gentrification?) Each of these is a model asking "where does it look most like the places that already worked?", which is to say: find the mode, and put another one there. Congratulations: your neighbourhood has been classified as Gail's-compatible, and is starting to look a bit more like all the others. I ran the numbers on my own future street in The Hague. Luckily, Pret and it’s highly mediocre coffee hasn’t yet decided to move the Netherlands yet.

My previous pub-closures work gets at the same loop from the other side. I trained a model to predict which pubs were most likely to close and, in doing so, effectively produced a profile of the kind of pub most likely to survive. The counter-intuitive bit is that many of the things that make a pub unusual, local or hard to categorise can also make it look risky. The features that make a place difficult to predict are often the same ones we later describe as its character.

Test 2: culture

The high street flattens in physical space. I assumed culture would too, so I checked, and the answer turned out to depend entirely on where you point the loop. I started with lyrics: every year-end Billboard Hot 100 hit from 1950 to 2015, just over four thousand songs, measured for how the words changed.

Within a song, the flattening is unmistakable. Vocabulary richness, distinct words relative to length, has fallen by more than a quarter since the early 1960s, while the share of repeated lines has climbed by nearly a third. The modern hit says less and says it more often, because the hook that works gets repeated. Btw, in the data AI music hasn’t even entered the scene here yet…

But zoom out to the chart and the loop reverses. Hits resemble one another less than they used to, the vocabulary across the chart as a whole has widened, and the biggest study of cross-national streaming charts finds music diversity across countries rising since 2017, not falling. I fully expected to find monoculture here, but it handed me the key to the whole argument. Prediction flattened the one dimension something was optimising on, the hook, the singable mode, and left every other dimension free. Nothing was minimising a similarity loss across songs, so nothing pulled them toward each other. Something was very much minimising a boredom loss within songs, so they collapsed inward. The flattening is a choice about which dimension we point the loss function at. Which means the real question is never whether prediction flattens the world, but which dimension we’ve aimed it at, and who did the aiming.

And on people, we’ve aimed it straight at consumption. A recommender shows you the handful of things it predicts you’ll like, you engage because they were selected to be engaging, it retrains on that, but it only ever learns from what it already showed you. Everything it didn’t surface decays out of consideration, untested. So, we have more data points than ever, but they’re a censored sample.

MovieTweetings shows the endpoint: nearly a million public film ratings since 2013, across roughly 38,000 films. If everyone watched a bit of everything, the line would run along the diagonal. Instead it shoots almost straight up. The top one percent of films, about 380 titles, pull nearly forty percent of all attention; half of everything anyone watched sits in the top two percent; thirty-eight films, a tenth of one percent of the catalogue, account for more than an eighth of all activity on their own.

That’s what “more choice than ever” looks like from the inside: a vast catalogue you can technically reach, and a near-vertical wall of sameness almost everyone actually climbs. We call ending up at the top of that wall “knowing your taste”, but it’s just the menu that narrowed.

Test 3: synthetic people

The same problem showed up in an experiment of my own, one that happens to also be a small research industry right now. Everyone from survey shops to LLM labs is trying to build synthetic respondents: "personas" meant to behave like a population before you spend money asking actual humans. It's a genuinely useful idea. It also fails in exactly the way this whole essay is about, which is why I couldn't resist poking at it.

The obvious way to do it is to give each person some attributes and let a model decide what they think. But real populations aren’t collections of independent variables. Age predicts education, education interacts with income, values cluster with attitudes, friends influence friends. In other words, people come in lumps.

So I tried progressively harder to put the lumps back in. I started with a deliberately dumb generator that rolls each characteristic independently, then added latent psychological traits, recognisable archetypes, and eventually social networks in which similar people cluster and influence one another. I did the same on the response side: independent sampling through correlated, conditional and causal models.

A problem arose: the fancy models produced more elaborate stories about why people differed, but the population-level answers stayed remarkably smooth, on many questions spread almost evenly, close to maximum statistical diversity. In one comparison the dumb independent model was the best at matching the target distribution.

And it’s the same theorem as the recommender, just standing further back. A synthetic population trained to get the averages right buys that average-rightness the only way any optimiser can, by suppressing variance. The marginal comes out perfect and the structure, the clusters, the correlations, the strange combinations, slowly dies. Which is a genuine problem, because a population can have perfectly respectable averages while representing nobody in particular. Fifty percent left and fifty percent right describes a country split into two furious camps and a country full of mild centrists equally well. And this is now (occasionally and by some) being sold, with a straight face, to campaigns and market researchers as a substitute for asking actual voters. (I may or may not have argued exactly this on a panel in London a few weeks ago.)

Why it matters

The obvious objection is: so what? If people like Gail’s, Spotify gives me music I enjoy (I’ll admit that I recently moved back to Spotify after getting way too frustrated with the algorithm of Apple Music, Apple, seriously, step this up), and Netflix saves me forty minutes of scrolling, maybe predictability is just convenience.

But variety is where new options come from. The strange Georgian-Japanese street-food fusion restaurant nobody can classify, the pub that doesn’t fit the template, the song that sounds unlike what came before, most will never matter. But some will. A world that keeps removing the unlikely also removes the things that could have become the next likely. Ecologists have known this for a long time already and even priced it: they call it the insurance value of biodiversity, the standing reserve of rare species that cost energy and contribute nothing, until the climate shifts and one of them turns out to be what survives. A monoculture is efficient right up until the weather changes.

And there’s a second cost, harder to see, and it’s the one that actually a bit scared of. Once prediction shapes the choices in front of us, we lose the ability to tell the difference between what people wanted and what the system made easy to want. If an algorithm feeds me Scandinavian crime dramas for ten years because I once watched two, my viewing history becomes airtight evidence that I love Scandinavian crime dramas. Admittedly, I have watched two Scandinavian crime dramas in my life. Netflix has since decided this is my entire personality. We risk mistaking a world shaped by prediction for a world shaped by preference, and the more these systems decide what survives, the more power moves from the people choosing among options to the systems deciding which options get a chance to be chosen.

So what do we do about it?

Here is the good news, and it’s the most hopeful finding I have: the fix is known, it’s old, and it’s cheap, it just requires a small change. The whole emerging political economy problem is a version of the single most studied trade-off in computer science, exploration versus exploitation. Any system learning from feedback has to divide its effort between exploiting what already scores well and exploring options it hasn’t tried, in case they’re better. This is the multi-armed bandit problem (yes, I actually paid attention in my undergrad minor), and the maths is unambiguous: an agent that only exploits gets trapped on the first decent option it finds and never discovers the best one. Pure exploitation is a known failure mode.

So the entire fight is forcing exploration back in, and when I simulate it, the result is actually encouraging. The only thing I vary is how much exploration I force, and diversity goes from near-total collapse to fully preserved. But, and this is the finding regulators need, it’s a threshold. Two or five percent, roughly the level of a token “discovery” tab, does almost nothing. You have to clear a cliff, around a fifth of the entire system’s attention, before variety survives at all (just as in the nature).

Which means this cannot be left to voluntary good behaviour, because no engagement-maximising firm buys a fifth of its own attention out of principle, and the maths guarantees that whoever does spend it will look, in the short term, like they’re leaving money on the table. This is a textbook case for regulation (aka failed markets), and the encouraging thing for those of us watching from this side of the North Sea is that the scaffolding already exists on both sides of it. The EU’s Digital Services Act already forces very large platforms to offer at least one recommender feed not based on profiling, and to open their systems to vetted researchers, which is precisely the audit access you’d need to measure whether a feed clears the exploration threshold.

The UK, post-Brexit, is writing its own version and could go either way. The Online Safety Act and the new Digital Markets, Competition and Consumers Act hand Ofcom and the CMA real leverage over how the largest platforms rank and recommend, but so far the framing is almost entirely about harm and competition, not diversity. That’s a gap, and a chance to lead: Britain has form here, the New Economics Foundation’s “Clone Town Britain” work named high-street homogenisation two decades ago. Interoperability is the other lever the CMA in particular could pull, because the deepest discipline on a feed that has trapped you is the credible ability to leave it and take your data with you. And at the level of the high street, friction turns out to be a feature: if a handful of chains does that much homogenising work, then protecting the independents around them, business rates relief, planning powers over frontages is the easy fix.

At the level of the self, sure, force your own exploration, the book from the shop that might close, the restaurant with four reviews instead of four thousand. But I want to be honest that this is the weakest lever, and I’m suspicious of every version of this argument that ends there, because “just be a more adventurous consumer” is exactly the conclusion that lets the systems off the hook. The exploration has to be mandated at the level of the machine, because that’s where real change can happen.

Going back to the café, and the question I still can’t answer. I used to think my problem was having no home, too many countries, no single belonging. The truth is though that the world is building me a home everywhere, frictionless and familiar and mine, and I should be more frightened of how good it feels than I am. Belonging everywhere and belonging nowhere turn out to be the same condition, and a model that can manufacture the first guarantees the second. The home I can find in any city is basically just a prediction about me that came true. I’d rather, I think, be somewhere that surprises me, even if it means being, for once, a stranger.

Subscribe now

Share

Thank you for reading! I have some cool, new ideas I’m working on. If you want to keep my Substack alive, buy me a coffee and/or subscribe:)

Then adult life happened: four countries, a great number of moves, and somewhere across all that motion I lost the part of me that was a local somewhere. I’ve become very good at arriving and fairly hopeless at belonging. To be fair, I’ve been approached a couple of times if I wanted to coach a local sports team in London (sorry Peter I could not commit since I’m in the process of moving, again, I know…).

But I’m now doing the closest thing: see if there is any way I can contribute with data and see where the bottlenecks of people joining in are. I expected to write some kind of Bowling Alone narrative which is Robert Putnam’s account of Americans who kept bowling but stopped bowling in leagues, the visible edge of a long collapse in clubs and the trust and social capital they generate. But the data told a different story.

Putnam’s image was of people who still bowled but no longer bowled in leagues. In Britain, the organisations are often still there and some new ones are still being founded. What has changed is the social form of community itself. In one Britain, civil society still looks like membership: the choir, the rowing club, the amateur dramatics society, the sports club, the place you join. In another more deprived Britain, civil society increasingly looks like provision: the food bank, the advice centre, the mental-health charity, the youth intervention project, the place you turn up to because something has gone wrong.

The register refused to cooperate

First I had to sort charities into the convivial kind (the bowls club, the brass band, the scout troop) and the provision kind (food banks, mental-health support, youth services), I did it three ways and only trusted what all three agreed on. A blunt keyword rule (does the description say “bowls club” or “food bank”?); an unsupervised clustering that ignores my labels and lets the text sort itself into families; and the fun one, a conviviality axis, i.e., I turned each charity’s self-description into a point in space with a language model, drew a line between a “members gather” pole and a “we provide a service” pole, and measured which way each charity leaned. (Still about a fifth of the register refuses to classify cleanly, so read everything below as direction, not decimals.)

Then I tracked the founding rate of the pure Putnam archetype such as the bowls club, the working men’s club, the sports and social club. The line over time is flat (see below). It is also tiny: the genuine members’ club is a couple of percent of the register and has barely twitched in twenty years. So, Britain didn’t stop founding the members’ club…

Zooming in, what the register does show is a reshuffle, from more community and youth sport, more arts and festival trusts, more mental-health charities; less education, training and youth work (see the figure below). I’ve seen this shape before. When I mapped Britain’s lost pubs, London looked like the one success story, until you noticed it hadn’t so much kept its pubs as swapped them: the £2.50 working men’s local for the £7.50 craft taproom. The charity register pulls the same trick at national scale, because a charity gets registered when there’s a reason to formalise, and those reasons differ wildly by field. A community sports club now has to constitute itself to get a grant; a group of friends who play bowls has never had to register anything. So the register is really a map of where it became worth incorporating, which is to say it measures the funding rules and the tax code, with extra steps.

The charity register is bad at measuring whether Britain still gathers, but it is good at showing what kinds of community infrastructure become visible to the state. That null is the first honest finding (which wouldn’t reach your academic journal!).

So I asked whether people actually turn up

If you want to know whether people still show up, you have to measure this directly. The Community Life Survey, the nearest thing England has to a running pulse of civic life, records formal volunteering falling from 45% of adults in 2013/14 to 28% by 2024/25 (see below) from nearly half the country to under a third, with regular, at-least-monthly volunteering down from 27% to 17%. Perhaps more surprising, the slide started before anyone had heard of covid furlough (it was already down to 37% by 2019/20); the pandemic simply shoved it off the ledge, and it hasn’t climbed back. Charitable giving has gone the same way, 82% of adults down to a low of 63%.

A small caveat: the participation surveys have seams, namely, the Community Life Survey changed method partway through, the pandemic muddies the middle, so treat the percentages as direction. And the causes of the volunteering slide are genuinely contested (time, secularisation, an ageing volunteer base, the phone in everyone’s pocket).

Sport says it in miniature. England is not less active, but Sport England now openly describes a shift away from organised, club-based sport towards flexible, individual activity: the treadmill, the solo run, the app instead of the team and the league. Fewer than one in five over 55s belongs to a sports club at all. We didn’t stop moving; we stopped doing it together (I run, cycle and swim often alone, so I have to admit that I am part of the problem - or I just blame it on Strava;)

Put the two halves together and the resolution is nearly the opposite of the piece I sat down to write. Bowling Alone did come to Britain but it’s just invisible in the place I went looking, because what’s dying is not the founding of organisations but the turning up to them. Which is the point the whole detour was really about. We measure civil society by counting its organisations, because organisations are the tidy thing the state knows how to count. But an organisation is a shell, and the shell outlives the people in it, which is what Theda Skocpol called “management without members”: bodies that still file accounts and keep a logo, and have almost nobody who belongs.

In richer places, community means joining. In poorer places, it means need.

There’s a second question the register can answer honestly, because it comes with postcodes attached: not just whether Britain gathers, but where. To unpack the aggregate national picture, I mapped every convivial body and every provision body to its local authority, divided by population, and lined the country’s three hundred odd districts up from least to most deprived. They pull cleanly apart. In the most comfortable tenth of England, convivial clubs are a slight majority of associational charities (about 52%) and provision the rest. By the most deprived tenth it has flipped hard: convivial bodies are barely a fifth (~22%), and provision charities such as food banks, mental-health support, youth services, are nearly four in five (see the image below). The lines cross around the third decile. Gathering, it turns out, is something the comfortable do for pleasure; provision is what grows where the state has thinned. Picture two high streets: the market town that still has its rowing club, its choir and its drama society, and the estate two postcodes over with a food bank, a youth project and a mental-health drop-in. Both are full of good people doing necessary things, but only one of them is somewhere you join.

Two checks, because a single crossing chart can fool you. The first asks whether this is a real spatial pattern or just a tidy-looking fluke. The standard tool for that is a lovely statistic called Moran’s I. It answers one question: do places that gather sit next to other places that gather, more than chance would allow? It runs from −1 (a perfect chessboard, every place the opposite of its neighbours) through 0 (random scatter) to +1 (everything clumped with its own kind). Convivial provision scores +0.11 which is modest, but with a shuffling test putting the odds of seeing even that by fluke at about one in a thousand. Thus gathering comes in geographic clumps: the leafy-suburb and market-town belt.

The second check is the objection you’re already forming: isn’t this just London, or just big cities, dragging the line? So I ran a regression that holds those things still, incorporating population size, and which region a place sits in, and asks what deprivation does on its own. It survives. Each one-standard-deviation step up the deprivation scale which is loosely, going from a middling district to a visibly poorer one, knocks seven to eight points off the convivial share, and the result is about as statistically unambiguous as this work ever gets (p<0.001). Population, once deprivation is accounted for, does nothing at all, so poverty is doing the work, not mere city size.

So the decline in showing-up isn’t spread evenly. Belonging is becoming a postcode lottery, thickest exactly where life is already easiest. A caveat first: this is a map of registration, not gathering, so poorer areas may simply incorporate their clubs less often. But the provision side is hard to wish away, and if the gap is even partly real, it matters, because clubs, teams and choirs are precisely where people meet across class lines, that is, the 'bridging' ties Robert Putnam argued let people climb and Raj Chetty's team, mining 21 billion friendships, found that the cross-class connectedness of the place you grow up in is the single strongest predictor of escaping poverty, ahead of school quality, family structure or local inequality, and worth on the order of 20% more in adult earnings for low-income children raised in well-connected places. So thinning the convivial life of poorer areas doesn't merely leave them lonelier; it strips out one of the few engines of upward mobility they had.

So what would I actually do about it?

The data narrows the advice three ways.

First, it says the bottleneck is not founding but turning up, so we should not make it easier to start organisations (which the register says we manage fine), but rather lower the cost of participation. The Community Life Survey’s own most-cited barrier to volunteering is time: work and life. That points at things such as paid volunteering leave, episodic and “micro” roles rather than the old time-bound committee seat, the flexible terms on which younger people are visibly still turning up.

Second, and this is where the map earns its keep since the intervention has a spatial element, and the policy actually arriving doesn’t match it. The 2026 English Devolution and Community Empowerment Act hands communities a genuinely strengthened Right to Buy the local asset such as the pub, the hall, the sports ground, now its own protected category, replacing the near-useless old Right to Bid with a real right of first refusal. Good, as far as it goes. But the Community Ownership Fund that used to put money behind those purchases has closed, and the Act’s own impact assessment expects only a minority of listed assets to change hands, because finding the money inside the moratorium is the hard part. A legal right you can’t afford to exercise is usable mainly where the money already is, and weakest exactly where the deprivation gradient says convivial infrastructure is thinnest. So, the missing half of the policy is capacity and funding aimed at deprived places.

Third, the meta-point this whole exercise kept arriving at: stop treating the count of registered organisations as the measure of civic health, it’s the number that stays reassuring while the thing it claims to track drains away. The honest dashboard already exists: the participation survey. Fund against that, target it by the geography of thinness, and stay sceptical of anyone promising community can be conjured by money alone.

I went looking for the dead clubs and found them mostly still standing. It’s the people who’ve left, and if I’m honest, I’m one of them, which is the harder thing to fix.

But these people haven't left evenly. The leaving has a geography: gathering holds where life is already comfortable and thins where it isn't, so we're not simply a country that joins less, instead we're splitting into places that still gather and places left with whatever the state can provide instead.

Thank you for reading! I have some cool, new ideas I’m working on. If you want to keep my Substack alive, buy me a coffee and/or subscribe:)

Share

I’d been meaning to go ever since I started writing this piece - Punchbowl is the kind of place that turns up in Sydney food writing as shorthand for something authentic and slightly inaccessible, the Lebanese heartland of the city’s south-west. I went, naturally, for booza: traditional Lebanese ice cream.

It was only after sending my mum this photo (obviously to make her jealous) that I found out this place had a personal history. The owners used to have an ice cream shop directly below where my taita (grandmother) lived in a small coastal town near Tripoli in Lebanon. She would lower a basket from her window with money in it and the owner would send up a tub of booza. It was only later buying pastries in Punchbowl that my grandmother suddenly recognised the owner. And now, by pure coincidence, I had bought my booza from that exact shop.

Walk through Punchbowl, Lakemba, Cabramatta, or Marrickville and the food tells you who arrived and when. Sydney is not unique in that, London also has Brick Lane, Brixton, Southall, Peckham’s Rye Lane. Every city with a significant migrant history has neighbourhoods where you can read the waves of arrival through the shop fronts. The difference isn’t whether these neighbourhoods exist, it’s whether they survive. Brick Lane, once the heartland of Bangladeshi settlement, now hosts a Cereal Killer Café and a Brewdog. The Vietnamese strip on Mare Street in Hackney has been thinning for a decade. In Sydney, the same communities that arrived in the 1970s and 1980s are still commercially present in the same suburbs.

That contrast became this piece: why is London slowly being replaced by chains while in Sydney locals can keep their doors open? I retrieved my scraped dataset of 17,127 London restaurants to find out.

The obvious explanation is that London has worse landlords, or worse food, or worse taste. I’m going to argue something different. London is losing its Brick Lanes precisely because it is the kind of city urbanists say you need: walkable, transit-rich, dense and mixed-use. And Sydney’s independent culture has survived because Sydney is, by almost every metric we use to measure good urbanism, badly designed. The car-dependent strip, the quarter-acre block, the Sunday penalty rate, just about every feature a planner would want to fix is exactly what protected the booza shop in Punchbowl.

I’ll show you the data and what we can do about it.

The meritocracy test

Let’s start with the obvious question: are chains replacing independents in London because they’re actually better?

To find out whether a restaurant is a chain or not, I flagged any restaurant brand appearing in 2+ locations across 2+ London boroughs, then manually checked the top 100 to catch false positives. (There are a lot of unrelated pubs called “The [Neighbourhood] Arms.”) This gave me 6,828 chain locations and 10,241 independents. Chains are roughly 40% of London’s restaurant fabric in this dataset, though with stark variation by borough. Hackney, Kensington and Chelsea, and Westminster are the most resistant; Havering, Barking and Dagenham, and Bexley the most captured.

To test survival, I briefly pretended London’s restaurant market was a meritocracy and Google ratings were reliable (this is a huge assumption, as I argued and even tried to correct for in my previous piece on Google Maps, now also out as a Guardian it’s complicated documentary), making survival purely a function of quality. I ranked all rated restaurants from lowest to highest and found where the major chains land. Subway: 10th percentile, McDonald’s: 18th, Greggs: 23rd. Even the “nice” chains such as Wagamama, Nando’s, barely crack the median. If ratings determined survival, these brands would be struggling to stay open. There are over 10,000 restaurants Londoners like better than Pret. Yet Pret has 200+ locations.

The horizontal bars tell another story: chains don’t just have mediocre averages, they have mediocre ranges. A Starbucks is a Starbucks wherever you go and they’re not selling good coffee, instead they’re selling the certainty that it won’t be bad coffee, at a location you’ll pass whether you meant to or not.

So chains don't dominate because they're better. They dominate because they can afford to be worse, which means the question is what kind of place lets them get away with being worse.

What chains actually need

I trained a gradient boosting classifier to predict whether a restaurant is a chain or an independent based on structural features (no names, no menus). Gradient boosting is the default for this job because it handles tabular data and messiness well, and it reveals which features matter most. Some of the features:

• Distance to the nearest Tube station (TfL published coordinates)

• Foot traffic (station usage stats)

• Estimated annual rent (Valuation Office Agency rateable values)

• How many other restaurants cluster within 200 metres

• Whether it sits on a main road or a side street (OpenStreetMap)

The model predicted correctly 78% of the time. A machine can tell, reliably, whether you’re walking into a Pret or a family-run Thai place just by knowing where it is. The top three features, by a distance: proximity to a Tube station, nearby restaurant density within 200m, and station footfall.

Read those features carefully, because together they describe a very specific kind of place. Not zone 1, where competition is brutal and customers are discerning enough to walk an extra block. Not the far edges of outer London, where the pedestrian volumes simply aren’t there. The chain sweet spot is the in-between band: walkable enough that footfall clears margin, transit-connected enough that customers arrive in pass-through commuter mode, dense enough that a 200m radius generates customers but not so distinctive that being unmemorable becomes a liability. That’s exactly the pattern in the borough map. Hackney and Westminster resist because their customers are choosing; Havering and Bexley get captured because their customers are just passing through.

The unifying claim is this: chains follow legibility. A city becomes legible to a site-selection algorithm when it has been organised into walkable, transit-connected high streets with predictable pedestrian volumes. That legibility is what good urbanism produces and simultaneously what makes a city capturable.

The strongest predictor of chain resistance is the absence of the conditions that make a place computable. Areas with very high station density have fewer chains because the competition is too good. Areas beyond reliable transit have fewer chains because the volumes don’t exist. Everywhere in between is exposed.

See this for yourself by looking at the chain saturation index or “character” index I build for every London borough: the percentage of restaurants that are chains. I’ve also added a few extra rabbit holes for exploration.

Welcome to the real estate game

Take Blank Street. You’ve seen them: the tiny shopfronts, the mint green branding, the queue of people who needed coffee 30 seconds ago. Blank Street raised ~$120M in VC funding to sell... coffee. The pitch wasn’t “better coffee.” It was: small footprint, high-traffic locations, automated La Marzocco Modbar machines ($20k Italian espresso systems now operated by algorithm), minimal staff. They’re a real estate arbitrage play disguised as a coffee shop. The coffee is the justification for the lease, not the product.

VC money is patient in ways independent capital isn’t, e.g., they can run a location at a loss for two years to build brand density. They can sign a lease that makes no sense this year because they’re betting on the neighbourhood in five. Your local café has to make rent this quarter.

Critically, Blank Street’s model requires a continuous pedestrian corridor. It cannot exist in Houston. It cannot exist in Sydney. It can only exist in a city that planners would describe as functioning well. Walkability is what makes their model possible.

And chains don’t just appear in legible places, someone lets them in. I tried to investigate landlord behaviour directly using the UK Land Registry’s CCOD dataset (every property owned by a company). In practice it was a nightmare: addresses don’t align cleanly, and the big estates own properties through dozens of opaquely-named subsidiaries. I eventually admitted defeat (btw, if any GIS masochists want to take this on, please do.)

What industry reports make clear is how the game works. Commercial landlords aren’t just looking for the highest rent. They want covenant strength: chains can show corporate accounts, family businesses just show faith. Brand fit: yes, landlords use the word “curation” now, I wish I was joking. And tenant improvement: will you spend £500k fitting out the space, effectively giving them a free upgrade when you inevitably close? An independent Vietnamese restaurant with a 4.8 rating fails most of these tests. A VC-backed coffee chain passes all of them. This is why leases have shortened from 15–20 years to 3–5 today. Landlords don’t need your loyalty when the next legible tenant is already calling.

This is the mechanism by which London loses its Brick Lanes. Not in one dramatic moment, but through a hundred individual lease decisions on walkable high streets that capital has learned to read.

How Sydney got unlucky enough to stay independent

In 2000, Starbucks entered Australia with enormous confidence: 84 stores by 2008, then a single retreat that closed 61 of them. The remaining rump survives mostly in airports and tourist traps, propped up by jetlagged Americans who’ve never had a decent flat white (sorry!). This is usually told as a story about Australian taste but it’s actually a story about urban form refusing to become legible.

Post-war migration to Sydney followed the railway lines west. Lebanese families in Punchbowl and Lakemba. Vietnamese communities in Cabramatta. Greeks in Marrickville. Italians in Leichhardt. They settled there not because the suburbs were charming, but because they were affordable: the quarter-acre block pushed property costs outward, and the western train lines made it liveable without a car.

What they then built is the opposite of a walkable high street. These suburbs run on strips: low-density commercial corridors along arterial roads. A car tyre shop, a Lebanese grocer, a Vietnamese bakery, a café with a handwritten sign. The strip is structurally illegible: the pedestrian catchment that makes a Pret viable simply isn’t generated by a road designed for cars. Looking at Sydney, beyond 400 metres from a Westfield, chain presence essentially disappears. The strip belongs to the independents because no algorithm can read it.

Three compounding factors locked this in.

• Sydney’s population density is roughly 40% of London’s across a comparable geographic footprint. The pedestrian volumes that activate chain site-selection don’t exist outside the Westfields, which is exactly where the chains cluster and exactly where Australians use chains contentedly, when they want to.

• Sunday penalty rates mean a chain’s Sunday shift costs 175% of base rate, compressing the labour arbitrage the chain model depends on. The unit economics that make a Pret work in London don’t work the same way in Sydney even when the location does.

• And the duopoly paradox: over 65% of Australia’s grocery market sits with Woolworths and Coles. Australians use chains readily when the category doesn’t reward distinctiveness. Hospitality escaped that consolidation because food quality is discernible, and because the strip gave independent culture room to raise the standard before chains could establish themselves. By the time Starbucks arrived in 2000, the answer to “where would I even put one?” was already “the airport.”

The uncomfortable corollary: if Sydney decided tomorrow to upzone the inner west, build mixed-use development along the strips, and run good public transport down Canterbury Road, the booza shop’s days would be numbered.

What (if anything) can we do?

This is the awkward bit. The standard urbanist policy menu including denser zoning, more walkability, better transit, mixed-use everywhere, is good for almost everything we care about (climate, congestion, housing, public health) except for the independent commercial culture this piece is about. The answer can’t be “less walkability”. It has to operate on what fills the walkable spaces, not on whether to have them.

So the targets are landlords and capital, not pedestrians.

Regulate the landlords, not just the tenants. San Francisco has “formula retail” restrictions: chains need special planning permission in certain neighbourhoods. It’s imperfect and litigated constantly, but it exists. The UK has nothing equivalent. Our 2020 Use Class E changes went the opposite direction, giving landlords more flexibility to chase the highest bidder.

Make ownership transparent. The Land Registry data I used is free but buried. What if Deliveroo and Google Maps showed “This restaurant is on land owned by Shaftesbury Capital PLC” the way they show hygiene ratings?

Consider commercial rent stabilisation. Yes, I know… rent control is the policy that unites economists in opposition. But commercial rent stabilisation for small retail isn’t the same as residential. It’s been tried in New York for small retail and the sky didn’t fall. At minimum, we could restrict rent increases for existing tenants past certain tenure thresholds.

Protect fragmented ownership. This is the corollary of the whole argument. The neighbourhoods most resistant to chain capture in London tend to have messy, distributed landlord structures and Sydney’s strips have this by accident, Soho and Spitalfields have it by historical fluke. Planning policy could actively discourage portfolio consolidation on designated high streets, treating fragmentation as a public good rather than a market failure to be tidied up.

None of this is simple and it’s highly political. But “the market decides” is also a political choice, one that favours the people who already own the land, and that compounds, gradually and irreversibly, into a city where no one recognises you from your hometown 14,200 kms great circle or by air about 15,500 depending on the flight path and all the current diversions (yes, I actually spent quite some time calculating the distance between Punchbowl and the town in Lebanon).

I’m not per se saying that chains are evil though. I like reliable wifi and a guaranteed seat sometimes and consistency has value and convenience is real. But when a restaurant closes, we tell ourselves stories. Such as: the food wasn’t good enough, they didn’t adapt, the market decided, etc. The market didn’t per se decide though. The market is a landlord choosing between someone who makes great pho and a VC-backed coffee robot and taking the bigger number. The market is a site-selection algorithm that can only see a city once it has been organised into shapes it can read.

My grandmother lowered a basket from a window, and a tub of booza came back up. That is not just a charming family anecdote. It is a tiny model of what cities are for: proximity, trust, memory, repetition, recognition. A city goes beyond just being a housing market or a transport network. It is also the chance that, 14,200 km from where a story began, someone still recognises the taste of home.

The problem is that we keep building cities where the easiest thing to finance, lease, scale, and insure is also the least likely to remember us.

If you found this interesting, the same structural logic applies to pub closures, supermarket deserts, and the political economy of the high street. Subscribe so you don’t miss the next rabbit hole. And if you’d like to keep this Substack alive, buy me a coffee and subscribe:)

Share

So perhaps it is a little strange that I am sitting in the Cronulla RSL, running gambling models while surfers outside pick off the hardest waves and, thirty metres behind me, the actual machines hum.

I am not here to tell you gambling is bad. I want to write about something more specific: that in a small outback town called Balranald, gambling machines extracted AUS$3,961 last year from every man, woman, and child in a town of 2,209 people. Once my models failed to explain that number, I realised that I had misunderstood something important about Australia (despite being an Aussie national and having come here since I was three months old).

The ‘pokie’ room at Cronulla RSL, thirty metres from the bistro and the sea view.

The rooms that closed

Sociologists have a name for the spaces that are neither home nor work, but where community actually forms: the third place. The pub, the library, the bowling club, even the post office. The place you go because it is there, other people are there, and that is enough. For many people, that is exactly what the Cronulla RSL is.

Britain lost these at scale. Austerity closed the libraries, Amazon closed the post offices, property prices and private equity closed the pubs: 14,000 of them between 2001 and 2021, one every fourteen hours for two decades. I wrote about this a few months ago: “Britain Lost 14,000 Third Places. They Were Called Pubs. Is Your Local Next?” The statistic that stayed with me most was what replaced them. For most communities that was nothing: the room closed and stayed closed.

This is the baseline to keep in mind, because Australia made a different choice.

The deal

In the 1950s, Australian state governments cut a deal with registered clubs such as RSLs (Returned Services League, the veterans’ organisation, the equivalent of the British Legion), football clubs, bowling clubs. The deal was simple: you get the right to operate gambling machines, (referred to often as pokies) and in exchange, you fund community services and keep the doors open.

It worked: the rooms stayed lit. The RSL stayed open and the third place which Britain was losing, Australia kept. Nobody looked too hard at the extraction rate but the clubs were drawing 40 to 60 percent of their income from the machines, and had become so embedded in community life that any threat to the pokies read, politically, was a threat to the community itself. The machines and the rooms had become the same object: you could not remove one without destroying the other.

The Cronulla RSL has a bistro, sports bar, sea views, ANZAC photographs on the wall and a Sunday afternoon crowd so relaxed that you cannot for a second accidently think that you are in London. It is also, behind me, running a variable ratio reinforcement schedule which is the same psychological mechanism as the pull-to-refresh on your phone, resulting in people who came in for a quiet afternoon ending up behind a screen. B.F. Skinner already documented this in rats in the 1950s.

The machines are legally required to return at least 87% of turnover, but turnover is not the same thing as the amount you first put in. Every small win is usually fed back into the next spin, so over the course of a session a $50 note can be churned through the machine several times. The machine is therefore returning 87% or more of that larger recycled total, not 87% of your original $50.

Now, Australia loses more money per capita to gambling than any other country on earth: roughly AUD $1,300 per adult per year, nearly double the next highest. Over 620,000 Australians are classified as problem gamblers and a further 2.9 million are at-risk gamblers: together about one in six adults. You could see it as the bill for keeping the third spaces open.

So I opened my laptop and tried to measure the bill.

Where the gambling takes place

The first question I wanted to answer was simple: where does the money actually go?

I downloaded the NSW Independent Liquor and Gaming Authority’s full venue register: 2,184 licensed venues, machine counts, and six months of net profit figures. ‘Net profit’ is the industry’s preferred term for ‘money extracted from the public,’ which is excellent branding if you are the industry. I joined this to Australian Bureau of Statistics deprivation and rurality scores by Local Government Area.

I first let the data sort places into a few broad types. It is good at telling you what kinds of places exist but less good at telling you why a particular place sits where it does, or whether something unusual is happening inside an otherwise familiar pattern. To answer that question, I needed a different tool.

I then trained a gradient boosting model on deprivation, rurality, density, and income to see how much of the pattern the obvious structural factors could explain. It explained about 71% of the variance in extraction rates. The more interesting part is the 29% it can’t explain though. I turned that unexplained remainder: the gap between what the model predicts and what actually happens, into a vulnerability score. It measures how much a place is being extracted above and beyond what its observable circumstances would predict. A high score means that something the model cannot see is making that place unusually exposed.

Balranald scored 98 out of 100. Every structural variable said it should be close to average, instead the extraction rate was $3,961 per resident in six months, and no feature in my model explained it.

At that point the model had done its job, and I had to do mine. The explanation brought me straight back to the idea of the third place. In a town of 2,209 people, the club is the only room where the lights are on on a Friday night. The machine has no competition because the club has no competition. It is not merely somewhere to gamble, it is the only place left to be. In other words, the machine does not only go where people are poor, it goes where people are captive.

Once I knew where the money was going, the next question was: who was taking it? The LGA (Australia’s version of a local authority area) picture above tells you where the losses are concentrated while the venue picture tells you which institutions are collecting them.

To look into that, I ran a sentence-transformer across all 2,184 venue names, sorting them into five types: independent hotels, large commercial groups, RSL and ex-services clubs, general community clubs, and ethnic and faith community clubs. Sentence transformers are language models that turn text into numerical vectors capturing semantic meaning, so “Slovenian Cultural Club” and “Polish Community Centre” end up close together in the mathematical space even if they share no words.

I found that small independent clubs, community anchors, RSL and ex-services clubs have near-identical medians, clustered very tightly. Then the ethnic and faith community category: median $73,000 profit per machine, nearly double every other type.

Dooleys Lidcombe Catholic Club, Cabra-Vale Ex-Active Servicemen’s Club, Triglav Mounties Group in St Johns Park, a Slovenian community club in Fairfield, are places where people speak their first language, eat their heritage food, maintain connections the broader city does not provide. Which is why the machines inside them extract so efficiently: the social cost of leaving mid-session is self-exclusion from the community itself. Again, I find that people are walking away from a gambling machine, would be actually walking away from the community space.

By this point, three things were becoming clear. The first model had shown the shape of the gambling landscape. The gradient boosting model had shown what that landscape could not explain. The venue classifier had shown which kinds of institutions were most effective at turning social belonging into revenue.

There was one last question left: which individual venues looked strange even by those standards?

For that, I used the bluntest of the three models: an Isolation Forest. It doesn’t try to predict anything or explain anything, it rather learns what normal looks like across the full dataset: normal machine counts, normal revenue per machine, normal venue type and flags the observations that are hardest to fit into that picture. The logic is that you repeatedly cut the data with random splits until each point is isolated in its own partition. Points that are genuinely unusual get isolated quickly, in very few cuts, because they’re far from everything else.

It kept pointing at Markets Hotel in Homebush West which has thirty machines which earned $228,000 profit per machine in six months. This is nearly four times the rate of Mounties Group, which has 603 machines and I could not explain it with structural features.

By then, all three approaches were converging on the same conclusion from different directions: the machine doesn't go where people are poor, it goes where people are captive. Captivity, it turns out, looks exactly like a third place, the room that stayed open because the machine was inside it. Before I tell you what Britain already learned, I want you to see the pattern for yourself. Here is the interactive gambling map I developed: the NSW gambling map.

Toggle to revenue per machine and watch the map reorganise: the Western Corridor fades, the bright spots shift west and north: smaller towns, thinner dots, places that don’t appear in the headlines. Then drag the vulnerability slider to 80 and above: Sydney nearly empties and the outback shows. Find the dot on the flat plain west of the Riverina: twelve machines with a population 2,209 and $3,961 extracted per head. That’s Balranald.

What Britain already learned

Now that I hopefully convinced you to see the machine as an captivity optimisation system, the next question is: what happens when you regulate it? Britain already ran that experiment, and the result is very useful for Canberra.

Britain’s version of the machine was the Fixed Odds Betting Terminal (FOBT): a touchscreen in a high-street bookmaker letting you stake £100 every twenty seconds on virtual roulette. At their peak, 35,000 FOBTs generated £1.7 billion annually. A single machine in a Ladbrokes on Bethnal Green Road could take £1,000 in an hour. One study found machines in the most deprived 10% of English neighbourhoods generating twice the revenue of those in the least deprived, mainly because the sessions were longer and the losses higher.

Where Australia’s machines sat inside community institutions funded by nostalgia and social obligation, Britain’s sat in shops with blacked-out windows funded by footfall and proximity to deprived neighbourhoods. After years of campaigning and parliamentary inquiries, the government cut the maximum FOBT stake from £100 to £2. Bookmakers closed shops, and to many people who had worked very hard for a very long time, it felt like a victory.

Betting shops fell 42%. Yet, online gambling yield nearly quadrupled: from £1.8 billion in 2013 to £6.6 billion in 2023. The lines cross almost exactly at April 2019, the month the reform came into force.

Due to this reform, the industry did not lose revenue though, instead it migrated the customer and kept the money. It even added a welcome bonus and became available at 3am in bed while the shops had closing times. Basically, the reform didn’t fail strictly speaking, it actually worked but in working, made the underlying problem structurally worse. It constrained supply in one channel and demand found another.

The extra worrying thing? Online sports betting in Australia is already legal, growing, and marketed during every NRL broadcast I’ve seen including those sponsored by the clubs whose pokies produced the numbers above. The NSW cashless gambling card pilot is a solid reform, worth fighting for, but you should take into account that if it reduces physical machine revenue, demand does not disappear.

Side quest: can I beat it?

Britain’s reform pushed gambling online, and Australia’s is about to. Which raises an another curious question for someone with a laptop, a VS Code window, and too much time on a Sunday: if the machine has moved to an app, can it be treated less like gambling and more like a pricing problem?

Online bookmakers are, in one narrow but important sense, a market. Each bookmaker publishes odds that imply a probability. If Bet365 offers 2.10 on an NRL team, it is effectively saying that team has a 47.6% chance of winning. Add up the implied probabilities across all outcomes and you get a number above 100%, usually something like 103 to 108%. That excess is the overround, which is the bookmaker’s built-in margin. The house does not need to be right every time, it just needs to be consistently less wrong than you.

The arbitrage scanner I am building does the following. It pulls live odds from six bookmakers at once: TAB, Bet365, Sportsbet, Ladbrokes Australia, Ladbrokes UK, and William Hill, then converts them into implied probabilities, and looks for moments when the combined overround across different books on different sides of the same event drops below 100%. When that happens, you can bet both sides and lock in a profit regardless of the result. The window usually closes within minutes as the books adjust, so the scanner runs continuously and flags opportunities in real time.

I found, using several weeks of data on Australian and British markets, that the edges are there, consistent, and small. Margins between TAB and Bet365 on NRL markets typically ran from 0.5% to 2.1%. On Premier League lines between Ladbrokes UK and Sportsbet, I even found gaps as high as 3.4%. I would say that is more than just noise.

Arbitrage doesn’t per se tell you whether you are actually good at reading the market though. For that, the useful metric is closing line value (CLV). The closing line is the final price a bookmaker offers before kick-off. It is the closest thing these markets have to a consensus estimate of true probability, because by that point the maximum amount of information, including sharp money, has been absorbed. If you consistently beat the closing line, you were ahead of the market when you placed the bet, whether or not the individual bet won.

So I also built a second model. It estimates true probabilities from a weighted combination of closing lines across markets, then flags any live line that sits outside a confidence interval around that estimate. In plain English, it looks for prices that are better than they should be and checks whether I got there before the rest of the market did.

My model works and that, it turns out, is the problem.

There is one category of bettor the industry cannot tolerate: the one who is actually right. Sharp bettors, that is, people who identify mispriced lines and consistently bet into them, are restricted or banned as soon as they are recognised as profitable. A former industry employee told me recently about a friend who made serious money this way before the accounts dried up, one by one, as each bookmaker noticed the pattern. The system is perfectly happy for you to gamble but is much less enthusiastic if you treat it as a pricing problem.

That is what makes bookmakers different from prediction markets. In a prediction market, informed money is the point. If someone with a better model bets into a mispriced line, the price improves and the market becomes more useful. The sharp bettor profits, and everyone else gets a better estimate. The bookmaker model works the other way around: it depends on most customers being wrong in roughly predictable ways.

(As a side, with caution Polymarket can be used as a good proxy for several things since people with money on the line arguably have a stronger incentive to be right than pollsters, pundits and anyone responding to a survey. The sharp money mechanism means informed bettors push the price toward truth and profit as they do so. We could see this in the 2024 US elections and even the timing of Israel’s attack on Iran. But my own work shows that it didn’t work for predicting the next pope though;)

This is the online version of the same logic that governs the machines. The physical system extracts most effectively from the people who cannot leave. The digital system removes the people who can calculate. In other words, it is a different channel with the same principle: the market is open only to the extent that you lose in the approved way.

What to do about it?

If the machine is funding the room, then the solution cannot just be to attack the machine and hope the room survives. Australia needs a transition plan that pays for third places directly. That could mean a public Third Place Transition Fund for clubs that reduce machine dependence, a buyback scheme for machine entitlements in the most captive towns, and a captivity index that targets reform where the club is functionally the only remaining room.

My personal favourite option though would be to create a second room. Think of late-opening cafes, youth spaces, outdoor cinema nights, subsidised sports halls, cheap transport to nearby towns, all of these are more motivating than gambling reform, but they go directly at captivity. They would create competition (an economist’s favourite remedy) for the club that currently has none.

My side quest also suggests another lesson. Online betting is not some cleaner, more rational successor to the pokies. Just like the pokies, it behaves like a market until someone is actually good at it, who then gets banned. So, please policymakers, tackle online betting markets simultaneously and don’t treat this reform in isolation. Britain has shown why. A simple start would be to prohibit advertising and sponsoring (apologies to my favourite cycling team, the Unibet Rose Rockets, but you are not helping) and introduce affordability checks and identity-linked loss limits across the various operators.

The machine is friendly

Balranald is a long way from Canterbury-Bankstown and a long way from my beloved Islington. It is a small town whose population has been declining for thirty years. On a Friday night, the club is the only room where the lights are on. In six months, its machines extracted nearly $4,000 from every man, woman, and child in the town.

Driving through small towns between Sydney and Melbourne, I kept seeing the same pattern: empty main streets, shops closing early, and one club still lit up and full. Australia did not solve the problem of disappearing third places, it financed them. The people who run the club are not villains, and the people who use the machines are not fools. They are operating inside a system now so load-bearing for community life that nobody in politics can touch it without being seen to close the room.

Britain lost its rooms and got nothing. Australia kept its rooms and hid the cost inside them. The system expels the people who find the edge and keeps the people who cannot leave. The machines are still humming behind me. Outside, the surfers are still taking the difficult waves.

The machine is friendly. That is the point.

If you found this interesting, please share it. And if you want to support my Substack project while I’m travelling through Australia and turning somewhat questionable side quests into posts, you can buy me a coffee or become a paid subscriber.

Share

I’ve written about London’s pub closures, its restaurant algorithm, its segregation patterns. The whole time I’ve been circling the same question from different angles: is London’s divide just a feeling everyone recognises, or is it stable and structural enough, that you can actually draw it?

So I tried. I built a model using neighbourhood-level data across London: education, occupation, ethnicity, house prices, cafés, pubs, restaurants, ratings, etc. and asked a simple question: if there really is a boundary in this city, can a machine find it?

It could (and you can check where your favourite neighbourhood sits at the end). And once it did, the more interesting question became who put it there. The answer is less about the market than we've been told, and more about three policy decisions that are still sitting on the tab.

Sydney named it. London aestheticised it.

I was just a few months old when I landed in Sydney for the first time. What I didn't notice then was that Sydney was sharply and structurally divided. Sydney has named its divide: the latte line. The line between those who make the latte and those who drink it. Once you name a line, you have to argue about it. You have to explain who drew it, who benefits, and who’s been paying for it. Sydney has been doing exactly that, in newspapers, planning documents, parliamentary inquiries, for twenty years.

London has the same divide. Charles Booth, a Victorian social cartographer who walked every street in London and drew what he found mapped it in 1889. Wealth clustering in the west, deprivation running east along the Thames. The gradient he found is still visible in today’s data and has barely shifted in 130 years.

But London never named it. Instead, London called it character. Called it up and coming. Called it regeneration. A vocabulary carefully chosen to describe a structural inequality as if it were an aesthetic phase, something a neighbourhood passes through on its way to becoming somewhere worth living. We have been looking at this line for longer than Australia has existed as a federation.

So I wanted to do the impolite thing and draw it.

How do you find a line nobody drew?

The obvious way is to stare at a deprivation map until a story presents itself. I did something only slightly more dignified.

I assembled a neighbourhood-level dataset from all data in my previous Substack posts (google map’s algorithm, pub closures, segregation and inequality measurement) at Lower Super Output Area level of roughly 1,500 residents each, combining measures such as education, occupation, ethnic composition, house prices, café density, pub density, restaurant density, reviews and ratings. In other words, both the deep structure of the city, and the softer cultural signals people usually use to talk about class without saying class.

With thirty-odd correlated variables, the first task was compression. This is where Principal Component Analysis comes in. Highly educated areas tend to have higher house prices, more managers, more coffee shops, PCA notices that and summarises it. The result is one clean score per neighbourhood: essentially a poshness gradient. Once every neighbourhood has a poshness score, K-Means asks a deceptively simple question: if you had to split London into exactly two groups, where would the cut go? The result is a binary label for every neighbourhood. No geography was used. The final step was validation. A gradient boosting classifier which is basically an ensemble of shallow decision trees, each one learning only from the mistakes of the last, was trained to predict which side of the divide an LSOA falls on. It achieved 95% accuracy.

The point was to let the algorithm tell me where the line was, not to pick a line and ask the data to flatter my intuition. If the model could reliably classify areas as above or below the divide, a real boundary existed. If it couldn’t, the divide was a story we were telling ourselves.

The model found a line. Unfortunately.

Broadly, the line does what you’d expect. West is more affluent, east more working class, with the Thames doing its usual work as both river and sorting device. It catches parts of outer west London that the usual west-equals-wealthy story ignores. It puts parts of inner east London on the prosperous side earlier than the gentrification discourse usually starts paying attention.

Then I looked at what the model had actually used to draw it.

Beyond the usual suspects

I looked at what the model had actually used to draw the line. The café variables I'd spent considerable time scraping and quietly feeling proud of such as pub density, restaurant reviews, average menu price, were sitting near the floor. The model did not care about the flat white.

Then I looked at the top.

And there it was. Median house price: dominant to the point of embarrassment. Of course it was. House prices are where cities compress decades of decisions into a single number. They absorb transport access, school quality, planning restrictions, public investment, demographic sorting, speculative belief and inherited advantage.

But that’s exactly why house price is analytically useless as an explanation. It is an outcome masquerading as a cause. Asking the model to explain London’s divide using house prices is like asking why some people are rich and being told: because they have more money. Technically true. Analytically useless.

So I stripped them out.

What emerged in the house-price-free version was that now no qualifications jumps to the top. Ethnicity follows. The other factors that house prices were summarising all along become the lead driver the moment you stop letting the property market absorb the signal.

I checked this with SHAP values (my favourite method from cooperative game theory), which confirmed that the pattern was not being driven by a few outlier neighbourhoods.

The line wasn’t natural. It was built.

The model found the line. What draws it: house prices first, then credential sorting. That’s the model’s answer. But a model that tells you where the line is, and even what marks it, still hasn’t told you why it’s there.

Three decisions in particular turned that old geography into something durable. It all started with the wind though. London’s wealthy merchants did not choose the west out of aesthetic preference. The prevailing westerlies carried smoke from the City eastward, so they built upwind. That is not a metaphor. It is literally where the wind blew. And it created a geography that 400 years of policy have since worked very hard to preserve. That history is less elegant than a wind direction. It is also more hopeful, because policy can be changed.

The 1947 green belt froze London’s spatial structure just as postwar migration was reshaping who needed housing and where. Right to Buy, from 1980 onward, transferred council housing wealth to its beneficiaries, but the sold stock was never replaced. Then, in 1981, the London Docklands Development Corporation stripped democratic planning oversight from a large section of the East End and handed it to private developers. The DLR was built for Canary Wharf. The Jubilee line extension was built for Canary Wharf. Tower Hamlets, which surrounds Canary Wharf on three sides, was left with some of the highest child poverty rates in the country.

The map above shows what happened next. The working-class share of London’s population fell almost everywhere between 2001 and 2021: down 5.5 percentage points inside the Latte Line, and 8.6 outside it. But the revealing part is not who left. It is who replaced them. Inside the line, departing working-class residents were replaced by professionals; investment followed, schools improved, and the streets got the coffee shops. Outside it, the working class shrank just as fast, and fastest of all in neighbourhoods 2 to 4 kilometres from the Canary Wharf towers. Those areas absorbed the displacement pressure of a financial sector that needed nearby land and nearby labour, without receiving much of the investment that sector generated.

That is the uncomfortable corollary of blaming policy: the divide is not natural, inevitable, or the residue of some neutral market process. It was built. Which is, depending on your disposition, either the most depressing or the most encouraging thing about it.

Send the invoice

Three decisions helped build the divide wall. Three decisions, or their reversal, could begin to dismantle it.

First, undo what the green belt locked in. The 1947 Act froze a geography of affordability that hardened into a geography of deprivation. The answer is not to abolish the green belt, but to build aggressively within and around the boroughs it stranded. Social housing in Newham, Barking, Tower Hamlets and Southwark should be treated as infrastructure, not as a residual category for people the private market does not want.

Second, replace what Right to Buy sold and never replaced. The policy transferred housing wealth to its beneficiaries, but the stock itself was allowed to disappear. The result is an ever-smaller supply of social housing concentrated in exactly the places private capital has been least willing to go. One-for-one replacement - legislated, funded, enforced - is the minimum.

Third, send Canary Wharf the bill. The Docklands experiment was allowed to bypass democratic planning on the promise that its gains would spill outward. The data suggests something closer to the opposite: surrounding neighbourhoods absorbed the displacement without sharing in the uplift. In the areas just beyond the towers, the working-class share fell by 8.6 percentage points in twenty years. That figure should be in front of a Select Committee. A retrospective community benefit levy on Canary Wharf landholders, ringfenced for Tower Hamlets and inner Newham, would be politically controversial. It would also be one of the most analytically defensible things City Hall could do.

Which side are you on?

I can now place exactly where Terry, Steve, Brenda, and Olivia from the cafés I visited stand. I'll probably never see them again though (although I’m lowkey curious if Steve is still seeing that girl).

Enter your postcode at laurenleek.eu/latte_line_map and it will tell you which side your coffee place sits on, how it scores, and what variables are doing most of the work.

Despite my urge to try a coffeeshop in every London neighbourhood, I’m leaving London in a few days. We are cycling parts of Australia's east coast, starting in Sydney, where I intend to test the latte line empirically, from both sides. (I've been warned the flat whites can beat London. I'll report back.)

But Sydney isn't just a coffee benchmark. It named its divide. Naming it like Sydney is all this piece is trying to do for London. Not solve it. Not flatten its complexity into one model. Not pretend a gradient boosting classifier can explain everything that Charles Booth, planning law, migration, austerity, and capital have conspired to produce over four centuries.

Just name it. So we have to argue about it.

One last thing: I’ve just turned on paid subscriptions. Everything stays the same though. I’m still aiming to publish at least once a month, but if you’ve enjoyed this kind of work and want to help keep it going, consider subscribing or buying me a coffee. It means more maps like this one, more of the datasets behind them, and more of the ideas I’m genuinely excited to build next.

Share

So I did what every little sister would do: instead of finally getting my payback for two decades of him beating me at every sport imaginable (…he was just taller), I opened VS Code. One evening later, I have 49,840 pubs in a dataframe, an NLP classifier on pub names, a pub vulnerability model, and an interactive tool that lets you check whether your own local looks structurally at risk.

His requested pub crawl - my original goal - is in there somewhere. But first: what I found isn’t really a story about beer. It’s a story about what happens to a country when the places where people belong become assets on someone else’s balance sheet.

Why I Care About This (despite being a morning person)

I’ve lived in four countries in the last few years. I am not, by temperament, a pub person. But some of the most at home I’ve ever felt has been in exactly the kind of places this piece is about.

In Brooklyn (not New York), a tiny town north of Sydney on the Hawkesbury River, the kind of place with one pub and one opinion about outsiders, we entered the quiz as "The Wookies" (yes, that's a Star Wars reference) and somehow, against all odds, narrowly beat the local elderly ladies from Mooney Mooney. When we passed our prize back to them, something shifted. They started talking and suddenly we weren't tourists. We felt at home.

Or in Tuscany, just after I’d moved and didn’t speak a word of Italian, I was cycling behind a local club ride and held the wheel for 40km, eventually pulling on the front for them. After the ride, with some very Italian hand gestures, they invited me to a bar in a tiny village on the piazza. Over a vino rosso we exchanged a few broken sentences about the best routes. I felt at home.

Neither moment was planned. Neither required an app, a membership, or a booking. They happened because a place existed - open, informal, cheap enough that the price of entry was just showing up.

Ray Oldenburg called these spaces third places: not home, not work, but the pubs, cafés, and piazzas where community happens without anyone organising it. Robert Putnam, in Bowling Alone, measured what happens when they vanish: trust collapses, civic life withers, democracy frays. No places, no trust. No trust, no society. But neither asked the question that matters most for Britain in 2026: who is doing the dismantling of these third spaces, and why?

The Decline (and why the “people stopped going” story is too easy)

First, let’s put some more numbers to this. I pulled the UK business register snapshots for SIC 56302 (public houses and bars), from 2009 to 2022. Britain went from ~54,000 registered pubs to under 40,000. Over 14,000 gone. A quarter of the total. The trend is relentlessly downward.

(The ONS methodology captures businesses registered for VAT and/or PAYE. Very small pubs below the VAT threshold, the kind of micro-local that matters most, may be undercounted. In other words, the real number is probably worse.)

The timeline is a political economy in miniature: the beer duty escalator (scrapped 2013 after years of above-inflation hikes), the 2017 business rates revaluation, COVID, the 2022 energy crisis. But the point is: the decline predates the shocks. This isn’t one bad year, it’s structural.

Every region is losing. The North East, North West, Yorkshire, and the Midlands fastest - 25-30% of their stock. London least. The correlation between regional pub decline and the 2016 Leave vote is r = 0.67. I won’t claim causation with twelve data points, but the comfortable reading is Putnam's passive erosion of social capital. I’ll soon come to the harder reading of this.

The Brooklyn pub quiz happened because the pub existed. The cycling club aperitivo happened because the village bar existed. What happens in a town where neither does?

The number that changed how I see this

So I took all surviving pubs and every pub that has closed since 2016, and asked a simple question: what's structurally different about the ones that died?

I trained a random forest model (an algorithm that builds thousands of decision trees and asks each one what matters most) on several structural characteristics of every pub in the dataset to predict which pubs closed between 2016 and 2024. Then I decomposed the feature importances to see what mattered most. The overwhelming winner was spatial isolation: how far a pub stood from its nearest neighbour and how many other pubs existed within walking distance. The model was telling me something simple. Pubs don't die because of what's inside them. They die because of what's around them.

How exactly did I measure isolation? I calculated the distance to each pub’s nearest neighbour using a haversine BallTree. Translation: straight-line distance over the Earth’s surface and not walking time. (And yes, I used a BallTree instead of a KDTree. The curvature of the Earth matters when your dataset spans from Cornwall to Caithness. I will not be taking questions.) Then I did the same for every pub that has closed since 2016, using archived snapshots of the same dataset.

The pattern is as follows: median nearest-neighbour distance for a surviving pub: ~280 metres. For a pub that closed: ~640 metres. Simply said: pubs that were isolated died while pubs in clusters survived. The red dots on the map are the next to go. They are disproportionately rural, disproportionately northern, disproportionately in the same communities that have already lost the most.

And it’s not just correlation in a static picture. It’s dynamic. Every pub that closes increases the isolation of the remaining pubs, pushing more of them into the danger zone. This is the spatial death spiral.

Who owns your local?

As promised, the harder reading of the decline of pubs. Brett Christophers (Rentier Capitalism) and Grace Blakeley (Stolen) can offer an answer. The UK economy increasingly extracts value from assets such as land, housing, infrastructure, rather than producing it. The pub is a perfect case study: it sits on land, it has a freehold, it generates modest cash flow. For a private equity firm, that’s not a community asset, it’s a leveraged buyout opportunity. Acquire, load with debt, extract, underinvest, and when the pub becomes unviable, sell the land to developers at a multiple.

We have already seen that isolated pubs die. But pub isolation isn’t a coincidence, someone creates it. When a pubco owns thirty pubs in a region and decides to cut the five least profitable, the remaining twenty-five don’t just lose neighbours. They lose the clustering effect that was keeping them alive. In other words, the just identified spatial death spiral has an author.

The regions that lost most pubs are the regions where ownership was most consolidated under leveraged pubco models. When Ei Group raised rents to service its debt, it was tenants in Sunderland and Burnley who couldn’t pay, not tenants in my dear Islington.

Britain’s largest pub company is called Stonegate: roughly one in eleven falls under this. You know the brands: Slug & Lettuce, Yates’s, Walkabout, all owned by TDR Capital, a private equity firm that also owns Asda. Its corporate structure runs through the Cayman Islands.

Stonegate carries over £3 billion in debt, mostly from its leveraged acquisition of rival Ei Group in 2019, completed months before the pandemic shut every pub in Britain. In 2024: £455 million in interest payments, a £214 million loss, and a citation from the Department for Business for failing to pay minimum wage to 3,650 workers.

It’s not alone. Greene King (~2,700 pubs) was taken private by Li Ka-shing’s Hong Kong property empire for £4.6 billion. Punch Pubs (~1,300 pubs) was bought by Fortress Investment Group, a US private equity firm formerly owned by SoftBank; now majority-owned by Mubadala (Abu Dhabi). Between them, PE-backed and overseas-owned companies control roughly a quarter to a third of every pub in Britain.

So here’s the political economy of pub closures. It is not: people stopped going. It is: pubs became collateral in leveraged buyouts, debt costs were passed down as higher rents and lower investment, and the pubs that couldn’t sustain the extraction closed, while the ones that could were reshaped into higher-margin branded concepts serving a wealthier clientele.

The Bricklayers Arms didn’t die of natural causes. It was killed by a balance sheet in George Town. And if you think London escaped this, look closer…

London’s illusion (and what pub names reveal)

London looks like the success story. Smallest decline nationally. If you live in Hackney or Peckham, you might think the pub is thriving: new taproom on every corner, natural wine bar where the laundrette used to be.

London didn't lose its pubs. It swapped them. And this is where the spatial story meets the ownership story. In Burnley, private equity closes a pub and nothing replaces it since the land isn't worth enough. In London, the land is worth plenty, so the site reopens: refitted, rebranded, repriced. The count stays stable. But the name changes, and so does who walks through the door. This is why the random forest flagged spatial isolation as the dominant predictor: it's not geography acting alone, it's the financial logic of extraction expressing itself spatially. PE firms don't close pubs at random. They close the ones that sit on cheap land with thin margins and replace the ones that sit on expensive land with higher-margin concepts. The map of pub death is a map of where extraction was profitable and reinvestment wasn't.

I used the pub names to expose it. I classified every surviving London pub as either traditional: such as Red Lion, Kings Arms, Bricklayers Arms, names encoding centuries of heraldry, trades, and coaching roads or as modern, such as The Yard, The Social, Tap Room, names that strip away history for something brandable. Then I mapped them.

Traditional pubs scatter across all of London: inner, outer, rich, poor. Modern pubs cluster in zones 1-2 and the gentrification corridor from Hackney through Peckham. London's count looks stable because craft taprooms in gentrifying postcodes replace closures in outer boroughs. But a £7.50 hazy IPA in Dalston is not the same third place as a £2.50 pint in a working men's club in Barking.

I didn’t want to rely on my own keyword list, so I ran a sanity check using machine learning. (ML people: TF-IDF on character n-grams of every London pub name, K-means clustering projected via PCA. I know, TF-IDF on character n-grams is not state of the art. But when your unsupervised model rediscovers the same class structure that took historians centuries to document, you don't need BERT/ fancy fine-tuned LLM). Simply said, I fed every pub name into an algorithm that breaks words into small character fragments, think of it as teaching a model to recognise the texture of a name rather than its meaning. Then I asked it to sort all names into groups, with no information about history, geography, or what “traditional” even means.

It found the same divide I did. And when I plotted those algorithmically-discovered groups back on the map, they had geography: old-textured names in the suburbs, new-textured names in the centre. The algorithm sees what the eye sees - naming eras are a spatial signature of class.

Teal: the old Britain: heraldic, royal, rural, trades, naval. Pink: the new Britain: bars, kitchens, tap rooms. The countryside is teal and the cities are turning pink. Both are valid places to get a drink. But only one is a third place in Oldenburg’s sense: a space where the full cross-section of a community meets without requiring a particular income, education, or cultural fluency to walk through the door.

Check your local

So what do you do with this information? I built a companion tool: Is Your Local At Risk?

The model behind it is deliberately simple and that's the point. For every one of Britain's pubs, I calculate two things: how far away is the nearest other pub (nearest-neighbour distance, via haversine BallTree), and how many pubs exist within a walkable radius (pub density). Those two spatial signals get combined into a continuous risk score, then bucketed into green (clustered, resilient), amber (thinning), and red (isolated, vulnerable).

That's it. No pricing data, no opening hours, no operator quality scores. Just structure and space. Because the finding from the closure analysis is stark: the single strongest predictor of whether a pub closed between 2016 and 2024 is how alone it was. Not how good the beer was or who owned it. But whether it had neighbours.

Enter your postcode and see every pub within walking distance, scored green/amber/red by the vulnerability model. The risk scores, not coincidentally, map onto the worst loneliness scores in the ONS wellbeing data.

The tool won't tell you btw whether your local does a decent Sunday roast (I’m working on that model, it just requires some more field research). But it will tell you whether the spatial conditions around it look like the conditions that preceded every closure in the dataset.

Fighting back

The fatalist reading of this all is that: pubs close because economics, because supermarket alcohol, because young people don’t drink as much (pls don’t blame Gen Z for this one, we inherited the wreckage). These are real factors. But they’re the cover story for a series of reversible policy choices. Britain chose to tax a pint in a pub more heavily than wine in a restaurant. Chose permitted development rights that let landlords flip pubs into flats without planning permission. Chose to cut council budgets by 40%, hollowing out the only institution that could have said no.

So here’s my policy brief. I’ll keep it short.

Map it. Build a national Third Place Index. Every neighbourhood scored by density, diversity, and walkability of its social infrastructure. I built a version with free data and a laptop. The ONS has better resources than I do. Presumably.

Protect it. Use the vulnerability model to fast-track the 500 most isolated pubs for Asset of Community Value status (a legal designation that gives communities 6 months to bid when a pub goes up for sale). Require planning permission for any pub conversion, not just where communities have the lawyers and the social capital to fight. The current system protects pubs in Richmond and lets them die in Rochdale.

Fund it. The government spends serious money treating loneliness downstream: social prescribing, befriending services, community connectors. Good programmes. But it’s like funding ambulances while defunding the road barriers. Redirect 10% toward keeping existing third places open. A pub costs less to save than a community centre costs to build, and it comes pre-loaded with something no government programme can manufacture: centuries of accumulated meaning.

Measure it. Add third place access to the Index of Multiple Deprivation (IMD). What gets measured gets managed. Right now a neighbourhood can score perfectly on the IMD while having zero places for its residents to sit together.

Tax it. If TDR Capital wants to own 4,500 British community assets from the Cayman Islands, it can fund the mechanism for communities to buy them back. A levy on offshore-held pub freeholds, ringfenced for community ownership.

Because the counter-model already exists. Community-owned pubs have grown 63% in five years. The model is simple: a community raises shares to buy the freehold. Ownership stays local. Profits get reinvested. No absentee landlord, no debt servicing, no Cayman Islands. The Ivy House in Nunhead became the first pub in Britain listed as an Asset of Community Value, saved from developers by locals who crowdfunded the purchase. It’s still open. The people who own it are the same people who drink in it.

So, the Asset of Community Value framework exists and the co-op model works. What’s missing is money and a government willing to ask where it should come from.

The pub crawl

Oh right. My brother.

After all that, here’s what I’ve actually built him: a pub crawl through London that traces the naming eras. I start at a medieval heraldic pub in the City (there are still a few that have served beer since before the Great Fire). Walk east to a Victorian trades pub in Whitechapel - a Bricklayers or a Carpenters. Cross into Hackney for a modern craft spot to see what’s replaced the old guard. South to a coaching inn in Borough. End at a proper South London local that hasn’t changed its name or its prices since Thatcher.

Five pubs. Five centuries. One brother who doesn't know he's about to get a lecture on the financialisation of community infrastructure with his fifth pint. I’ll let you know if he ever forgives me.

Oh, and next month I'm cycling through Australia for six weeks. I want to see whether a country that built its third places differently has built something more durable than what England is losing. Australia's social infrastructure runs on not-for-profit clubs such as RSLs, surf lifesaving clubs, bowls clubs and leagues clubs, all member-owned, community-governed, and structured so that profits stay local. It's essentially the co-op model that Britain is only now discovering, except Australia built thousands of them. The trade-off is that many are funded by poker machines (“pokies”), which is its own political economy story. But the basic architecture of community ownership, democratic governance, no Cayman Islands balance sheet, is what I've been arguing Britain needs. I want to see how it holds up in person. I’ll promise that I’ll bring the dataframe to the pub quiz.

Team name: still The Wookies.

If this was useful, share it - especially with your MP who loves a pint. Hope they love the data too. Also consider buying me a coffee to keep the project alive:)

Share

That pattern mattered. Because what looked like “Airbnb taking over” wasn’t sprawl - it was pressure pooling in very specific places.

So I did what I usually do when something feels off: I pulled the data. I scraped every listing, mapped the coordinates, and let the spatial patterns speak.

What emerged wasn’t a story about Airbnb acting as a market-maker (as I’d previously found with Google Maps). It was something quieter and more uncomfortable.

The takeaway, upfront: Airbnb didn’t overwhelm London. It concentrated in places where housing scarcity, wage pressure, and planning gaps already made short-term letting the rational response. This isn’t a defence of Airbnb though. It’s a claim about causality: regulate the interface all you want, but if you ignore the underlying housing pressure, the problem will simply reappear in another form.

The platform didn’t create the problem. It found the gaps. At the individual level, Airbnb often does increase efficiency (call me an economist, just for this take…). Spare rooms get used. Travel gets cheaper. Households patch income gaps without moving. The conflict isn’t efficiency. It’s distribution - who bears the cost when individually rational behaviour aggregates into neighbourhood-level harm.

To see this, I built a dashboard map. It answers one question: where does short-term Airbnb letting cluster? In other words, where is the pressure of Airbnb felt the most?

Not to moralise. To diagnose. Because you can’t intervene in a system you can’t see - and right now, policymakers are regulating blind. And because individual choices still matter, I added an escape hatch: a layer that surfaces listings in low housing-pressure areas. Places where Airbnb still looks like housing, not infrastructure. High ratings. Reasonable prices. Outside the tourist-trap clusters.

Call it the guilt-free filter - or, perhaps more accurately, the here can you book to “not make things worse” layer.

Here’s what the map actual does and reveals - and why it changes the question from “How do we stop Airbnb?”

The Law Regulates Time. The Problem Is Space.

Before we go further: London - like most major cities - does regulate short-term lets. This entire debate goes wrong because London regulates Airbnb as a temporal problem when its impacts are fundamentally spatial.

Since 2015, Londoners can rent out their entire home for up to 90 nights a year without planning permission. The intent was reasonable: let people monetise a flat while on holiday, ease mortgage pressure, add flexibility to the tourism market.

Beyond 90 nights, you're no longer operating a dwelling - you're running "temporary sleeping accommodation," which requires explicit borough permission. In high-pressure areas, boroughs like Camden and Westminster, and Hackney routinely refuse these applications, with rejection rates around 60-80%. And Airbnb does enforce this. Since 2017, entire-home listings auto-lock at 90 booked nights. On paper, this is tidy. Cities love tidy.

But the system is porous: it tracks bookings, not actual occupancy. It can't catch hosts listing across multiple platforms (Airbnb + Booking.com + Vrbo = 270 nights, technically legal), creating duplicate accounts, or false claims about permissions. And while Westminster has dedicated enforcement officers, most boroughs don’t.

You can have a street where every flat is listed for exactly 89 nights. Every host is compliant. And yet the street has functionally become a hotel corridor. Rolling suitcases. Keypad lockboxes. A constant churn of strangers. The law sees individual compliance; residents experience collective transformation.

This is the governance vacuum: the metric being regulated (time) isn't where the disruption is happening (space and density). It's why seeing and correcting the map matters more than counting the nights. The city is enforcing the law perfectly - it’s just enforcing the wrong law.

The Map Nobody Shows You

The public Airbnb map is fine if you want to book a weekend and ignore context. It shows availability, not impact. It tells you where listings exist, but nothing about why they concentrate - or what that concentration means for a neighbourhood.

So I built an adjusted version: the Airbnb pressure Map - showing where Airbnb pressure is high. The question this map asks is: where does that perfectly rational, legal behaviour aggregate into something that starts behaving like hotel infrastructure?

What the map does and why it matters:

• It normalises density by housing stock. Raw listing counts lie. Shoreditch with 500 listings and Barking with 500 listings are not the same problem. One is a dense, already-saturated neighbourhood; the other is a sprawling borough where 500 is merely a rounding error. The map weights listings against actual housing units so you’re comparing pressure, not pins.

• It discovers high Airbnb density clusters without assuming them. Rather than drawing boundaries by borough or branding (“creative quarter”, “vibrant hub”), the map lets the data speak. I use DBSCAN to create clusters which is non-parametric and indifferent to administrative lines. Parameters: eps = 0.003 (~330m at London’s latitude), min_samples = 5: roughly the scale at which repeated short-term lets change how a street feels to residents. Change the parameters and the labels move; the underlying pattern does not. Clusters render as convex-hull polygons with centroid bubbles sized by listing count - readable at city scale without drowning in ~80,000 individual pins. For texture, I sample up to 120 points per cluster and aggregate the rest. This all allows me to reveal the real reasons why clusters form.

• It surfaces low-pressure listings. Finally, I added an escape hatch from the dense clusters: a pressure score combining ratings, price, and distance from high-intensity clusters highlights places where Airbnb pressure is low and still looks like housing, not infrastructure. The top ~1,500 appear as a toggleable layer when you zoom in. If Airbnb is efficient, this is where that efficiency stays local - without tipping neighbourhoods into churn. Worth your money, without the guilt. As a friend of mine says: no matter where you are in London, your destination is always 45 minutes away. Might as well stay somewhere that isn’t making things worse, e.g., Tower Hamlets perhaps?

How to read it: Darker shading = more pressure. Bigger bubbles = larger clusters. Toggle “low pressure listings” and zoom in to find the escape hatch.

What The Map Actually Shows

Once you stop looking at raw listings and start looking at pressure, the story changes.

Airbnb is not "everywhere." The disruption is sharply concentrated. Most of London remains relatively stable; this is not a citywide takeover.

Hotspots don't track regulation. If the 90-day rule were meaningfully shaping behaviour, density would be lowest where enforcement is tightest: Camden, Westminster, Hackney - boroughs that are explicitly restrictive and vocal about it. The map shows clusters there anyway. It also shows clusters where enforcement is fragmented and rules exist mostly on paper. Strict or lax, the outcome is the same. Regulation isn't absent. It's aimed at the wrong thing.

Time-based rules fail in a density-driven world. London caps nights per property. What actually transforms a street is how many units operate as short-term lets at once. You can have perfect individual compliance - every host stopping at 89 nights - and still produce a hotel corridor. The law sees atoms; residents live the aggregate collective transformation emerging from behaviour that is entirely “legal” in isolation.

Airbnb is filling an economic gap; it doesn’t create one. Overlay income-rent pressure and the clusters line up with places where short-term letting works as a patch: for households servicing mortgages that wages no longer cover, and for councils who'd rather tolerate visitor spend than fund enforcement. A listing in a borough where median wages cover median rent is usually a lifestyle choice. A listing where they don't is more likely a survival strategy - someone turning their only income-scalable asset into cash. That doesn't make the platform benign, but it does make it endogenous: ban it and the pressure finds another outlet, possibly a worse one. The pressure didn't start with Airbnb. Airbnb is the shape it took. Platforms don’t have morals. They have optimisation functions.

The real problem isn't rule-breaking. It's that the rules describe a London that no longer exists.

So What Do We Do About it?

Run the story backwards and Airbnb isn’t the first mover here. The gaps already existed; the platform just figured out how to monetise them efficiently.

Very roughly, the mechanism looks like this: Stagnant wages made housing the only income-scalable asset most households have. Austerity-era budgets made councils quietly dependent on tourism spend. Broken planning and low housebuilding baked in scarcity.

Airbnb then behaves like any well-designed platform: it routes demand through the path of least resistance. If you want to respond with something more serious than “ban Airbnb,” policy has to line up with how the system actually behaves. One thing I like to always emphasize: bans are emotionally satisfying. They are not a theory of change. If you want to respond seriously:

Regulate the right dimension. If the disruption is about density and clustering, regulating nights per property is necessary but not sufficient. Instead, introduce street- or block-level caps: beyond a certain share of dwellings in a micro-area, no new short-term lets. And tie permissions to local saturation: a borough with 0.2% Airbnb share is not the same as one at 3%.

Stop flying blind on data. Right now, councils are outgunned. Platforms have real-time operational data; and regulators only have freedom of information requests (or fragments via audits, complaints). (Guess who moves faster…) Instead, require platform-level reporting of short-term let activity to a central register (this is coming at national level, but only slowly…). Also, make aggregated, anonymised data public so independent researchers (hi) can stress-test the effects. If you can’t see the problem in something with minimal lag, you can’t manage it.

Align incentives instead of pretending they don’t exist. If residents are using Airbnb as a wage patch and councils are quietly enjoying visitor spend, bans alone are not serious policy. Some options: (i) hypothecate a portion of short-term let tax/fee income to local housing funds in the same borough (hint: perhaps lower the council tax in Islington?). (ii) Offer fast-track planning or tax relief for landlords who convert short-term units back into long-term rentals in high-pressure zones. In short, don’t just fight the platform at the front door while cashing the receipts at the back.

Fix the underlying housing and wage story (yes, the hard part). None of this replaces fundamentals. Sorry. Build more genuinely affordable housing (this is the abundance agenda 101: scarcity in housing isn’t natural) and reduce reliance on housing assets as the primary middle-class safety net.

None of this is radical. What is radical is continuing to regulate a spatial problem with a temporal ruler. I’m not naïve enough to think a Substack post changes planning policy. But I am stubborn enough to think that making the gap visible is the first step and cheaper than another consultation.

Can We Extrapolate?

In short, Airbnb isn’t blanketing London. It’s pooling in the gaps - places where the housing system was already failing before a single listing went live. The platform didn’t create the problem. It just gave the problem an interface.

But is this a London story or a universal one?

Florence - my previous home - just banned all new short-term lets in the UNESCO historic centre, prohibited keyboxes entirely, and now requires in-person guest check-in. The national registration system wiped out 20-30% of listings overnight; many turned out to be offices and warehouses masquerading as apartments. Aggressive intervention, real results. But Florence has ~10,500 listings in a city of 360,000 - the problem is concentration in a tiny walkable core.

Amsterdam - the closest big city South of where I grew up - went even harder: 30 nights maximum (not 90), mandatory permits, pre-notification of every booking, and a ban on short-term lets in parts of the old town since 2019. The centre and De Pijp may drop to 15 nights from April 2026. Yet despite all this, long-term rents have still risen 37% since restrictions began. The pressure underneath - housing scarcity, capital flows, tourism-first urbanism - doesn’t care which rules you write.

Three cities. Three regulatory regimes. One underlying dynamic. If yelling at the interface worked, Amsterdam would have solved this by now. It hasn’t. The system underneath - the one that made platform income perfectly rational - is what needs redesigning. My map won’t do that. But it makes the gap visible. And once you see the gap, the question changes: not “How do we stop Airbnb?” but “What is the city no longer doing for itself - and what did platforms step in to replace?”

If you like data-driven stories that go beyond yelling at apps, do share this, subscribe and consider Buying me a coffee to keep the project alive:) If you are interested in the all the code and data of this Substack post, I’ve made it available in my Github here.

Share

It started as a very reasonable problem. I was tired of doom-scrolling Google Maps, trying to disentangle genuinely good food from whatever the algorithm had decided to push at me that day. Somewhere along the way, the project stopped being about dinner and became about something slightly more unhinged: how digital platforms quietly redistribute economic survival across cities.

Because once you start looking at London’s restaurant scene through data, you stop seeing all those cute independents and hot new openings. You start seeing an algorithmic market - one where visibility compounds, demand snowballs, and who gets to survive is increasingly decided by code.

Google Maps Is Not a Directory. It’s a Market Maker.

The public story of Google Maps is that it passively reflects “what people like.” More stars, more reviews, better food. But that framing obscures how the platform actually operates. Google Maps is not just indexing demand - it is actively organising it through a ranking system built on a small number of core signals that Google itself has publicly acknowledged: relevance, distance, and prominence.

“Relevance” is inferred from text matching between your search query and business metadata. “Distance” is purely spatial. But “prominence” is where the political economy begins. Google defines prominence using signals such as review volume, review velocity, average rating, brand recognition, and broader web visibility. In other words, it is not just what people think of a place - it is how often people interact with it, talk about it, and already recognise it.

Visibility on these ranked lists determines foot traffic. Foot traffic determines how quickly reviews accumulate. Review accumulation then feeds directly back into the prominence signal. The system compounds. Early discovery generates demand. Demand generates data. Data generates future discovery. This creates a cumulative-advantage dynamic that looks remarkably similar to the way capital compounds in financial markets. This is essentially Robert Merton’s Matthew Effect applied to kebab shops - ‘unto every one that hath shall be given.’

This disproportionately rewards chains and already-central venues. Chains benefit from cross-location brand recognition. High-footfall areas generate reviews faster, meaning venues in those zones climb the prominence ranking more quickly even at identical underlying quality. By contrast, new independents face a classic cold-start problem: without reviews they are hard to find, and without being found they struggle to accumulate reviews at all. What looks like neutral consumer choice is therefore better understood as algorithmically mediated market design.

In economics, this dynamic closely resembles the logic of a market maker: an intermediary that does not merely reflect underlying supply and demand, but actively shapes liquidity, matching, and price discovery. Platforms like Google Maps perform an analogous function for local services by controlling visibility rather than prices directly. In the language of digital economics, ranking algorithms act as attention allocators, steering demand toward some firms and away from others.

Building a Counterfactual City with Machine Learning

If Google Maps now acts as a kind of market maker for urban demand, the obvious next question is: what would the city look like without that amplification layer? In other words, how do you separate a restaurant’s intrinsic performance from the visibility effects of the platform itself?

To get at that, I built a machine-learning model - a gradient-boosted decision tree (for the ML crowd: HistGradientBoostingRegressor from scikit-learn) - to predict what a restaurant’s Google rating should be, given only its structural characteristics. This class of model is designed for large, messy, mixed-type tabular data and is particularly good at capturing interaction effects, without me having to specify those by hand. Features include how many reviews it has (log-transformed to reflect diminishing returns to attention), what cuisine it serves, whether it is part of a chain or an independent, its price level, broad venue types (restaurant, café, takeaway, bar), and where it sits in the city via a spatial grid.

Quick aside: for a subset of places I also scraped review text, languages, and photos. But for this first full-city run I stayed within the Google Maps API free tier - partly for reproducibility, partly because previous grid-scraping adventures taught me that cloud bills compound faster than review counts. So, for future versions, more features will only improve things. In particular, who is doing the reviewing matters. A five-star review of an Indian restaurant written in Hindi probably carries a different signal than one written by someone ordering chips with everything. (No judgment of white British people ofc…)

One practical problem I ran into early on is that Google Maps is surprisingly bad at categorising cuisines. A huge share of restaurants are labelled vaguely (“restaurant”, “cafe”, “meal takeaway”), inconsistently, or just incorrectly. So I ended up building a separate cuisine-classification model that predicts cuisine from restaurant names, menu language, and review text where available. In other words, the cuisine filters in the dashboard are not just Google’s tags - they’re machine-learned. This matters more than it might sound: if you misclassify cuisines, you misread diversity, clustering, and who actually competes with whom on the high street. Btw, I briefly considered classifying Pret A Manger as French, just to see if it would make any French people angrier at me than they already are. I didn’t. But I thought about it.

Before any modelling happens, all features go through a standard preprocessing pipeline - imputation, encoding, the usual. Crucially, the model is trained only to learn the mapping between observable platform-visible features and ratings. This allows me to generate a counterfactual expected rating for each restaurant - what the platform would typically assign under those structural conditions. The difference between a restaurant’s real rating and this predicted rating is what I call the rating residual. A positive residual means the restaurant performs materially better than its platform baseline would suggest. A negative residual means it underperforms relative to what the algorithm normally rewards. This is not a perfect measure of food quality - but it is a powerful measure of algorithmic mispricing: where social or culinary value diverges from what the platform structurally amplifies.

One caveat: some restaurants pay for promoted pins or local-search ads. Because paid visibility isn’t publicly disclosed, I can’t estimate how many - which is itself a sign of how opaque platform influence has become. My residuals may partly reflect ad spend I can’t observe.

Introducing: The London Food Dashboard

To summarise this, I built the London food dashboard. The dashboard currently allows users to search by name and filter by underrated gems (identified by my machine learning algorithm), cuisine, borough, price level, min rating, and review volume. It is still very much a version-one prototype - but it is already a working microscope into London’s algorithmic food economy.

If you want to explore it yourself, you can find it on my personal website at: laurenleek.eu/food-map .

Naturally, I immediately stress-tested it on my favourite part of London: Islington (maybe all this promo - also in my previous Substack on UK segregation - makes me qualify for a council tax rebate? - I’m looking at you Jeremy Corbyn…). I switched on my “underrated gems” filter - that’s the ML residual at work - set a minimum rating and review count, exclude the eye-wateringly expensive options, and let the bubbles guide me. Bigger, darker bubbles mean places my model thinks the algorithm is undervaluing.

And just like that, I had dinner plans. Do try it yourself.

Btw, this is very much still a beta version - which means bugs, blind spots, and lots of room to grow. If something looks odd, missing, or wrong, please leave feature ideas and suggestions in the comments or via the comments on my website. Unlike the VS Code GitHub tracker and its 13.8k open issues, I really do read them.

From Individual Restaurants to Algorithmic Neighbourhoods

But restaurants don’t fail alone - they fail in ecosystems. I also wanted to understand what happens when platform dynamics scale up from restaurants to entire neighbourhood food ecosystems. So I added a second modelling layer.

First, I aggregate restaurants into small spatial cells (the hexagons you see on the maps - because squares are for people who haven’t thought hard enough about edge effects) and compute summary features for each area: restaurant density, mean rating, mean residual, total reviews, chain share, cuisine entropy, and price level. I then standardise these and run principal component analysis (PCA) to compress them into a single continuous hub score that captures overall “restaurant ecosystem strength” in one dimension. Finally, I apply K-means clustering to the same feature space to classify areas into four structural types: elite, strong, everyday, and weak hubs.

At first glance, the patterns look comfortingly familiar. Central London dominates. Of course it does. But what matters is not just where the hubs are - it’s what kind of hubs they are. Using the full hub score rather than raw ratings alone, I identify the five most structurally powerful restaurant hubs in London. They are the places where density, algorithmic attention, independent survival, and consumer spending power all line up at once. They are labeled on the maps. I am deliberately refusing to rank them loudly in prose in order to avoid starting neighbourhood wars at scale (and to not disappoint Islington) - but the visual story is already extremely clear.

Overlaying this with the cuisine density panels reveals something even sharper. London’s culinary diversity is not evenly distributed across its platform economy. Migrant cuisines cluster strongly in parts of the city where algorithmic visibility is structurally weaker. Italian, Indian, Turkish, Chinese, Thai, British, Japanese, French, American, and fish-and-chips all trace distinct settlement histories, labour networks, retail formats, and relationships to capital and rent. Some cuisines form long, contiguous corridors. Others appear as punctuated clusters tied to specific high streets or income brackets.

Cuisine diversity, in other words, is not just about taste. It is about where families settled, which high streets remained affordable long enough for a second generation to open businesses, and which parts of the city experienced displacement before culinary ecosystems could mature. (If this part especially speaks to you, I go much deeper into it in Food for thought: local restaurant diversity meets migration).

The Take-Away and Some Unwanted Policy Advice

This project started as a search problem and ended as something more. The most important result isn’t which neighbourhood tops the rankings - it’s the realisation that platforms now quietly structure survival in everyday urban markets. London’s restaurant scene is no longer organised by taste alone. It is organised by visibility that compounds, rent that rises when discovery arrives, and algorithms that allocate attention long before consumers ever show up. What looks like “choice” is increasingly the downstream effect of ranking systems.

For policy, that shifts the frame. If discovery now shapes small-business survival, then competition, fairness, and urban regeneration can no longer ignore platform ranking systems. Councils can rebuild streets and liberalise licensing all they like - but algorithmic invisibility can still leave places economically stranded. Platform transparency and auditability are no longer niche tech debates; they are quietly becoming tools of local economic policy. At minimum, ranking algorithms with this much economic consequence should be auditable. We audit financial markets. We should audit attention markets too.

For a navigation app, Google maps has a remarkable amount of power.
Just saying.

I’m also working on other maps (including a map of the best cycling and running routes with excellent cafés along the way, because I have needs). More broadly, I’m investing more and more time into building higher-quality public data projects. If you have an idea you’d like to see built - pitch it to me. And if you enjoy this kind of work, you can always Buy Me A Coffee or subscribe to help fund the next round of over-engineered maps.

Share

Anyway. Time to discuss a topic that absolutely hasn’t been talked about enough recently especially in finance… oh wait.

Since late 2022, one word has reorganised half of finance, a good chunk of global industrial policy, and pretty much every podcast I listen to: AI. Markets now rerate companies based on GPU access, VC throws money at anything labelled “AI-native,” and hyperscalers are spending like they’ve been mobilised for war. At this point, a firm only has to whisper “AI” on an earnings call and the stock jumps upward (someone please run this event study - otherwise, I will).

So here’s the question that even people with zero money in markets keep asking: Is this an AI boom - or a bubble waiting to pop?

And, of course, the ceremonial disclaimer: nothing here is investment advice (or any other kind of advice).

A Simple Toolkit: Three Questions to Diagnose Booms vs Bubbles

Booms and bubbles aren’t opposites; they often ride together. Transformative technologies tend to arrive with real investment and speculative excess. The trick is parsing one from the other.

A simple diagnostic - pieced together from my old notes of Bresnahan–Trajtenberg, Jovanovic on general-purpose technologies, Minsky on credit cycles, and Shiller on narratives - comes down to three questions:

(1) Are we building things?

In a boom real investment accelerates (data centers, chips, model training, hiring), you can see it in capex (capital expenditures - which is the money a company spends to acquire, build, upgrade, or maintain long-term physical assets).

(2) Are the numbers catching up?

In a boom revenues, adoption and productivity gradually catch up to expectations while in a bubble they are detached from these same fundamentals.

(3) Is adoption spreading?

In a boom we see diffusion of general purpose technologies across sectors from services to manufacturing, government and SME’s. In a bubble the gains remain concentrated in a few firms or niches - most of the economy experiences no change.

Btw, in theory the closest approximation would be to ask whether AI valuations are “rational” by backing out the implied future productivity gains from current market caps. In practice, doing that cleanly requires a full structural model of diffusion, rents, depreciation, competition, and adoption, i.e., an entire PhD dissertation just to justify Nvidia’s P/E. The simpler takeaway is enough for now: if valuations imply growth that reality can’t plausibly deliver, then expectations have run ahead of fundamentals.

So, using the simple diagnostics, let’s run the test.

THE BOOM CASE: Signs AI Is the Real Thing

(1) Are we building things? Yes - and at extraordinary scale.

Data-centre capex has gone vertical: Global data-centre spending has roughly doubled in three years. Hyperscalers (Microsoft, Google, Amazon) account for the bulk of this, with each planning tens of billions in annual AI-related capex. This is visible investment we can actually measure, unlike the internal AI adoption happening inside firms that remains largely invisible in economic statistics.

Chip and semiconductor investment is following the same pattern: Foundry expansions (TSMC, Samsung, Intel), GPU production, and advanced packaging facilities all show multi-year investment commitments measured in the hundreds of billions. Nvidia alone is on track for multiple trillion-dollar valuation because its chips are treated as a core constraint on global compute capacity.

Electricity and grid upgrades are now part of the story: AI training and data-centre cooling are pushing local grids to their limits in the US, Europe, and parts of Asia. Utilities are reporting multi-year demand commitments from cloud providers - a hallmark of real-economy pull-through rather than hype.

(2) Are the numbers catching up? Not yet - but investment is visible in GDP.

There are early signs:

• S&P Global estimates the data-centre buildout boosted US GDP by 0.5 percentage points in Q2 2025.

• IMF and OECD scenarios show meaningful potential TFP gains if adoption spreads.

But the macro picture? Not yet. Only patchy productivity gains so far; many firms remain stuck in “pilot purgatory.” So, the numbers aren’t fully catching up yet - instead it is: capex now, productivity later.

(3) Is adoption spreading? Yes - but unevenly.

Adoption is spreading, but unevenly. In the US, abundant capital has enabled a hyperscaler-led supply boom. Europe shows strong interest but slower deployment, held back by regulatory friction and more cautious capital mobilisation. China is advancing through cost breakthroughs like DeepSeek and sheer scale, albeit under different structural constraints. Much of the rest of the world faces compute scarcity, cloud dependence, and late adoption.

THE BUBBLE CASE: Where the Froth Has Clearly Formed

(1) Are we building things? Yes - but money is moving even faster

The real economy is building data centres and fabs - but the financial economy is piling into AI even faster. Venture capital is unusually concentrated, and public markets are rewarding “AI exposure” as much as actual earnings.

We can see that through:

• VC share: AI now takes roughly half of global venture capital - an extreme allocation outside peak hype cycles.

• Nvidia: A 4× market-cap jump from 2023 to 2025 and the first firm to hit $4tn market cap (July 9, 2025)

• S&P 500 concentration: AI-linked megacaps now make up ~44% of index weight.

(2) Are the numbers catching up? Not remotely

Markets are already pricing future monopoly rents and multi-decade AI leadership. Meanwhile:

• Productivity gains are not yet visible.

• Earnings growth lags far behind valuation growth - though measuring AI’s true productivity contribution remains notoriously difficult since most AI work happens inside organizations, not in traded outputs.

• The DeepSeek event showed how fragile the narrative is: one cheap competitor and the whole sector re-priced instantly.

This is where bubbles tend to form: in the timing gap between investment and observable performance.

(3) Is adoption spreading? Yes - but too slowly for these valuations

Adoption is happening, but unevenly: US hyperscalers are sprinting. Europe is slower, more regulated, more fragmented. China is innovating cheaply but within a walled garden. Everyone else is compute-constrained.

And organisational adoption inside firms is even slower: many are stuck in pilot purgatory. Capital scales instantly; workflows, regulation, and managerial change do not. That’s a structural warning sign: valuations are global; adoption is local.

The Scorecard

So, I let my favourite LLM summarise my entire analysis into one visual:

Take-away? This doesn’t help make up my mind. So, to further quantify that gap and make a final judgement, I built my own metric: the Hype-to-Investment Ratio.

Introducing the Hype-to-Investment Ratio

A deliberately simple, but surprisingly informative, way to track where the narrative runs ahead of capital formation.

To measure the mismatch between AI “attention” and physical AI investment, I constructed a macro-narrative indicator designed to move when discourse outruns deployment. Like most AI metrics, it’s built from proxies (Google Trends, hyperscaler capex) because we lack direct measures of ‘AI capital services’ or compute utilisation.

1. Attention/hype Index

For attention, I used Google Trends for “Artificial Intelligence” (worldwide, 2018–2025). It’s a high-frequency proxy that moves with media noise, investor chatter, policy debates, and earnings-call buzzwords. Annual averages give a neat, unit-free baseline; I index it to 2018.

2. Investment Index

For the investment side, I pulled annual capex from the three firms building the entire modern AI stack: Microsoft, Google, Amazon and Meta. Their SEC-filed cashflow statements (via Yahoo Finance) capture the bulk of global GPU procurement, data-centre buildout, and cloud infrastructure. Capex is recorded as a negative cashflow, so I flipped the sign and indexed the aggregate to 2018.

3. The HIR Formula

The Hype-to-Investment Ratio is then defined as:

Conceptually:

• HIR > 100 → hype rising faster than physical AI investment

• HIR < 100 → investment rising faster than attention

• HIR falling → investment is catching up

• HIR spiking → narrative ahead of fundamentals (bubble dynamics)

Let’s look at the results:

A few notable patterns stand out:

1. 2023: HIR spikes exactly when GPT-4 drops. Classic narrative shock: attention surges, investment lags.

2. 2024: HIR falls sharply as hyperscaler capex ramps. The “build real stuff” phase finally catches up to the story.

3. 2025: with limited data, I forecast a mild further HIR decline. The DeepSeek event is shown as a prospective shock, because cheaper training could push attention up or capex expectations down - both of which would push HIR in opposite directions.

The underlying indices make the dynamic clear: attention accelerates early, but capex overtakes around 2024, and the two diverge again only if shocks (technical or narrative) perturb the system.

Btw, one could, in principle, build a proper “AI Capital Services” measure using GPU-hours, model-parameters-trained, cooling loads and a shadow price of compute. I’m mentioning this mainly so someone else does it.

A Note on Nvidia or why we should slightly downgrade investment

Before anyone gets too comfortable, let’s talk about Nvidia. Nvidia alone could justify an entire blog post on bubble mechanics. I haven’t focused on individual-firm valuations here, but you simply cannot understand the full AI cycle without grappling with the most valuable chipmaker in history.

Nvidia’s dominance is real - CUDA, ecosystem lock-in, and first-mover software tooling - but the pricing of that dominance increasingly relies on assumptions that look… optimistic. GPUs are not magical scarcity goods. You can train frontier-scale models without them (Google does, on TPUs). Competitors exist and are ramping fast.

Also something awkward happened recently. Nvidia has pledged roughly $100bn in financing to OpenAI, tied to OpenAI’s commitment to buy Nvidia chips. In other words, Nvidia is - at least in part - selling chips to firms whose ability to buy chips depends on Nvidia financing them to buy chips. It’s not fraudulent; it’s not even uncommon in big tech infrastructure markets. But it is the kind of accounting geometry one normally sees in late-cycle exuberance, where growth becomes self-referential. A company can look like it’s expanding because it is, in effect, paying its own customers to be customers.

This doesn’t invalidate the AI boom. It simply adds an important caveat: not all “demand” is organic, and not all capex is a clean signal of fundamental adoption.

So what will make or break this cycle?

Whether this becomes a durable boom or a painful repricing depends on five constraints:

1. Revenue growth vs capex growth

If earnings converge to the investment curve, the boom is validated. If not, valuations reprice hard.

2. Broad-based productivity

The single most important macro question: does AI show up in productivity - especially in services where most people work?

You can actually get a back-of-the-envelope sense of how heroic current expectations are with a toy valuation exercise. Imagine an “AI megacap basket” trading on a P/E of 40, versus a boring big-tech benchmark on a P/E of 20. Plugging this into the textbook Gordon formula with a discount rate of 8% says the market is effectively assuming long-run earnings growth of about 5.5% a year for the AI basket, versus roughly 3% for the baseline. In other words, today’s prices only make sense if AI-exposed profits grow around 2½ percentage points faster every year, indefinitely. After a decade that implies earnings about 30% higher than a non-AI benchmark – and that’s before you ask whether the rest of the economy can absorb, pay for, and productively use that much AI. It’s not a full model, but it gives you a feel for how much future productivity the current narrative is quietly baking in.

3. Organisational adoption

AI’s payoff requires firms to rewire workflows, processes, and management structures.
That’s slow. If this stalls, the boom loses its downstream engine.

4. Energy + grid constraints

If power infrastructure can’t keep pace, the boom hits a physical ceiling. If the grid adapts, the cycle extends for years.

5. The next cost shock

Another DeepSeek-style breakthrough (cheaper, faster, open) could reset valuations instantly - especially if monopoly-rent assumptions prove fragile. However, so far what Blanchard on Twitter mentioned about DeepSeek being “probably the largest positive tfp shock in the history of the world” seems a bit exaggerated now.

My final verdict (for now)

If you force me to collapse all the info I have to date - the capex curves, valuation heat, adoption frictions, and my own slightly unhinged Hype-to-Investment Ratio into a single call, it’s this:

AI today is neither a speculative mirage nor a stable general-purpose revolution. It is a boom running slightly ahead of its own fundamentals.

The physical economy is delivering; the financial economy is extrapolating. The gap between the two is narrowing, not widening, which matters far more than whether markets occasionally overshoot.

If that pace holds, the cycle resolves into a sustained expansion.
If it slips, the correction will be sharp, but the infrastructure base will remain.

Share

Btw, in case you are still feeling too confident by now: the entire AI boom or bubble still ultimately rests on electricity, Linux, and whoever maintains libcurl this week.

For decades, the European Central Bank has been treated like the Vatican of central banking: no votes, no politics, no peeking behind the curtain. Unlike the Fed or the Bank of England, it won’t even tell you who voted for what.

So you’d think political influence couldn’t possibly get in. I thought so too… until my PhD traineeship on the 34th floor in Frankfurt - four floors below the Executive Board. Close enough to share the lift with them, nowhere near close enough to know what they were actually plotting. But my suspicions were definitely raised.

This Substack is about how I tried - and gloriously failed - to prove the ECB is politically responsive in the “usual” way. So I got creative. Armed with nothing but public speeches and some in my humble (and slightly biased) opinion very cool methods (first-mover graphs, Markov chains, and time-series), I built a way to measure responsiveness inside an institution designed to hide it.

The punchline? The ECB may be independent (even too independent, if you listen to John Cochrane…), but it is definitely not alone. Its agenda is quietly shaped by the national central banks (NCBs) - by many rendered obsolete since monetary policy moved to Frankfurt in 1999. Yet, I find that they still act as intermediaries of the member states, feeding national and public priorities upward.

Let me take you along how I came to this conclusion.

The Problem: How Do You Study Influence in a Black Box?

The challenge was obvious: you can’t measure responsiveness the usual way when the institution refuses to show you any of the usual indicators. No roll calls. No dissent. Not even a hint of who pushed which idea (with the exception of some very vague meeting summaries published since 2015). So instead of hunting fingerprints that weren’t there, I turned to the only thing the ECB consistently leaves behind - its speeches - and started treating them as a conversation rather than isolated monologues.

And the surprising part? The ECB wasn’t leading the conversation - it was following it. The national central banks, stripped of monetary power in 1999, were there bringing up new topics like climate, crisis management, supervision… and the ECB started picking them up. Which I could detect thanks to my BERTopic model (built through a slightly over-engineered but undeniably cool multi-step modular pipeline) on 4,659 speeches, I could actually map that sequence. Once you see who speaks first, the whole idea of the “ECB as lone intellectual beacon” starts to be questionable.

Next, I followed the chain - who echoed whom, who shifted topics after whom - using Markov transitions. (Reviewers politely suggested I move my sliding-window, 30-day moving-average, time-indexed transition-probability Markov formulas to the appendix… so I’ll spare you too.). It sounds tedious, but once you line up timestamps and topics, the structure of NCBs driving the agenda of the ECB in the short-term practically reveals itself.

Why does this happen? Because NCBs never actually became irrelevant. Even without formal monetary powers, they’re still embedded in national politics, plugged into domestic debates, and surrounded by journalists who actually quote them. (Or if your name is Klaas Knot, you just give your opinion on every other talkshow). They gather the granular data, feel the national pressure points, and pick up on emerging issues long before Frankfurt does. In a system that performs technocratic neutrality, they’re the ones with their ears closest to the ground - and the big NCBs even have more staff for certain tasks than the ECB itself (e.g., they practically run the forecasts while the ECB only verifies, reviews and aggregates).

This is where the three-way interaction terms in my cross-sectional time-series models came in. The seemingly null results - the ones that made me question my dataset, multi-modular topic model, my sanity, and repeatedly delete the draft from Overleaf in a fit of despair - turned out to be the clue. They revealed that the ECB isn’t responsive by default. It’s responsive only under specific conditions. Namely, independence isn’t absolute; it’s contextual.

So under what circumstances does the “independent” ECB start following the pack?

• Salience. During crises (hello euro crisis my old friend), when everyone is paying attention, ignoring NCB chatter becomes costly - so we see more positive significant interactions during crisis times.

• NCB coalitions. When several national banks push the same line, it effectively becomes an internal veto point - Frankfurt falls in step. Coordinated signaling helps.

• Major events. High profile gatherings as I show is the case for Sintra - the ECB central banking conference - provide a focal coordination point to synchronise communication across the Eurosystem. These events don’t “cause” responsiveness, but they concentrate attention and align the timing of signals.

I did not stop there…

To be fair, there are more innocent explanations you could give for these patterns - and I tried very, very hard to shoot my own argument down. (Including spending a full day writing permutations and simulations in Python to prove that the Markov chain results weren’t just random noise. They aren’t. And yes, JEPP, I left this part out of the submission because I didn’t want to reveal quite how nerdy I am...)

Maybe, I thought, the ECB isn’t “following” NCBs at all - maybe they’re just talking about adjacent issues at the same time. You know, NCB says “climate,” ECB replies “monetary policy is important for climate,” and suddenly it doesn’t look like responsiveness. So I reopened my Python script and computed topic-to-topic transition matrices for every central bank. If issue linkage were driving the pattern, the Markov chains would show lots of cross-topic jumps: crisis → monetary policy, banking → supervision - rather than the clean within-topic followership I was seeing. But they didn’t.

Fine, then maybe the ECB isn’t only responding to the NCBs - maybe the NCBs are responding to the ECB. So I re-ran the models with leads instead of lags, and the whole pattern became a lot weaker.

Still unconvinced (my brand), I checked whether it was just generic convergence inside the Eurosystem. I computed text similarity within topics over time - because if everyone’s just talking past each other, “responsiveness” is basically an optical illusion. But for several topics, the agreement scores made sense. Then I tried different time windows - 7, 14, 30, 60, 90 days - and nothing changed.

At that point, I decided to stop doubting myself (for once), accept that the pattern was real, and submit the article.

Final thoughts (for now)

In the strict political-science sense, I failed. No political capture, no partisan fingerprints, no governments whispering into ECB speeches - the institution is as insulated as advertised from regular politics. My early nulls weren’t errors; they were confirmation.

But independence isn’t pure government isolation. Beneath the surface, the ECB absorbs signals from the national central banks embedded in their member states - especially when things get messy. Because the ECB can hide its votes and blur its minutes - but its speeches give everything away.

So yes, ECB independence survived. But I still found something more interesting: a conditional form of responsiveness that boosts legitimacy and draws on the informational advantages sitting in national capitals. Responsiveness isn’t a threat to independence; it’s how the Eurosystem works.

This project followed me to the ECB and was the first PhD paper idea I had but also (for the highlighted reasons) the last I finished. It taught me if nothing else, how “null” results can become a theory and a toolkit for studying institutions that work very hard not to leave fingerprints.

Development, it turns out, isn’t just about money. It’s about learning - knowing how to turn opportunities into projects, how to write proposals that actually get funded, and how to navigate the Kafkaesque maze of national calls and EU procedures. Some local governments have mastered this craft; others are still learning the language.

Walking through Florence these days, you can see this geography of capacity up close. Around the city - from San Niccolò to Novoli - plaques announce that a renovation, a playground, or a digital upgrade was “financed by Next Generation EU.” These are the physical breadcrumbs of Europe’s groundbreaking €750 billion Covid-19 recovery fund experiment - and they raise a deceptively simple question: why here, and not there?

We examine this in our new article (forthcoming in Regional Studies but see the preprint here) in which we leverage the case of Italy’s National Recovery and Resilience Plan (NRRP), part of the Next Generation EU post-covid recovery fund. We show using this case that the map of development projects isn’t random - it follows invisible lines of learning. Let’s unpack this.

Italy first: a stress test for Europe’s grand recovery plan

We use the case of Italy - not just because it’s dramatic (though it is), but because it received more funding through the Recovery and Resilience Facility than any other national plan under the Next Generation EU framework: nearly €225 billion, once national co-financing is included.

The plan, known as the National Recovery and Resilience Plan (NRRP), has dominated Italy’s public debate. In February 2021, disputes over how to manage these funds helped bring down the Conte II government - a unity cabinet meant to steer the country through the pandemic. Many believed that such an enormous budget demanded technocratic precision, so Italy turned to a familiar face: Mario Draghi, the former European Central Bank president. Since the right-wing government led by Giorgia Meloni took office in October 2022, the NRRP has become a litmus test of the new administration’s capacity to govern.

To see how this plan actually unfolds on the ground, we analyse nearly 240,000 projects across all of Italy’s 7,890 municipalities. Instead of tracking euros spent, we count projects secured - each one representing a local team that managed to design, draft, and win approval for a proposal. It’s a measure not of wealth, but of bureaucratic skill.

A log-scaled map of municipal NRRP projects makes the pattern clear: dense clusters around Rome, Milan, and Naples - but also bright pockets in Tuscany and Emilia-Romagna. Despite the government’s formal rule that 40 percent of funds go to the South, local variation is dramatic: neighbouring towns can differ by an order of magnitude.

Our argument revolves around these neighbours. Let’s zoom in a little on what we actually mean by “neighbours.” Using our favourite city - Florence (not biased cause this article was mainly written there, of course… could you imagine?) - as the prime example, we show the municipality in yellow surrounded by different 7 definitions of neighbours in blue - all of which we use. We choose the most clean version as baseline, i.e., we define neighbours as all municipalities sharing a physical border.

And then there’s the human side. We also (read: Lorenzo) talked to people - local officials, entrepreneurs, and cultural organizers - who live inside this machinery of funding calls and project deadlines. (Who knew people had stuff to say beyond filling out surveys?)

Challenging the status quo

Most discussions of regional development still assume a simple formula: money flows from the center to the periphery, and underdeveloped places catch up. The implicit model is hydraulic - pour in funds, wait for growth. But what if that’s the wrong metaphor? What if development doesn’t flow at all; it diffuses. It spreads through networks of learning, imitation, and experience. Municipalities don’t just receive projects - they learn how to make them happen. And those that have already mastered the process are best positioned to do it again.

We call this the learning model of development. We argue that places learn in two ways: spatially, from their neighbours who are already winning projects, and temporally, from their own past experience. Both forms of learning are amplified - or constrained - by local administrative capacity, especially the competence of local political and bureaucratic actors who know how to translate opportunity into action. This framework challenges the idea that uneven development is a mere product of economic structure or political favoritism. Instead, it points to a subtler inequality - the unequal ability to learn, remember, and act within the machinery of modern policy.

What we find: development indeed spreads through learning

The data confirm what our theory predicted. Across all of Italy’s 7,890 municipalities, we find strong evidence that development projects cluster in space and time - not because of chance, but because of learning.

Two learning patterns emerge clearly:

1. Spatial learning:
   Municipalities surrounded by active neighbours were much more likely to win projects.
   👉 A one-standard-deviation rise in neighbour performance = +12-16% more projects locally. This pattern holds across multiple definitions of “neighbourhood,” from shared borders to 30 kilometer radii, and even when accounting for politics, income, and population size.

2. Temporal learning:
   Past experience counts even more.
   👉 A one-standard-deviation rise in past EU Cohesion Policy projects = +40% more NRRP projects today.

Administrative capacity amplifies these effects. Municipalities with more competent, better-educated political leaders - the “political capacity” component of the Municipal Administrative Quality Index - show the steepest learning curves. Where local elites are capable, temporal learning doubles in strength; where they are not, experience is easily lost. The coefficient plot below also shows this:

But why? We dive even deeper…

To prove that municipalities genuinely learn from one another (and don’t just compete), we compared text similarity across thousands of project descriptions. The logic is simple: if you compete you would not have exactly the same info while if you cooperate and learn you will. The result was striking: proposals from neighbouring municipalities used more similar language, sometimes literally the same phrases and structure, than those farther apart. True evidence of learning.

The plot below shows this: each dot depicts the average Jaccard similarity in project descriptions. Within-neighbour pairs consistently score higher than between-neighbour pairs - strong evidence of shared templates and knowledge diffusion. (Basically: local bureaucrats plagiarize smarter than you think - and we have the similarity scores to prove it.)

However, this isn’t a story of corruption or “copy-paste laziness.” It’s collective learning. Local administrations borrow, remix, and repurpose successful examples - a deeply human process that economists rarely capture but every public servant will recognise.

And as promised, we also talked to people. The main takeaway? Development spreads through networks, not manuals. In Trento, officials described how new calls for funding “arrive through the Consortium of Trentino Municipalities” - a steady local information loop. In Lecce, a cultural cooperative relied on Puglia Creativa and other professional networks that share templates and tips across towns.

Experience builds trust too. As one respondent put it, “If you already have a track record, they know you won’t mess up - it makes evaluation easier.” That institutional memory is what allows some municipalities to keep winning projects while others never quite break in. And in Reggio Calabria, an entrepreneur summed up the limits: “If you have fewer than nine employees, your problem is invoices, not planning future investments.”

In short, the mechanisms behind the statistics are social: a quiet infrastructure of relationships and reputations that shapes who learns, who gets funded, and who’s left waiting.

What we can learn - beyond Italy

If learning drives participation, then inequality can deepen over time. The same towns that mastered EU funding in the 2010s now dominate NGEU allocations. In Florence, you can see the result in the uneven geography of construction signs. Meanwhile, municipalities with weaker administrative memory remain stuck - not for lack of need, but for lack of know-how. This path-dependence creates what you might call a bureaucratic Matthew Effect: to those that have (learning capacity), more shall be given (projects).

And this story doesn’t stop at Italy’s borders. Across Europe, governments are channeling unprecedented green and digital funds through competitive, project-based mechanisms - from Spain’s PERTEs to Germany’s climate transformation funds. Access increasingly depends on writing convincing proposals, building partnerships, and passing complex administrative checks. Yet few ask the obvious question: who actually knows how to apply?

Italy, then, is less an anomaly than a warning. If the EU wants its massive recovery and climate programmes to close gaps rather than reproduce them, it must invest not only in projects but in project-makers: training, staffing, and institutional continuity for local administrations. Because in the end, development policy doesn’t move through formulas - it moves through people who know how to make them work.

Share

So, fascination established. Understanding? Not quite. Which is why I decided to dig deeper. What do quant traders actually do all day - and could I, with my VS Code, a few Python scripts, and open data, even begin to replicate that world? Or are these people really worth the money they make? Let’s find out.

(Disclaimer: none of this is investment advice. I’m probably the last person you should ask. About 77 percent of retail day traders lose money - and I’m not betting I’d be in the lucky 23.)

What quant traders actually do

A quant’s day isn’t a string of “buy!” moments - it’s cleaning data, testing ideas, debugging code, and debating whether a weird number is alpha or just an error. They start before markets open, updating models and checking overnight risk; the rest of the day is spent running backtests, tweaking features, and hunting for small, repeatable edges that quietly add up.

Quant traders sit where math meets markets. Their job is to find patterns in chaos - inefficiencies in how prices move, how information spreads, or how investors react. Some chase microsecond price gaps; others design slower, factor-based strategies for pensions or central banks.

Before diving in, let me share a few key ideas that helped me decode this world. Beta is what you get by simply riding the market - how high the wave is you are riding. Alpha is what’s left when you strip that away - the extra edge of return that everyone is fighting for. Quants obsess over Sharpe ratios (return per unit of risk), drawdowns (how far a portfolio falls before recovering), and overfitting - when a model looks great on paper but collapses in real life. Some chase momentum (buying what’s been going up), others trade mean reversion (betting prices snap back to normal), and the boldest feed thousands of signals into machine learning models to squeeze out extra basis points of alpha.

These are the ideas I kept in mind as I built my own 3 models - starting from a simple rule any student could code to something resembling a laptop version of what a hedge fund might actually run.

Model 1: The Simple One - Momentum

The first model I built is a finance classic: buy what’s been going up, sell what’s been going down. Economists call it the momentum anomaly (Jegadeesh & Titman, 1993). The logic is simple and still underpins a lot of the professional trading industry - investors chase winners and dump losers.

I ran it on 30 big names - Apple, Microsoft, Coca-Cola, the usual suspects - from 2015 onward. Each day, it checked the past year’s returns (skipping the most recent month, when markets tend to cool off). If momentum was positive, it bought; if negative, it shorted - equal weight, no emotions, just math.

Model 2: The “Smarter” One - Pairs Trading

The second model I test is pairs trading. The idea dates back to the 1980s, when Morgan Stanley’s newly minted “StatArb” (statistical arbitrage) desk realised that some stocks moved so closely together that their prices behaved like elastic bands - stretch too far, and they’d snap back. It looks for two stocks that usually move in sync (say, Visa and Mastercard, or Coke and Pepsi), waits until one drifts too far from the other, and bets on the gap closing.

The code tests every possible pair for statistical co-movement, picks the best one, and constantly recalibrates their relationship with a rolling regression. When the price gap gets unusually wide, the model shorts the rich stock and buys the cheap one - betting on mean reversion.

Model 3: The Ambitious One - Machine Learning Alpha

By the third model, I wanted something that looked and felt closer to what real hedge funds run though still lightweight enough for my laptop. So, I built a machine learning model (a Ridge regression, to keep it fast and stable) that tries to predict tomorrow’s stock returns using patterns from recent momentum and volatility.

Here’s what’s under the hood:

• Smarter features: Instead of just looking at past returns, the model now builds multiple “momentum fingerprints” - 5-day, 21-day, and 63-day returns - plus measures of trend (moving averages) and risk-adjusted momentum (Sharpe-like metrics). In simple terms, it asks: which stocks have been winning, how steadily, and compared to how risky they were?

• Market context: Each stock’s signals are adjusted relative to the market (SPY), to separate stock-specific alpha from general market moves.

• Ranking across stocks: Each day, it ranks all stocks in the universe (about 45) by their predictive signals - long the top 25%, ignore or lightly short the bottom few.

• Mega-cap weighting: I added a realistic hedge-fund twist - boosting positions in the biggest, most liquid names like Apple and Microsoft. These mega-caps dominate the S&P 500, so overweighting them gives performance closer to how institutional money actually behaves.

• Regular refits: Every six weeks, the model retrains itself on new data - a walk-forward approach that keeps it adaptive but fast enough to run on a CPU.

Finally, the model’s portfolio runs fully long, like a traditional equity fund.

Going a bit more in-depth, here are my (naive) assumptions across all models:

• The market assumptions: I’m assuming momentum keeps working, tech dominance continues, and markets aren’t fully efficient - meaning machine learning can still find small, exploitable patterns. I model transaction costs as minimal (1e-4 per trade, or about one basis point).

• The data assumptions: I assume perfect data - no survivorship bias (every failed company still exists in my dataset), no look-ahead bias (no cheating with future information), and perfectly adjusted prices.

• Model assumptions: Linear relationships hold, features stay relevant, and the statistical properties of returns don’t shift over time (stationarity).

• The execution assumptions: Flawless execution - trades fill instantly at the closing price, positions can scale infinitely without moving the market, and mega-caps stay mega.

• The risk assumptions: No black swans, no regime shifts, and no tech crash taking down my overexposed portfolio.

So, how much “alpha” did I find?

The first chart shows the cumulative returns of my three models against the market. The simple Momentum Strategy (blue) tracks close to the major indices, proving that “buy what’s been going up” works to a certain extend. The Pairs Trading model (purple) barely moves - a flat line (its brief heyday in the 2000s has long faded under high-frequency competition). Then there’s the Machine Learning Alpha strategy (green), shooting up over 7× since 2017, beating every benchmark (even the Nasdaq).

The second chart plots risk versus return. Here, ML Alpha sits firmly in the top-right corner - high returns, high risk, and the largest Sharpe ratio bubble, meaning it delivered the best return per unit of volatility. Momentum sits closer to the middle -solid but conventional - while Pairs Trading is in the low-return, low-risk corner.

Finally, the performance breakdown: ML Alpha posted a 25.7% annual return with a Sharpe (returns per unit of risk) of 0.96, outpacing every index; Momentum followed with a 10% CAGR (smoothed annual return), while Pairs Trading slipped into mild losses. The trade-off is visible too: ML Alpha had the biggest drawdowns (how much you’d lose before the portfolio recovers).

A fair critique: Of course, comparing these strategies side by side has a big caveat: they don’t all carry the same beta - meaning, they don’t all move with the market to the same degree. A strategy that swings wildly with every market move (high beta) will naturally show higher returns in a bull market, while one that’s market-neutral (low beta) will look dull by comparison, even if it’s producing steadier, risk-adjusted gains.

In practice, quants and academics adjust for this using factor models like Fama-French. Instead of assuming there’s just one market beta, these models break returns into several systematic risk factors: size (small vs. big companies), value (cheap vs. expensive stocks), and later, profitability, investment, and momentum. What looks like “alpha” in a simple market model might just be exposure to one of these well-known factors.

What makes the quants still outperform me?

I did my best - but there are reasons the pros still win (without potentially overfitting the model… perhaps I’m slightly guilty). Their edge isn’t luck or clairvoyance; it’s data, infrastructure, and discipline.

1. They work with better data

• Market microstructure data: Millisecond-level order books, bid–ask flows, quote imbalances, and dark-pool prints. This data captures the real mechanics of price formation - where every trade, cancel, and hidden order leaves a trace.

• Alternative data: Credit-card transactions, satellite imagery, shipping manifests, web-scraped news, and ESG disclosures - all cleaned and linked to tickers through entity-resolution pipelines. Btw, I’d love to try this myself: track AI mentions in earnings calls or scrape every investment podcast transcript… I’m sure I can make some returns here.

• Feature depth: Thousands of engineered variables - liquidity metrics, momentum decays, volatility surfaces, crowding indices - that collectively drown out the noise of public daily prices. Retail data simply lacks the signal-to-noise ratio for durable alpha.

And now, an entirely new layer is emerging. Hedge funds are experimenting with large language models - not to predict prices outright (yet), but to read the market’s language. Using transformer architectures fine-tuned on transcripts, headlines, and even tweets, these systems convert text into high-dimensional embeddings that quantify tone, uncertainty, and sentiment drift in real time. Some funds now map these embeddings to factor returns or volatility spikes; others use them to detect stress in central bank communication (a bit close to home for me - there’s always space to plug my PhD project centralbanktalk.eu, right?).

2. They build ensemble pipelines - not single models

Instead of one grand model, they run hundreds of small, specialized ones.
Each targets a distinct horizon:

• Milliseconds: market-making and order-flow prediction

• Hours: short-term flow and liquidity shocks

• Weeks: trend and relative momentum

• Months: valuation, factor rotation, or regime-switch models

Some firms operate with a higher-level “meta-strategy” which allocates capital dynamically across these sub-models, smoothing performance across time and market conditions.

3. They invest in execution and infrastructure

The same alpha, executed 10 milliseconds faster, can be worth millions. While I play around in my spreadsheet they have to really execute.

• Execution algorithms (to hide their footprint and reduce market impact - since dumping the order in the market would move the price against them) like time weighted average price, volume weighted average price, and percent of volume slice orders to minimize market impact.

• Real-time risk engines throttle exposure when volatility spikes, i.e., traders dial down how much risk a trading system is taking when volatility suddenly comes in - so a bad day doesn’t become a catastrophic one.

• Compute: GPU/FPGA clusters that backtest (simulating how a trading strategy would have performed historically) decades of tick-level data in hours rather than days. For the non experts like me: FPGA is a field-programmable gate array which is hardware you can literally rewire for your exact trading logic making it extremely fast.

4. They operate under a completely different regulatory regime

Quants can run large long-short books, borrow stock at scale, and use synthetic exposures through prime brokers - options that retail investors (or me and my python file) simply don’t have.

Retail faces strict margin requirements, limited shorting (especially hard-to-borrow names), and almost no access to synthetic shorts like swaps or portfolio margin. Rules like Reg SHO and conservative broker risk models cap leverage before it becomes useful.

Quants, by contrast, use:

• portfolio margin based on net risk,

• institutional borrow facilities, and

• futures and swaps to hedge cheaply and precisely.

This lets them build market-neutral, factor-neutral strategies, while retail is mostly stuck long-only, absorbing factor cycles and drawdowns that pros systematically hedge away.

5. They optimize for risk, not raw returns

Quants don’t chase the biggest returns - they chase the most reliable ones. Every strategy is volatility-targeted: positions are scaled so that portfolio risk stays roughly constant (say, 10% annualized).

Capital flows to models that are:

• Efficient - high Sharpe ratios (strong returns per unit of risk),

• Independent - low correlation with others,

• Resilient - small, stable drawdowns.

To keep returns smooth, they use dynamic leverage and hedging. When volatility is low and signals are strong, they scale up; when risk spikes, they cut exposure or hedge with futures and options.

6. They enforce research discipline

Quants live by process:

• Continuous out-of-sample validation, cross-market replication, and paper trading before deploying real capital.

• Bayesian model updating: when a signal decays, weight it down automatically.

• Feature orthogonalization: constantly searching for uncorrelated alphas rather than doubling down on crowded ones so that if one signal underperforms the other keeps the overall strategy stable.

My lessons learned?

Quants today are the bridge between the old world of handshakes and the new world of computation - proof that in finance, intelligence has quietly migrated from the trading pit to the server rack. Firms like Citadel, Two Sigma, and AQR aren’t just trading - they’re engineering entire ecosystems of execution algorithms, factor models, and risk engines that run around the clock.

The difference between me, my Python scripts, and a hedge-fund quant isn’t insight - it’s scale. They have cleaner data, faster pipelines, and infrastructure that turns risk into opportunity. But the core logic - turning patterns into probabilities - is something you can reproduce with a few hours in VS Code, a free API, and some linear algebra.

And all of this plays out inside a macro storm. A decade of near-zero rates made cheap capital the oxygen of quant finance. Then came inflation, the fastest tightening cycle in history, the Russian invasion of Ukraine, and now Trump-era volatility 2.0 - election uncertainty, tariff talk, and meme-stock déjà vu. Suddenly, liquidity wasn’t free, volatility returned, and the real alpha became survival. Emerging markets promised higher returns but came with political risk; U.S. tech soared again on AI hype; Europe quietly muddled through.

So yes - in bull markets everyone looks like a genius (especially Nancy Pelosi’s portfolio - I briefly considered scraping Pelosi’s trades for features, but even I have limits.). But in the long run, what separates quants from the rest of us isn’t foresight - it’s discipline. The real skill isn’t predicting the future; it’s building systems that don’t break when the future changes.

If I had a bit more time, I’d shove the pipeline into DuckDB, parallelize backtests with Ray, and JIT-compile my indicators with Numba - just enough to pretend I’m running a hedge fund from a laptop. After all, I didn’t take those data science classes during my PhD for nothing.

Btw, Warren Buffett (who just announced to retire at age 94) would roll his eyes at all this, of course. He once quipped that “you don’t need to be a rocket scientist. Investing is not a game where the guy with the 160 IQ beats the guy with the 130 IQ.” He prefers businesses with moats, not models with momentum. And maybe he’s right: while quants chase microseconds, Buffett’s been compounding at 20 percent for six decades - without a single line of Python.

Share

What if segregation today doesn’t look like fences or riots - but like us? What if it’s about who shares your coffee queue, your running route and your algorithm? Segregation now isn’t about walls - it’s about worlds. The trouble is, our data still look for walls. We measure diversity in ten year census snapshots, in neat census polygons, pretending people don’t cross streets or scroll feeds. The official numbers count where you sleep, not where you socialise and commute. They were designed for the Britain of 1951, not the one of Deliveroo, Zoom, and ever-rising London rents. So when the headline data tell us segregation is falling, perhaps they’re capturing the wrong kind of closeness. Let’s explore.

Collecting data

I built a dataset using the 2001, 2011, and 2021 UK censuses to track how segregated Britain has become across ethnicity, education, and class. To my knowledge, census data is the only data that reliably offers LSOA-level detail on all three. For the non ONS UK data crowd: LSOAs (Lower-Layer Super Output Areas) are tiny neighbourhoods of roughly 1,500 residents - to my knowledge the smallest units available that make social geography visible. (Btw, dear ONS: next time, please release an aggregated file. Merging three censuses with different indices, naming conventions, and column structures was, let’s say, “educational”…) Fun fact: the final merge script ran for 19 minutes.

Using this freshly cleaned dataset, I computed a Composite Segregation Index on a 0-1 scale - combining unevenness in ethnicity, education, and class into one measure of multidimensional spatial sorting. A score of 1 indicates areas that are highly homogeneous and diverge sharply from the national average, while 0 represents neighbourhoods whose social make-up closely mirrors the country as a whole. The (normalised!) results are below.

Between 2001 and 2011, segregation rose sharply and has since plateaued. Class segregation fell, ethnic segregation grew, and education-based segregation stayed high. In other words, we’ve mixed more by income, but sorted more by ethnicity and credentials.

Zooming in, the geography of segregation sharpens. For visualisation aesthetics - which, in my opinion, should count for at least 40% of every academic decision - I used quintiles.

The 2021 map reveals where Britain’s social distinctiveness concentrates. Lightest areas - Islington, Southwark, parts of Manchester and Leicester - are sorted. So, the fact that me and my friends all live together in an Islington LSOA is proof that we as roughly the same class and educational group are very segregated from others (I’d add ethnicity here too, but I only discovered today that in the UK census I’d need to additionally tick “Mixed or Other - Arab” since I’m ethnically half Lebanese. The joy of growing up in the Netherlands, where nobody ever asked what box you fit into…).

But let’s take a step back and not just trust what I’ve spun up in my VS Code console at face value.

How Do You Measure Segregation?

Economists and sociologists have spent seventy years trying to quantify something inherently messy, namely, who lives with whom. The first wave of studies emerged in the U.S. during the 1950s to capture racial segregation - when it was literal and still legal. Modern Britain is more complex: multi-ethnic, multi-class, multi-credentialed(? - not sure if that’s a word). The classic formulas strain under that complexity.

But before diving into the technicalities here are two fun facts. Fun fact 1: economists largely ignored segregation until the 1980s - when they realised the math behind inequality and segregation was almost identical. Theil’s entropy index, first used to study income dispersion, quietly became one of sociology’s favourite tools. Fun fact 2: British researchers only started publishing segregation indices in the 1970s - mainly to compare council estates and post-war new towns. Until then, most “segregation” work in the UK was about class rather than race.

Let’s now walk through some core measures:

1. Dissimilarity Index (D) – The Classic Unevenness Measure

• Intuition: the share of one group that would need to move to achieve perfect mixing.

• Range: 0 = perfectly integrated, 1 = completely segregated. Thus, a high *D* means “degree islands” surrounded by degree deserts.

• Most commonly used for: education, i.e., degree-holders vs. everyone else.

This early generation of segregation index treated segregation as binary and spatially static. The Dissimilarity Index can’t handle beyond binary cut-offs: for G groups, you need G(G−1)/2 pairwise D-scores, and weighting them is arbitrary. Then, in the early 2000s, two sociologists Sean Reardon and Glenn Firebaugh (2002) introduced information-theoretic measures like the Entropy and Divergence indices. These could capture multi-group segregation, treating each group’s share as a contribution to an overall diversity measure:

2. Divergence Index (Dᵣ) – The Information-Theory Upgrade

• Intuition: how much each area’s composition diverges from the national average - basically a Kullback–Leibler divergence.

• Range: ≥ 0, often normalised to [0, 1].

• Most commonly used for: ethnicity and social class (NS-SEC).

This formula, borrowed from information theory, reframed segregation as a loss of diversity - a reduction in diversity relative to the national benchmark.

However, if we want a summary measure of how much overall diversity is between vs. within areas, rather than how different it is from a a national average, we need a different measure, namely:

3. Entropy Index (H) – Segregation as Lost Diversity

• Interpretation: the share of total diversity that occurs between areas rather than within them.

• Fun fact: It’s mathematically the same entropy used in thermodynamics - segregation as a kind of social order.

Yet even these advances stayed one-dimensional: one index per variable (race, class, education). They rarely asked what happens when these hierarchies align.

That’s where my approach comes in.

The Composite Segregation Index I propose combines three domains - ethnicity, class, and education - each measured through its own entropy or divergence-based unevenness, and scales them to a common 0-1 range. Averaged together, they form a measure of multi-dimensional alignment.

4. My Composite Segregation Index

Each dimension operates on a different scale, so I rescaled and normalised them to [0, 1] within each year and took their simple average:

Higher values mean the same areas concentrate high education, high income, and certain ethnic groups.

But does it really matter which index you take? Short answer: yes. Academia likes to pretend equations are objective, but every formula is a mirror. It reflects what you think matters. Or perhaps more unconventionally said: segregation indices are like horoscopes: everyone has a favourite, and they all claim to explain the world.

The figure below compares the indices. Each bar represents how “segregated” the same twelve neighbourhoods appear. The Dissimilarity Index mostly flags deprived, low-education areas; the Entropy Index highlights homogeneous, low diversity enclaves; the Divergence Index captures any area that looks “atypical” compared to the national average - including wealthy, monocultural ones. The Composite Index, in contrast, identifies places where ethnicity, class, and education all align - clusters of both privilege and disadvantage. The figure shows that the choice of index doesn’t just change the score; it changes the story.

The Field’s Big Problems

Despite all the math and using a composite index, the measurement of segregation is still not straightforward. Here’s why:

1. The Boundary Problem

Census areas are administrative, not lived. The Dissimilarity Index assumes impermeable borders: you either live in one LSOA or another, no middle ground. Yet real life is fuzzy - people cross streets, commute, and mingle. Geographers like O’Sullivan & Wong (2007) proposed spatial kernels that weight nearby neighbourhoods by distance, smoothing sharp boundaries. Economists rarely adopt this because it complicates inference - but the result is that we’re still modelling social life as a checkerboard.

2. The Scale Problem

Segregation depends on the spatial scale you choose. A city can look integrated at the regional level and deeply segregated at the block level. Sociologists call this the modifiable areal unit problem (MAUP). Economists call it “aggregation bias.” It’s the same mathematical sin - we pretend one spatial scale fits all.

3. The Temporal Problem

We tend to freeze segregation as a cross-section. But neighbourhoods churn: people move, age, or are priced out. Dynamic segregation, i.e., the flows rather than stocks - remains mostly invisible. Economists have the panel tools, but not the micro-spatial data; geographers have the maps, but not the identification strategies.

Thus, even the best indices can’t capture that:

• Living together doesn’t equal interaction. Walk through a “mixed” borough like Tower Hamlets or Camden and you’ll meet Bangladeshi families, French expats, and Polish professionals within the same postcode - yet rarely crossing social or economic boundaries. Indices treat that as integration; in reality, it’s proximity without interaction. Sociologists call this superficial diversity: everyone is present, but few are connected. That’s why cities like London can be both hyper-diverse and socially stratified. Our metrics conflate coexistence with cohesion.

• Mobility: segregation can fall statistically while social mobility collapses. If higher-income households move into deprived areas and displace lower-income residents, the census will record more “mixing” - even as inequality deepens.
  In other words, gentrification looks like integration in the data. Static indices can’t tell if neighbourhood change reflects inclusion or replacement. Economists call this the compositional fallacy: confusing a population’s makeup with its trajectory.

• Networks: social mixing may happen online, not offline. Current segregation metrics can’t see digital mobility or online homophily - yet both shape how ideas, norms, and opportunities circulate. Ignoring this is like studying 21st-century society with 19th-century cartography.

In sum, the composite index tells us who lives near whom - not who lives with whom. Our current data architecture - decennial censuses and coarse spatial units - is not fit for the social realities we’re trying to measure.

Instead, we need to enrich our data:

• Higher-frequency microdata capturing population turnover between censuses. Administrative registers (tax, NHS, school) could be safely anonymised and linked for yearly population panels. In my previous post on Economic inequality I also make this call since this problem is even more prominent for measuring economic inequality. Even better: dynamic linkage between census, housing, and mobility datasets. We need to see how neighbourhoods evolve, not just who occupies them at one moment.

• Spatially aware surveys that ask who you interact with, not just where you sleep. Imagine a redesigned “Understanding Society” at an LSOA level that records friendship networks, workplaces, and mobility patterns. Meta’s Social Connectedness Atlas is, ironically, the closest we’ve come to that vision - albeit confined to digital networks.

• Open APIs for anonymised mobility data to link neighbourhoods to networks. Platforms, telecoms, and transport agencies already collect micro-mobility data at scale. Secure, aggregated access could reveal the functional geography of everyday life - the places people actually share. And yes, Strava, my one and only favourite social network, I’m looking at you - cycling and commuting data could do for mobility what census data did for residence.

Last, I’m still trying to make my mind up about agent based modelling (ABM) to predict how societies further segregate. ABM lets virtual individuals “move” through stylised cities based on income, education, or preferences. The appeal is obvious: you can watch segregation unfold in real time, test how housing markets or school catchments amplify sorting, and see whether small behavioural biases snowball into macro-level divides. It’s social physics at its most seductive. But that’s also the risk. These models can turn rich social processes into oversimplified algorithms - pretending people move like particles when, in reality, they move through stories, identities, and constraints. Perhaps ABMs are more powerful for exploring mechanisms, not predicting futures. They show what could happen, not what will. Still, in a world where our data lag behind reality, maybe simulation is the next best thing: a sandbox for testing how the invisible lines of class, ethnicity, and education might thicken - or blur - in the decades ahead. I’m not sure yet - but perhaps intrigued enough to soon open VS code and start modelling with some sketchy arbitrary priors…

For now, perhaps we’ll keep mapping a fluid world with mainly static instruments - measuring twenty-first-century segregation through the eyes of the mid-twentieth. Every dataset I build seems to end with the same conclusion: we need better data. At this point, it’s less a research finding than a personality trait.

Btw, I’m far from being a sociologist - this is my first proper venture into that field - so feedback is more than welcome. Consider this piece a learning exercise: an curious political economist wandering into sociology’s terrain.

Subscribe now

Share

So how do you measure local inequality in practice? Measuring inequality sounds easy until you try doing it locally - and over time. The World Inequality Database gives national snapshots, but below that the data thins out fast. Which sources can we actually trust? Which measures hold up?

I think it’s fair to say that measuring inequality is itself unequal. Let me explain.

Sensitivity to Data Sources

The UK doesn’t ask residents about income or wealth in the census. That means we lack the kind of granular, universal data you’d want to measure local inequality properly. Surveys are too small at the local level and underreport the rich, and administrative records are incomplete (many households never file tax returns and the informal sector is left out).

In short: garbage in, garbage out. Below, with simulated data I show that inequality looks very different depending on the data source. Household surveys (blue) understate top incomes and show a flatter trend. Tax data (orange dashed) captures the top better and suggests inequality is rising faster. The underlying “true” inequality (black) is never directly observed. The choice of dataset alone can change the story.

Data quality

As you know from the previous rants of mine (see The quiet collapse of surveys: fewer humans (and more AI agents) are answering survey questions & The Quiet Collapse of Surveys Part II: adding some nuances), even if the data is there, the reliability due to low response rates can be questioned. Survey response rates in the UK have collapsed since before the pandemic - from around 50-60% down to 20-30% for major surveys like the Labour Force Survey. That matters. The missing respondents aren’t random. The rich are less likely to respond; the poorest are harder to reach. This skews the distribution, making inequality look more muted than it often is. Add in risk of misreporting or AI/bot responses, and you have a data problem that goes beyond just missing values.

Recently, the UK’s statistical agencies even had to delay or revise key publications due to quality concerns. For example:

• The ONS recently postponed its retail sales release citing quality issues.

• Its Labour Force Survey (LFS) response rate is still well below pre-pandemic levels, and reforms in survey methods are underway - but full reliability isn’t expected until 2027.

• Official wealth data lost its “accredited status” due to concerns over declining response rates and survey reliability.

Time Comparability Issues

Tracking inequality over time is harder than it looks. Survey methods change (e.g. the UK’s Family Expenditure Survey → Living Costs and Food Survey). Tax reforms alter what’s counted as “income” (capital gains, pensions, self-employment). These breaks in series often get swept under the carpet, but they can radically alter long-run trends.

Focus mostly on Income, not wealth (or consumption)

Most inequality stats are income-based. But inequality in wealth is far higher, and much harder to measure. Alternatively, consumption may be closer to welfare - but consumption data is thin, outdated, and often unreliable. So when we talk about “inequality,” we’re really talking about one slice of the pie.

Too Many Measures, Too Little Agreement

Then, assuming we have data, we need to use it wisely. Economists love indexes - Gini, Theil, Palma, Atkinson - but they don’t agree. One measure says stable, another says inequality is soaring. See below a little simulation: normalising inequality measures shows how differently they evolve - the Gini is steady, Palma and Atkinson barely move, while the top 1% share surges, underscoring how the story depends on which metric you choose (a fact that politicians know very well…).

Inequality vs. Poverty vs. Affluence

Even if we set on one measure, a change in the measure can mean different things. For instance, a rising Gini can mean: the poor got poorer or the rich pulled away. Or both. But most measures don’t distinguish. That makes them blunt instruments for understanding social change. Below three scenarios that mean that provide similar Gini coefficients with different underlying stories:

Ignoring Composition Effects

Who counts as a “household” or “unit” matters. Migration, aging populations, more single-person households - all these shift inequality measures even if underlying distributions of resources stay constant.

Let’s take the aging populations: more retirees means higher inequality: suppose workers earn around £40k and retirees around £20k. Within each group, incomes stay constant over time. But as the population ages and the share of retirees doubles, the overall Gini coefficient rises - purely because of demographics. Below a simulation with dummy data:

A second-best analysis: local UK inequality over time

So, what to do? Give up? Not quite.

But I did my best with what I had and created a measure of income inequality and wealth inequality exploiting MSOAs being embedded into local authorities as the source of variance. For income inequality, I leaned on the best ONS data available, though only for a relatively short period (2014-2020). For wealth inequality, I used house price distributions - a surprisingly strong proxy, thanks to the Land Registry’s exhaustive coverage of every property transaction since 1995.

The figures below show the change over time at the Local Authority Level for income (top) and wealth (bottom). Once you start comparing indices - Gini, Palma, 10/90 ratios - the picture fragments. Sometimes the Gini is flat, while the Palma or the 10/90 ratio suggests growing divides. This isn’t just academic nitpicking: it shows how easy it is for policymakers to pick whichever statistic best supports their argument.

How do these measures compare over time? Across 2014-2020, local wealth and income inequality are positively correlated, but wealth measures consistently run higher and diverge more strongly over time, showing that where you live often looks more unequal in assets than in income.

When mapped geographically, local wealth inequality reveals striking variation across England and Wales. Some local authorities show persistent high inequality, others much lower. But when you look at change over time, much of the South lights up red: wealth inequality has increased in most areas since the mid-1990s.

Finally, I compared wealth inequality (house price Ginis) to measures of social mobility from the Opportunity Index of Sutton Trust. Why make this comparison? Because inequality matters most when it affects opportunity. Classic economic theory (shoutout to the Great Gatsby Curve!) may tell us that high inequality can translate into barriers to mobility - fewer rungs on the ladder, or rungs spaced further apart. Wealth inequality is particularly relevant here: housing markets shape where families live, what schools children attend, and what networks they can access. If house prices are highly unequal within an area, poorer households may be priced out of high-opportunity neighborhoods, reinforcing disadvantage. (The counterargument may be though that high inequality can offer strong incentives to mobility but let’s just ignore for now…).

The weak correlation in the data below between those in the top of economic mobility and wealth inequality suggests that, at least in the UK context, the story is more complicated. Inequality is rising, but it doesn’t translate neatly into local mobility outcomes - perhaps because other factors (schools, transport, policy, social capital) intervene.

In sum, local inequality in the UK is hard to measure - and my analysis is at best a second-best attempt. Still, the patterns are clear enough to raise questions and merit further research with better data: inequality has risen in many places - even more when you look at wealth, but the relationship with opportunity is weaker than you might expect.

How to move on/improve?

Beyond the brief analysis I just did, there are some slightly more time-consuming data-driven ways forward to improve from what I just did:

• Small-area estimation: Borrow ONS’s trick - combine surveys with census covariates in hierarchical Bayesian models to map inequality down to the MSOA level using richer census covariates for sparse data. But do remember: garbage in, is still garbage out!

• Distributional national accounts: Align survey microdata with ONS macro totals for consistent, spatially differentiated inequality measures (e.g., by reweighting and scaling household survey data).

• Spatial econometrics: Model inequality with spatial lags (e.g., spatial weight matrix using contiguity or k-nearest neighbors), because Croydon and Surrey aren’t islands - apparently.

And perhaps being a bit creative here, some of my ideas beyond the usual toolbox:

• AI/NLP (how can I write a substack without mentioning an AI option?): The idea is simple: scrape the stuff local life produces in text - council reports, housing ads, planning applications, job postings. Then train and fine-tune a LLM to score how often words cluster around “affordability/struggling/basic” versus “luxury/exclusive/premium.” Aggregate those scores across space and time at the MSOA/LSOA level, and you suddenly have a perceived inequality index: what landlords, councils, and employers are signalling about local economic conditions. It’s noisy, sure, but as a complement to hard numbers it captures the “vibe” of inequality that surveys and tax records can’t.

• Computer vision on street-level imagery. With Google Street View or Mapillary frames, you can train object-detection and classification models (YOLO, ResNet, ViT) to quantify visible inequality: housing type and façade condition, roof and window upkeep, tree cover, cars (old hatchbacks vs new SUVs), retail frontage (pawn shops vs cafés), even security cues like shutters or CCTV. Aggregated to MSOA/LSOA level, these features become a visual deprivation index or streetscape asset index, which can be tracked over time by pulling older vs newer imagery. Validation against IMD, house-price Ginis, or FSM rates shows whether the proxies work. It’s not perfect, but it turns what you can literally see on the street into measurable local inequality.

• Transaction data: Anonymised card spending or retail footfall feeds (think Visa/Mastercard, bank APIs, or mobile location data) can be aggregated to MSOA/LSOA to capture consumption inequality - how often households eat out, travel, or buy discretionary goods versus only essentials. With large-N, privacy-preserving aggregation (k-anonymity thresholds, differential privacy noise), you can track local consumption divides at monthly frequency. This provides a high-resolution, behaviour-based proxy for inequality.

📌 And credit where it’s due: the only real UK local inequality measure I’ve found in practice is Joel Suss’s one-off Zoopla-based study - proof that creative data proxies can work.

Why this matter - and moving forward

Inequality numbers look neat in a textbook, but measuring them in practice is messy, contested, and often misleading. Inequality measures shape policy debates, yet most of the tools we use are blunt, inconsistent, and too often divorced from how inequality is actually lived. Without better data, we’ll keep patching together proxies and hoping for the best - and politicians will keep cherry-picking whichever measure suits them.

Still, the toolbox is expanding. From Bayesian small-area estimation to computer vision and AI-based perception proxies, we can now push beyond national averages and get closer to how inequality plays out locally. But let’s be honest: the real breakthrough would be direct access to granular tax data, combined with other sources, to anchor everything else.

So if you’re reading this at ONS: unlock the data, and I promise I’ll turn it into public insight rather than private mischief. Think of me as a responsible borrower - lend me the data, and I’ll return insight.

Subscribe now

Share

At the heart of this is a simple idea: the scoreboard says who won (albeit after a penalty shoot out final) - the underlying pitch data shows whether it was deserved and ‘how’ and ‘why’.

From the pitch to the dataset

I lean on two simple ideas in football analytics. A common concept in the literature is total expected goals (xG) which estimates how good a chance looked before the strike. Slightly less common and often used in proprietary models and internal club analytics is Off-Ball Scoring Opportunities (OBSO) which captures how the moment actually unfolded. I argue in this Substack that when those two disagree, that’s where the fun lives - decision-making, pressure, angles, chaos.

Before we get there, a quick note on process. In a typical football-analytics workflow, you’d turn match video into a usable dataset with a computer-vision pipeline - e.g., a Python package called YOLO - to detect players and the ball, tracking to link frames, and a lot(!) of manual cleanup. It’s powerful, but painfully time-consuming (speaking from experience). Thankfully, StatsBomb have done the heavy lifting and shared an analysis-ready dataset in their GitHub repo which is just waiting to be scraped and formatted.

With the data loaded in Python, I first looked at where and how each team took their shots.

The geometry of attack

The chart below compares shot distances and angles which shows a subtle split in style: England’s shots came from slightly closer in; Spain’s from a touch wider angles. Small, but telling.

If that visual is nice, density maps are even better. Spain’s shot density shows more attempts from wider and slightly higher positions in the box - the half-spaces -reflecting their patient ball progression before shooting. England’s density is more central, pointing to a direct route to goal. The differential map makes this crystal clear: Spain owned the flanks (red zones), England pressed centrally (blue zones). These small differences that hint at contrasting attacking styles.

These attacking fingerprints give us the canvas. Now let’s paint in the match’s story using xG and OBSO together.

xG vs OBSO - opportunities vs execution

xG (expected goals) is a supervised probability model trained on historical shot data using only pre-shot features: location, angle, distance, body part used, defensive pressure, game situation, pass type, build-up context, and so on. In my workflow, I compute the core geometry features exactly as most xG models do - shot distance (straight-line to the goal centre) and shot angle (lines from shooter to each post). A classifier - often logistic regression or a tree-based model - estimates the likelihood of a shot becoming a goal. After calibration, that probability is the shot’s xG.

OBSO, on the other hand, measures how often a team or player creates a dangerous off-ball scoring opportunity - being in position to receive a pass with a high likelihood of scoring. In tracking data, this comes from player positioning, passing lanes, defensive pressure, and angles to goal.

Why does this matter? Looking at the two both matters because xG captures the quality of the shot as taken, while OBSO captures the quality of the opportunity before the shot (or even if no shot happens at all). Because football isn’t just about the shots you take - it’s about the chances you could have taken.

From this plot (using all data rather than match data), the “best” depends on what you’re optimising for:

• Most consistently dangerous off the ball: María Francesca Caldentey Oliver (Spain) and Alexia Putellas Segura (Spain) are standouts. They have the highest OBSO counts (well above the trend line), meaning they were constantly in threatening positions, even if their xG wasn’t the absolute highest.

• Most efficient at turning opportunities into shots with high xG:
  Alessia Russo (England) and Chloe Kelly (England) are notable here. They have high xG from relatively fewer OBSO counts, meaning they capitalised well on the opportunities they did get.

• Balanced creator-finisher profile: Patricia Guijarro Gutiérrez (Spain) sits nicely above the trend line with both high OBSO and a solid xG, suggesting she was both creating danger and taking quality shots.

If we define “best” as both active in creating dangerous positions AND turning them into quality shots, Alexia Putellas Segura probably takes the crown in this match - high OBSO, high xG, and a healthy pass completion rate.

Aside from looking at individual players (over all matches), the cumulative xG timeline is the cleanest read of how the match breathed.

Spain built a steady lead in chance quality for most of the game (solid orange), while England’s xG climbed in bursts - long plateaus punctured by sharp steps. Both sides exploded late, but those final surges were due to the penalty shoot-out not a sudden flood of open-play chances. The dashed lines tell the scoreboard story: Spain struck early and managed the game, forcing England to rely on set-piece lifelines to close the gap. Strip away the penalties, and Spain’s underlying chance creation stayed ahead throughout - a reminder that the better open-play team doesn’t always win the box score.

But football isn’t just about the chances you take. It’s also about the chances you don’t allow.

Defence wins finals too

Also using StatsBomb event data, we can measure defence in two main ways: shot prevention - total and xG of shots conceded and pressure and disruption - the volume and success of defensive pressures.

The defensive radar tells a clear story: Spain edged England in most defensive metrics, especially in pressure rate and danger zone pass prevention. Their back line consistently disrupted build-up play and limited England’s access to high-value shooting areas. England, meanwhile, matched Spain closely on xG conceded, suggesting their defensive shape still blocked quality chances, but their lower pressure and higher allowance of dangerous passes gave Spain more control in critical moments. In short, Spain’s defence was not only reactive but proactive - forcing England to work harder for every attacking opportunity.

The final proved that in football, controlling the flow of chances beats chasing the scoreboard - until penalties arrive that is…

Can these classifiers and analysis be improved? Absolutely. And let’s be clear - we should be building entirely different models for men’s and women’s football. The patterns, contexts, and even physical dynamics aren’t the same, so neither should the models be. The bias and variance in evaluating players using the wrong model for women’s football could definitely be a Substack in itself - building a xG model from scratch accounting for gender and many more factors.

Having said that, in recent years, football analytics has moved well beyond basic xG. From the NFL, we can borrow concepts like route trees and separation metrics - adapted for football to measure off-ball runs, pressing triggers, and defensive spacing. Also, they use one of my favourites: Markov Chains to look at transitions. American football’s situational decision models (e.g., 4th-down calls) have clear parallels for when to counter-press, switch play, or take a shot.

Yes, the maths-and-physics crowd (especially the ML specialists) can push the technical side forward. But on a separate note, I’d argue we also need behavioural scientists in the mix. Just imagine pairing the data we’ve just explored with information on player stress, decision-making under pressure, or fatigue - not only could we recruit (even) smarter, but we could tailor coaching to the psychological as well as the tactical. The real opportunity lies in merging these technical advances with human data and I promise that I’m not only saying this because I am a social scientist.

Do I have ideas for how to do this? Definitely. All I need is the data - rather than vibes. And if any club happens to be reading this - if you’re short of an extra bench player, I’m here - good xG and OBSO are not guaranteed though.

Subscribe now

Share

So what did I do?

1. Baseline. I use YouGov’s June 2025 multilevel regression and post‑stratification (MRP) seat model - the most transparent, seat‑by‑seat projection currently in the public domain.

2. Add the teenagers. The latest ONS mid‑2023 estimates supply the exact head‑count of 16 and 17 year‑olds for every English constituency.

3. Turnout scenarios. I scrape constituency level official turnout from July 2024, then scale it down to three teenage cases: Low = 70% of adult turnout, Medium = 80%, High = 90%. This range brackets the Electoral Commission’s 46 % validated turnout for 18‑24s in 2024 and the 75 % teenage turnout recorded in Scotland’s 2014 referendum (where 16 year olds could already vote).

4. Party preference. In the absence of standalone polling for 16‑17s, I borrow YouGov’s 18‑24 cross‑tab (a necessary proxy until 16‑17 polling exists; sensitivity tests suggest ±5 pp swing moves the flip count by ±3 seats): Green  31% • Labour  27% • Lib Dem  22% • Reform 8% • Conservative  5% • Plaid  1% • Others  6%.(Focaldata/Merlin show similar skews; I will revisit this once age‑specific data appear - but this age category is notoriously difficult to poll and past voting behavior cannot be used).

5. Re‑tally winners. For each constituency I add teenage votes to the MRP baseline, recalculate party totals and declare the new winner.

What follows is an early‑stage calculation, not a final forecast. The exercise is deliberately rough: it ignores local variation in registration drives (now folded together with turnout), assumes teenage party preferences mirror the wider 18‑24 cohort, and confines itself to English constituencies. A fuller, region‑sensitive simulation - with confidence intervals - will appear in a subsequent post. For now, this first cut is enough to show the direction of travel.

What moves - and why

Let’s dive into the results. Below, each blue dot marks a constituency that would change hands once teenagers vote. Reform loses every one of its knife‑edge gains; Labour picks up the lot. Conservatives, Greens and Lib Dems neither win nor lose a seat in this single‑scenario run because their existing strongholds are either already safe (Greens in Brighton Pavilion) or still out of reach.

The 12 constituencies that flip once 16 and 17 year‑olds are added have two things in common: a thin Reform (or “other”) leads and a critical mass of teenagers concentrated in sixth‑form colleges or university feeder towns. A brief profile of each seat helps explain why the youth vote is decisive.

• Bournemouth East with 1100 extra teen ballots erases Reform’s 59 vote edge.

• Bradford West West Yorkshire: a 19‑vote “Other” majority is swamped by new voters from four FE colleges.

• Derby North East Midlands with University of Derby campuses sit inside the seat; ~1200 teen ballots overturn Reform’s 191 vote lead.

• Gateshead Central &  Whickham: the reform margin of 189 is no match for a similar‑sized youth cohort.

• Mid Cheshire is North‑West commuter belt where sixth‑forms yield 1170 teenage voters - double Reform’s 54 vote cushion.

• Reading West &  Mid Berkshire South‑East has University of Reading halls plus two colleges add 1450 votes, flipping a 209 vote Reform lead to Labour.

• Rochdale Greater Manchester is one of England’s youngest boroughs; 1 500 teen ballots swamp a 250‑vote Reform majority.

• Southampton Itchen is a South Coast Solent sixth‑forms and docks apprenticeships deliver 1 100 first‑time voters; Reform’s 154‑vote lead disappears.

• Wigan North‑West Leigh & Wigan College feeds 1 140 teenagers into the roll, overturning Reform’s 146‑vote edge.

• Worcester West Midlands has University of Worcester and local sixth‑forms provide 1290 new voters - 24 times larger than Reform’s 55 vote margin.

• Worsley & Eccles is a Greater Manchester Salford City College cohort wipes out an 89 vote Reform lead.

• Monmouthshire is a Welsh border Rural but ringed by comprehensive sixth‑forms; 1230 teenage ballots cancel Reform’s 84 vote edge.

  ---

Margins on the brink

A scatter of “teen votes added” vs “new majority” (see figure below) shows hundreds of seats sitting just below the 45‑degree line - teenagers out‑number the winning margin even though the seat does not flip in this run. In other words, if the blue dot is below the line, teenagers can cast more ballots than the difference in majority is while if it’s above the line the teenage cohort cannot in any way impact the results. Thus, one can see that one well‑targeted registration drive or a small shift in youth preference could push many of those constituencies over the edge. And remember those 3 turnout scenarios? It doesn’t even change that much - the low turnout scenario has 11 switch seats and the highest 12.

Provisional verdict

Under somewhat plausible assumptions, enfranchising 16 and 17 year‑olds would shift a dozen English constituencies from Reform (or other) to Labour, each on the strength of fewer than 1600 additional ballots. Many more seats move within striking distance. The net effect - for now - looks modest in absolute numbers but consistently tips the scale towards Labour.

Next steps

• Incorporate local registration completeness (Electoral Commission accuracy audit).

• Swap in age‑specific polling once reputable 16‑17 cross‑tabs emerge. Or in any case do some better imputation since 16 year olds are still in school and their priorities and media diets are not identical to first year undergrads. Some machine learning may come in handy in this step.

• Not assume uniform registration and turnout: a sixth‑form in the Wirral will not mobilise like a sixth‑form in Tower Hamlets. Perhaps college density effects on the turnout multiplier will do a great job here.

• Run a full Monte‑Carlo with constituency‑level draws for registration, turnout and preference (e.g., with a few thousand draws). Presenting single‑number flips masks the fact that a ±3 pp swing in youth preference moves the result by dozens of seats.

• Extend to Scotland, Wales and Northern Ireland where very different party structures apply.

• Perhaps some tactical voting? A teenage Green vote in a Labour‑Green marginal behaves differently from a teenage Green vote in a Con‑Lab fight. I can incorporate a tactical component by regressing BES tactical‑intent on seat‑level majority structure.

  ---

Rigorous refinement may change the count, but the direction is unlikely to reverse. Campaign managers, local authorities and the Electoral Commission would be wise to plan for the administrative - and electoral - implications now.

Good to note is also that lowering the franchise would not just tweak the next election; it would plant seeds for many elections to come. Political scientists have shown that casting a first ballot while still in school - surrounded by peers and family who discuss the campaign - fosters durable voting habits. Early enfranchisement embeds turnout in adolescents’ social networks and primes them to remain engaged as they age.

I’m still fairly new to England (two years in London spread over some time and counting), so if I’ve missed any local quirks or fresher data, do let me know.  I’ll incorporate many corrections into the next, more granular update… (hopefully soonish).

Subscribe now

Share

Bonus: you’ll also learn why the yellow jersey is yellow.

Fast forward 112 editions, and the race has become a highlight of human endurance, team strategy, and high-altitude drama. In this post, I’ll preview what makes this year’s Tour special - from its brutal mountain stages to the riders and teams most likely to shape the race - and offer my own predictions as to who’ll be standing on the podium in Paris.

But before we dive into what’s coming next week, let’s rewind to where it all began: a newspaper war, a political scandal, and a publicity stunt.

The Tour de France has its origin from an important French political event: Captain Alfred Dreyfus, a Jewish artillery officer, was wrongly convicted of spying for Germany in 1894 which resulted in two camps: republicans, secularists and most of the intelligentsia (the “Dreyfusards”) demanded a retrial, while the army, Catholic hierarchy and nationalist right (“anti-Dreyfusards”) insisted the verdict protected national honour.

This fracture ran straight through the French press. One of the loudest Dreyfusard voices was Le Vélo, an influential sports daily printed on green paper and edited by the liberal journalist Pierre Giffard. But its biggest advertisers - including car magnate Jules-Albert de Dion, tyre tycoon Édouard Michelin, and bicycle manufacturer Adolphe Clément - stood firmly on the other side. When Le Vélo openly backed Dreyfus, the industrialists withdrew their funding and decided to bankroll a rival paper.

That rival, L’Auto-Vélo, launched in October 1900 under the fiery former cyclist Henri Desgrange. Its yellow newsprint was a deliberate jab at Le Vélo’s green. The paper claimed to be “above politics,” but few readers were fooled. Worse, a court ruling in 1903 forced it to drop “Vélo” from the name, leaving just L’Auto - and revealing how badly it lagged behind its competitor in circulation.

Desgrange needed a stunt - something so spectacular it would bury Le Vélo for good. His industrialist backers, thriving in the golden age of bicycles and motor cars, wanted something nationalist and modern. The idea came on 20 November 1902, over lunch in a Paris café. Desgrange’s young cycling reporter, Géo Lefèvre, proposed a crazy idea: a race around the entirety of France. The tour was born.

The first Tour de France, announced in January 1903, included six monster stages - Paris to Lyon, Lyon to Marseille and so on - totalling 2,428 kilometres. The winner was often the one who actually managed to finish - the last man standing. The entry fee was fixed at a single franc and riders were forbidden of any outside assistance so even a factory worker could participate.

Maurice Garin, the “Little Chimney-sweep”: the 32-year-old Italian-born Parisian dominated, winning three stages and leading for all 19 days to pocket 6128 francs - three years’ wages for a labourer. But the true victor was L’Auto. Its sales jumped from 25,000 to over 300,000 during the race, while Le Vélo never recovered and folded the following year. The Tour’s leader wore a yellow jersey from 1919 onward - an enduring echo of L’Auto’s yellow pages.

So, in short, the race’s DNA still holds the imprint of a divided Third Republic. The Tour de France was born not merely as a bike race but as a circulation-boosting marketing masterstroke - a risky bet that fused nationalism, new-century technology and raw human grit. The gamble paid off so spectacularly that 122 years later the whole world still tunes in each July to watch history’s longest newspaper promotion ride by.

This year - anything special?

After last year’s detour to Nice, the 2025 Tour de France starts in Lille on 5 July and, three weeks and 3,338 km later, returns to the Champs-Élysées on 27 July. It is the first edition since 2020 to stay entirely inside France’s borders, visiting 11 regions and 34 departments along the way.

The route mixes seven flat, six hilly and six mountain stages with two time trials. Five summit finishes (Hautacam, Luchon-Superbagnères, Mont Ventoux, Col de la Loze and La Plagne) and an Everest-scale 52,500m of climbing tip the balance toward the climbers. Let me try to summarise this in a graph (excuse me for using double axis please):

The opening northern block is anything but a ceremonial parade: three circuits through the exposed plains of Nord-Pas-de-Calais, a cross-wind run down the Channel coast to Boulogne-sur-Mer and a classics-style finish in Rouen give the sprinters early targets but could also spring echelon chaos for the overall contenders.

The Tour’s first real sorting-hat arrives in the Pyrenees. Over three consecutive days the race tackles Soulor-Hautacam, Col de Portet en route to Luchon-Superbagnères and then the Peyragudes mountain time trial - just 11 km, all uphill, and likely to penalise any GC hopeful whose engine prefers long flat efforts.

Mont Ventoux headlines the third weekend: stage 16 climbs the bald, wind-scoured giant from Bédoin. The queen stage comes two days later: 171 km from Vif over Glandon and Madeleine before the brutal 26km grind to Courchevel’s Col de la Loze. With more than 5,000m of elevation and gradients hitting 20%, this day is expected to create the biggest time gaps of the race.

To celebrate the 50th anniversary of the Champs-Élysées finish, organisers have spliced the Olympic road-race loop into Stage 21: the peloton will climb the cobbled Rue Lepic up Montmartre three times, the final ascent just six kilometres from the line. GC nerves will stay until the last lap.

Who to watch out for?

So who’s most likely to tame Ventoux and conquer the Champs-Élysées this year? Let's look at the contenders - and the teams behind them.

I expect there to be a familiar heavyweight duel between two-time champion Jonas Vingegaard and Tadej Pogačar, with Remco Evenepoel and a resurgent Primož Roglič lurking. Power-data previews still rate Pogačar as slight favourite after a dominant Dauphiné, yet Col de la Loze training sessions this month show all three primary challengers testing one another’s limits on the very climb that could decide July. The following graphic contains my bets (which I based on bookmakers and UCI points):

Then it’s time to look at support - because I cannot stress enough that cycling is a team sport. My chart below breaks down each top team’s core strengths (according to me) across three domains: climbing support, time-trial power, and sprint depth. UAE and Visma stand out for their mountain engines - crucial for shielding their GC leaders on Ventoux and the Col de la Loze - while Soudal–Quick‑Step dominates on the time trails, and Lidl–Trek leans more toward sprinting.

Another chart below shows how those same components contribute to each team's overall GC support score. With 50% of the weighting from climbing ability, it's no surprise UAE tops the chart, bolstered by its all-star climbing trio (Yates, Almeida). Visma and INEOS follow closely, thanks to their balance of climbing and TT depth. This graph summarises it:

After having watched the Tour de France from start to finish (even a couple of times at the Champs-Élysées), I can say that a lot can happen in the race which is only more reason to get hyped up for tomorrow!

A final note: while this preview focuses on the men’s race, we shouldn’t forget the Tour de France Femmes, which has rapidly become one of the sport’s premier stage races. Since its revival in 2022, it has showcased fierce competition, world-class climbs, and rising stars like Demi Vollering and Katarzyna Niewiadoma. The 2025 edition - held in August to avoid Olympic clashes - will feature expanded mountain stages and a summit finish in the Alps. As the women’s peloton gains depth and visibility, this race increasingly stands on its own merits.

Subscribe now

Share

Recently, however, populist showdowns (think Trump vs Powell on and off Twitter and Milei vs just about every central banker) grab the headlines claiming the end of central bank independence. Nevertheless, my PhD thesis (see my website centralbanktalk.eu for all papers, data and dashboards) - and months roaming the corridors and sitting in my 34th floor office of the European Central Bank - suggest that there are also deeper, slower-burn threats to independence other than populism:

1. Financial pressures. Markets now assume the “independent” bank will catch every liquidity fracture and these financial pressures only increase with more independence!

2. Societal pressures. Central banks no longer lean on a permissive consensus but feel societal pressure to act on longer term issues - climate change and inequality are suddenly on the central bank agenda.

3. Fiscal fault-lines. Statutory separation masks the growing need for permanent monetary-fiscal coordination. Central banks cannot keep solving fiscal problems.

These conclusions aren’t just vibes - they’re measured vibes. I ran 18 826 speeches from 119 central banks (including many developing economy and central banks in non-democratic countries!) - given between 1997 and now - through fine-tuned large-language models to systematically detect these threats. Below a preview of a basic LLM embedding-based topic model on the speeches which already hints at various threats.

Despite that just eye-balling the topics in the graph already gives a gist of threats, let me dive into the three threats one-by-one below (beyond descriptives), share some thoughts how bad they are, how I identified them and what the silver bullet is to save central banks from them.

Challenge 1: Independence is a magnet for financial pressures

The more untouchable the central bank, the louder markets demand it become lender-of-first-resort for every wobble - from regional bank runs in California to climate-stress backstops in the Sahel and the more they have to cover for governmental financial leniency in search of alternative ways - other than monetary policy - to boost the economy. Legal distance from politicians and fiscal policy morphs into tighter financial dependency.

To show this, in Chapter 2 (published in EJPE) , we scraped all central bank speeches from 1997 to now, fed every sentence to a fine-tuned Gemini LLM. The pattern is crystal: the more independent a bank becomes, the more it talks like Wall Street’s emergency hotline. The causal link is tight: a 0.10 jump on the classic Romelli independence scale (0-1) decreases inflation obsessed speak by a quarter and doubles talk about financial pressures. We also show that it isn’t the 2008 crisis doing the work - this shift starts years before Lehman and keeps climbing.

Why? Freedom from ministers doesn’t buy silence; it buys expectations. Once politicians can’t twist the rate dial but can make financial rules more lenient to boost the economy and markets assume the technocrats will cushion every shock. So independence packs a hidden cost: it drags the central bank out of the ivory tower and straight onto the trading floor - whether the statutes mention “financial stability” or not.

The figure below shows that talk about monetary pressures - dubbed as a measure of monetary dominance for measurement sake - decreases in line with expectations with independence. Financial pressures - dubbed financial dominance - increase after a delay (which is in line with the government needing time to change policies). In the paper we show the exact mechanisms.

Btw, a note for the applied LLM nerds among us, we have a pretty cool and innovative 7 step LLM classification and validation pipeline benchmarking Gemini to other state-of-the-art LLM models - so do read the paper! For those causal inference nerds, I think we must hold some kind of record of the most robustness checks for our staggered difference-in-differences and instrumental variable models - our close to 100 page appendix is a great leisurely read, I promise!

Challenge 2: societal pressures - the slow burn fuse under CBI

If populists are a noisy fire-alarm, society’s long-horizon demands are the smoldering wiring behind the panel. Chapter 3 shows that even the world’s most statute-armored central bank cannot pretend the room isn’t filling with smoke.

Using 4 654 speeches since 1997, I trace how “non-monetary” issues - climate risk, inequality, digital money, even geopolitics - travel up from national central banks (NCBs) to Frankfurt.

The classic CBI bargain was negative: politicians may not meddle. Societal pressure forces a positive reading: the bank must proactively show usefulness on issues voters care about. As my Markov chains reveal, in crisis years 2010-15 the ECB was twice as likely to echo Italian or German speeches on “crisis management” or “inequality” than in tranquil years. Silence is no longer neutral; it looks like negligence. Independence becomes contingent on visible responsiveness - a very different animal than insulation. My within-topic similarity scores show monetary-policy language converging (markets demand a single voice) while climate talk stays heterogenous - evidence that societal items are where fragmentation, and hence vulnerability, lives.

The good news? Central banks do not systematically budge under this societal pressure - as I show in the figure below in the case of the European Central Bank responsiveness of the European Central Bank to economic and public issues playing in national member states (public attention, trust in the ECB, inflation and GDP growth) is random at best. Rather, communication is a way to embrace emerging epistemological consensus or policy issues. Thus, societal pressure is the quietest yet widest threat. It does not sack central bankers; it re-writes what “doing their job” means, expanding the evaluative arena until the old independence armour no longer covers all vital organs.

For those who love methods, in this paper I also pioneer some cool Markov Chains to look at topic responsiveness in the short-term and combine it with some more traditional cross-sectional timeseries models to examine more systematic responsiveness of central banks.

Challenge 3: Statutory walls can’t bridge the monetary-fiscal policy chasm.

The insistence on the artificial fiscal-monetary separation means there is almost no permanent architecture for joint crisis action. When shocks hit - pandemic swaps, war-time energy subsidies - improvisation replaces rule-making which potentially can blur independence. Additionally, when shocks do not occur - there is also need for coordination to achieve the optimal policy-mix for a currency area to thrive.

Chapter 4 shows that speeches explicitly calling for joint monetary-fiscal (or monetary-supervisory) action have doubled for all central banks world wide - from 22% in 2000 to 44% in 2014 - yet no formal coordination has thus far taken place. The talk of coordination has raced ahead of the machinery to deliver it, so every new shock still forces ad-hoc improvisation - eroding the very autonomy the statutes were meant to safeguard.

Again, for those applied LLM nerds (I think by now you must have noticed my methods passion): in this measurement paper, we systematically show the most accurate and efficient ways to prompt and use ChatGPT and explore some human uncertainty measures.

Now, it’s time for solutions! and yes I’m hopeful

So, I think it’s fair to say that central bank independence is certainly at risk - but the enemy is not only aggressive politicians. The quieter threats are financial, fiscal and societal pressures: independence was conceived as an one‑off institutional fix; the evidence in my thesis shows it behaving more like a renewable lease - continuously re‑negotiated.

The data in my dissertation don’t just map the threats; they also point to fixes that work. None requires rewriting constitutions; all build on the tool central banks already possess: central bank communication. Simply said: global inflation is expectations, expectations are formed by good and bad vibes which can be shaped by the narratives. Trump and Milei can scream what they want (as long as they do not formally overhaul CBI), as long as the credible central bankers are there and the narrative strong: CBI lives on! Especially in these difficult times, a strong voice of a capable technocrat is what we need (we see you doing your thing in Canada, Mark Carney!).

Thus, I argue that in 2025 it isn’t raw macro numbers or textbook technocracy that will rescue independence - it’s narrative discipline. Central banks themselves once before already recognised this through a major transformation from 1990-2010, moving from silence - once shrouded in "temple-like" secrecy - to open communication within just two decades. As I quote in my thesis:

“If I seem unduly clear to you, you probably misunderstood what I said.” (Alan Greenspan as Chairman of the Fed, 1987 as quoted in the Guardian on November 4, 2005)

“The effects of monetary policy depend critically on the public getting the message about what policy will do months or years in the future.” (Janet Yellen as Chairwoman of the Fed, 2013, in a speech at the Society of American Business Editors and Writers’ 50th Anniversary Conference)

These quotes capture how quickly the profession pivoted from mystique to message management. Communication can be used in the following ways against the threats:

• Markets now equate independence with a standing promise to backstop every wobble. However, because expectations, not statutes, set the speed-dial to the central bank, liquidity support with clear talk can become a rule-based product with a clear menu and calendar, not a vibes-based hotline. Independence should be framed as predictability, not generosity.

• Climate, inequality and digital money have entered the central bank agenda, but mandates stay monetary. Silence looks negligent; activism looks mission-creep. Therefore, clear narratives draw the mandate perimeter to deal with societal pressures.

• Crises and uncoordinated fiscal action force ad-hoc actions; every improvisation invites the charge that independence is melting. Central bank communication can enable that before the swap line, the story of the swap line is told. Narrative discipline is the cheapest coordination architecture that can build overnight: coordination is pre-narrated and therefore feels rule-bound, not improvised and critics can see exactly where the firewall stands.

Let me also be a bit controversial and suggest the following: 2025 demands the next central bank communication pivot: real‑time vibe management enabled by large‑language models. LLMs can be used to carefully curate these messages and offer real-time indicators of economic variables. I would suggest:

• Real time indicators of vibes rather than delayed responses. Continuously scrape speeches, social feeds, and headlines; translate them into sentiment indices that update as fast as markets themselves.

• Use LLMs to vibe test messages and improve the clarity before they're public. By repeatedly generating multiple alternative phrasings, the LLM can detect if a message consistently triggers negative sentiment or misunderstanding. If the model flags that most phrasing variations provoke negativity - say, over 90% - the message is refined or rewritten entirely, ensuring clarity and preventing unintended negative reactions before reaching the audience.

• Use LLMs to offer and scale up differentiated yet targeted unitary messages. By extending the database to 119 central banks in this thesis, I uncovered communication logics that diverge sharply from G7 norms. Nigeria’s 2024 diaspora‑bond roadshow, sold in New York to stabilise remittance flows, would be inconceivable at the Bank of England. LLMs can help see these differences.

Thus, instead of bureaucracies running after the facts and banning the use of LLMs, they could be used (with caution) to their full advantage! In-house AI units should sit inside the bank, not outsourced to defence contractors or megacorps. Obviously, it also is important that humans remain in the loop comprising economists, behavioural scientists, and linguists - ensuring that capable technocrats remain the final voice. And yes - I’m being provocative and have made various warnings about the use of LLMs in previous posts (e.g., while LLMs offer powerful tools for rapidly analyzing vast amounts of text, their inherently probabilistic nature means results can vary between runs, potentially introducing subtle inconsistencies or biases. Thus, careful validation and transparency are essential to ensure the reliability and reproducibility of findings derived from these models.)

But, overall, the good news is that the same tools which exposed the drift also show how to reverse it. None of the above requires reopening constitutional texts. The tools already reside in central‑bank press rooms and GitHub repos. The task is operational will: to treat narrative discipline as important as the policy rate. Independence was never meant to be absolute; it was meant to be functional. By refocusing communication, central banks can renew the political bargain that underpins their autonomy.

Full papers, code and dashboards at centralbanktalk.eu (Tentative PhD defence date: 7 October)

Subscribe now

Share

Now that I have some attention drawn to this topic, it’s time to move past headlines and into nuance. I have now spoken to practitioners with way more expertise and knowledge than myself. Out of those conversations come five nuance-heavy take-aways: when response rate doesn’t tell the whole story, why length hurts more than bots, how cash incentives plateau, what panel conditioning really does, why volume itself is a risk, and how modern bot-detection is catching up.

Let’s unpack each one, debunk a few myths along the way, and end with fixes that hopefully come to use or might somewhat inspire. Stick with me; the details matter.

Nuance 1: Probability vs non probability sampling response rates

Something I somewhat neglected in the previous Substack is the distinction between probability and non-probability sampling - a crucial starting point for surveys! Probability panels start with a random address or phone draw; whereby only a small share (in the case of Pew their panel averages 3%) - the recruitment rate - say yes. Specifically, researchers draw a sample of addresses or phone numbers from a complete frame (e.g., every residential address in the country). Because the draw is random, everyone in the population has a known, non-zero chance of being contacted. If you multiply the recruitment with the average wave completion and retention rate over time you have your cumulative response rate which is often very low.

Opt-in panels flip the model: the sample is self-selected from day one, so per-wave completion can exceed 70%. Opt-in samples over-collect “easy” groups (high education, high broadband) and miss lower-income and limited-English households - but on paper these non-representative samples have high response rates.

Okay, so raw response rates do not tell us the whole story it seems, but should we care? Fluent, broad attitudinal measures (e.g., brand tracking) often tolerate some error - especially if they field daily and look for relative shifts. But if you need a point estimate that will drive funding or seat allocation, you still want a probability backbone. Especially, if that bias lines up with your outcome (say, political engagement), weights won’t rescue you no matter how hard you try and how high your response rate is!

Let’s look at some data. Using Data of Pew’s 2023 panel comparison which benchmarked each sample against 28 gold-standard population margins, we can observe strong differences in the absolute mean error rates between non-probability samples and probability samples. Even after weighting and calibration we can see that there is double the amount of error in the non-probability samples.

The rules of thumb I would use? If you need trend precision (e.g., approval ration of plus/minus 2 percent), you should definitely stick with probability or hybrid in which you have a probability seed and add a touch of Bayesian blending. If you need segment discovery and speed? Go with large non-prob samples as long as you occasionally validate with probability pules. Want to reach a hard-to-reach niche? start with non-prob sampling oversampling the niche and make sure to do some post hoc propensity matching to a small probability check.

What do industry players actually do? To give a few UK based examples I am familiar with:

• YouGov admits selection is non-random, but argues its sampling-plus-modelling stack (stratified quotas, sample-matching, and finally multilevel regression-with-post-stratification) recovers population truths at a fraction of the cost and in 24-hour field windows. The trade-off: larger mean absolute error versus benchmarks (≈ 4½ percent vs. 2 percent for Pew’s ATP) and heavier reliance on model assumptions if the world drifts.

• The ONS is the archetype of classical probability work: systematic address sampling, five follow-up waves, and expensive interviewer call-backs. It delivers sub-percentage precision on employment but suffers from falling co-operation (wave-1 down ~16 percent in a decade), forcing costly sample boosts and rapid-redesign projects.

Nuance 2: Survey length reduces data quality more than bots do.

Hoerger (2010) found that any online survey loses about 10% of respondents almost immediately, then sheds another 2 percentage points for every 100 questions that follow. Peytchev (2009) sharpened the point: an open-ended text box makes a respondent 2.5 times more likely to quit.

The survival-curve below shows what that looks like in practice: even with a perfectly human sample, a 200-item form shaves the effective N from 100% to ~86%. That 14-point hit translates straight into wider confidence intervals, heavier post-strat weights, and - when designers over-correct - bias creep that no bot-detection filter can fix.

So what to do? Avoid ruthlessly prune grids and replace them with short, single-topic items or card style questions could be a starter. Swap open-ends for quick taps (a single option comment box is often more than enough!), and move low-priority psychographics to re-contacts. Keeping the core survey under 50 items often delivers more usable data than the fanciest respondent-validation stack.

In short, bots may grab headlines, but question overload does more everyday damage. Trim the form first; fight automation second.

Nuance 3: Cash bumps response, but costs climb faster than budgets.

I mentioned in the previous Substack that respondents should perhaps be paid more, I now looked into this a bit more. The idea behind prepaid cash is straightforward: include money with the invitation and people feel obliged to reciprocate. The instinct is simple: tape more cash to the invite → nudge more people to answer.

Yet the latest large-N experiments show we’re already into diminishing-returns territory. The chart below uses the 2022 Survey of Consumer Finances randomized trial (around 8,600 sampled households). Moving the enclosed bill from $5 → $10 buys you a visible bump (6.5% → 8.6% completions). Doubling again to $15 barely matters (9.3%) – you add three dollars to get just 0.7-point extra reach. This is the case while the budget impact can be immediate: most agencies budget incentive lines months in advance. If a study scales from 5,000 to 50,000 addresses (common for labour-force surveys) an unplanned shift from $5 to $10 can add $250k overnight.

My advice? Obviously it heavily depends on the task at hand, but overall, prepaid cash works, but $1–$5 works best. Above that, you are spending faster than data quality improves - and drawing budget and compliance risk in the process. You can also boost under-represented strata by layering post-paid bonuses targeted to strata after survey completion for a fraction of the cost. Also perhaps to avoid “professional respondent” churn: replace straight cash with a lottery or charity top-up once base response is stable. This maintains reciprocity signal without steady escalations.

Btw, combining nuance 2 and 3 there seems to be a trade-off between making the study longer and paying more. Since the costs of the study grows rather linearly (Hoerger’s rule) and the returns to give higher incentives seem to be concave (the SCF experiment): there’s an unique interior minimum that can be reached - the “sweet spot” where an extra dollar or an extra question makes the cost per usable interview worse, not better! I’ll leave it up to some economist with extra time to calculate this and fit the real parameters though;)

Nuance 4: panel conditioning - when the sample starts gaming the test

The longer the same people stay in a panel, the more their answers change - not because their views shift, but because they adapt to the questionnaire.

The heat-map below shows how answer quality erodes as the same people answer successive waves. Heavy responders dominate panels - and their shortcuts, not bots, are bending your data. The straight-lining rate is the share of grid questions where a respondent simply clicks the same column all the way down (e.g., “Strongly agree” × 10). It’s a classic fatigue/shortcut signal - not a measure of learning. In panel studies it climbs because frequent respondents “remember the drill” and speed-click.

• In the German PaCo project (2024), straight-lining - ticking the same box down a grid - creeps from ≈ 5% at wave 1 to 17% by wave 12.

• Pew’s U.S. experiment adds nuance: knowledge-based questions inflate by +7 pp after just four waves, yet attitudes drift only +1 pp - thus conditioning is topic-sensitive.

My advice would be to rotate sub-samples or insert ‘rest’ waves so the same people are not answering, down weight serial responders or add practice items that absorb their learning without contaminating target questions and randomise the grid order and limit grid length, breaking the pattern discourages auto=clicking.

Frequent respondents are valuable, but only if you blunt the conditioning effect. Otherwise the panel starts gaming the test - and the bias belongs to you, not the bots.

Nuance 5: bot-detection is getting smarter (and more layered)

In the previous Substack, I showed how easy it was to set-up a bot yourself - even I can manage with some vibe coding. But, for almost every new trick that me and GitHub Copilot together invent, the major survey platforms now field at least one counter-measure - and the evidence shows it works to identify bots like mine.

To give some examples, a few platform-level defences have already emerged:

• Qualtrics pipes every respondent through Google’s Invisible reCAPTCHA v3 and stores a Q-Recaptcha. Scores below 0.5 are automatically flagged as “probable bot” and can be filtered in real time. Internal benchmarks on large B2C trackers put the share of completes that trip this flag at 1–2%, with a false-positive rate below 0.1%.

• Prolific relies on a multi-step identity gate: e-mail, SMS, and photo-ID verification at sign-up; IP/ISP and VPN checks on every login; a short, hand-graded writing task before a new account sees live studies. The company reports that fewer than 2% of would-be participants ever make it past onboarding but are later removed for quality.

• SurveyMonkey stress-tested its own system by unleashing a ChatGPT-powered bot across an incentivised sample. The bot could evade simple honeypots and “speeding” flags, but was eventually caught by the platform’s Build-with-AI response-coherence engine plus duplicate-ID checks; none of the synthetic completes reached the final dataset.

The graph above also shows some other evidence: a 2024 Frontiers in Research Metrics & Analytics study evaluated 31 separate fraud indicators - from keystroke timing to semantically-aware text entropy - on two California agriculture surveys. Their best six-indicator ensemble reached 96% precision and 0.92 AUC, ultimately flagging about 8% of otherwise “clean” completes as fraudulent. Crucially, the authors show that no single indicator clears 90% recall.

Besides these established platforms, many start-ups have also entered the field specializing on identifying bots - I briefly talked to the founders of a start-up called Meaningful and get a chance to go through their platform - it looked very promising!

Some tips? Layer your screens. CAPTCHA → IP blocklists → latent-semantic checks catches more than any single rule. Also, keep raw paradata: millisecond timestamps, focus-change events, and typing cadence give auditors the evidence needed to justify exclusions. Perhaps a budget for manual review wouldn’t be a bad idea either. In any case, update the mix every quarter. Bot tactics now evolve almost as fast as LLM APIs; what works this spring may leak by autumn.

In short, sophisticated detection is already here and demonstrably effective, but only if we treat it as a living part of survey operations rather than a one-time checkbox.

The number of surveys are increasing, so more can also go wrong…

The figure above shows that sheer volume of surveys is exploding. The blue series tracks every individual survey question ever logged in the Roper iPoll archive. Since 2010 they have gone from roughly 600 000 to 1.1 million questions in field - an +83% jump. Further, industry money is keeping pace. The orange line uses ESOMAR’s annual global revenue totals for market-research (all modes). Turnover rises from $69 bn to $140 bn (+71%) over the same window. So budgets are there to keep asking - but not necessarily to keep fixing the quality problems created by that extra volume.

Naturally, the error surface scales with traffic. Every additional questionnaire adds potential for:

• Non-response - more invitations means more ways to overload panels or email lists.

• Bot infiltration - when payouts and route IDs multiply, so does the attack surface.

• Measurement drift - panellists see more waves, triggering the conditioning you saw in the previous heat-map.

• Weight-instability - post-stratification matrices that worked at 600k items may buckle at 1 million because minor groups fragment across more surveys.

In sum, yes the problem is real. Yes, there is ‘enshittification’ of surveys. But let me be the typical academic and point out that there are also many nuances and solutions: I just pointed out 5 of the many. Do reach out and share this if you interested to hear more or want to follow-up on any of these ideas.

Share

Subscribe now

But there is a fundamental problem: fewer and fewer people are answering - and more and more of those who do are AI agents.

I explore these two converging trends below. Then, I’ll show that anybody (including me) can easily set-up an AI agent to earn some money with taking surveys. I’ll then estimate the impact of this further down the line in three main fields and propose some solutions.

Problem 1: The increase of non-response rates

If you use survey data, it probably hasn’t gone unnoticed: survey response rates have plummeted. In the 1970s and 1980s, response rates ranged between 30% and 50%. Today, they can be as low as 5% .

To give some (shocking) examples: the UK's Office for National Statistics (ONS) experienced a drop in response rates from approximately 40% to 13%, leading to instances where only five individuals responded to certain labor market survey questions. In the US, the current population survey dropped from a 90% response rate to a record low of 65%.

Below, based on three sources, I slightly extrapolated and created a continuous graph displaying the decline.

Problem 2: The increase of AI agents

How difficult is it to build an agent? So… I did what any overcaffeinated social data nerd would do. I built a simple python pipeline for my own AI agent to take surveys for me (don’t worry I promise that I didn’t actually use it!). The pipeline I built just requires me to:

• Access to a powerful language model (I just used OpenAI’s API - but perhaps for research representativeness of the distribution an uncensored model is way better!).

• A survey parser: this can be as simple as a list of questions in a .txt file or a JSON pulled from Qualtrics or Typeform. The real pros would scrape the survey live though!

• I prompted it with a persona. The easiest is to built a mini “persona generator” that rotates between types: urban lefty, rural centrist, climate pessimist, you name it.

Overall how long did this take? Not too long at all, the most difficult and time consuming part is making it interact with the interface of the survey and tool/website.

That’s it. With a bit more effort, this could scale to dozens or hundreds of bots. Vibe coding from scratch (see my previous Substack on how to do vibe coding ) would work perfectly too.

Don’t worry btw, I didn’t deploy it on a real platform. But other people did. Below, I extrapolated the trends of AI agents based on data points in existing research since data is very hard to find (if someone has some better data (e.g., on the before and after the introduction of ChatGPT), please share!!).

Downstream problems

Let’s explore how this impacts three main fields in which surveys are used: political polls, market research and public policy.

Political polls. Many polls depend heavily on post-stratification weighting to correct for underrepresentation in key demographic groups. But when response rates fall and LLM answers increase, the core assumptions behind these corrections collapse. For instance, turn-out models become unstable: if synthetic agents overrepresent politically “typical” speech (e.g., centrist or non-committal), models overfit the middle and underpredict edges. Similarly, calibration failures increase: AI-generated responses often mirror majority-opinion trends scraped from high-volume internet sources (like Reddit or Twitter), not the minority voter. This results in high-confidence and stable predictions that are systematically biased.

Market research. AI-generated responses are, by design, probabilistic aggregations of likely human language conditioned on previous examples. That’s great for fluency and coherence, but not good for capturing edge-case consumer behavior. Real customer data is heteroskedastic and noisy: people contradict themselves, change preferences, or click randomly. AI, in contrast, minimises entropy. Synthetic consumers will never hate a product irrationally, misunderstand your user interface, or misinterpret your branding. This results in product teams building for a latent mean user, resulting in poor performance across actual market segments, particularly underserved or hard-to-model populations.

Public policy. Governments often rely on survey data to estimate local needs and allocate resources: think of labor force participation surveys, housing needs assessments, or vaccine uptake intention polls. When the data is LLM generated this can result in vulnerable populations becoming statistically invisible and lead to underprovision of services in areas with the greatest need. Even worse, AI-generated answers may introduce feedback loops: as agencies “validate” demand based on polluted data, their future sampling and resource targeting become increasingly skewed.

So what can we actually do about this?

Unfortunately, there’s no silver bullet (believe me - if there were, my start-up dream would be reality and I’d already have a VC pitch deck and a logo). But here are a few underdeveloped but in my humble opinion promising ideas:

1. Make surveys less boring.
We need to move past bland, grid-filled surveys and start designing experiences people actually want to complete. That means mobile-first layouts, shorter runtimes, and maybe even a dash of storytelling. TikTok or dating app style surveys wouldn’t be a bad idea or is that just me being too much Gen Z?

2. Bot detection.
There’s a growing toolkit of ways to spot AI-generated responses - using things like response entropy, writing style patterns or even metadata like keystroke timing. Platforms should start integrating these detection tools more widely. Ideally, you introduce an element that only humans can do, e.g., you have to pick up your price somewhere in-person. Btw, note that these bots can easily be designed to find ways around the most common detection tactics such as Captcha’s, timed responses and postcode and IP recognition. Believe me, way less code than you suspect is needed to do this.

3. Pay people more.
If you’re only offering 50 cents for 10 minutes of mental effort, don’t be surprised when your respondent pool consists of AI agents and sleep-deprived gig workers. Smarter, dynamic incentives - especially for underrepresented groups - can make a big difference. Perhaps pay-differentiation (based on simple demand/supply) makes sense?

4. Rethink the whole model.
Surveys aren’t the only way to understand people. We can also learn from digital traces, behavioral data, or administrative records. Think of it as moving from a single snapshot to a fuller, blended picture. Yes, it’s messier - but it's also more real.

No single fix will solve this overnight. But a combination of better design, smarter detection, fairer incentives and creative methods can help. Having said that, surveys luckily aren’t dead yet and the good survey companies and researchers are finding creative (sometimes old-school) ways around these problems. If we want surveys to survive the twin identified threats, we need to collectively put full effort into increasing data quality.

Share

Subscribe now