Author's note: In 2023 I wrote AI Isn't Coming for Tech Jobs—Yet for Asterisk, along with a companion piece laying out the model. The core thesis held: Whatever the disruptions of AI, the number of tech jobs grew on net. My simulations suggested a median of 5.1M jobs in Computer and Mathematical Occupations by 2025. BLS came in at 5.26M. I still think AI will create more jobs than it eliminates in the near term, but I am less certain now than I was then, and my uncertainty grows the further out I look. This article is my best guess at why we will not see mass unemployment in white-collar work before the end of the decade.

On July 2, Scott Alexander published “The AI Superforecasters Are Here.” Humans had taken the top two places in a recent tournament and an AI system came in third. Given the luck involved, Scott called the best humans and AIs “a statistical dead heat.”

A week later, I saw an announcement from a platform where they paid top forecasters to tackle policy-relevant questions that they were closing down the platform and moving to a project-based model.

While they didn’t mention AI in the announcement, you don’t have to be a superforecaster to suspect it might have had something to do with it.

So if AI can replace forecasters, why won’t other jobs follow? Looking at why forecasting work is being disrupted is instructive.

Forecasting Was Never the Bottleneck

Give a group of top forecasters a clear question and they quickly converge on a reasonable probability, but forecasting didn’t catch on as part of organizational decision-making because it requires the client to do the hard part of figuring out the right questions to ask in the first place.

An executive wants to know: “Will the war spread?” A forecaster needs a precise question with a deadline and resolution criteria. Unfortunately, making the question resolvable can move it away from the thing the executive cares about. This might be potential shipping disruptions. They could be caused by a blockade, an invasion, or a cyber attack. Writing the right question requires understanding the decision it is supposed to inform, the options the organization has, and what information would cause it to change direction. Often, no one person knows.

Bringing in top forecasters creates work and mental overhead. It’s like hiring an intern for a job you do not know how to perform. By the time you specify the task, provide the context, and explain what a useful answer looks like, you might discover that you have higher value options than doing the essential pre-work to make the task possible in the first place.

Automation Moves the Bottlenecks

Unlike forecasting, most jobs unblock bottlenecks.

Consider radiologists. Even if AI interprets scans perfectly, someone still has to resolve ambiguous findings, communicate results, and take responsibility for the report. The job shifts toward the decisions and exceptions that remain.

Software follows a similar pattern. Producing code is only one step between an executive saying “improve the customer experience” and customers using a working product. As code gets cheaper, requirements and organizational approval become the constraints. Developers spend less time writing code and more time clearing whatever blocks the project.

Jobs are bundles of tasks built around bottlenecks. Automating one task may reduce headcount, increase demand, or both. It does not remove the constraint so much as reveal the next one.

Human Data Warehouses

Most organizations are like giant, inefficient databases whose most important information exists only in people’s heads.

Consider this hypothetical business situation:

You need the Watkins contract? Debbie worked on it last year. She knows the person at the client who can approve a change. Do not use the official portal; nobody checks it. Debbie gets too many emails, but if you walk by her desk she will tell you everything.

The portal is the official process. Debbie is the actual process.

Management theorists call this the garbage-can model: organizations are not machines translating goals into action. Problems, solutions, people, and opportunities slosh around until some combination produces a decision.

Even perfect requirements elicitation from AI does not solve this. Senior leaders are often “big picture” people. They say “improve the customer experience,” delegate the project, and move to the next meeting. They are not going to spend an afternoon playing twenty questions while an AI converts that direction into a specification.

The people willing to answer may not have authority. The people with authority may not know the constraints. The requirements may not exist in any one person’s head, but only emerge when legal, engineering, and the client react to one another. To replace a human role, AI needs to decide who to overrule, when to contact clients, and be trusted to take action when there’s a real risk of being wrong. Organizations built around human accountability will be slow to delegate those powers.

Sometimes that friction protects the organization from its leaders. Imagine leadership decides to raise prices on the company’s largest client. The contract has renewed automatically for a decade because nobody at the client company has to think about it. An employee knows changing the price would trigger a client review, and the client would likely discover that competitors are better. An employee who slow-walks the increase until its too late to make the update and management moves on might save the business. On a process map, that employee looks like an obstacle. In reality, they’re part of the organization’s immune system.

Conclusion

To replace a human role, an organization needs to give AI the appropriate context, access to the right people, permission to make tradeoffs, and the authority to act

AND

the right people will have to respond when AI requests things from them.

Most organizations are not even remotely set up to do that.

This is why AI may transform millions of jobs while eliminating very few of them.

QueensLink would have been a signal of confidence in New York City's future. By reactivating an abandoned rail corridor, it would have connected southern Queens to the subway and complemented the future Interborough Express, giving Queens something closer to the network Manhattan already takes for granted. That network is why I supported it: every station added to a connected system creates value beyond its own ridership, because it links to every other station in the city.

In May, the Mamdani administration, which had supported QueensLink during the mayoral race, told advocates it now supports the alternative park proposal, QueensWay, which is expected to break ground by year's end. Instead of restoring rail, the corridor would become something closer to the High Line, with no transit component beyond a bike path.

I wanted the train, but, after building a model for a retreat about the next 100 years of NYC, I think I understand, at least from a risk-minimizing / pragmatic point of view, why the administration came to this decision. Rail on this corridor means three to six billion dollars and probably more than a decade before a single rider is served. Any mayor who starts the project will experience only the headaches from construction delays / rising costs and wouldn’t see any of the benefits during their administration. They would take all the blame and a future mayor would be the one at the ribbon cutting.

Fortunately, the choice doesn’t have to reduce to rail-or-park. A third option captures much of what QueensLink promised, including a real share of that network value, at a fraction of the cost, on a timeline measured in years instead of over a decade, and it is the only option that keeps the door open for rail later on.

Under the model’s default assumptions, rail eventually serves more people per year than a busway alone, but it loses a crucial decade before opening. A busway-plus-bike corridor begins serving people in years two to four, costs roughly $145M rather than billions, and remains competitive with rail on discounted person-years even over a century. Its advantage is not that buses are better than trains. It is that a preserved right-of-way can deliver useful transportation now while keeping the rail option alive.

That is the real shape of the choice. A busway on a preserved right-of-way is a rail line waiting for its ridership case. A park forecloses it, because cities do not un-build beloved parks. The busway is the only option that doesn’t require a correct forecast about southern Queens: it wins if the city stagnates and keeps rail open if it booms. The park works only in futures where this corridor never grows, and if that is the city’s bet, it should be made in the open rather than arrived at by default.

The ask is that the park the city is about to build keep a transit envelope in its design: the busway now, and the option for rail conversion later. I still believe New York will eventually need trains on this corridor. But the fight worth having this year is smaller and winnable. Before ground breaks, the administration should commit to a QueensWay that moves people as well as hosts them. That is a decision a pragmatic mayor can make in one term and be thanked for over the next hundred years.

After arguing about AI at LessOnline last weekend, I want to register my beliefs.

The short-timelines people are seeing something real, but the coming years probably won’t manifest as a rapid AI takeoff with an explosion in GDP.

More likely we’ll see uneven automation with rapid progress in some domains and failures in others paired with a steady accumulation of changes that make ordinary life materially better.

GDP will remain within historical norms1 until 2030, but it will understate the transformation because many of the most important gains will show up as time saved / pain avoided / risks reduced / choice expanded / everyday friction removed.

The economy will grow, but welfare will grow faster.

AI Remains Spiky

AI progress already looks spiky, and I don’t expect that to change in the next few years.

The same systems that can one-shot a working app or find a needle in a haystack of data can still fail at tasks that look insultingly simple. Ask for a map of all U.S. states with the letter “R” in their name and you get a miniature version of the whole problem.

Not even a year ago, models produced maps full of hallucinated state names and obvious false positives.

They’re a lot better now, but still struggle with reliability. When I tried it today, the result was almost right, but it still missed North Carolina.

It’s not that AI can’t ever get this right.

Even in the present, there’s a workflow that works much more reliably: run a deterministic program to look up the fifty states, filter for names containing “r,” then generate the map from that verified list.

In this case, however, the system chose an unreliable method. Rather than handling the problem with its strengths, I can see from the chain of thought it went through multiple rounds of guess-and-check only to still come up short.

Although it’s gotten better, we still can’t rely on AI for tasks like this yet. Even after it improves again so that it works most of the time, and even with more robust scaffolding, it will still need someone to double-check the final output until we get a few more 9s of reliability.

This is an area of rapid improvement, but there are many areas like this at various capability levels that are all progressing at different rates and the point is, I don’t expect them to all converge to human level in the next few years.

The final nines can be the difference between “nifty” and “useful” and it’s why I expect AI to keep transforming work before it cleanly replaces existing systems. The capability will be there in pieces. The hard part is making the system choose the right tools, check its own work, and know when to call for help.

AI will continue to be superhuman in more and more domains, useful but unreliable in many, and still weirdly fragile in others. The next few years will be a jagged frontier: astonishing successes next to embarrassing failures.

No Takeoff Required

I still expect the coming years to be transformative. AI does not need to become perfectly reliable to make life much better. It only needs to make enough things easier.

Some improvements will be directly AI-driven: faster software, better tutoring, cheaper content creation, more useful search, better medical triage, more capable small teams, faster customer service, more automated compliance, and less paperwork. Others will come from technologies that AI helps accelerate or coordinate: autonomous vehicles, better logistics, drug discovery, personalized medicine, fraud detection, and more public administration capacity.

It won’t be that everything suddenly becomes automated; friction will slowly melt away.

Autonomous transport is the easiest example. It can start city by city and still return time, reduce crashes, expand mobility, and lower costs.

Health is similar. Although not from advanced AI, GLP-1s point to the kinds of improvements that are possible. They’re not just weight-loss drugs. If the category keeps improving, it could help with multiple issues: diabetes, sleep apnea, joint pain, and cardiovascular disease.

GDP may capture some of that through higher labor-force participation and lower downstream medical costs, but the real benefit is mostly uncaptured: people feel better and live longer.

GDP Undercounts Welfare

The internet changed daily life more than GDP suggests.

The late 1990s saw a real productivity acceleration, but the full welfare gain from email, maps, online shopping, digital media, and instant access to information was larger than what showed up in the national accounts.

The coming years may rhyme with that, but more broadly. The internet transformed information, communication, media, and commerce. AI touches those, but also reaches into many more areas like software, medicine, transportation, and research.

Still, there are good reasons not to expect GDP to go vertical.

Humans will remain in the loop longer than the demos suggest. Some capabilities will remain unreliable, and much of the surplus will show up as lower prices, free services, better coordination, time saved, and pain avoided rather than as measured output.

So my take is the actual improvement in daily life will be larger than measured growth implies.

Suppose you are bitten by a rattlesnake. Would you rather be treated by a medical toxicologist, or by a faith healer who chews up herbs and spits them into the wound?

Because snake bites usually aren’t fatal, most patients survive either way. In noisy domains, almost anyone can claim some surviving patients. The question is not whether you can survive a bad process. The question is which model actually improves your odds.

In public reasoning, we choose the faith healer constantly. Most people form beliefs using heuristics like narrative fit or social identity that were good enough for natural selection. This might work for navigating social environments, but it isn’t suited for developing accurate world models. Without external feedback loops, it’s too easy to reinterpret contrary evidence and explain away failures. The deeper problem is that most people have no reliable mechanism for even noticing when they’re mistaken.

The Exam

A story can be crafted to explain anything after the fact, and a theory that doesn’t make falsifiable predictions can survive indefinitely. Forecasting brings the right incentives for beliefs to move from trying to sound good to trying to be predictive.

When I’m teaching, I want my students to do well, but I also want them to comprehend the material. I can’t just give them the exam key to study from or most of them would just overfit to the test. The purpose is to find out if they understand the material well enough to solve new problems.

World models should be judged the same way.

A model that only explains the past may be nothing more than a story fitted to the answer key. A model that anticipates new cases has captured something real. Forecasting demonstrates whether a world model holds up outside the training set.

Forecasting is an exam for world models. A strong track record is compressed evidence about whose beliefs are disciplined by reality. Top forecasters have demonstrated that their expectations are aligned with how the world actually is.

Top Performers

If forecasting is the exam reality gives to world models, forecasters who consistently outperform the crowd are the students at the top of the class.

Marginal advantages matter. An investor does not need to beat the market every day to have skill. Similarly, a good forecast usually isn’t “the crowd is at 20%, but I alone know it’s 100%.” More often, the advantage looks like slightly better calibration over time.

Forecast scoring separates people with reality-shaped models from the rest. These people have demonstrated one of the rarest and most valuable intellectual skills: the ability to form beliefs that reality vindicates.

The Leaderboard

Building a meritocratic leaderboard was the promise of forecasting platforms.

They would identify people with unusually good judgment, measure their performance over time, and make that talent useful. The idea was to discover people whose beliefs repeatedly made contact with reality, then route them toward decisions where reality-contact mattered.

On the first part, forecasting platforms largely succeeded.

They showed that probabilistic judgment is not just noise. Some people consistently outperform others. They are better calibrated, better at using base rates, better at updating, and less likely to let a preferred story dominate their expectations. Across many questions, these small advantages compound into a meaningful edge.

The Opportunity

The ideas from the forecasting community have started to take hold, but the market for truth is still smaller than the market for reassurance, and many of the most important concepts haven’t been widely adopted yet.

In many cases, the faith healer model limps along simply because of inertia. When you combine entrenched institutional advantages with the natural appeal of a good narrative and drop it all into a noisy world, the faith healer can survive a very long time before anyone starts questioning their track record.

A compelling story will always be easier to absorb than a calibrated probability. But that blind spot is exactly what makes the opportunity so valuable.

Forecasting platforms have identified people whose beliefs have been repeatedly tested against reality. The next step is to route that judgment toward decisions where being wrong is expensive, giving organizations access to something public reasoning usually lacks: a measured record of who is more likely to be right.

About two weeks ago, my friend Varun threw a farewell party. He’s moving to the Bay Area. The apartment was full of researchers, founders, and the kinds of people who get it when it comes to how much AI is about to turn everything on its head, the kind of people where every conversation makes you think I want to talk to this person again.

I left the party with a familiar feeling. I’d just spent an evening with exactly the kind of people I want more of in my life, and I knew that, in all likelihood, I would never see most of them again.

We exchanged a few numbers, sure, but I’ve done this enough times that I know follow ups are rare and when contemplating the cognitive overhead of coordinating two people’s scheduling and location preferences, I usually just file it under the “I’ll think about this later when I have bandwidth” which usually never happens.

This time though, I wanted to try something different. This felt like a software-shaped problem, so, a few days later, I started building.

Epistemic Density

There’s a reason I live in NYC and not the Bay Area. I like having at least some friends with interests outside of technology. But I also sometimes want a particular kind of conversation. I have Claude, ChatGPT, and Gemini subscriptions and actually know what the latest models are capable of and how fast they’re improving. I’m building forecasting bots, experimenting with agents, and working on research. I spend my days thinking about how the world is about to change, and I want to talk about it with people who are thinking about it too.

People talk about the Bay Area as the place to be for this, and there’s something to that, but I think it’s less about geography and more about what I’d call epistemic density: the odds that the person next to you at a coffee shop has updated on what AI means for everything.

In New York, I’m surrounded by brilliant people, but most of them don’t spend their weekends building agents. That’s fine. They have rich lives and I appreciate the conversations I have with them, but I also want conversations where I don’t have to start from scratch explaining why this matters.

My friend Ben Turtel, who runs an AI startup here in NYC, put it perfectly: his friends treat the things he says about current reality as predictions… that are wrong! That’s my experience too. I want to talk to more people who “get it”, but, right now, it feels like they’re scattered around the city and I don’t know how to find them. Maybe they’re not so scattered though. Maybe they’re just around the corner and we just haven’t found each other yet.

Closer Than You Think

Steve Jobs said “If you are willing to travel around the world to meet a teacher, one will appear next door”.

The people I want to know are often hidden in plain sight. Varun lived five minutes away, yet we were strangers until we met at a Dwarkesh Patel party. Another friend lives even closer, we’d known each other for years, but we only started getting coffee in the last few months after discovering we lived in the same neighborhood at a conference on the other side of town. I have multiple other NYC friends who I only really got to know from events in the Bay Area.

The irony is painful: I travel to niche events to meet people living on my block.

Reducing Friction

The concept was like Calendly, but for coffee meetings. I started working on coffeemeet.xyz the next day, made it live after a few days of testing, and by the following weekend, I was already meeting someone who booked time with me.

The concept is simple: I pick my favorite coffee spots, I set when I’m free, and anyone can book a time to meet me. No back-and-forth texting. No “where works for you?” Just pick a spot, a time, and show up.

For my profile, I picked coffee shops nearby. This isn’t because my time is more important than anyone else’s. It’s because between work, teaching, forecasting, and life with my partner I don’t have many free time slots. If meeting someone new requires an hour-long commute, I know from experience that “let’s do this again” turns into “maybe next month” turns into nothing. The friendships that actually develop are the ones where getting together is easy. Varun was five minutes away and we still only managed coffee once or twice a month. Proximity is what makes consistency possible, and consistency turns conversations into friendship.

Software-Shaped Problems

Here's the thing: the gap between an idea and a working tool has collapsed. Friday night, it was just a feeling at a party. A few days later, it was an app people were using.

I could have built coffeemeet without AI, but “could have” and “would have” are different things. AI didn’t spit out a completed app from a single prompt for me, but it compressed a project that might have taken a couple of weeks into a few hours. And that speed changes the calculus on what’s worth building. Problems that used to feel too small or too personal are now worth a Sunday morning.

I think a lot of people are sitting on little problems like this, software-shaped holes that could be solved if they knew where to start. Not every idea needs to be a startup. Some just need a clear idea, a few hours to work out the kinks, and a willingness to ship something imperfect.

Come Have Coffee With Me

If you’ve read this far, I want to meet you. Seriously.

That’s the whole reason I built this and the whole reason I’m writing about it.

If you’re in or around Brooklyn, or willing to make the trip, pick a spot and a time on coffeemeet.xyz. I’d love to talk about forecasting, AI, what you’re working on, what you’re reading, or whatever’s on your mind.

Some of my best friendships started with a coffee. Maybe ours will too.

When I worked in government, an experienced colleague told me something that’s stayed with me: “Ever since ancient Rome, politicians who build roads win elections.” It’s an achievement people can see, and impossible for opponents to deny.

Housing reform is the centerpiece of the Abundance agenda, and the theory makes sense: America’s cost-of-living crisis stems from a shortage of housing where people need it, and the solution is to build more.

The problem is, if reforms work and rents only rise 2% instead of 6%, no one will notice. Worse, by the time housing construction translates into rent relief (probably 5 years later), someone with a different agenda will be in power. Either way, “Rents didn’t increase as much as they would have” isn’t a ribbon-cutting moment.

The Visibility Problem

Consider Obama’s stimulus tax cuts.

According to Tax Policy Center estimates, 96.9 percent of households enjoyed a tax cut that averaged almost $1,200. Just one measure—Obama’s Making Work Pay tax credit—put more than $116 billion into people’s pockets in 2009 and 2010. Yet, a Times poll found that fewer than 10 percent of those surveyed had any clue. Remarkably, fully one-third thought their taxes went up—even though the actual number was about zero. 1

The Obama administration’s theory that spreading tax cuts through withholding would be more effective stimulus than lump-sum checks was probably right on the economics, but politically, it was a disaster. It may have helped Republicans win the House in 2010, even as voters were receiving tax cuts they didn’t know about.

Housing reform faces the same problem. If rents grow 2% instead of 6%, will anyone connect it to a zoning change from eight years earlier? I doubt it.

Housing Reform’s Invisible Results

Minneapolis is the poster child for YIMBY success.

Minneapolis’ success in building new apartments has enabled the city to substantially add to its housing supply and keep rent growth low. From 2017 to 2022, Minneapolis increased its housing stock by 12% while rents grew by just 1%. Over the same period, the rest of Minnesota added only 4% to its housing stock while rents went up by 14%.2

But notice the timeline. The Minneapolis 2040 plan was adopted in late 2018, with major zoning changes taking effect in 2020. The dramatic results took years to materialize, and this is the optimistic case.

Paradoxically, abundance reforms can actually make things worse before they get better. For example, when Chicago implemented targeted up-zoning from 2013-2015, there was no discernible increase in housing construction in the impacted areas. Even more concerning, property values actually increased potentially due to developer anticipation of projects that have yet to materialize.3

Over time, housing reform will make people better off and it’s the right thing to do, but history suggests the median voter will take the results for granted.

Unlocking the Housing Shadow Supply

I heard about a family who recently moved from NYC to Buffalo because they had a remote job situation that enabled it. The median house price there is around $200K4 and houses sell for around $100K in certain neighborhoods. Ample underutilized housing exist across the country, but under today’s setup, the commutes are unworkable. Remote work is a structural reform of the housing market. The crisis is a location mismatch, not a lack of housing. Workers are being artificially forced to compete for a limited pool of housing within a commutable radius.

By unbundling work from location, we effectively bring a massive “shadow supply” of housing onto the market. When a worker moves from a high-demand city center to an under-utilized home in a smaller town or a distant suburb, they aren’t just improving their own life; they are de-pressurizing the urban core.

Counter-intuitively, this is the best thing that could happen to “on-site” workers. When the “laptop class” stops outbidding the nurse, the firefighter, and the service worker for the apartment next to the hospital, rents for essential workers actually stand a chance of falling. Remote work relieves rent competition.

The Salience of Remote Work

If the goal of the Abundance movement is to deliver visible improvements to people’s lives on a timeline that matters, we need to look beyond the slow churn of construction. When a worker gains ten hours a week back by skipping a soul-crushing commute, they notice. When a parent no longer has to juggle childcare drop-off, rush to an office miles away, and then pay for extra after-care just to cover time spent sitting in traffic, they feel it. This kind of flexibility is the most immediate housing policy we have. It delivers direct, daily relief to working families in a way new construction cannot, because it changes how people live right now, not years later.

The Real Force Behind Return-to-Office

Return-to-office mandates are often framed as business decisions: executives wanting collaboration, culture, or accountability. But there’s another force at play that rarely gets named: political pressure from city governments.

Cities like New York have built their fiscal models around commuter presence. Commercial property taxes, transit revenue, and downtown retail sales tax all depend on office buildings being full. When remote work emptied those buildings, it didn’t just hurt landlords, it threatened municipal budgets.

City officials have responded by pressuring employers to bring workers back. Former Mayor Adams was explicit about this, calling remote work a problem to be solved and urging companies to return to offices5. This isn’t subtle. It’s a policy priority.

Employers, in turn, face a choice. Some genuinely believe in-office work is more productive, and if that’s true, they’ll pay the premium in higher salaries or lost talent and the market will sort it out. But others are responding to political pressure, not business logic. They’re making decisions about where their employees spend 40 hours a week based partly on what city officials are asking of them, and the cost falls most heavily on families.

When a parent is forced back into a commute, that’s hours lost to childcare shuffling, missed dinners, and the logistical grind that remote work briefly relieved. It’s the single parent who now needs extended aftercare. It’s the father who used to do school pickup and no longer can. It’s the couple who moved further out for space and now faces an impossible commute.

These families are being used to plug the holes in a municipal revenue model.

An Opportunity for Pro-Family Politics

Some of the same officials who claim to support working families are actively supporting policies that make family life harder. The same governments that worry about declining birth rates and childcare costs are pressuring employers to impose commutes that exacerbate both problems.

This is an opportunity.

Pro-family advocates should recognize remote work as one of the most tangible, immediate quality-of-life improvements available to working parents. Unlike housing construction, which takes years, or tax credits, which disappear into paychecks unnoticed, flexibility is felt daily. It’s the kind of visible, concrete benefit that builds political loyalty.

A political coalition that positioned itself as defending family time against the demands of city budgets built around commuter dependence would be offering something real. Not a promise of future affordability, but an immediate improvement in daily life.

A City of Choice

As a resident of New York City, I love the density, the community, and the culture of urban life. But a city propped up by captive commuters is a fragile one.

The Abundance agenda should strive for a city of choice. If we stop forcing people to live in NYC just for a desk, we make it possible for the people who actually want to be here to afford it. By allowing the “unwilling participants” to exit the market, we create a more resilient, affordable, and vibrant urban core.

Housing reform is a noble long-term goal, but it needs a political shield. Remote work provides that shield. It delivers the immediate, visible “win” that buys a movement the time it needs to build the future.

When I started building a forecasting bot for the Metaculus AI Benchmark, I was spending my own money. That meant using smaller, cheaper models and squeezing out every bit of signal I could through methodology.

I built a structured pipeline that approximated how a human forecaster would approach a question. On my initial tests, this process-driven approach outperformed the raw model by a healthy margin, and it felt like I was really onto something.

Then Metaculus gave me API credits, and suddenly I could afford frontier models.

I ran my pipeline against these frontier models, expecting my clever methodology to stack on top of their superior reasoning. Instead, I crashed into what so many before me have experienced: the Bitter Lesson.

The Bitter Lesson

The Bitter Lesson is Rich Sutton’s observation that in AI, methods that scale with computation tend to outperform approaches that rely on human-designed structure or domain knowledge. My forecasting pipeline was my human-designed structure, and it didn’t add anything to what the frontier models already provided.

What the Data Showed

My process-driven pipeline, which had shown promise with smaller models, didn’t add noticeable value with frontier models. In my testing, a better use of tokens was simply polling larger models more times to build a larger ensemble, rather than spending tokens on classification, routing, or decomposition.

Why This Happens

My best guess is that, with weaker models, the pipeline helped because it imposed structure on reasoning that the model didn’t reliably produce on its own. It was scaffolding for limited capability.

Frontier models don’t seem to get much value from this scaffolding. They can reason through base rates, consider multiple scenarios, and synthesize information without being explicitly told to. The pipeline was just consuming tokens that could have been better spent on additional polling.

What I’m Doing Now

I’ve simplified. The elaborate pipeline has been replaced with more polling, and I have new theories I’m testing against the crowd. My testing suggests there may be a way to elicit latent knowledge from models that meaningfully improves forecasts, but I need more data before I’m confident the effect is real.

Testing and Validation

I used Metaculus community probabilities as a calibration benchmark. A baseline frontier model might correlate at r ≈ 0.65, while aggregating multiple frontier models correlated at r ≈ 0.74. Including the pipeline did not provide additional uplift over ensembling.

While not perfect, this provided a useful sanity check. This kind of empirical validation is what makes forecasting bots useful. We can actually test what works instead of arguing about theory.

The Lesson

I started this project trying to stretch small models with clever methods, but it turns out the winning approach was just using better ones. Sometimes the boring answer is the one that actually scales. For now, with frontier models, keep it simple: ensemble aggressively and validate empirically.

This is the fifth post in a series on building an AI forecasting bot. In previous posts, I covered the Broken Leg check (short-circuiting on breaking news) and classification and method selection (routing questions to the right forecasting approach) and decay functions (updating as time runs out).

Forecasting failures are often obvious in hindsight. A lot of them could have been caught by a brief moment of structured skepticism. Conceptually, there are two parts: a pre-mortem and a sanity check. They are related, but they catch different classes of mistakes. The pre-mortem asks “what could go wrong?” while the sanity check ensures the forecast doesn’t violate basic reasoning.

What Could Go Wrong?

The pre-mortem assumes the forecast is wrong and asks why.

What plausible unconsidered path could change the outcome?

Consider a question like:

Will Company X release Product Y by December 31?

Suppose the pipeline outputs 65%, based on past delivery timelines, current hiring signals, and executive statements. This may be directionally reasonable, but it embeds a quiet assumption: that the product is released at all, cleanly, under the same name, and under the same definition implied by the question.

In practice, questions like this commonly fail in predictable ways.

Maybe the product is canceled. Maybe the company is acquired invalidating the “Will Company X” portion of the resolution criteria. Maybe the release happens, but in a form that doesn’t meet the specification of the question.

A forecaster might try to account for many of these implicitly, but being explicit makes it less likely that something important will be left out. These adjustments are often modest, but they materially improve the forecast.

Wait, What?

The sanity check asks a different question:

Does this number make sense given what we know about the world?

Sometimes, even for a human, following a sound process can still produce obvious failures.

Consider the question:

Will it snow on July 4th, 2026 in New York City?

Suppose the pipeline outputs 3%.

It is easy to see how this could happen if the bot grabs the wrong base rate. NYC gets snow ~12 days a year ≈ 3.3%, but to anyone with lived experience, this is obviously absurd.

If a forecast implies something wildly inconsistent with ordinary experience, one of its assumptions is probably wrong. The sanity check does its job by catching and correcting obvious problems.

Completing the Pipeline

The pre-mortem and the sanity check serve different roles. The pre-mortem looks for missing paths to failure. The sanity check looks for structural mistakes that make the forecast incoherent even if no single step was obviously wrong.

If either of these fire, the forecast needs a second pass, and possibly a different approach altogether.

Once these checks are complete, the forecast is ready to submit.

This is the fourth post in a series on building an AI forecasting bot. In previous posts, I covered the Broken Leg check (short-circuiting on breaking news) and classification and method selection (routing questions to the right forecasting approach).

Now we’re in the final stages of the pipeline: taking a raw method output and adjusting it to account for how probability decays over time.

The Problem

Forecasting methods typically output a single probability: “There’s a 30% chance this happens before the deadline.” But that number hides an important question: how is that probability distributed across the time window?

A 30% chance spread evenly across two years is different from a 30% chance concentrated in the final month. If we’re six months in and nothing has happened, these two forecasts should update very differently.

This is where cumulative probability comes in. The method output represents the total probability that the event occurs somewhere in the window. As time passes without the event occurring, we consume portions of that probability. The decay function tracks how much probability remains based on where we are in the window and where the probability mass was concentrated.

Decay Pattern

The decay classification portion of the pipeline buckets questions into a decay pattern based on how probability is distributed over time:

Constant Hazard

Some events could happen at any time with roughly equal probability. Earthquakes might fall into this classification. The chance of a major earthquake in Seattle next month is about the same whether we’re asking in January or December.

For constant hazard events, probability decays at a steady rate as time passes. Each month that passes without the event consumes a proportional slice of the cumulative probability. If 25% of your time window has elapsed without the event, you’ve consumed roughly 25% of the original probability.

Increasing Hazard

Some events have their probability concentrated near the deadline. The classic example is a debt ceiling negotiation. Neither side wants to blink early since maximum leverage comes from holding out. Maybe 10% of the probability is spread across the first eleven months, 35% is in the final two weeks excluding the last day, and 55% is on the last day before default.

For increasing hazard events, probability decays very slowly at first. Early time passing without the event is expected—you’re not consuming much of the cumulative probability because there wasn’t much probability there to begin with. If eleven months pass without a deal, you’ve only burned through 10% of the total probability mass. As the deadline approaches and you enter the high-probability window, the decay rate accelerates.

Decreasing Hazard

Some events have their probability concentrated early in the window. The classic example is a ceasefire. If it’s going to collapse, it’s most likely to collapse in the first few days or weeks. The probability of collapse is front-loaded.

For decreasing hazard events, probability decays rapidly at first, then slows. Early survival is expensive. It burns through the front-loaded probability mass quickly. If a ceasefire holds for six months, most of the “collapse probability” has already been consumed without the event occurring. You’re now in the long tail where collapse was always unlikely, so further time passing costs you less.

Event-Driven Hazard

For completeness, I’ll mention another decay type that I use in forecasting, but didn’t implement due to complications. Sometimes, key events with a lot of probability mass happen at times that are not particularly close to either extreme.

An example might be a previously scheduled military exercise that could quickly pivot to an attack. If no attack materializes by the time the exercise is over and no further exercises are planned, it might be appropriate to expect the probability to decay rapidly afterward. As I mentioned, I did not implement this since it would be too easy for an AI to over-update on events.

How It Works

The decay analysis takes the method output and the hazard pattern, then calculates how much cumulative probability remains given how much time has elapsed.

Conceptually:

1. The method outputs a total probability P for the entire window

2. The hazard pattern defines how P is distributed across time

3. Given elapsed time t, we calculate how much of P has been “consumed” without the event occurring

4. The remaining probability is our updated estimate

For constant hazard, this is straightforward: linear decay. For increasing and decreasing hazard, the decay follows the shape of the probability distribution. Back-loaded distributions decay slowly then quickly; front-loaded distributions decay quickly then slowly.

Why It Matters

Decay functions matter most for questions with long time horizons where we’re partway through the window. If a question resolves in two weeks, there’s not much room for temporal dynamics to play out. If a question resolves in three years and we’re eighteen months in, the decay adjustment can meaningfully shift the estimate.

Consider a debt ceiling question at the start of the year with a December deadline. Your method might output 70% chance of a last-minute deal. In March, that 70% should barely have decayed as almost all the probability mass is still ahead of you. In November, it should still be close to 70%, because you’re just now entering the window where deals actually happen. Only if December arrives and passes without a deal does the probability finally collapse.

Contrast this with a ceasefire question. If your method outputs a 40% chance of collapse in the first year, and six months pass with the ceasefire holding, you’ve burned through most of that probability. Your updated estimate might be 10% or lower, not because new events have unfolded, but because the high-risk period has passed without incident.

Classification

How does the bot know which pattern applies? It’s part of the classification step. The LLM looks at the question structure and identifies cues:

• Negotiation dynamics → increasing hazard

• Random or memoryless processes → constant hazard

• Stability as evidence of equilibrium → decreasing hazard

Most questions default to constant hazard unless there’s clear structure suggesting otherwise. It’s the most conservative assumption. You’re not claiming to know something about temporal dynamics that you might be wrong about.

The Limits

Decay functions are a refinement, not a revolution. They adjust estimates at the margins based on temporal structure.

And the classification can be wrong. Some questions look like constant hazard but have hidden deadline pressure. Some look like decreasing hazard but have sleeper risks that compound over time. The decay function is only as good as the pattern recognition that assigns it.

As with everything in this pipeline, I’m tracking whether decay adjustments improve calibration.

This is the third post in a series on building an AI forecasting bot. In the first post, I argued that forecasting bots represent an opportunity to test methods empirically at scale. In the second post, I got the bot running and cut costs by ~30x to enable experimentation and rapid prototyping. In the third post, I covered the Broken Leg check, the first step in the pipeline, and a short-circuit for questions already resolved by breaking news or overwhelmingly likely to fall on a particular side.

Now we get to the core of the pipeline: figuring out what kind of question you’re dealing with and applying the right method.

Why Classification Matters

Not all forecasting questions are the same. “Will Bitcoin drop 25% from its peak before 2030?” is fundamentally different from “Will the US Constitution be amended to allow a third presidential term by 2028?” The first has historical data you can extrapolate from. The second is asking about a rare, discrete event with no direct precedent.

A good human forecaster recognizes this immediately and adjusts their approach. They don’t apply the same mental model to every question. Neither should a bot.

In real life, I might have 50 ways of classifying and forecasting a question, but I don’t have time to encode them all into a bot. Fortunately, I think 4 approaches capture ~80% of the value.

My pipeline classifies each question into one of four types, then routes it to a specialized method:

• BASE_RATE: Questions about events with identifiable reference classes

• TIME_SERIES: Questions about quantities with historical trends

• CONDITIONAL_CHAIN: Questions requiring multiple independent conditions

• NOVEL_EVENT: Questions about unprecedented situations

I’m not the only one taking this approach. On the Metaculus AIB Resources page, mmBot describes a similar strategy:

My bot first does research planning by decomposing the question and generating targeted search queries, scraping and summarizing the most relevant articles (llm-driven). It then classifies the question and routes it (also llm-driven) to one of several specialized context pipelines (eg. stock, sport, election, etc). Finally the optimized prompt and context is used to prompt an llm for a prediction several times which are then ensembled into the final forecast.

The intuition is the same: different question types benefit from different approaches. The classification step costs a few tokens but determines the appropriate method.

The Four Methods

Base Rate

Most forecasting questions are base rate questions in disguise. “Will X happen?” is really asking “How often do things like X happen, and is there any reason to think this case is different?”

The method:

1. Identify the reference class: what category does this event belong to?

2. Find the historical base rate for that class

3. Adjust for specific conditions that make this case more or less likely

4. Update based on current evidence

For example: “Will it snow in New York City on December 25th, 2026?”

Reference class: December 25ths in New York City. Historical rate: about 20%1 of Christmas Days in NYC have measurable snowfall since records began. Adjustment: current climate trends (slightly warming), La Niña/El Niño conditions for that year, whether there’s existing snow cover. The base rate anchors the estimate; the adjustments move it.

The danger is picking the wrong reference class. “Christmas Days in NYC” gives you one rate. “December days in NYC” gives you another. The skill is in choosing the class that’s most predictive for this specific question.

Time Series

Some questions have quantitative historical data that can be projected forward. “Will Bitcoin drop 25% from its peak before 2030?” is asking about a price trajectory with years of history.

The method:

1. Extract relevant historical parameters (volatility, trend, mean reversion)

2. Build a simple model of the process

3. Run Monte Carlo simulations forward to the resolution date

4. Calculate the probability of crossing the threshold

This works well when the underlying process is relatively stable and you have enough history to estimate its parameters. It works poorly when the process itself might change or when you’re extrapolating far beyond your data.

For crypto prices, the historical volatility is high enough that a 25% drawdown over a is very plausible and the time series method captures this by simulating many possible paths and counting how often they cross the threshold.

Conditional Chain

Some questions are really asking about a conjunction of independent events. Consider: “Will JD Vance be president on January 20, 2029?”

This isn’t a single event, it’s a chain of conditions that all need to happen:

1. Will he run for president?

2. Will he win the Republican primary?

3. Will the general election happen as scheduled?

4. Will he win the general election?

5. Will nothing prevent him from being sworn in?

Each step has its own probability conditional on the previous step which can be multiplied together to produce the forecast. Even if each step is reasonably likely—say 92%, 66%, 98%, 45%, 99%—the product is about 27%. Much lower than any individual step.

The method:

1. Decompose the question into necessary conditions

2. Estimate each condition’s probability (often using base rates)

3. Assess independence and adjust for correlation

4. Multiply through for the final estimate

This method can be brutally deflationary. Humans chronically underestimate how unlikely conjunctions are. A chain of “probably” events quickly becomes “probably not.”

The skill is in the decomposition. Too few steps and you miss crucial dependencies. Too many and you’re multiplying noise. You want the load-bearing conditions that actually determine the outcome.

Novel Event

Some questions have no reference class and no historical data. “Will a U.S. or U.S.-ally satellite be permanently disabled by another country or organization before January 1, 2027?” is asking about something that hasn’t happened yet (at least not in a confirmed, public way).

For genuinely novel events, I use a Laplace prior. The intuition: if something hasn’t happened in N attempts, your best estimate for it happening next time is 1/(N+2). This comes from Laplace’s rule of succession, a principled way to reason about events with zero observations.

The method:

1. Pick a reasonable starting point for when the event could have begun happening. For satellite attacks, we might select 1995 since that’s when GPS became fully operational and space infrastructure became a meaningful military target. That gives N=30 years of opportunities with zero confirmed events.

2. Calculate the per-period probability: 1/(N+2) = 1/32 ≈ 3.1% per year.

3. Apply it over the remaining window. With about 1 year until the deadline, the cumulative probability is 1−(1−1/32)^1 ≈ 3.1%.

4. Adjust for evidence not captured in the base calculation. There are more satellites now than in 1995. There have been suspected cyber attacks that might have temporarily disabled satellites. Tail risks like Russian escalation or Taiwan conflicts could change the calculus. Nudge the estimate as appropriate.

The key insight is that “unprecedented” doesn’t mean “impossible.” It means you should be uncertain and should flag that uncertainty clearly.

Classification in Practice

The classifier is a prompted LLM call that takes the question text and outputs one of the four categories plus a brief justification. It’s fast, reasonably accurate, and provides real uplift for forecasts in empirical results so far. Better instructions let us get away with using smaller, cheaper models without loosing too much in performance.

The tricky cases are questions that could plausibly fit multiple categories.

“Will the US recognize Taiwan before 2029?”

This could be:

• BASE_RATE: How often do countries make major diplomatic recognition changes?

• CONDITIONAL_CHAIN: Requires US decision + willingness to accept China’s response + no reversal

• NOVEL_EVENT: The US-Taiwan-China situation is historically unique

In ambiguous cases, the classifier picks the method most likely to produce a calibrated estimate. Usually that means defaulting to base rate with adjustments, since that’s the most robust approach when you’re uncertain about structure.

Why Split It Out

Having the methods separated lets me track what works and what doesn’t. Each question in my dataset records which classification it received, how the method-specific logic processed it, and what probability came out. As questions resolve, I’ll be able to answer:

• Which classifications are most accurate?

• Are certain methods systematically overconfident or underconfident?

• Do some question types benefit more from structured reasoning than others?

• Where should I invest effort in improving the pipeline?

This is the whole point of running at scale. Instead of debating whether conditional decomposition “should” work in theory, I can measure whether it actually works in practice. The tournament generates the feedback loop.

This post is part of a series on building an AI forecasting bot. In my last post, I described getting the bot running and cutting costs by almost 30x. Now I want to start walking through the pipeline itself.

The first step mirrors what a skilled human forecaster would do: check if you can skip the work entirely. Not only does this save time for a human forecaster, it saves computation and therefore money if a bot does it correctly.

The Broken Leg Rule

The concept comes from psychologist Paul Meehl, who spent his career studying when statistical models outperform human judgment. His finding was consistent and humbling: simple actuarial formulas beat expert intuition in domain after domain, but Meehl identified an exception.

Suppose you’re predicting whether someone will go dancing on Friday night. Your statistical model, trained on their past behavior, says 99%. But you happen to know they broke their leg this morning. You should override the model.

This “broken leg” represents information so decisive that it renders the baseline analysis irrelevant. The model can’t account for it because it’s rare, specific, and outside the training distribution. But when you have it, you should use it.

Meehl’s point wasn’t that intuition beats statistics. It was the opposite: trust the model almost always, but recognize the rare cases where you have information the model doesn’t.

Why Check First

For a forecasting bot with a complicated pipeline, there’s a practical reason to check for broken legs before doing anything else.

If the question is “Will Nicolás Maduro cease to be president of Venezuela during 2026?” and the news says U.S. forces just captured him and flew him to New York, you don’t need base rates. You don’t need time series analysis. You don’t need to decompose the question into conditional chains. The answer is already overwhelmingly clear.

A human forecaster would recognize this immediately and move on. The bot should do the same. Spending tokens on a full analysis when the outcome is already determined is wasted computation.

The Broken Leg check is the first step in my pipeline because it’s the cheapest way to potentially avoid all subsequent steps. A quick news lookup costs a fraction of what the full pipeline costs. If it fires correctly, you save everything downstream.

The Danger

Here’s the problem: being confidently wrong is catastrophic in a forecasting tournament.

Proper scoring punishes confidence asymmetrically. A broken leg check that fires incorrectly can sink your entire tournament. One blowout erases the gains from dozens of correct shortcuts.

I’ve seen human forecasters make this mistake. A headline feels definitive. Congress passes a law. A major announcement drops. The forecaster updates hard, confident they’ve found a broken leg. But the resolution criteria are more nuanced than the headline suggests, and the “decisive” news turns out to be merely suggestive.

The instinct to update on breaking news is correct. The instinct to update all the way is dangerous.

The Trade-off

So why include the check at all?

Because when it works, it’s extremely valuable. Not just for accuracy but for efficiency. You’ve spent minimal resources to correctly skip an expensive pipeline. Those saved resources compound across questions.

The trade-off is straightforward: spend a little up front to potentially save a lot. But only if your broken leg detector is reliable enough that the expected savings exceed the expected cost of the occasional blowout.

This is an empirical question. You can’t know the answer in advance. You have to run the bot, track when the check fires, see how often it’s right, and calculate whether you’re coming out ahead.

Right now, I’m still gathering data. The check is implemented and I’m monitoring its performance. If the blowouts start outweighing the savings, I’ll tune it, add confidence dampening, improve the news filtering, or disable it entirely until I can make it more robust.

Implementation

The actual implementation is simple. Before running the full pipeline, the bot queries a news API with the question text. If the returned summary contains information that overwhelmingly resolves the question in one direction, output a high-confidence probability and stop.

The hard part isn’t the code. It’s defining “overwhelmingly resolves.” That’s where the judgment lives, and where the misfires come from.

A president being captured and jailed? Overwhelmingly resolves “will they cease to be president.” A law being passed requiring document release? Maybe. Depends on enforcement, interpretation, timing, and exactly what the resolution criteria specify.

The more I can teach the bot to distinguish between these cases, the more value the Broken Leg check provides. That’s an ongoing refinement.

What’s Next

The Broken Leg check is step zero—the escape hatch. Most questions won’t trigger it, and the bot will proceed to the main pipeline.

In the next post, I’ll cover what happens when you don’t have a broken leg: how the bot classifies questions and selects the appropriate forecasting method.

My forecasting bot is now live in the Metaculus AI Forecasting Benchmark.

In my last post, I argued that this tournament represents forecasting’s “ImageNet moment” - a chance to test methods empirically at scale. Now the work begins.

The Forecast Pipeline

Most AI forecasting bots take a similar approach: feed the question to an LLM, maybe do some web search, ask for a probability. Some get fancy with multi-agent debate or ensemble averaging across models.

My bot runs each question through a structured pipeline: a sequence of steps designed to mimic how good human forecasters actually think. The pipeline details will come in future posts as I reveal each stage, but the flow mirrors how I operate as a superforecaster.

Codifying it into a reproducible system that can be tested and refined? That’s the experiment.

Top Priority: Reduce Costs

I applied for LLM credits from Metaculus, but I haven’t heard back yet and I’m currently paying out of pocket for LLM usage (AskNews generously provided free API access for news retrieval over the weekend - thank you!).

To make this work for my purposes, I need lots of iterations. The tournament has hundreds of questions. That requires running the bot many times. And running LLMs many times gets expensive fast. I want to test different approaches, compare results, and figure out what actually works. In practice, that means before I could iterate on methodology, I needed to make iteration affordable.

The unmodified template bot cost:

$0.109 per question

After my cost-reduction updates:

$0.004 per question

That’s a 27x reduction.

At the old rate, running the bot across 500 questions would cost ~$55. Now it’s ~$2. I can test dozens of different approaches for what one used to cost.

The Trade-off

I expect a performance hit from these changes. Cheaper models, fewer tokens, less elaborate reasoning. That’s fine for now.

The goal at this stage isn’t to win, it’s to learn. Which parts of the pipeline matter most? Where does additional compute actually help? What’s signal versus noise?

Once I understand what works, I can selectively add compute back where it counts.

What’s Next

The bot is running. Data is accumulating. Over the coming weeks, I'll share results, break down each pipeline step, and adjust based on evidence.

This is forecasting about forecasting using the tournament as a feedback loop to refine the methodology itself. The first real test: does a structured pipeline, even a cheap one, beat naive approaches?

We’ll find out.

Since its beginning, forecasting has been more craft than science. There were theories about what works (e.g. base rates) but testing them was slow, expensive, and small-scale. Forecasters could disagree about method and never get to a resolution because the feedback loops were long and sample sizes were too small.

That’s changing.

Forecasting bots are automated systems that take questions - “Will X happen by Y date?” - and output probability estimates. They can run 24/7, answer hundreds of questions, and get scored against outcomes. The Metaculus AI Benchmark is now running tournaments with hundreds of questions and real-time feedback. We can finally test forecasting methodologies empirically at scale.

This is forecasting’s ImageNet moment. ImageNet was the dataset that transformed computer vision from an academic backwater into the foundation of modern AI. Before ImageNet, researchers argued about which approaches to image recognition were best. After ImageNet, they could just run experiments and see.

The same thing is about to happen to forecasting. We’re going to learn what actually works.

I’ve written before about simple forecasting instructions that get you 80% of the way there, but building a bot that performs at the frontier requires going deeper. The simple heuristics are table stakes.

I’m building a bot. And I’m going to reveal my methodology in a series of posts. After the tournament ends, I’ll release everything.

The field is about to take a leap forward. Might as well document it in real time.

A driver with 5,000 rides and a 4.9 rating on Uber? Counts for nothing on Lyft.
A host with a 5 star rating on Airbnb? Try convincing guests on VRBO to book without offering a steep discount. A seller with 800 successful transactions on Ebay? Good luck when you try to break in to the Amazon marketplace.

It’s not a technical limitation, It’s a business model.

Platforms trap you because rebuilding reputation is costly. Each move to a new platform applies a reputation tax. You start at zero, and the platform keeps your identity in a walled garden.

In theory, cryptographic identity verification could address this, but, until now, the infrastructure has remained largely inaccessible.

Cipherblock.io changes this by making reputation a portable, verifiable identity that you control.

Portable Trust

In the abstract

• You carry a single keypair that represents your public identity.

• Anyone can publish an attestation about you that is digitally signed, timestamped, and published publicly.

• Any platform can verify an attestation in constant time without asking us or trusting a third party.

In practical terms

• A rideshare driver with 5,000 rides asks riders to sign attestations

• Each attestation is cryptographically signed and timestamped

• When joining another, the driver presents these attestations and users can see that these attestations are very likely coming from actual people and not fake accounts because the software can typically tie them back to someone they know or the signals are too costly to fake

• The platform verifies the signatures instantly no phone calls, no manual review

• The driver starts with established credibility, not from zero

Reputation stops being a private resource and becomes a set of verifiable claims you can take anywhere.

Technical Foundation

Everything runs in the browser meaning no downloads and no setup required:

• Generate your keypair client-side in seconds (private key never transmitted to any server)

• Anyone can run an operator that records attestations

• Operators maintain interoperable chains and compete for users without lock-in

• Users store signed receipts as cryptographic proof; even if an operator disappears, you keep the attestations

• Sites can integrate cipherblock for identity management with a few lines of code (an npm package)

• Open source and verifiable

The infrastructure is live at cipherblock.io. Read the technical specification to understand how it works.

Why It Just Might Work

• New entrants can bootstrap trust networks without years of building reputation systems

• Smaller platforms can compete with incumbents on service quality rather than network effects

• Regulatory pressure for data portability is increasing and users increasingly value platforms that respect their autonomy

Why It Might Not Work

• It introduces trade-offs that many people are unaccustomed to and might therefore hate

• Depends on network effects as well, if there aren’t enough adopters no one will want to support it perpetuating the problem

• It depends on someone caring about this enough to do something

Get Started

**Important: This is a functional prototype and active research project.**

The infrastructure works and is live at cipherblock.io, but expect rough edges, breaking changes, and ongoing experimentation. Think of this as an invitation to explore the concept, not production-ready software.

For Users: Build Your Portable Identity

1. Visit cipherblock.io - Generate your keypair in seconds, entirely in your browser

2. Grab tokens while you’re there or if the faucet is out contact me directly and I’ll give you some.

3. Request attestations - Ask people you’ve worked with to provide signed ratings

4. Store your receipts - Save the signed attestations locally as cryptographic proof

5. Take them anywhere - Present your portable reputation when joining new platforms or applying for opportunities

Your identity. Your reputation. Your control.

Even if every housing reform went into effect tomorrow, construction timelines mean it could be years before we see meaningful improvements in affordability or availability. To improve quality of life and opportunity in the near term, we need high-leverage, low-friction interventions that reshape how people live and move right now.

Mobility is the highest leverage near-term option we have.

Mobility improvements offer immediate and high-leverage benefits, but are just one component of addressing our housing affordability crisis. Mobility enhancements can deliver tangible benefits while housing developments are still navigating approval processes.

What Abundance Means

By "abundance," I’m referring to a material prosperity that enables greater life choices and opportunities. In its simplest form, abundance means having enough affordable housing, efficient transportation options, and economic opportunities so that people can thrive rather than merely survive. This concept explains why many are leaving traditionally progressive states like New York and California for places like Texas and Florida, where housing costs are lower and day-to-day living is more affordable. This exodus has political consequences: as population shifts, so does electoral representation. If New York wants to preserve its importance, we need tangible policies that improve material well-being, starting with mobility solutions that can be implemented quickly while we work on longer-term housing reforms. When people can access jobs, education, and amenities affordably and efficiently, we create the conditions for genuine abundance.

Mobility Improves Affordability

Housing affordability isn't just a function of how many units exist, it's also about where those units are, and how easily people can access jobs, schools, services, and social networks from them. Mobility is a force multiplier: when people can get where they need to go quickly, reliably, and safely, it effectively increases the supply of livable housing, even if the number of units stays the same.

There are three core ways transportation improves affordability:

Expanding Access to Undervalued Neighborhoods

Across New York, thousands of housing units remain underutilized simply because they're isolated by inadequate transit options. A faster, more flexible transportation system opens up these areas to new demand and investment, raising utility without having to wait for new supply to come online. It also has the added bonus of incentivizing further investment improving the total housing supply.

Reducing the Combined Housing-Transportation Cost Burden

When reliable transportation options exist, families can make strategic trade-offs between housing and transportation costs. Improved mobility allows households to access more affordable housing farther from city centers without sacrificing job opportunities or dramatically increasing commute times. This shifts the traditional calculus of "drive until you qualify" to a more sustainable model where transportation doesn't consume the savings gained from lower housing costs.

Creating Economic Mobility Without Physical Relocation

Robust transportation networks connect people to a wider range of employment, education, and service opportunities without requiring them to move. This is particularly powerful for lower-income communities where residents can access better jobs and amenities while maintaining crucial social networks, community ties, and cultural connections. Transportation becomes a ladder to opportunity that doesn't require abandoning affordable housing or supportive communities. If better transportation means families have access to better job opportunities, it improves affordability because they have higher incomes.

Solutions for Near-Term Impact

New York already has the tools needed to dramatically improve mobility in the near future and some of these solutions are already underway, but they need our continued support. I’ve tried to highlight solutions that can be implemented within months to years, as opposed to years to decades, providing tangible benefits while longer-term housing reforms take shape.

The focus here is primarily on low-friction improvements that don't require massive capital outlays like building new rail lines or other major infrastructure projects. Instead, these solutions maximize existing resources, repurpose underutilized assets, and implement policy changes that can deliver near-term benefits at relatively modest costs. By prioritizing these high-leverage, low-friction interventions, we can achieve significant mobility improvements without waiting for complex, expensive infrastructure projects to complete their lengthy planning and construction cycles.

Expanding Congestion Pricing

Expanding Manhattan's congestion pricing to high-traffic areas in Brooklyn and Queens could reduce congestion while generating revenue for transit improvements. By thoughtfully designing fee structures, we can ensure access while discouraging unnecessary traffic.

Repurposing Street Space for Dedicated Bus Lanes

New York can learn from London's successful transformation of its bus network through strategic repurposing of road space. In 2021, Transport for London (TfL) made 85km of bus lanes on its busiest roads operate 24 hours a day, seven days a week. This bold move significantly improved journey times and service reliability, particularly in central and inner London areas.

The results speak for themselves: bus service became more predictable and faster, making it an attractive alternative to driving. As Louise Cheeseman, TfL's Director of Buses, noted, "Shorter journey times will help deliver a more reliable bus network," which in turn helps "make buses an attractive travel option." Most importantly, these changes created a positive feedback loop: more reliable service attracted more riders, which justified further service improvements, creating a virtuous cycle of transit growth.

This stands in stark contrast to the current situation in New York, where bus unreliability has decimated ridership. Most New Yorkers don't consider buses a viable transportation option precisely because of unpredictable arrival times and crawling speeds when stuck in traffic. Nobody wants to wait 20 minutes for a bus that then inches along in congestion, turning what should be a quick trip into a frustrating ordeal.

By converting underutilized street parking into dedicated bus lanes, we can break this cycle of decline. Continuous, camera-enforced bus lanes that operate 24/7 could be implemented relatively quickly by repurposing existing infrastructure, delivering immediate benefits to hundreds of thousands of New Yorkers daily. The evidence from London1 shows that these measures can dramatically improve service without significantly impacting other traffic, providing a rare win-win in urban transportation policy.

Demand-Based Parking Reform

Strategic adjustment of parking pricing based on demand has proven remarkably effective at reducing traffic congestion in urban areas. San Francisco's pioneering SFpark program demonstrates the power of this approach. By implementing demand-responsive pricing at parking meters, San Francisco reduced "cruising for parking" — drivers circling blocks looking for spaces — by 43%, which translated to a 30% decrease in vehicle miles traveled in SFpark areas.

The program works through a simple application of supply and demand economics: when parking demand is high on a particular block, prices increase slightly ( e.g. $0.25 per hour); when demand is low, prices decrease by the same amount. This creates a natural distribution of parking across neighborhoods while ensuring that spaces remain available on every block.

Contrary to initial concerns about increasing costs, the program actually resulted in lower average parking rates (down 4% for on-street parking and 12% for city-owned garages) while simultaneously increasing business activity. Sales tax revenue rose 35% in SFpark areas compared to less than 20% in other parts of the city.

By implementing similar demand-responsive parking policies in New York, we could immediately reduce the estimated 30% of traffic caused by drivers searching for parking, improving mobility for everyone while supporting local businesses and reducing emissions.2

Repurposing Street Space for Bike Lanes

New York has made significant progress in expanding its bike lane network, but too many routes remain disconnected, unprotected, or both. By strategically converting misallocated street space into protected bike lanes, we can create a comprehensive, safe cycling network that serves both commuters and recreational riders.

Protected bike lanes have been shown to substantially increase cycling volumes within just months of installation. This dramatic growth occurs because many potential cyclists cite safety concerns as their primary reason for not biking. When people feel safe, they ride.

My own experience confirms this. Since protected bike lanes were installed connecting Downtown Brooklyn to Williamsburg, I've made them my preferred mode of transportation during much of the year. While driving this route might take 17 minutes on a good day, it's wildly unpredictable – today it took 45 minutes (it was rainy so I foolishly opted to take my chances driving), and I've experienced trips longer than an hour before. By contrast, my bike commute is a consistent ~17 minutes every time, plus I avoid the hassle of finding parking. This reliability is why I often choose cycling over other modes even in winter when protected lanes are available.

Beyond just increasing cycling, protected bike lanes benefit the entire transportation ecosystem. They reduce injuries for all road users (not just cyclists), decrease traffic congestion by encouraging mode shift, and improve business for local retailers along bike routes. Many neighborhoods have seen retail vitality increase after protected bike infrastructure was installed.3

Implementation can be remarkably swift - a basic protected bike lane can be created using paint and flexible bollards in days or weeks rather than months. More permanent versions with concrete barriers can follow as funding permits, but the immediate safety and mobility benefits begin as soon as the lanes are marked.

With thousands of miles of streets in New York City, dedicating just a fraction of this space to safe, protected bike infrastructure could create an accessible network.

Embracing Technological Innovation

Creating a clear regulatory framework for robotaxis and other emerging mobility technologies can accelerate their safe deployment. These services can fill gaps in the transit network, reduce car ownership needs, and free up valuable urban space currently dedicated to parking. While full-scale deployment will take time, establishing the groundwork now ensures these benefits arrive sooner.

Improving Transit Information Accessibility

New York's public transit system, while extensive, often fails at the most basic function of helping riders navigate with confidence. These informational barriers create unnecessary anxiety and discourage transit use, even among regular riders.

Anyone who has tried to navigate New York's subways has experienced the frustration of missing or inadequate wayfinding tools. Maps are scarce throughout stations, making quick orientation difficult. While other global metro systems prominently display route maps in multiple locations within each car and throughout stations, New York riders often find themselves straining to glimpse a distant map or relying on smartphone apps – assuming they have service underground.

Compare this to the Bay Area's BART system, where airport-bound trains are clearly designated with distinct signage. In New York, the train to JFK is indistinguishable from the other on the same line, with no special markings. For visitors and residents alike, this creates needless uncertainty about whether they're heading to their flight or somewhere entirely different.

The critical function of announcing upcoming stops is rendered nearly useless by poor audio quality throughout the system. Garbled, unintelligible announcements compete with the train's noise, leading riders to miss their stops entirely – especially when crowding prevents visual confirmation of location. This basic communication failure creates anxiety for riders who must remain hyper-vigilant rather than using travel time productively.

Perhaps most frustrating is the unpredictable nature of service changes and skip-stop patterns. Riders can never fully trust that their train will make all expected stops, forcing them to constantly monitor announcements (which, as noted, are often unintelligible) rather than using transit time productively for reading, working, or resting. This unpredictability erodes confidence in the system and discourages ridership among those who have alternatives.

Implementing clear, consistent, and accessible information systems would dramatically improve the rider experience at relatively low cost compared to infrastructure changes. Digital displays in every car showing upcoming stops, clear airport route differentiation, improved audio systems, and real-time service alerts visible throughout stations would transform how New Yorkers interact with their transit system. These improvements would make transit more welcoming to occasional users while reducing stress for regular commuters.

By addressing these basic informational shortcomings, we can make transit not just physically accessible, but cognitively accessible as well – a critical step toward encouraging greater ridership and reducing dependency on private vehicles.

Prioritizing Mobility for Immediate Impact

These solutions share three critical advantages over housing reforms alone:

Implementation Speed

These mobility interventions can be deployed within months to years, not decades. London's bus lane transformation and San Francisco's parking reforms demonstrate how quickly well-designed policies can reshape urban mobility patterns.

Leverage Existing Infrastructure

Rather than waiting for new construction, these approaches maximize what we already have. By repurposing street space, optimizing pricing mechanisms, and improving information delivery, we can dramatically increase the utility of our current transportation network.

Proven Success

Each proposed solution builds on documented successes in comparable urban environments. The data from London and San Francisco isn't theoretical - it shows concrete improvements in reliability, accessibility, and even economic vitality.

New York stands at a crossroads. While we must continue pursuing long-term housing solutions, we cannot afford to neglect the immediate mobility crisis affecting millions of residents daily. By implementing these high-leverage mobility interventions today, we create breathing room for housing reforms to take effect, improve quality of life in the near-term, and build momentum toward a more accessible, affordable city.

The path to abundance starts with mobility. Let's take the first steps.

Imagine betting your life savings on a weather forecast, only to discover the meteorologist was relying on a groundhog's shadow instead of analyzing atmospheric data. This is essentially what we do when addressing global existential threats without scrutinizing our underlying world models.

From nuclear war to bioterrorism, from authoritarian regimes to pandemic risks, humanity faces numerous existential threats. Yet we struggle to agree not only on how to address these challenges, but even which ones deserve priority. With our future at stake, we need better methods to distinguish genuine risks from distractions.

The solution may be simpler than we think: systematic forecasting that puts models to the test.

World Models Described

Everyone has a world model, whether they realize it or not. It's an internal representation of reality – the beliefs, assumptions, and mental shortcuts used to make sense of a complex world. Essentially it’s a framework for understanding how the world functions.

When you wake up in the morning and plan your day, you're using your world model. You assume the sun will rise, your lights will come on when you flip the switch, and the world outside your house will exist when you go out the door. These aren't just random guesses – they're based on your understanding of how the world operates.

The problem is, our world models can be flawed, particularly when we don’t get direct feedback on them. They can be based on incomplete information, biased by our personal experiences, or simply outdated. It’s not a big deal when the stakes are low and we find out that while we had no trouble explaining how an airplane can fly normally, we can’t explain how it can fly upside-down. Unfortunately when we're dealing with global challenges, the limitations of our world models become critically important.

Evaluating Competing World Models

The first challenge is that there are a lot of competing world models to consider, so many in fact that evaluating them all would be an intractable problem. In virtually any given field, even if there are many points of consensus, you'll still find areas where there is wild disagreement. This diversity of models reflects the complexity of the world we're trying to understand, but it also makes it difficult to determine which models are most accurate or useful.

On top of that, we face the problem that every now and again the prevailing paradigm is flawed, but gathering the data to overturn it could take a long time to collect. This is challenging because data collection can be inherently difficult, especially for complex global issues, and the feedback loops can be very long, making it hard to quickly validate or invalidate a model. Moreover, existing paradigms often become entrenched not just due to scientific inertia, but also because of ego investments, tribal affiliations within academic or professional communities, and power structures that benefit from the status quo. These social and institutional factors can create significant resistance to new ideas or evidence that challenges established models, further complicating the process of correcting flawed paradigms.

Additionally, even if a model has very good explanatory power, it might still be spurious. Consider the classic example of the correlation between ice cream sales and drowning incidents. A model based on this correlation might appear to have strong predictive power, but it misses the true causal factor (warm weather) that influences both variables.

Evaluating Models with Instrumentalism

It probably goes without saying that some models produce better results than others. Consider this passage from Sapiens.

In 1744, two Presbyterian clergymen in Scotland, Alexander Webster and Robert Wallace, decided to set up a life-insurance fund that would provide pensions for the widows and orphans of dead clergymen. They proposed that each of their church’s ministers would pay a small portion of his income into the fund, which would invest the money. If a minister died, his widow would receive dividends on the fund’s profits. This would allow her to live comfortably for the rest of her life. But to determine how much the ministers had to pay in so that the fund would have enough money to live up to its obligations, Webster and Wallace had to be able to predict how many ministers would die each year, how many widows and orphans they would leave behind, and by how many years the widows would outlive their husbands.

Take note of what the two churchmen did not do. They did not pray to God to reveal the answer. Nor did they search for an answer in the Holy Scriptures or among the works of ancient theologians. Nor did they enter into an abstract philosophical disputation. Being Scots, they were practical types. So they contacted a professor of mathematics from the University of Edinburgh, Colin Maclaurin. The three of them collected data on the ages at which people died and used these to calculate how many ministers were likely to pass away in any given year.

How do we differentiate between competing world models? It depends on the goal, of course. For some, it's about forming a consistent paradigm, for others it's a grab bag of ad hoc rules that work well enough to get by without thinking too hard, and for others it may be more about finding a set of beliefs that allows for minimum conflict within their tribe. For most readers here, however, I suspect it is more about practical utility: using models that help us accurately predict and navigate reality.

This pragmatic approach, known as instrumentalism, judges models not by whether they're "true" in some abstract sense, but by whether they're useful tools for understanding and interacting with the world. Under instrumentalism, the best model is the one that most reliably generates accurate predictions and enables effective intervention.

Consider how we evaluate scientific theories: Newton's laws of motion aren't considered valuable because they represent some perfect, ultimate truth, in fact, we know they break down at quantum scales. They're valuable because they allow us to predict with remarkable accuracy how objects will behave under everyday conditions. Similarly, when evaluating world models for global challenges, we should prioritize predictive power over ideological purity or intuitive appeal.

This is where good forecasters enter the picture. These individuals and teams have empirical track records of successfully predicting future events across diverse domains. Their consistent success isn't theoretical, it's demonstrated repeatedly through verifiable results. While we'll explore the cognitive mechanisms behind their success in a future article, the important point is that their methods work. By identifying who consistently makes accurate predictions, we gain a shortcut to identifying which world models are most effective for understanding reality.

The instrumentalist view liberates us from endless philosophical debates about the "true nature" of reality and focuses our attention on what matters: which models help us anticipate and address real-world problems most effectively. When facing existential risks, this practical focus isn't just intellectually honest, it's essential.

The Cheat Code

One might reasonably ask: How do we know forecasters aren't just identifying spurious correlations themselves? After all, we've seen that correlations can appear predictive without revealing true causation.

What sets good forecasters apart is their consistent performance across diverse domains and time horizons. Unlike the ice cream-drowning correlation that breaks down when conditions change, skilled forecasters maintain accuracy through shifting circumstances. They don't simply identify statistical patterns, they build flexible models accounting for causal mechanisms and context-specific factors.

When forecasters maintain predictive accuracy whether analyzing geopolitics, economics, or public health, they demonstrate something beyond coincidental association. They actively combat the spurious correlation trap through techniques like base rate analysis, scenario testing, and continuous recalibration. Their success persists precisely when simple correlations fail, demonstrating a resilience that validates their methodology as capturing something fundamentally useful about reality.

Indeed, research shows that forecasters with good track records tend to maintain their high performance across various domains, making well-calibrated predictions even outside their areas of expertise. This consistency isn't merely theoretical, it's an empirically verified phenomenon that gives us a powerful tool for assessing world models.

Even more impressive, groups of forecasters consistently outperform individuals. These collaborative efforts combine diverse perspectives and analytical approaches to produce remarkably accurate predictions, often surpassing traditional domain experts. Their strength lies not in specialized knowledge, but in their ability to effectively synthesize expert findings and arguments into coherent, actionable forecasts.

Consider this passage from my fellow forecaster Molly Hickman’s recent article that she wrote in conjunction with Rajashree Agrawal:

The foundational literature (e.g. Goldstein et al, Karvetski et al, Tetlock's Expert Political Judgment) in Tetlockian forecasting seems to point at the top forecasters having better understanding of how the world works than domain experts (if you're on board with the idea that proper scores, in this case Brier, are a reasonable proxy for "understanding of how the world works"). Despite lacking the years of knowledge, experience, and gut-feelings that experts have, forecasters are better at putting numbers to whatever knowledge, experience, and gut-feelings they do have—implying that high-quality expression of shallow information > low-quality expression of deep information. This makes some intuitive sense; for instance, despite farmers' having much deeper understanding of agriculture, it doesn't seem like they're better at crop futures trading than traders.

This observation captures an important insight: the quality of the forecasting process can outweigh the depth of subject-matter expertise. This helps explain why forecasters can succeed across diverse domains: they've optimized the methodology of prediction itself rather than focusing solely on accumulating domain knowledge. The agriculture/trading example illustrates how even in highly specialized fields, good forecasting techniques often triumph over experience-based intuition.

Importantly, these forecasters have the right incentives. They stake their reputations on their predictions, aligning their interests with the pursuit of truth rather than supporting particular agendas or maintaining the status quo.

By tapping into this collective wisdom, we can navigate complex issues more effectively, identify which world models are most accurate in practice, spot emerging trends or potential threats earlier, and make more informed decisions in the face of uncertainty.

As we confront the array of challenges facing civilization, from nuclear risks to climate change, from biosecurity to governance failures, this forecasting approach offers our best path forward. Rather than endless theoretical debates or waiting years for definitive evidence, we can look to these forecasting groups as our reality-checking mechanism and a practical means of determining which world models truly deserve our attention and resources.

In a world of competing narratives and complex threats, good forecasting may be our most valuable compass.

I haven’t had a lot of time for writing lately as teaching, work, forecasting, and life have kept me busy, but I’ve got a few new articles coming soon, including one that’s going out tomorrow morning.

In other news, I’ve been invited to LessOnline this year and I’m looking forward to seeing many of you there. If you’re attending, feel free to say hi or reach out in advance. I’ll be around for the full weekend and always up for a good conversation, especially about forecasting, epistemics, or AI progress.

Finally, Samotsvety is holding a forecasting conference and retreat this June the week after Manifest. While the conference is invite-only, we’re holding a public hangout in Fort Greene Park in Brooklyn on the evening of Tuesday, June 10. If you’re in NYC and interested in forecasting, rationality, or a casual reader of this blog, come by and say hello—everyone’s welcome.

Making a forecast isn’t always the best use of time when making a decision. Most of the time, we’re better off either planning for the worst-case scenario or relying on rules or experts with proven track records. However, there are specific scenarios where investing effort into a structured forecasting process can lead to significantly better outcomes. Let's explore a framework to determine whether to fortify, follow, or forecast when faced with an important decision. By identifying when forecasting truly matters, you can apply it effectively to achieve better results.

Fortify: Prepare for Worst-Case Scenarios

Some decisions are too critical to leave to chance. Take, for example, planning your route for a job interview where traffic could take anywhere from 30 to 60 minutes. Even if you’re 90% confident you can get there in 45 minutes, the stakes are too high to risk being late. In such cases, the precise forecast doesn’t matter, whether the worst-case scenario is a 5% or 50% likelihood, you still plan for it because the stakes are too high. You give yourself 60 minutes to ensure punctuality because the cost of failure is unacceptable. Whether it’s emergency preparedness or ensuring you’re on time for an important event, the goal is to mitigate risks and safeguard against unacceptable outcomes.

Follow: Rely on Rules and Experts

In some domains, experts provide highly reliable forecasts, such as weather predictions. Meteorologists are well-calibrated experts with an empirical track record of accuracy. For example, if you’re planning a beach trip for the weekend and deciding between Saturday and Sunday, it’s practical to trust the weather report to make your choice. Their reliability makes it unnecessary to attempt to outperform them yourself.

Similarly, when the stakes are low and you have personal experience to rely on, following rules that have led to positive outcomes in the past can simplify decisions. For instance, if your favorite restaurant is usually uncrowded on Wednesdays and often features your preferred special, you might choose to dine there based on this reliable pattern. Such rules offer a practical and time-efficient way to navigate everyday choices without overanalyzing.

In both cases, empirical track records guide the decision, offering a reliable foundation for making choices efficiently and effectively.

Forecast: Use Best Practices to Make the Best Decision

Unfortunately, many questions in life are nuanced and involve complicated tradeoffs that don't come with convenient expert solutions. Sometimes the best path forward isn't obvious. Major life decisions, like whether to move to a new city, require personal foresight and deliberate analysis. Follow the most important rule of forecasting and start by gathering base-rate information: What are typical salaries in your field for that city? How happy are people who live there?

For example, if 70% of people in your field who moved to Austin saw their real income increase within two years, that's more informative than your gut feeling about the city's prospects.

Use these data points to anchor your forecast, as base-rate statistics about others’ experiences are often more reliable than your own subjective feelings. People frequently overweight their own predictions about how they’ll feel ("I'll definitely love the weather!"), but trusting what has been true for others is usually a better guide for decision-making.

Why This Matters

Most people are already pretty good at fortifying—planning for worst-case scenarios where the stakes are high (okay, fine, maybe they’re not great with tail risks, but that’s a topic for another post). And unless you’re part of the crowd who thinks wrestling is real, but the moon-landing is fake, you’re probably already mostly following empirically-validated strategies where you should too (another post about how to know which experts to trust is coming soon… HINT: empiricism).

The real opportunity lies in that third category: people rarely take the time to forecast deliberately even when doing so could lead to much better outcomes. Think of the last time you made a major career move - did you look up the base rates for success in similar transitions? Or consider someone choosing between grad school options - how often do they gather systematic data on career outcomes versus just following their gut? By recognizing these opportunities to forecast more carefully, you can make significantly better decisions in areas that really matter.

Forecasting isn't merely about predicting the future; it's a tool for making smarter decisions today by reducing uncertainty and aligning actions with desired outcomes. The next time you face a decision, ask yourself: Should you prepare for the worst case, trust reliable experts, or dive into forecasting? By using this framework and leveraging available data, you can approach uncertainty with clarity and confidence, ultimately improving your results.

Recently, my neighbor Daniel Golliher, author of Maximum New York, proposed a book club. I decided to participate, so I’ve written my reflections on the revised 2020 edition of Robert Putnam's Bowling Alone.

Bowling Alone presents a compelling narrative discussing the decline of social capital in America and supports its case with in-depth analysis of the factors that may have contributed to its decline while endeavoring to compensate for confounding factors and control for spurious correlations.

Social Capital and Its Decline

In Putnam’s framework, social capital refers to the networks of relationships among people who live and work in a society, enabling it to function effectively. Putnam posits that social capital can be broken down into two main categories:

1. Bonding capital refers to connections within homogeneous groups fostering a sense of community and shared identity.

2. Bridging capital refers to connections between heterogeneous groups. The theory proposes that bridging capital can link people to broader opportunities and resources outside their immediate social circles.

Proponents argue that social capital can reduce friction in social interactions and daily life and suggest that higher levels of social capital might correlate with more efficient societies, as trust and established networks could facilitate smoother transactions and collaborations.

As per the emblematic chart above, Bowling Alone argues that social capital in the United States has been declining since the 1950s by presenting decreases in a wide set of indicators of social capital, including civic engagement, political participation, religious involvement, workplace connections, informal social ties, philanthropy, and trust in institutions and other individuals. Putnum provides numerous charts depicting declining participation across a variety of institutions and attributes this to a combination of factors, including increased time spent watching television, two-career families, suburban sprawl, and generational change. He contends that this erosion of social capital has significant implications for community well-being, democratic participation, and overall societal health.

Putnam's approach seems robust. He takes a comprehensive look at the potential causes behind the decline of social capital and provides a well-reasoned rationalization for whether or not he thinks each meaningfully contributed. For example, in evaluating a factor, he might look at trends in comparable countries to determine if the same factor is present there and whether the magnitude of the effect was similar. While Putnam put range estimates on each of the factors he found likely to be responsible as shown in the image below, he left room for unexplained other factors.

Putnam's argument sparked considerable debate and inspired further research into the state of social connections. While his ideas are influential, the mechanisms and applicability of social capital is the subject of ongoing debate. As with many sociological concepts, social capital evades straightforward measurement and quantification.

As a professional forecaster, I found Putnam's approach well-considered and worthy of discussion, and, in my personal judgment, probably directionally accurate. However, with observational evidence, even with extensive controls, it’s hard to draw conclusions with high confidence.

While Putnam's work is thought-provoking, and there are many interesting arguments over the relative importance of the various factors he discussed, I don't want to focus on reanalyzing them since that has been done elsewhere. Instead, I want to look into the unexplained portion of the precipitous declines observed in many of the social and community organizations the book addresses. In the rest of this post, I’d like to explore how network effects might explain an important part of that mystery.

The Power of Connections: Network Effects

Network effects occur when the value or impact of a product or service increases as more people use it. Think of how much more useful a telephone is when the person you want to call also has a telephone. A phone wouldn’t be very useful if you were the only one that had one, or if the only other person who had one was someone you didn’t care much to talk to, but the more people that have them, the more valuable they become. In the context of social capital, we can think of community organizations, civic groups, or even informal social circles as networks whose value grows with increased participation.

One way to quantify network effects is through Metcalfe's Law, which states that the value of a network is proportional to the square of the number of connected users1. Even if the value of a social network doesn’t scale exponentially with the number of users, it seems reasonable to expect that many people would find larger organizations, at least up to a point, to be more valuable since they offer more opportunities to establish both bonding and bridging capital.

Presuming that, for most participants, participating in an organization depends on the cost-benefit balance, in the unfortunate event where a network loses a participant, perhaps due to moving, even if nothing else changes, it could lead to the next marginal member leaving as well as the decreasing value of the network moves their participation from net-positive to net-negative. This could lead to a snowball effect where more and more participants leave. Once enough people stop participating, there is no escaping the vicious cycle.

Combine this concept with rising opportunity-costs such as better at-home entertainment or longer commute times, and we can see a promising explanation for why the decline in organizational participation might have accelerated beyond what Putnam’s factors alone would predict.

This compounding effect could explain why Putnam observed such steep declines in participation across various social organizations. It's not just that people are choosing not to join these groups; it's that the groups themselves become less attractive as they shrink, creating a self-reinforcing cycle of decline.

To better demonstrate this idea, I created a simulation of a community center with a capacity of 200 members where every potential member is randomly assigned a benefit value that they get from the community along with an opportunity-cost. As long as the benefit is greater than the cost, and as long as there is room, the person becomes and stays a member (I also set some other parameters like having the value they get from membership increase slightly over time as they grow more deeply engaged, a random chance of quitting, a maximum monthly join and leave rate, and other variables, but those aren’t important to the overall narrative and you can play with the assumptions if you want to build on my simulation). However, if the cost exceeds the benefit, the member quits.

The simulation starts out with 150 members in 1940. Membership as a percentage of capacity is represented by the blue line, the calculated value of the community (as determined by the network effect, but maxing out in value after reaching saturation at 150 participants) is represented by the orange line. Finally, the baseline societal opportunity-cost is represented by the green line. In my simulation, I added two step-changes in opportunity cost, one after the wide-spread adoption of television and the second after the advent of the internet.

As the simulation starts out, new members rapidly join filling the community cent to its maximum capacity. This remains very stable until the scaled adoption of TV after which some members quit because they find staying home is a better alternative, but they are quickly replaced by other community members who put a higher value on what they get from the community. The network remains relatively stable until the arrival of the internet. For a few years, the value of the network remains high and new members continue to join as other members leave, but as more and more people stay home, the network quickly reaches a point of no return.

Understanding this dynamic is crucial for efforts to rebuild social capital. It suggests that simply addressing individual factors like time constraints or competing entertainment options may not be enough. Instead, strategies to revitalize community engagement might need to focus on reaching a critical mass of participation quickly, leveraging technology to reduce barriers to entry, and creating clear, immediate value propositions for potential members to overcome the inertia of non-participation.

In essence, the power of network effects works both ways – it can amplify growth in social capital when conditions are favorable, but it can also accelerate its decline when various factors align against it. Recognizing this can help us better understand the dramatic shifts in social engagement that Putnam documented and inform more effective approaches to fostering community connections in the future.

It’s easy to underestimate the capabilities of current-generation AI, but with a few adjustments in approach, current models are capable of much more than most people realize.

Better Prompting -> Better Responses

The quality of the prompt significantly affects the result. Providing detailed, step-by-step instructions or better context can dramatically improve results. For example, instructing the model to "think step-by-step" or to "ask clarifying questions" can guide it to produce more accurate and useful outputs.

For example, here’s a prompt I gave to GPT-4o:

Prompt: Jim is Bob's brother. Jim's wife is Sarah who has no siblings. Bob has no other siblings. Sarah's daughter's aunt is named Susan. What is Susan's relationship to Bob?

I gave Claude Sonnet 3.5 the same prompt and it nailed it out of the gate, but this was GPT-4o’s incorrect response:

gpt-4o:

Given the relationships described:

1. Jim is Bob's brother.

2. Jim's wife is Sarah.

3. Sarah has no siblings.

4. …

… the most plausible relationship, without contradicting the given information, is that Susan is Bob's sister-in-law, married to either an undisclosed brother of Jim (and therefore Bob) or related to Sarah through another form of marriage connection.

Yikes!

This response makes me feel kind of sympathetic to the AI is useless crowd, but we can do a lot better. To show what better prompting can do, I created a new session and appended the bolded text below to help GPT-4o get to the right answer on its own:

Prompt: Jim is Bob's brother. Jim's wife is Sarah who has no siblings. Bob has no other siblings. Sarah's daughter's aunt is named Susan. What is Susan's relationship to Bob? Let's think through this logically step-by-step and be careful to make sure you get this right by double checking your answer for inconsistencies with the provided information.

gpt-4o:

To determine Susan's relationship to Bob, let's break down the given relationships step by step:

1. Jim and Bob are brothers.

2. Jim's wife is Sarah, who has no siblings.

3. Bob has no other siblings…

…Given that Susan is identified as an aunt and can only be Bob's wife due to the lack of other siblings and Sarah having no siblings, we conclude:

Susan's relationship to Bob is that Susan is Bob's wife.

This example demonstrates the power of prompt engineering. By providing explicit instructions to think step-by-step and double-check for inconsistencies, we guided the AI to arrive at the correct conclusion. This approach not only improves accuracy but also demonstrates how AI can be steered towards more reliable and logical reasoning.

The takeaway is that this is a skill that can be honed. Just as a manager learns to give clear instructions to their team, users can learn to craft prompts that elicit more accurate and useful responses.

Unfair Comparison: Context and Tuning Matter

Humans often ask models challenging questions they themselves know little about, leading to frustration when the LLM fails on the first attempt. This often results in an unfair comparison, as humans would generally seek additional information or ask clarifying questions before tackling such complex requests. Most consumer-facing LLMs are tuned to provide answers without seeking further context, unlike their human counterparts who instinctively probe for more information or, even more likely, already have sufficient context from past experience specific to the requester. While this kind of single-fire response does well in the context of the chatbot ranking arena, it can lead to frustration in the real world.

For example, look at how two different models respond to a line of questioning that lacks sufficient context.

Prompt: Who is the President?

gemini-1.5-pro-api-0514 : Joe Biden is the current president of the United States.

deepseek-v2-api-0628: As of my last update in October 2023, the President of the United States is Joe Biden. He took office on January 20, 2021.

Prompt: I work for McDonalds. Who is the President?

gemini-1.5-pro-api-0514 : Whether you work at McDonald's or anywhere else doesn't change the answer: the President of the United States is still Joe Biden.

deepseek-v2-api-0628: I work for McDonalds. Who is the President? As of my last update in October 2023, the President of McDonald's Corporation is Chris Kempczinski. He was appointed to this position in November 2019, succeeding Steve Easterbrook.

As the example above illustrates, both models attempt to respond to this ambiguous situation without seeking clarification from the user. The prompt could be interpreted in several ways: Is the user asking about the president of McDonald's, the president of the United States, the president of another country, or even some other president entirely? Instead of asking for more context, the model makes its best guess and proceeds with the response. While it’s technically possible to adapt models to ask clarifying questions at each step, this approach might be less appealing to the average user, who typically expects quick and straightforward answers.

It’s surprising how quickly some users are willing to give up and write these tools off as useless, but I suspect most of them have never experienced the challenges of working with an overseas development team. Although the human-level intelligence of a dev team far surpasses that of any existing LLM, most users would still find it frustrating to accomplish tasks using only asynchronous written communication. For me, the LLM experience, while not the same, is reminiscent of this. When you’re careful to fill in the missing context that most in-person communication takes for granted, you get results.

Overcoming Unreliability

A common objection I hear from the "AI is overhyped" crowd is that AI outputs are unreliable and therefore useless since everything they produce has to be checked by a human anyway. Where this objection falls apart is, in many cases, verification can be automated. For example, in software development, AI can generate code while automated systems verify its correctness. Modern development practices already employ Continuous Integration/Continuous Deployment (CI/CD) pipelines that automatically run unit tests, integration tests, and even performance benchmarks. These same systems can be applied to AI-generated code, ensuring its reliability without human intervention, so developers can focus on higher-level tasks like architecture design and complex problem-solving, rather than writing and testing every line of code manually.

Even when human oversight is necessary, the efficiency gain can still be substantial. AI rapidly produces initial solutions, allowing humans to refine and make high-level decisions rather than starting from scratch, dramatically accelerating the development process. While some may suggest that AI code quality is so buggy that it’s more trouble to use than it’s worth, it’s useful to remember that this is the worst it will ever be going forward and not every company employs a staff of elite programmers like yours does, and in some (most?) cases, the AI is still going to write higher quality code than the human who otherwise would have.

Additionally, using scaffolding techniques, where multiple passes or agents generate a variety of possible answers, can also enhance the output quality. This method allows a terminal model (or potentially even you) to select the best result from a set of responses, leveraging the model’s ability to generate diverse solutions. Incorporating a variety of models into the solutioning and refinement pipeline improves outputs even more.

As an example, here’s an interaction with GPT-4o mini applying just the simple action of performing multiple passes to allow this small model to self-correct (I had to use mini because GPT-4o and Claud kept getting my questions right on the first try).

Prompt: please give me five sentences with exactly five words each and number them 1 - 5.

gpt-4o-mini:
1. The cat chased the mouse.
2. She enjoys reading mystery novels.
3. He runs every morning before breakfast.
4. They traveled to Paris last summer.
5. Coffee helps me stay awake.

Prompt: which of those sentences actually have 5 words? Please just output the numbers.

gpt-4o-mini: 1, 2, 3, 4, 5

Prompt: which of those sentences actually have exactly 5 words and no more? Please count the words from each one and display it after the sentence.

gpt-4o-mini:
1. The cat chased the mouse. (5)
2. She enjoys reading mystery novels. (5)
3. He runs every morning before breakfast. (6)
4. They traveled to Paris last summer. (6)
5. Coffee helps me stay awake. (5)

Prompt: now tell me which actually have 5 words please.

gpt-4o-mini: 1, 2, 5

In essence, the "unreliability" argument underestimates both the power of automated verification and the value of AI as a force multiplier for human productivity. As AI systems continue to improve, the balance of effort will increasingly shift from verification to strategic application of AI-generated outputs, leading to significant efficiency gains.

Conclusion

By enhancing our interactions with large language models—through better prompting, scaffolding, and feedback—we unlock their true potential. Feedback is critical for improving model performance. Unlike humans, who naturally validate their answers against real-world feedback, deployed models have to rely on explicit feedback mechanisms. For instance, when writing a technical report, a human might cross-check data against reliable sources to ensure accuracy. Implementing similar feedback loops for models can help refine their outputs. This approach allows us to appreciate the capabilities of AI and the efficiency gains achievable when humans and AI work together effectively.