But where are all the AI-designed drugs?

The world we now live in:

• A vocal number of commentators who spent the last two years waxing enthusiastic about AI-designed drugs are now publicly wondering, in apparent confusion, where the AI drugs actually are. Or stroking their chins and sagely noting that, of course, the bottleneck was always data, or clinical development, or some other thing they failed to mention back then.1

• The lauded founders of a darling biotech suddenly announce they are departing to start a new venture. What actually happened in the boardroom is not announced. However, industry figures and experts praise the timing and strategy of the transition, studiously ignoring the desultory trial results and bleeding cash reserves sitting behind the curtain.

• “Anthropic is now a biotech” zombie posts bounce around LinkedIn, bereft of any explanation of what it means or why anyone should care.

• Consultancy whitepapers explain that AI will revolutionize drug development, with detailed charts and numbers attached, but no real account of how these were arrived at.

• A pharma company or biotech announces some blue sky target (100 drugs in our pipeline, doing $30B in revenue) and the commentariat praises them for their vision and ignores that it’s mere press release puffery, designed to short up the stock price.

• Investors and analysts talking up a new drug class or modality with the certainty of people describing something that has already happened, rather than something that might, while ignoring their earlier and equally certain deliberations on the previous new class or modality.

Most of this doesn’t qualify as news in any real sense. It’s speculation dressed up as reporting, commentary about commentary. There isn’t enough genuinely new, true information in biopharma on any given day to fill the volume of posts and newsletters and opinion pieces claiming to deliver it. Necessarily, most of what circulates has to be noise.

Of course, none of this is new, and none of it is unique to biotech. William Goldman, who spent decades writing and doctoring Hollywood scripts, had a blunt aphorism about the movie business: nobody knows anything. Every time someone explained why a film worked, or didn’t, they were making it up after the fact. The reasons given were stories, built to fit the results.

Biopharma commentary runs on the same trick. Something happens, and within a day there is a tidy, authoritative explanation circulating on LinkedIn, in an investor newsletter, in a trade press quote. It’s not that biotech and drug development is utterly random and facts are irrelevant. It’s that the immediate, neat explanations are almost always a construction, put together after the fact to fit an event nobody saw coming2. It’s that press releases aren’t news but an attempt to set a narrative3.

Subscribe now

The AstraZeneca-BMS rumor

Recent weeks gave a clean example of the narrative machine running in real time. Reports surfaced that AstraZeneca and Bristol Myers Squibb had held preliminary merger talks4. The commentary that followed was instant and confident. Analysts explained why the deal made sense for BMS (patent cliffs on Eliquis and Opdivo, a need to offset revenue erosion) and why it made much less sense for AstraZeneca (a market-leading pipeline, no obvious need for financial engineering) although maybe it did (a US-listing play tied to AstraZeneca’s recent move to the NYSE). Some called it perplexing5. AstraZeneca’s shares fell as much as 7% on the news, one of its worst single-day drops in years. Perversely, BMS rose about 6%.

Some of this may turn out to be approximately right. But nobody commenting had detailed knowledge of the actual negotiations, the actual motivations of the two boards, details of the deal, or whether the deal will happen at all. Within a day, the market had a fully formed narrative, built on a single-sourced vague report of ill-defined discussions held at some point in recent months6.

Or put it this way: you, I or any one of a number of people could have started that rumor, with no basis in reality. So why are we all talking about something that is near-on phlogiston?

The commentary economy

A messy, uncertain, multi-causal event is hard to write about and hard to read about. A clean story with a villain, a hero, a trend, a moral is easy to produce and easy to consume. Journalists have deadlines. Analysts have clients who want a view. The thought-leaderati need material for posts. They all need to show that they know things. The system rewards confident takes over “beats me.” Having opinions, loudly and often, gets you attention7. We’re so ready to hear stories and explanations that we hunt them down whether they’re true or not.8

There’s a pattern here worth calling out: the people with the most knowledge tend to say the least. The boards negotiating a merger, the C-suite that knows why a founder really left, the scientist who knows a drug didn’t get enough time in preclinical, they’re under NDA, managing legal or professional exposure, or busy doing the actual work. The people producing the most voluminous, confident commentary are often the ones with the most time on their hands, the ones trying to impress and sell themselves, the ones farthest from the information.

And there’s no real cost to being wrong. Nobody audits a LinkedIn post from eighteen months ago. The people who were bullish on AI drugs in 2024 aren’t posting corrections now, they’ve moved on to a new hot take, and it costs them nothing. An industry with ten-year development runways gives bad predictions a long time to fade away. Where being wrong costs something, in an actual R&D program with your name on the failed trial, people hedge more and say less.

None of this requires anyone to lie or to be malfeasant. We’ve just incentized over-certainty and de-risked being wrong.

What to do with this

“Don’t believe everything you read” was true, is still true, tempting, and not very useful. There are better questions to ask.

First, in the face of such a volume of low-value information, scepticism is a useful initial stance. “How do you know this? What is it based on?” Very few of the most vocal commentators show their work, instead just making bald assertions.

Second: how fast, and how confident, is this explanation? A true causal account of something genuinely messy - a merger, a departure, a trial failure - usually takes months to surface, if it surfaces at all. A confident explanation that shows up fully formed within a single news cycle was likely assembled to fit the news rather than to explain the news. Plausibility is cheap and plentiful.

I frequently recommend Neil Postman, because he was prescient as regards the modern media and information landscape. My final point comes from him: if this is true, so what? Would it change anything you actually do?9 Most commentary fails this test even when it’s accurate. The founder’s exit story, the consultant’s whitepaper, the conference paper claiming a small increase in accuracy on a limited dataset, none of it would change a single decision any of us make, whether or not it’s true.

There are exceptions, and it’s worth being honest about them. Merger rumors like AZ-BMS are exactly the kind of information that would change behavior for a certain group of people (investors) if they were reliable. Sometimes we do have technical innovations that change the whole field (e.g. AlphaFold). Which is exactly why they get produced and consumed so eagerly. Perhaps the better version of this second question isn’t just “would this change what I do?” but instead “if I acted on this and it turned out to be manufactured, what would that cost me?” This framing separates the potentially useful from the perpetually white noise.

Anthropic is now a biotech? So what. China biotech is getting stronger? Interesting but so what. AZ is merging with BMS? Huh, so what. A big pharma executive said that their data mesh, digital strategy and AI systems are going to accelerate drug development? So what, they all say that. GSK is moving to Cambridge?10 It’ll take years if it ever happens, and even if, so what11. Why did that drug trial fail? We really don’t understand the biology; anything could have happened, so what. Eli Lilly and NVIDIA have announced a co-innovation lab? The spend is less than 1% of their operating cash flow, this is a minor project, so what. Those founders that left? Had made a huge amount of money and failed to deliver; they were probably pushed out, so what. A pharma company has announced a strategic partnership with an AI company? This happens all the time, so what, did they say exactly what the partnership would do …

Pharma, aerospace, defence, big tech, construction: pick your sector but the pattern of failure is always eerily similar. If you’ve been working in the sector for more than a few years, it’s easy to let cynicism slip in. Things Don’t Work. Timelines slip. Budgets balloon. Teams burn out. The Big Project that promised to solve everything is delayed, refocused, adjusted and finally allowed to fade away unmentioned.

I once asked a bunch of pharma veterans about how many projects they’d been on that had actually worked. Where I defined a project loosely as a substantial piece of work, that involved multiple teams and cooperation across units, on a timescale of at least months. Developing a drug, digital transformation, building a pipeline for daily integration of data, setting up a lab-in-a-loop: non-trivial and non-routine work with significant outcomes. Where success meant a positive outcome, doing roughly what it was supposed to do, without an absurd blowout in time and money.

Most said that successes were far and few between. A few said that they’d never worked on anything successful1.

If you have any curiosity, you want to know why. If you have any pride in your work, you want to know why. How can large, expensive projects, the business of a business2, be so inefficiently executed? Surely, it’s in the organisation's interest to understand and fix this? But many people see project failure as something external like the weather. It just happens and we are helpless against it.

Of course, there’s no shortage of methodology and experts and books, none of which are obviously correct or successful. But some years ago I stumbled upon a expert and a book that, for me, captured a lot of the symptoms and obvious problems, and expounded a set of solutions that seemed to be going in the right direction. William Livingston was a distinguished engineer at the Lockheed Skunkworks, overseeing many demanding but successful projects. He also saw a lot of failures and wrote about them in Have Fun at Work. It’s one of the most brutally honest dissections of how complex organisations generate messes, automatically, predictably, and without anyone intending harm3. If you work anywhere near science, engineering, or large-scale R&D, you’ll recognise almost every page.4

I do not recommend that you read it yourself. It was written in the 80s, is hard to find, every copy seems to be a scan of a poorly typeset original, the tone is eccentric, angry, cynical, and meandering. I also think he’s right. Here’s my summary to save you the read.

Complexity always wins

A system becomes “complex” the moment it exceeds one person’s cognitive capacity to understand and direct it. Complexity isn’t exotic; it’s normal. It’s the way things naturally are because of:

• too many parts

• too many interactions

• unfamiliar technology

• long timelines

• cross-disciplinary overlaps

• high stakes

• tight constraints

Hello, modern biomedical R&D.

When complexity crosses a threshold, organisations start doing what humans always do when they’re overwhelmed: guess, react emotionally, retreat to routine work, reject novelty, and cling to whatever looks like control5. That’s where the trouble starts.

Why no one is to blame

We’d like to believe project failures come from:

• poor planning

• the wrong tool

• the wrong vendor

• a lack of funding

• “not enough time”

But in reality, none of these are root causes. Per Livingston:

Most disasters aren’t caused by bad people or bad tech. They’re caused by misaligned structure and culture.

Organisations fall into failure because of their culture, because of the way they are built:

• implicit rules (unwritten norms) overpower explicit rules

• feedback get suppressed, dissenters are socialised into silence

• specialised teams carve the work into pieces no one can integrate, responsibility is diluted6

• leadership and management are rewarded for optimism rather than clarity

This is why the same patterns occur across every sector even though the technologies are wildly different. The system behaves the same way because the social mechanisms are the same. And these social mechanisms are not conducive to completing complex projects.

The Universal Scenario

Livingston documents a repeated cycle, a kind of institutional Groundhog Day, seen in countless large projects:

1. Ready–Fire–Aim: Complex problems get ignored until they can’t be. People latch onto the first oversimplified solution.

2. Feeding Frenzy: Everyone wants a piece of the action. Nobody understands the problem, but work gets divided up anyway.

3. Party Time: Leaders celebrate. Optimism fills the gap where clarity should’ve been.

4. Camouflage & Drift: Sub-projects and teams drift apart, then hide the drift. Nobody knows who’s in charge.

5. Compounding Errors: Communication degrades. Teams throw their misaligned piece of the work “over the fence.” Whistleblowers are punished.

6. Crisis & Blame: Mismatch can no longer be denied. Leadership churns. Consultants arrive. The innocent suffer.

7. Run–Break–Fix: Operations inherit the barely working mess and patch it indefinitely.

8. Recycle: Later, a new team retries the project, pretending the previous attempt didn’t happen.7

Every major R&D failure you know maps onto this model frighteningly well.

The usual fixes don’t work

When organisations hit trouble, they pull the same levers8:

• more processes

• more standards

• more people

• more reports

• more funding

• stricter specs

• tighter controls

• new software

• reorgs etc.

And these never work. In fact, they often make things worse. Because they attack the symptoms of complexity, not the complexity itself. They concentrate on what is happening and often getting a better picture of what is happening, over why it’s happening. Worse, they generate additional coordination overhead which actively saps away attention and resources.

When a remedy is unrelated to the problem, adding more remedy just creates a bigger problem.

So what actually works?

Livingston’s recipe for success (and he had many successes throughout his career) is refreshingly pragmatic and mostly contradicts standard behaviour.

1. Structure the problem (properly)

A well-structured problem is half-solved. This means:

• clarify boundaries

• map interfaces

• identify constraints

• build a working system description

It’s like drawing the shape of where a solution will fit in. Most organisations hurry through this step and just get started because it’s slow and boring.

2. Protect feedback loops

Fast feedback is the single most powerful tool against complexity. Slow or no feedback means guaranteed failure. Most bureaucracies suppress feedback. Have you ever suggested that a project should was way off course and should be retooled / cancelled / redesigned / taken away from the current team? Optimism is rewarded over clarity.

3. Match variety with variety

Livingston loves quoting Ashby’s Law:

To control a complex system, you must match its complexity.

That means having

• diverse talent

• multiple response modes

• adaptive processes

• tolerance for ambiguity

• genuine interdisciplinary integration

It’s the opposite of “standardise everything” or having an “AI team” or “software team”. You have a big gnarly multi-disciplinary problem; you need a diverse team that doesn’t get culdesac’d into particular solutions.

4. Use the Skunkworks team model

In the experience of Livingston, myself and lots of other people, many people say they are doing innovation. In reality, organisations tend to restrict and crush innovation. You almost have to fly under the radar to get new things done. Innovation only survives in protected pockets:

• small team

• flat hierarchy

• cross-functional

• shielded from corporate antibodies

• free to iterate

• tightly coupled to reality

In R&D, this is often the only place where genuine progress happens.

5. Keep specs loose

Rigid specs in a changing environment are a trap.

Loose, adaptable specifications outperform in any domain where uncertainty is high.

You don’t eliminate messes, you navigate them

So, to sum up Livingston’s bleak but useful thesis:

• Failure is the default in complex organisations.

• The system’s true values are revealed by behaviour, not PowerPoint slides9.

• Success requires conscious design.

• Culture outweighs intention.

• Feedback beats optimism.

• Innovation comes from individuals, not committees.

Seen through that lens, “fun at work” isn’t about perks or positivity. It’s about meaning; the satisfaction of doing real work, solving real problems, and creating an environment where clarity, feedback, and structure make progress possible. If you’ve worked in big pharma, biotech, engineering, or public-sector digital, you already know how rare that can be.

What can we do?

It’s a bleak read. If Livingston had to fight tooth-and-nail to get Good Work done, what hope do we have? But I’m an optimist, a believer in positivity. We’ve done good, useful things done before. How can we do this again? How can we do more of these? So let me translate Livingston-ese into some thoughts and action points for our next projects:

1. The default result for any non-trivial innovative project is chaos and failure. Let’s accept that and act to avoid it.

2. Spend a lot of time thinking about what you’re actually trying to achieve and how you might get there.

3. But hold these beliefs loosely and be ready to change.10

4. Negative feedback and failure tend to get hidden because of politeness / culture / fear of punishment, etc. You have got to encourage it and listen to it or you are running blind.

5. A small, focused, multi-disciplinary team beats all others. Give them the responsibiity and authority to solve the problem and stand back.
   

Introduction

For much of my career, I felt that drug repurposing was the most obvious idea that everyone in the industry was studiously ignoring. But times changed, I changed my mind, new players entered the arena, and I changed my mind again. It’s still a good idea, but as always with drug development, things are more complicated than they seem. So where does that leave us?

For those new to the subject, here’s a brief definition and introduction:

Warning - the terminology around this field gets a bit murky, with different people using different terms (drug repositioning / rescuing / redirecting / re-tasking …) for the same or overlapping ideas. I’m talking in the most general sense:

1. You tried to make a drug D for disease X

2. Maybe you were successful, maybe not

3. But now you’re going to see if drug D works on other, non-X, diseases

Prima facie, this might seem ridiculous. Why would you try to use a (say) anti-fungal drug against cancer?1 But there’s a logic behind it. As compared to de novo drug discovery, a pre-existing drug:

• Exists

• Can be synthesized

• Can be delivered into a patient

• Can be tolerated by a patient

• Has a pre-existing body of data and studies (safety profiles, pharmacokinetics)

• … and thus should be quicker to develop

• Does something, has some useful biochemical action

So the problem is reduced to effectively redirecting that biochemical action to another target. (Or, more often, finding what other targets that biochemical action can be redirected to.) Taking into account that many drugs already have multiple modes of action and that diseases often share mechanisms and pathways, that we do indication expansion all the time, this task is possibly not as difficult as it might seem.

Thus, we have a healthy number of drugs that have been pivoted from their original intended use to wholly new indications: thalidomide (sedative and anti-nausea → myeloma & leprosy), sildenafil (hypertension → erectile dysfunction), itraconazole (anti-fungal → oncology).

So, a good idea? Now, hold on …

Subscribe now

The actual track record

Ask around for examples of successful repurposed drugs and you’ll hear the same examples, time and again. And these are good examples. They are also, almost without exception, serendipitous: discovered through side effects, clinician observation, or pure chance.

This is not for lack of systematic searches for repurposed drugs. Benevolent.AI used its knowledge graph tech to look for COVID treatments. Every Cure is a company looking at checking “every drug against every disease”. You can find a host of academic papers exhorting algorithms for repurposing.

In reality, there has been little success. One of my former students2 spent considerable time looking for these systematic approaches and found, broadly, a lot of confident language and very little actual methodology underneath it. This is not a niche observation. A 2020 paper in ACS Medicinal Chemistry Letters reached essentially the same conclusion: most drug repositioning cases occur more by chance than design. More recently, my colleague Francis Osei has pointed out that the evidence proffered in repurposing is often patchy, fragmented and noisy3. It’s evidence but is it good evidence?4 The deliberate successes exist, but are rare enough to prove the rule.

Problems that won’t go away

Like most things in drug development, getting a repurposed drug into a patient isn’t a single problem and it’s not just a medical or biochemical problem. It’s a set of tangled, interconnected issues.

The biology problem. Repurposing rests on an implicit assumption: that we understand target-disease relationships enough that deploying a known drug in a new context is meaningfully de-risked. But mechanistic understanding of most diseases remains poor. One estimate of how much we know about biology5 places it at 0.1%. If de novo drug development fails because the biology is hard, repurposing approaches the same hard incomplete biology, just from a different angle. Incomplete biological knowledge is still the bottleneck.

The methodology problem. As above. There is no validated, systematic framework for identifying repurposing candidates that consistently outperforms chance. Computational approaches (network analysis, transcriptomic signatures, phenotypic screening, «gestures broadly at the latest hotness») generate hypotheses. They do not reliably generate the right hypotheses. We know how to look; we do not know how to find.

The economics problem. This has been raised before but is usually swept due to lack of a solution. Where do the compounds for repurposing come from? Most attractive candidates for repurposing are often off-patent, which means limited exclusivity, which means limited return on investment. Like it or not, a drug has to make money for it to be sustainably developed. Alternatively, companies have a lot of information on drug candidates that they have partially or fully developed and shelved. But they have little incentive to pursue indications outside their core franchise, or let anyone else have access to these candidates. This is a massive structural roadblock.

The clinical execution problem. Baricitinib was a flagship case for AI-driven repurposing. BenevolentAI identified it as a candidate for COVID-19 treatment; the biology made sense; it reached approval. This story was repeated as a triumph of AI and of AI-driven repurposing.

Look more carefully, and a different lesson emerges. Identification was fast (reported as “48 hours”), but everything downstream (trials, regulatory review, manufacturing, all the “making a drug and putting it into patients” steps) ran at exactly the speed it always runs. It took 9 months to get Emergency Use Authorization and more than 2 years to get full approval. The most celebrated AI repurposing story is a demonstration that hypothesis generation was never the bottleneck, a candidate is just the start of drug development and that no matter how many NVIDIA GPUs you have, clinical validation runs at the same speed and financial sustainability is a long way away6.

A new wave and new hope?

I am an optimist7. I want us to develop more and better drugs faster, I want repurposing to work. But scepticism is not an obstacle to progress. It just has to be applied honestly.

There is a fresh wave of interest in repurposing, energised by better AI tooling and some genuine business model creativity:

• Ignota Labs are doing interesting work but the details are the point: acquiring distressed assets that failed on safety grounds and using AI to unpick the toxicity mechanisms, then rehabilitating them. This is a more coherent economic proposition precisely because they acquire and own the IP.

• Axovant initially focused on repurposing drugs developed by other companies, but following a series of clinical trial failures and the departure of its CEO, it experienced roughly a 90% slump in market capitalisation and gradually transferred its focus from repurposing.

• One of the best recent examples of drug repositioning is mirdametinib, a shelved Pfizer asset. When urged to take another look at the drug by the Children’s Tumor Foundation, Pfizer spun out another company, SpringWorks Therapeutics, which is now developing it for neurofibromatosis and other conditions. But that required an external advocacy push and two years of negotiation with 200 volunteers to extract the asset from its original owner.

• BioXcel Therapeutics is using an AI-driven drug repurposing platform to advance a pipeline targeting neuropsychiatric disorders, a notoriously difficult area. The logic would seem to be that if repurposing can derisk a tough area in any way, it’s worth the punt. Their most concrete success is a reformulation of an existing sedative/anaesthetic drug, approved for acute agitation in schizophrenia and bipolar disorder, and they have Phase III drugs in other areas. And BioXcel is still unprofitable. Which is a depressing reality-check. Most approved drugs don’t make significant money8, and financial sustainability is necessary to keep a drug pipeline going. How well does a repurposing strategy have to be for the economics to close?

• Of course, there’s EveryCure which is getting money to systematically work through repurposing, treating almost like a public good and slightly side-stepping the economic issue.

What would actually work

Repurposing is still not a bad idea. But it’s an idea that requires honest execution rather than brilliant models and algorithms.

Hear me out: The most tractable cases are probably where we can do an end-run around the economic barriers. Say, in rare diseases and neglected indications, where nonprofit and public models can absorb the economic problem that the market cannot solve. Indication-expansion thinking embedded at the start of development, not bolted on at the end, would help. There are regulatory paths for this. And better mechanistic biology would help, the kind that lets you say with confidence why a drug will work in a new context. Network analysis that works by associative, friend-of-a-friend analysis can only take us so far.

I was a believer in drug repurposing. Recent developments have made me more cautious. Not cynical, but wanting to be precise about where the promise and problems actually sit. As with much of drug development, much of it comes down to money. We can make drugs, but can we make them sustainably? The economics may require a non-market solution. That may not be a popular conclusion. But it may be the right one.

How We Got Here

In a way, this all started with our attempts to infer over sequences, arrays of inputs or tokens that have an order and relationship to each other. Text, proteins, DNA, images …

While neural networks were invented in the 60s and had a heyday in the 80s and 90s, and were theoretically capable of approximating any function since the 1980s, they were limited in complexity for many years. When this complexity barrier was broken by improved training (along with GPU compute and large datasets) in about 2012, deep learning had its Big Moment. Suddenly, neural nets could handle big, real, interesting problems. As regards sequences, while the original neural architecture treated all inputs as equally related to (or equally independent from) one another, new architectures tackled the issue of related or adjacent inputs. Convolutional networks solved vision problems by representing neighbouring pixels as neighbouring connected inputs; recurrent networks and LSTMs tackled sequences by autoregression, inferring one position in the sequence and then feeding the results into the inference for the following position. But this approach is difficult to parallelise and has limited complexity.

In 2015, attention mechanism solved this by letting the model selectively focus on any part of the input at any step, allowing inference for any token to use the context of other parts of the sequence, a sort of selective spotlight. The transformer architecture (Vaswani et al., 2017) took this idea, further discarding recurrence entirely, and inferring over any token for using only attention plus position encodings (so the model knows where every token is). This scaled predictably, could be trained in parallel, and generalised across domains.

In a sense, two lineages subsequently emerge from this:

• BERT and related transformer architectures, that consume a sequence as a whole and then are good at inferring and classifying terms within the text (e.g. keywords, names)

• GPT, which would move along a sequence, left to right, working on predicting the next token. This is a considerably simpler task and inherently generative: predicting the next token is the same as generating it. And this is how we get to LLMs …

Truly Weird Machines

Scale a GPT to hundreds of billions of parameters (weights), train it on all the data you can get, fine-tune its communication style based on human preferences (to sound polite, coherent, positive), and you get a Large Language Model or LLM. And these models turned out to have some surprising and unexpected abilities: it can hold conversations, generate code, do arithmetic, (apparently) reasoning. So people could talk to them, ask them for opinions, to make decisions, to do work …

Then everyone lost their goddamn mind.

Before showing where LLMs lead us, it’s important to understand some aspects of how they work. Several properties are structural and cannot be simply patched away:

• An LLM is stateless, has no memory between calls. Every conversation is a series of independent inferences; the illusion of continuity is maintained by passing your previous conversation to the LLM, like the amnesiac from Memento looking at notes to remember where they are and what they are doing.

• LLMs, like all models, are lossy compressions, generalisations of the data they ingest. “Hallucinations” are simply errors, compression artefacts where there is no information.

• Fine-tuned on human approval, these models have learned to communicate in an agreeable manner. Push back and they may cave: not because you’re right, but because you expressed displeasure and they have been trained to answer in a pleasing way.

• They are prompt-sensitive and non-deterministic. Small wording changes in requests can produce different outputs. The same question asked twice can yield different answers.

• A weird thing about the context provided to a model in that attention biases toward the start and end. Important information buried in the middle gets less weight.

• Finally, context size is naturally limited, and as this is approached (by providing a lot of information, by a long history of interactions), performance starts to degrade.

An agent is an LLM in a loop

An agent is an LLM that is orchestrated or directed towards autonomously completing a multi-step task: Observe → Think → Act → Repeat. A request is made of the agent; the agent computes (thinks); then acts or generates something; it examines the result of what it has done, and so on and so on ...

What makes this different from simple chatting with an LLM is:

• Orchestration: there is a framework controlling the agent, handing off work between agents; dictating tasks and stopping conditions

• Tools: the agent can actually do things: search, execute code, query databases, call APIs, read and write files.

• Memory (via context window, vector stores, structured logs) provides the statefulness the model itself lacks.

Agents could do anything, but what should they be used for? The sweet spot is:

• Tasks that are too long for a single prompt, where multi-step decomposition is required

• Tasks that have to be done on demand, dependent on live or data

• High-volume, exacting but repetitive work, where expert time is the bottleneck

• Jobs with a clear success criterion that can be evaluated automatically

In biopharma this could be trial protocol review, literature curation, data QC, regulatory drafting, patient screening. A six-week literature review delivered in hours with citations is a reasonable near-term expectation. Imagine a system that designs clinical trials:

Best Practices

We are definitely still learning how to do this and LLM failure modes don’t disappear inside agents. Silent compounding failure is the characteristic risk, as with any automated system.

• Build evals (tests) before you build agents. Without scored, repeatable tests you are running blind.

• Start simple. Most ‘multi-agent’ problems can be solved with one agent and some tools. And not everything has to be an agent, but can be orchestration, simple code or logic gates. Add complexity only when you hit a real ceiling.

• Keep each agent’s task small and well-defined with a small number of steps. One agent, one job.

• Put humans in the loop for irreversible “failure is not an option” actions: sending, deleting, submitting, purchasing.

• Set hard limits on cost and steps. Unbounded agents find loops you didn’t anticipate.

• Log everything. Traces are your primary debugging tool.

• It’s a good idea, I think, to approach a lot of agent work as being a good way to get to your “shitty first draft”. Then a human can take over, or revise and hand back to the agents.

Vibe coding

If you can use agents to write documents, you can use them to write code. So people started doing that. Even at an early stage, the ability of LLMs was fairly impressive to generate or translate pieces of code. What’s changed is the use of agents to do a huge span of software development: generating & deploy whole projects, writing tests, writing documentation … there came the promise of literally knowing nothing about programming and being able to produce a fully working piece of software.

Then everyone lost their goddamn mind.

There are dangers there (some of the produced software is unmaintainable, contains grievous security holes, can’t easily be extended or modified because literally no one understands it). But there is also a bunch of interesting work to do with specification and test-driven development.

Interesting directions and advice

• There is so much going on that it’s impossible to keep up. You want to use these tools to get your work done, and not just endlessly tweak and configure LLMs all day. Perhaps you should stay one step behind the curb.

• Different LLMs have different strengths. It’s definitely worth looking around. The free levels are sometimes underpowered but lower paid tiers should be fine.

• For vibe coding look at Antigravity and ClaudeCode. Check out some of the work that Github is doing.

• MCP (Model Context Protocol) is an emerging standard for tool APIs

• LLMs carry a lot of useless (and possibly distracting) knowledge. Small models are an interesting new direction, with possibly more focused and predictable abilities.

• Read Andrej Karpathy who is an enthusiast but a balanced and reasoned one.

• Ignore the extremists & boosters who want you to go all in and spend all your time with the latest bleeding-edge tech.

OpenClaw: A Live Case Study

This has got so much air time, that I have to discuss it.

It used to be very complex to build agentic systems, with a wide span of unintegrated tools and libraries. This changed recently and unexpectedly. OpenClaw (formerly ClawdBot/MoltBot) is an open-source agent that combines tool access, persistent memory, code execution, and consumer messaging integration into a system that can run on your local machine, accept instructions via WhatsApp or Telegram, access and manage your mail, your calendars, the commandline …

So, of course, everyone lost their goddamn mind.

People started using it to run their lives, letting it post to social media, manage appointments … It hit 247,000 GitHub stars in months. It also illustrated every failure mode in this essay simultaneously: 500+ security vulnerabilities, 20% of marketplace plugins malicious, an agent deleted a user’s inbox. One maintainer warned it was too dangerous for anyone who couldn’t run a command line.

It’s incredibly interesting. But it’s definitely an early, early preview. The capabilities are real. The governance is not yet there. The distance between that chaos and a properly evaluated, human-in-the-loop system is the practical challenge of the next few years.

Based on a presentation to and discussion with the PSI Data Science group.

“Noise” is a ubiquitous idea within analytics and especially within biomedical analytics - unwanted, extraneous or corrupted data points that obscure the real pattern you want to model. That noise can come from measurement error, sensor imprecision, mislabeling, ambiguous clinical endpoints, qualitative scores, outliers, cross-site variability, (yadda yadda yadda) or - as is the case within biology - simple natural variation. Many, if not most, biological signals just seem to be naturally “fuzzy”, going up and down slightly for seemingly no reason. If you don’t handle this noise well, models can try to overfit on the noise, or collapse when applied to the real world.

Machine learning has spent the last decade chasing bigger architectures, bigger datasets, and bigger compute budgets while largely ignoring this issue. We pretend that all data points are equally informative, or that there’s “good” data and “bad” data and we simply have to decide which is which, purge the bad so we have only irreproachable data left.

Subscribe now

There’s a bit of back literature here that I’m going to elide, but the last several years have seen various attempts as at probabilistic approaches to training on noisy data, where the model infers which points are likely corrupted and reduces their influence. Instead of pretending noise doesn’t exist, they integrate uncertainty directly into the training loop. And this has had clear benefits: a study on EHRs showed that messy chaos of clinical practice made patient data noisy and noise-robust models significantly outperform standard ones on real hospital data.

Alzaraiee and Niswonger (2025) present an interesting take on this (see: https://www.sciencedirect.com/science/article/pii/S1364815224001944). They propose a probabilistic training approach combining classical ML with Markov Chain Monte Carlo (MCMC) simulation, aimed at detecting and “under-weighting” likely noisy data points during training. In effect, rather than treating all training examples equally, this method tries to infer which data are likely “noise / corrupted / low-confidence,” and reduce their influence on the model’s parameter estimation. 

An aside: this is not a question of classifying data as noise / not-noise. That just puts us back in the bad data / good data paradigm. What this and related methods try to do is detect which data is “noisy” (not pure “noise”), quantify the extent of noisiness and weight the data accordingly. In contrast to preprocessing-only noise removal, this integrates noise estimation into the training loop, takes out a human-in-the-loop step and make the noise handling more consistent and quantifiable.

For those of you with a Bayesian tendency, the MCMC approach will appeal. The approach, in a sense, is modelling the data, using a Markov process to “tour” the data distribution. This distribution is a set of plausible splits between the data (noise / not-noise), but because it’s an ensemble, points can be assigned a degree of noisiness rather than a binary good/bad.

(Thankfully, the authors actually validate the approach: first on some synthetic datasets with added noise and then on real-world public water supply data which “may contain authentic anomalous data and unknown noise caused by sensor errors, human data input errors, and errors in the incorrect association between water withdrawals and populations served”.

Some thoughts:

• Probabilistic noise-detection & re-weighting inescapably rests on assumptions about the distribution of noise and data. If those assumptions are violated (e.g. noise is structured, systematic bias rather than random errors), it’s unclear what would happen. This seems like an impossible point to fix.

• Noise isn’t always noise; sometimes it’s rare, weird data. And methods like this have a trade-off: under-weighting noisy data reduces overfitting but might also reduce sensitivity to these rare but real patterns (e.g. rare disease variants, outlier responses, sub-phenotypes). This also seems impossible to handle … but perhaps the correct attitude to take is that an outlier is an outlier, no matter what causes it.

• The computational complexity seems fine, but there’s no way this is handling omics data any time soon. There’s an inevitable overhead combining ML training with MCMC.

Some ideas:

• This might not be capable of handling most ‘omics data, but it seems like it could handle (say) patient data that was in the 10K range. Or perhaps after preliminary data filtering and feature selection, datasets would be small enough to handle (e.g. epigenetics)

• Perhaps it could be used federated or privacy-aware learning contexts (e.g. data pooled from different centres), where data quality is heterogeneous and discarding data is not desirable. Noise-aware weighting could improve model robustness across centres.

Something interesting for sure. And the Python code is available: https://github.com/aymanalz/outlier_detector

Aim

• I’ve been a dedicated user of Evernote for many years. Recently, circumstances have led me to look for alternatives. This is a long list of all the possibilities I checked out over 2025, left here in the hope it might help someone else.

• It’s incomplete (there are so many apps, with new ones every day) and uneven (some apps seemed right and got a closer look, others were obviously not right and so quickly dismissed). These notes are fragmentary and will remain so because they were for my use and are provided as is.

• What you’re looking for may not be what I’m looking for. My criteria are set out below.

• It was tough to understand or assess some candidates because key features were hidden behind a subscription, and I didn’t want to spend money on multiple apps just to test them.

• Likewise, there are a lot of enthusiasts pushing their favourite system, even for ill-fitting cases

Subscribe now

Desired capabilities

General, mile-high view of what I want to use it for:

• Effortless capture: can quickly save thoughts and ideas for later processing

• Contextualization: can add your own context and link to existing notes, to find associated notes

• Emergent structure: be able to piece this together or shape this across time and even change it, rather than having to get it right at the very start

• Search: ensure important information resurfaces when needed.

• Minimize clutter and complexity: so it is easy to use

Exclude or avoid

• Just / only notetakers

• Systems that could do a given feature ... if you only bought or connected another piece of software

• Abandoned systems, those with limited support, solo creators

Desired features

To get the capabilities above, what features are needed?

Must-haves

• Cross-platform, with a decent mobile app

• Task management, that cascades or is aggregated

• Most features should be there, rather than need a lot of endless configuration, scripting, or installation of plugins.

Nice-to-haves

• Tables / databases of decent sophistication

• Tabs, so multiple pages can be open at once

• Web clipper

• Templates

• Organisation

• Tags

• Multiple levels of folder nesting (which is like a universal, top-level set of tags that you could use for PARA)

• Project management

• Ways of shifting captured or daily notes easily into other categories

• Daily pages

• Web / app-less interface, so you can always get at it

• AI interface for creating and cleaning up

Other lists of notetaking apps:

• https://github.com/tehtbl/awesome-note-taking

• NoteApps.info: 41 best note taking apps analyzed over 343 features

The search

Types

Candidate applications tend to fall into a few different categories, with most apps emphasizing one to three categories:

• Notetakers: self-explanatory

• Scrapbooks and pretty notetakers

• Project management and task trackers

• Whiteboards, canvases, and visual notetakers

• AI-organized and searchable archives, summarizing and sorting

• Collaborative tools for teams

My interest is in the first two (notetaking and project management)

Candidates

Some systems are marked up with X (hard no) and ? (maybe).

Agenda ?

• Looks good

• Has categories/subcategories (works like folders)

• Somewhat weird but “Projects” act like documents; “notes” are sections

• Many features in premium, but the free version is very functional

• Integrates with calendar

• macOS/iOS only

• No web interface

• Has to-dos and aggregates them

• Good formatting options

• No text find-and-replace

Affine ?

• Like Notion, but better design

• Many document types

• Checklists/todos but no propagation or global collections

• Big on whiteboards

AmpleNote ?

• Does some note text recognition

• No folders, just nested tags

• Which makes PARA a bit of a problem

• Has most of the desired features

• Although still looks a bit ugly / non-native (like a webapp)

Anytype ?

• Object-based

• Has sets/queries

• A collection is basically a folder - you can select anything to be in one. Sets are queries

• Feels like it’s veering towards a chat/collaboration type platform rather than a PKMX

• Fyi: Uses Intel architecture

• Verdict: feels powerful but a bit raw and awkward

Appflowy X

• One document type with embedded views

• No smart collections

• Tasks: can’t be propagated or collected

Bear X

• Mac only

• Just a note-taker

Capacities X

• Weak web app

• No tables/databases

• No folders

• Feels like a simpler Anytype

• Daily pages — more journaling/notetaking-oriented

Coda X

• More of a project management tool

Constela

Craft X

• macOS/iOS only (no Android app)

• Tasks:

• Detected but not propagated - gives a list of documents that include tasks

• Creators not aiming for full task/project management

• Web clipper: only saves full pages

• Organisation: nested folders, no tags

• Daily pages: excellent implementation

• Projects: none explicit

• Performance: sometimes slow/freezes

• Looks good overall

Docmost X

• Wiki

• Only hosted

Drafts

Elephas

• AI / ChatGPT over “your own documents”

Evernote

• No nested folders - you can only have one level of subnotebooks

• What’s my beef with Evernote?

• Subscription going up and up

• Lagging slightly on features

• Frequently can’t find what I’m looking for, even though I know it’s in there

• Mobile experience is weak

Fluster X

• Uses extended markdown

• Needs you to install Ollama - implications for mobile use?

• Appears to have been written by a crazy person

• Uses an extended markdown

• Bleeding edge - not obvious how to use and doesn’t seem to pick up the test notes and tasks I created

Falcon

• Apple ecosystem only

• Several people talk about this one but I can’t seem to find it

FSNotes

GYST.fr

Heptabase X

• A visual notetaking tool

• Has tasks

Inkdrop X

• A markdown-centric Evernote

• Tracks tasks and tasks in documents

• But no nested folders - although documentation says you can (you can - need to create “subfolder”)

• Mostly it seems like a good markdown editor

Joplin X

• A markdown, self-hosted version of Evernote

• But no real features other than writing notes

JournalIt X

• One-man app, newish

Kortex X

• An AI-driven Notion

• Looks nice

• No tasks, just checkboxes and you can only find them by text searching

Laverna X

• Self-hosted

• Limited development, support

Logseq

• Does tasks

• PDF annotation

• Everything is a linked page in a graph

• A really interesting model - it would be a change in the way of working, but Logseq has something interesting going on here with the way it links disparate knowledge. The task systems is attractive as well.

Loop

• No OS X native app, just a web app

mem.ai X

• No task management

• Does notetaking and organizing

Nimbus Note X

• Like a supercharged Evernote

• Lots of templates

• Very corporate feel

• Verdict: why not just use Evernote

NotebookLM

• Not really a note collection app despite people using it as such

• No good quick capture or mobile

• Helps you remember & find things

Notejoy

Notebooks

• Mac only

NotePlan

• No Android app, web-only

• Somewhat expensive

• Folders can’t be reordered

• Includes templates

Notion X

• Very powerful but ...

• No offline mode

• Too much time spent fiddling - notion seems to almost encourage messing with notes rather than using them

• Model is too exposed/intrusive

Noteplan

• Notetaker and task manager/scheduler

• Daily notes

• Lots of plugins

• Seems to be markdown under the hood

• Gives you a PARA setup

• Look pretty

• Integrates with calendars

Notesnook X

• Cross-platform

• Tasks: yes, but don’t propagate or list globally

• Web clipper: unknown

• Organisation: folders, tags, topics

• Daily pages: no, but default titles to date/time

• Projects: none explicit

• General notes: nicer Evernote, supports private “monographs”

Obsidian ?

• A plain text / markdown in folders based system

• Open source

• Lots and lots of plugins with a big community.

• Conversely almost everything non-trivial needs a stack of plugins to be uploaded. And plugins slow the apps down, or

• A bit ugly

• Indifferent mobile experience

• Tasks semi-implemented without use of external task programs

• Very slow loading and searching ... but it later seems fine. Maybe some caching and indexing are going on

• But the cache seesm to get corrupted often

• Conversely the plain text means that you can do a lot of hacking on it

OneNote

• Mimics a physical notebook — a collection of pages

Opennote

• Helps you write and organize ideas and files

rabbithole.ai X

• Infinite canvas with notes

Radiant X

• Seems to originally be a meeting notetaker, that also acts as an agent

Recall

Research by Un.ms X

• An AI notes organizer

• No mobile

• Mac / Windows no mobile

• Works offline

Reflect ?

• A lot like a nicer LogSeq

• Backlinks and tags

• Nested lists

• Integration with calendars

• Integration with AI

Remnote X

• Very nice knowledge and note management

• But no tasks, just checkboxes

• Emphasis on flashcards, learning, and studying

Roam Research X

• Innovative in its time but now behind the curve

Saga X

• Looks like Notion

SiYuan

• No mobile app

• Looks like a very powerful system nonetheless

Simplenote

Standard Notes

• Stripped-down Evernote alternative

• No todos

Supernotes

• Fully cross-platform

• Tasks: yes, but don’t propagate

• Tables: markdown-based

• Organisation: tags, no folders (but can pin cards to multiple parents)

• Daily notes: yes

• Projects: no

• Tabs: no

• Web clipper: unofficial version available

• Includes graph view and sharing features

Tagspaces

• Document organsier with task management

• Sluggish performance?

Tana X

• A sort of super-outliner

• Has daily pages

• There are to-dos but as a line in the list, which you can sort, but not as a separate entity

Taskade X

• Project-centric

• I have no idea what this app does anymore. AI agents creating apps? It used to be about tasks but now I open it and it immediately tries to create an app

Tinderbox X

• Seems to also be a big hypernotes type app, a little retro and all about gathering information

TriliumNext

TheDrive.ai X

UpNote

• Cross-platform

• Web clipper

• Templates

• A better Evernote

• Looks good

Workflowy X

• Basically a big outliner

Xtiles X

• Cross-platform

• Web clipper

• Aims to replace Evernote, Google Docs, and OneNote

• A sort of scrapbook approach

Zettlr

• Markdown projects and editor

• Exports to various forms

Conclusion

These candidates seemed worth investigating in depth

• Amplenote: has almost all desired features (except folders)

• Evernote: has improved markedly over the last year

• Noteplan: pretty-looking notetaker with a decent tranche of advanced features

• Obsidian: low-tech and roll-your-own but potentially powerful

There’s so many candidates out there, I think driven by people deciding to write something for their own needs and by the GenAI wave. Most of these tools I expect to fall away.

AI in these apps can be useful - cleaning up documents, aiding research, search and so on. Some apps however have covered themselves with AI.

Flower Labs invited me to their recent AI day in Cambridge. While it was great to catch up with old colleagues (and visit Cambridge, which will always be a special place for me), it was also good to see so much exciting work in a field that I feel has been teetering on the edge of usefulness for years. Some bullet points for your edification:

– Federated learning is where multiple sites collaboratively train a shared ML model without exchanging their individual datasets. There’s a lot that’s attractive here - you can leave data in place, avoid tricky governance issues, defuse problems like sharing sensitive or commercially valuable data.

– Having said that, we’ve been hammering at federated learning over biomedical data for years, c.v. the many previous efforts like OHDSI / OMOP, Redcap, etcetera, etcetera. Yet, it’s always had a lot of friction. Designing decentralised ML algorithms is not easy. Managing a network of distributed nodes is not easy. Just getting a compute node into a hospital or registry—the site of much interesting data—is not easy. (Years ago, I attended a talk where some researchers waxed lyrical on their “amazing” framework and how it opened the door to all sorts of interesting analyses. Afterwards, I approached them and asked how they strong-armed hospital IT into installing a new compute node that was sending data to the outside world. They laughed and said they hadn’t. But when they did …)

– Flower Labs is a startup out of Oxford, the source of the flower toolkit for federated and decentralised AI. It’s framework agnostic (PyTorch, HuggingFace, etc.), solves lots of problems out-of-the-box, and scales to millions of nodes. And it’s already been used for several real use cases. At last.

– I was particularly impressed by the work of Ismail Moghul and eye2gene, aggregating eye imaging from around the globe as a way of proxying for tests for genetic defects affecting the eye. The whole idea is that this needs data, a lot of data, diverse data, and getting hospitals / medics / whatever across the global to release this data is an intractable problem. So don’t try to get it released, leave it in place and query it there, with full privacy and security ensured …

– The problem of placing a node in a secure and conservative IT environment that is probably disinterested in your work (i.e. hospitals) isn’t solved1 … but there were a few clever solutions showcased, including thin clients residing in their own AWS subnet.

– All of this, of course, requires a lot of infrastructure work and maintenance and costs. There are some ideas about how best to do it (one good idea was to use resumable pipelines on cheap spot instances—if it fails or you get kicked off, the pipeline just picks up where it left off), but I think this will remain a keen point. And a lot of this work is being done by people who aren’t skilled at MLOps2.

– Data imbalance also remains a keen issue. It’s almost a universal attribute of biomedical data and we don’t get to ignore it3. Federation offers some solutions (i.e. as a way to gather more diverse data), but many promising sets become a lot smaller once you examine distribution.

– The flipside of this is also true and I know Kevin Garwood has spoken about it often. Researchers tend to accumulate data, like magpies, with little consideration for marginal value—how much does this add to the analysis? Is it just more of the same? Is it worth the curation effort?

Thanks again to Flower Labs and thanks to Pedro Mesa for the organisation.

Some thoughts:

They present what is essentially a take on the infamous Eroom’s Law, showing that the number of new drugs developed per R&D dollar has halved every nine years since 1950.

Src: EvaluatePharma, IRR analysis

Many have pointed this out before. It’s been one of the stated motivations of investment in AI (and every previous “disruptor de jour”). In some analyses, the line wobbles a bit at the bottom (about post 2018), and some have taken this as a good sign. But at best, it just means a bad situation isn’t getting worse1.

So what to do?

Most, if not all, drug hunting efforts have swung in behind the idea of genetic evidence. And why not - it’s a no-brainer. Drugs with genetic evidence have double the chance of success. But it’s not that easy. Everyone expected that the Human Genome Project would catalyze a torrent of new therapies but, crudely, genetics turns out to be really complicated. As Nuzhna observes:

Mendelian variants overwhelmingly describe rare disorders. The majority of the genetic variants are not rare though, and tend to have a small effect size …This leaves drug developers relying on genetic targets with a tough tradeoff: small populations with high-confidence targets, or big populations with low-to-medium confidence target

There is some nuance here. Studies on rare or unusual variants can lead to mechanistic insights or new therapies, perhaps by showing how a population is protected against a disease. (Look at some of the genomics of understudied populations.) But it’s a problem. You could try to develop drugs just for rare diseases, but that brings in a host of problems in running trials, getting approval and more.

Okay, so let’s repurpose old drugs or those that got discarded or dropped during development. Lots of people have suggested this, and some organisations have even made it their primary focus. Hell, I’ve talked about this many times before. There have been some amazing successes here; if it works, it could save much, if not most, of the development costs.

But, as my colleague Sanjay Buddheo has observed, there’s actually not a lot of systematic knowledge on how drug repurposing works. Historically, it’s mostly been driven by serendipity. But what approaches work, and how well do they work? We have no idea. Repurposing seemed and still seems like an effective approach. But we’re stuck in the land of “seem”2.

What about AI? Nuzhna observes:

the question of whether AI will be helpful in drug discovery is not as interesting as the question of whether AI can turn a 2-billion-dollar drug development into a 200-million-dollar drug development, or whether 10 years to approve a drug can become 5 years to approve a drug. AI will be used to assist drug discovery in the same way software has been used for decades

This I am not so sure of. There have been many and various claims that AI will accelerate development / slash development time. It’s been many years since John Overington quipped that this “slashing” usually added up to saving a few weeks or months off a particular stage of development. Given the long, multi-stage process of drug development, it may be foolish to expect a single innovation to make significant savings across multiple stages. Especially with so much of development friction being caught up in the complicated, messy (and hard to compress) world of clinical operations. And, so far, the evidence is weak that AI-designed drugs make much of a dent in clinical progression. It is more reasonable, IMHO, to expect AI to grant not revolutionary but evolutionary increases in efficiency, or to look for improvements not in time but in the probability of success at any stage. For example:

… being able to predict efficacy in humans is what will ultimately bend the painful economics of biotech. This is also the direction in which almost no progress is being made in the current AI/biotech landscape.

So what does that leave us? Quite reasonably, Nuzhna observes that (paraphrasing) the way you make a successful drug is to make one you can sell to a lot of patients3. But I’m not persuaded by what they nominate as the obvious market, longevity and aging. No doubt, it looms large for everyone, especially the rich and well-off4. But aging isn’t a single coherent condition, as much as a constellation of (waves hands) stuff going wrong, wearing out or breaking down. The epidemiology is complex as well, when you have a whole lifetime of habits and exposures to sift through.

Maybe, as they hypothesize, it’s a question of needing the revolutionary technologies to mature. Maybe AI needs to get much better, away from the “stochastic parrot” model to something that can really understand context, cause and effect. But that’s a discussion for another day.

For the last year or two, when people ask me how long BioinformaticsLondon has been running, I’ve hazarded about 10 years. In this context, “about 10 years” is practically equivalent to saying “a long time ago, back in the Dreamtime”. In reality, Meetup doesn’t make it easy for you to find these things but I finally tracked down the date that the first member (and founder) joined: September 5, 2015. So it’s our tin anniversary.

What is BioinformaticsLondon anyway?

You can find us here:

https://www.meetup.com/bioinformatics-london/

Our under-maintained description says:

For anyone interested in #Bioinformatics or discovering solutions to analyse #Biodata. We are omics flavor agnostic and #Reproducibility best practices curious. Discussions may include anything from Machine Learning applications to biological data and and the statistics of multi-omics integration to Rmarkdown tricks and #FAIR data. Bio-analysts, software developer and engineers of any level welcome!

As time went on, our audience expanded:

A social hub for bioinformaticians, bionanalysts, computational biologists, biostatisticians, genomicists, epi-informaticians, health data scientists, epidemiologists …

Or, as I like describing it to outsiders:

Every two months we get together in central London to listen to a technical talk, ask questions and discuss. Then we go to the pub and complain about our jobs.

Subscribe now

Some history

I arrived shortly after the start of the group, which was prompted by our founders (Nathan Lau and Mark Bartlett) noticing that there was really nowhere for this new “bioinformatics” thing to be discussed. Participants were frequently embedded in a larger lab with no support and no one who really understood what they were doing.

Bioinformaticians had no one to talk to.

And so it started. It took a while to work out what worked. Still, we eventually found the formulae: Every two months, somewhere in central London, a mid-length talk, aimed at the informed generalist, which acted as an excuse for the gossip session at the pub afterwards.

We’ve found new colleagues, new friends, new solutions and technologies to use or avoid. There were good, even great talks. (Mercifully, we’ve never had a downright bad talk. Something to look forward to?) We’ve seen rising stars and companies, and seen a few on the way down, too. The audience has changed too - at the beginning, we were dominated by research assistants from universities. Now, it’s more students or scientists from industry. Bioinformatics is no longer this weird, obscure thing, hidden away in university labs, but a recognised function and career path.

Our leadership has changed, too. Stephen Newhouse brought in a similar group he was running at Denmark Hill. The tireless Manuel Corpas bought into his group, which he runs in Cambridge. Andy Nuzzo has joined us from GSK. And of course, people have left - Mark for a new job, Stephen for the North and a better life, Nathan back home to Hong Kong. We miss them and thank them.

The challenges

Of course, there are the usual hassles of running a regular meeting.

• Sometimes, people are the problem1. There are the speakers who are difficult to schedule or who suddenly go radio silent and stop communicating without explanation2. There are the speakers who have been extraordinarily flaky3. There’s the standard meetup issue of members RSVPing but not showing up. But on the whole, people have been well-behaved.

• Finding a venue to host the meeting has been a constant battle. In our days at the university, we could usually find a room somewhere, at the constant risk of the university losing our booking or scheduling another meeting on top of us4. Ben van Zwanenberg was a generous host at Hays Life Sciences for many years. Post-pandemic, it’s been especially challenging. Currently, the University of Westminster is our base, but hosting arrangements always feel a bit precarious.

• Meetup was the logical choice for hosting the group, having the reach, necessary tools, a decent interface and good discovery for running a regular meeting series. However, the deterioration of service5, steady removal of features6, a series of bizarre decisions and pivots7, and frankly extortionate prices8 have us looking elsewhere9.

• How to publicise meetups has been a constant question. Everyone gets too much email, everyone is a bit numb to Yet Another Annoucement. At the beginning, we stuck physical signs on bulletin boards and mailed departments. With the growth and increased diversity of the field, this is an even bigger problem today. How do we reach people?

• Money has been a problem, although not a big one, thankfully. We’ve always run BL “off the books”, as collecting even a small amount of money incurs a lot of other issues. Occasional sponsors have paid for more lavish events, but primarily we’ve flown through on a bit of scheming, good connections and a little bit of money from our own pockets. I feel, however, that these days are over, due to Meetup fees, and the greater difficulty of finding venues.

The future

And there’s the question of group direction. 10 years is no small thing. It says a lot for Nathan’s original vision for the group (stated as “man, people just want to have a drink and bitch about their jobs”). We built a community where there was nothing, amongst a group of people who are not known for their social skills. We made an event that people wanted to come along to just to catch up, just to hang out with their peers. 10 years.

So, what’s next?

BioinformaticsLondon need not do anything more, it’s working well as a low-key networking talk group. But could it be that and more?

• “Bioinformatics” was once the obvious name for people who did mathy / statsy / computery stuff to biological data. It was the only word we had. Nowadays, it feels like there’s been a sort of linguistic drift and “bioinformatics” means - or implies - something a bit different. There are many people who we’re not reaching, not finding, because they don’t think what they do is bioinformatics.

• Likewise, there’s the question of location, in the large and small sense. London/Cambridge is a hive of activity these days. But how do we find those people, make it easy for them to come to the meetings? Transport between London and Cambridge is neither cheap nor easy and the issue is complicated by the location of busy but inaccessible campuses (Adenbrookes, Hinxton, various research parks). University venues have worked well for us, but they may be preventing us from reaching, again, that industrial audience. It would be great to be amongst the small biotechs or the Knowledge Quarter.

• What is it that the group should be doing? We’ve had proposed more networking, careers talks, hackathons and more. These are all good ideas, but I think we want them as well as the core social function. And we need the right people and the right venue to support these as well.

Here, there are many questions, and maybe not too many answers. But maybe this is a good time for change. Here’s to BioinformaticsLondon and its leaders. Here’s to another 10 years.

From a LinkedIn post, lightly edited.

I often receive requests for career advice or suggestions on finding jobs. This post is neither of those, but is connected.

These are grim days. There are a lot of solid, mid-level life scientists out of work, months into the job search. Or maybe stuck in a job that's less than them, or that's turned bad, but unable to find anything else. As said, I often get asked for career advice. These days, however, it feels less like advice and more like talk therapy. That’s what this post is.

The job market isn't just bad, it's malfunctioning. Jobs are posted and taken down within a day. Or posted then withdrawn. Hiring managers are swamped by 1000s of applications. Interview processes stretch into months, and even those who make it to the final stages are being ghosted. Permanent employees are being treated like disposable contractors, threatened by a perpetual wave of successive layoffs. C-suite feuds and ad hoc decisions are shuttering sites or triggering reorgs for no discernible benefit.

You cannot change this. But know this:

- In a long enough career, you're going to suffer at least a few reorgs, layoffs and downturns. This is not your fault. It's like a storm, an earthquake, or a plague, something that happens to you. Don't beat yourself up for things outside your control.

- You've been getting endless advice about how you should tune your LinkedIn profile, network, optimise your CV, get AI experience, give talks, do certifications, etc., etc. All of this is good advice. All of this is useful. None of this is a panacea or guarantee. You could do all of this perfectly and still not get a job. You could grind yourself into dust following all this advice, and it might lead nowhere.

- Well-meaning people will tell you to hang on, be positive, keep your chin up, something will surely turn up, to hope. Do what's necessary, but it's okay to stop hoping. Hope can be poisonous. It’s okay to stop hoping. But keep moving, keep doing something.

- It's also okay to give up. Sometimes giving up is the right decision. It's not failure, it's making a choice for yourself.

Best of luck to all you job-seekers. The market may improve; it may not. It's just the weather. Make decisions. Have options. Take care of yourselves.

A month or so ago, I was doing some consulting about AI and pharma strategy, and speaking at some conferences. During that time, this question was posed to me several times:

If you had to bet on a (drug / biotech) company, invest, set their strategy, what would you look for?

It's a fair challenge. There's plenty of people quarterbacking the industry, jeering at all the failures and misses. It's a lot harder to plot a positive course of action to be tested by the vagaries of the market. So, as an intellectual exercise, some answers, in no particular order.

Subscribe now

Whats the question, exactly?

• You could interpret it a number of ways, but there are two, somewhat incompatible ways:

  • How do we get a good return on investment in biotech?

  • What's the most effective and efficient way to develop therapies for patients?

• These are not the same thing. I'm biased towards the second. But sustainable success in the second depends to a large extent on success in the first. Although they often operate on different timescales.

What's the current situation?

• People have been trying to work out how to best run a pharma company and develop drugs for decades. If the answers were obvious, easy and comfortable, everyone would be doing them.

• Drug development as it stands is a casino economy. It's a series of long-shot bets. Almost all of these bets will fail. Even most of the "wins" will give only mediocre returns. The industry is driven and supported largely by the rare blockbuster wins.

• As a result, drug development is very much a game for those who are rich, patient, with a very healthy appetite for risk.

• As a further result, even the smartest people in the world with the best ideas making the best decisions can still fail through no fault of their own.

• Having said that, just because someone has spent years and billions of dollars doing something, doesn't mean that it's a good bet. The industry has a tragic habit of latching onto the latest "disruptor de jour" (c.v. Derek Lowe) and insisting that it will be the the solution to all our woes.

• To paraphrase a famous quote, no-one sets out to develop a drug that doesn't work, is too expensive, that crashes out of an underpowered trial, with unanticipated side effects. Still, here we are.

• Much is made of Eroom's Law, the phenomenon where the cost of developing a new drug doubles roughly every nine years, leading to ever-diminishing returns on R&D. The doomsayers point out that this has held for the last 50 years, with increasing amounts of money leading to fewer and fewer drugs. The hype-artists will point at the most recent years where it looks like the trend line has altered, and gesture at AI or improved efficiencies. But the course hasn't reversed, it's just plateaued. At best, a bad situation has stopped getting worse.

• There is a huge gulf between pre-clinical and clinical development, things that work in the laboratory and things that work in patients. Most promising drugs get wrecked upon this reef. As a consequence, a drug asset before this gulf is worth little, is nothing but potential. A drug asset that has passed even the lowest of clinical tests, is worth a lot more.

What does a "good" company look like?

• If a company can't explain what they're doing, that's a massive red flag. I mean really explain, at a level above an elevator pitch, that details the unique proposition, why they will succeed where others have failed. Many companies can't, and just fall back on slogans ("accelerating drug development with AI", "getting new drugs to the patients who desperately need them") and so are indistinguishable from dozens of competitors.

• Similarly, if a company is relying on how smart their staff are, or the value of their "proprietary technology", they are built on rotten foundations. There are lots of smart people out there. There are lots of proprietary technologies. That's not a distinct edge.

• I'm instantly suspicious of any AI-first company in this space, because they're trying to hand-wave away the complexities of biology and treat it just like a data problem. Biology isn't like analysing sales records or traffic song plays. It's a system that we're discovering and that we barely know which are the important things to study. Context is everything.

• However, a colleague cautions me that some "AI-first" companies may actually know and use a lot of biology, but are using the AI label for fund-raising purposes. So, one has to look beyond the hype. What are they actually doing?

• This is not a dismissal of AI. It's massively important, especially when we're looking for clues in vast and incomprehensible biological systems. But these are just clues, just hypotheses, that depend on the data we use and the assumptions we make. AI leads have to be explored and validated. We need biology in-the-loop, a back-and-forth between the analytics and the lab. We don't get to abstract it away.

Other directions

• I'm distrustful of panaceas, of companies that are going to solve all of drug development across all indications. I don't think the problem - the problems! - is that simple. I don't think there's just one answer. I don't believe a company can be sufficiently competent and informed across many indications, many therapeutic modalities to effectively develop a drugs within all or any of them.

• Conversely, if a company has a very focused disease target and cultivates expertise in the sphere, that's a green flag. It's an admission of complex biology, gathering the sort of domain knowledge you need to get past barriers in late development.

• Put another way, a company cannot do everything. It cannot be good at everything. This applies even to big multinational companies. Companies would do better to recognise what their core expertise is and partner with companies with different expertises.

• In a very real sense, the large pharmas have outsourced innovation and risk to the smaller companies. This doesn't mean that a big pharma don't make their own drugs and never make breakthroughs. But the dynamics and incentives are such that innovation and development is much easier and more likely in a small company. (See "Have Fun at Work" by WL Livingston.) And the industry knows this and relies upon it.

• In a gold rush, you can make a lot of money selling spades. Likewise, there's a lot of potential in selling services, tools and materials to the frontline drug-makers. Your returns are much more immediate and less high-risk. And people will always need antibodies, real-world evidence studies, local expertise in health systems, software engineering consultants ... Someone once quipped that there's a lot of boring work out there that still needs to be doing. You'll probably never be the lauded hero, but it's a lot surer bet than the drug development casino.

• I'm distrustful of those selling platforms who can't demonstrate successful outputs of those platforms - for example, 😷 AI drug discovery companies selling platforms for drug discovery that have never discovered a single validated drug 😷 ...

• Immunology is increasingly key. Cancer in many ways is a disease resulting from immune failure. Differential response to drugs often seem to lie in immunity. Some diseases not usually regarded as diseases of immunity (e.g. diabetes, Alzheimers) nonetheless have strong links to inflammation and the immune system. Understanding immunity even a little better could be the key to so many advances.

• There's a lot of unglamorous diseases, unmet needs that don't get attention relative to their impact on the population. Being willing to go off the beaten track could give outsize returns.

Let’s walk through some of the discussion and think about it.

• Straight out of the gate, how bold and correct to exclude candidate generation with AI - it’s the cliché that every AI drug startup trots out and not the roadblock that they make it to be. I suspect it’s just an attractive and easily measurable metric, and allows them to ignore the messiness and complexity of late dev.

• Jeremy Leipzig points out that even for approved drugs, there’s scant information. You shouldn’t be able to patent something and still try to be secretive about it.

• Sarah Constantine points out that the biggest failure point is still at PhIII, the final stage of development. Which is insane. Why aren’t we failing earlier?

• Blake asks for lowering the costs of trials … but I’m not sure a major reduction would be possible. Trials are complex, costly things, although perhaps more efficiency is possible.

• Roberto Munita asks for better animal models (including non-mice) and larger pre-clinical trials. We certainly could use better and more diverse models (e.g. the standard model for asthma is forcing mice to breathe smoke), but with the turn against the use of animals, this is a tough call. Sometimes you can only learn something from putting it inside a complex biological system. However, I’m completely onboard with having better and broader datasets, by whatever means.

• Carlos Costa asks for better surrogate markers to replace PFS and OS. This is a good call - OS may be the gold standard, but patients often have more nuanced ideas of what constitutes a good outcome (e.g. less time on therapy, more time to next therapy). Do we need better metrics?

• Aaron Edwards points out that the bottleneck is not discovery but the clinic. And relatively few people are looking at that. He also asks for more sharing and reusability of frameworks and approaches 👏 👏 👏

• Christopher Li disagrees about the bottleneck but points out (correctly IMHO) that saving and transferring knowledge inside companies needs to be improved. This is so, so true. So much information is locked up inside the head of individual scientists.

• Matt Schwartz asks for less focus on AI, more focus on AI and novel datasets. Much like how others have said that AI can’t be a differentiator, but AI and a unique dataset can be a moat …

• Alex Federation asks for less focus on tools and more on outcomes. Not “we have this tech, what can we use it for” more “we need this disease cured, how can we do it?”

There’s more. Go across and give it a read.

Last week, J&J announced a shift in their AI strategy, described as "The company is making a shift to focus on only the highest-value GenAI use cases and shut down pilots that were redundant or underdelivering". It would be easy to make fun of this - 'stop doing things that don't work and do more things that do work' sounds like the sort of cheap business advice that you'd find in an airport bookshop1. But there’s something very real and widespread here:

The majority of biomedical AI innovations rarely move beyond proof of concept into production, with most prototypes failing to translate into real business value.

Most anyone in pharma who has worked with or around AI has seen examples of this:

• IBM Watson for Oncology was supposed to help doctors recommend cancer treatments. It performed well in demos but failed in real hospitals, recommending inappropriate treatments. Watson also stumbled when used to mine scientific literature for drug discovery.

• Many, many AI-led drug discovery companies have struggled to produce assets that move into trials or even beyond basic pre-clinical tests.

• BERG Health's platform for finding oncology biomarkers and targets "discovered" biomarkers that weren’t reproducible in third-party studies.

• Many and endless statements have been made about how AI is going to replace radiographers / pathologists / etc. “next year”. To pick one example, Pathai promised to speed pathology slide interpretation with AI, but early examples couldn’t handle variability across labs and processes (stains, scanners, etc.)

• DeepMind’s Streams app was meant to predict acute kidney injury but integrated poorly with NHS workflows and ran aground due to ethical considerations, having played fast and loose with patient data.

And these are the examples that made it to production, only to fail visibly in public. We don’t see the remainder of the failure iceberg, where projects shamble along for years, always on the cusp of delivery, before being quietly cancelled.

Let's excavate this problem. As usual, note that I believe firmly in the potential and power of AI to revolutionise drug development and biomedicine. But potential doesn't mean practical or proven. Why is there a gap?

Is there a problem?

The concern is not new. Four years ago, Vibhor Gupta and myself held a workshop on this very problem. The response was muted. Most everyone saw the problem, agreed there was an issue, but felt it was a minor or transient, or that someone else, somewhere else would work it out. But is the gap real? In fact, it’s been studied multiple times across multiple industries:

• Only 15% of AI projects make it to production (McKinsey)

• 3 out of 4 AI projects fail to deliver ROI. 75% of executives say their AI projects did not yield substantial business gains (Boston Consulting Group / MIT Sloan Management Review)

• 85% of AI projects deliver “no measurable value” (Gartner)

• 80%+ of pharma leaders see AI as strategic, but say very few projects move beyond pilot phase (Deloitte)

• AI adoption plateaus around the proof-of-concept stage (MIT Sloan AI Adoption Reports)

• 85% of AI pilots in pharma are never deployed (BenchSci)

• 88% of AI pilots fail to reach production (IDC / Lenovo)

• 42% of businesses scrapped most of their AI initiatives in 2024, up from 17% in the previous year (S&P Global Market Intelligence)

There's a possible riposte to this, asserting that most projects or innovation initiatives of any kind fail, and this is just the natural attrition of experimentation. That's perhaps true, if non-falsifiable. But many of these experiments have promised much, consumed large amounts of resources over many years, been scheduled for production use … only to run into endless delays, excuses, reduced expectations and finally quiet cancellation. Even if you believe this failure rate is appropriate and just the cost of experimentation, it’s valuable to study what separates the successes and failures, to better understand and improve the process.

Why do so many AI projects fail?

Here’s my one-liner summary:

Biomedical ML/AI is performed by many different types of people with different knowledges, different skills, different incentives and different goals, leading to systemic misalignment.

Let’s step through this.

AI projects are usually misaligned

Most AI proof-of-concepts fail to translate into real value because they start life disconnected from business problems. It’s easy for projects to be initiated by the tech end of the business, wondering what’s possible rather than what’s useful, with poor scoping and requirements gathering. What is the actual problem this piece of software solves?

This can happen because the data science and AI teams are often stuck away in a silo, isolated from the actual stakeholders. Development takes place 'over there' and is done by 'computer people'2. So they end up solving problems that no one actually cares about, or solving them in ways that don’t actually reflect how the business does things.3

Complexity is added in spaces like healthcare and pharma, due to data note being abundant or easily available. Because of small patient populations, silo’d info-systems, legacy infrastructure, changing formats, etc. etc. Or the data is available but has to be accessed by or plugin into particular workflows or systems. We often say that pharma is data-rich. In reality, it isn’t.

In addition, compliance is mandatory and often complicated. You can’t just share data casually; that data has to run on validated and audited systems with stringent security requirements. This clashes with the blissful utopia that most AI apps are developed in, where data is freely available, limitless and yet also impactless, of no value or possible harm to anyone.

Many years ago, I was working with a group of epidemiologists who were obsessed with mosquito wings. There was a logic to it: different species of mosquitoes can be identified from characteristic patterns of veins in their wings. Different species carry different diseases. QED, epidemiologists would trap mosquitoes and then consult “The Big Book of Wings” to find out what they were dealing with.

I saw an opportunity. Why not digitise the wing images, use some fancy AI to match a new wing with the pre-existing corpus? Much better and faster than looking things up in some old-fashioned book, amiright?

The epidemiologists were polite, but demurred. They liked looking up wings in the Big Book. It didn’t take too long anyway - a few hours of page-turning, which they enjoyed. The answer wasn’t massively time-critical anyway, a few hours was as good as a few minutes. And to match a wing in my fancy 21st Century All-AI system, they would have had to photograph it, import it into the system, run the software …

No one needed my brilliant wing-recogniser system.

No one cares

Even a good model will die if users don’t trust, understand, or care about using it. And a lot of the time, no one has thought about change management - adoption and transition, how do we get people onto the new system? The actual users keep defaulting to the old systems they know and understand, while the new tool gathers dust.

Even if the end users were impressed by the prototype, if there’s no business unit, sponsor or budget committed to moving it forward, it's not going to happen. This happens so often in big pharma - an R&D or innovation team puts together a valuable prototype or sometimes even a finished system, but there’s nowhere for it to go and live, no one to be responsible for it. So the R&D team ends up taking care of it, even though they don’t have the skills or resources to do so.

We're just plain bad at writing software

The pandemic triggered a firestorm of applications, code and analyses, from all sorts of people, from all sorts of backgrounds. Some of them wanted to help, others just wanted “to help”, others to grab some of the limelight, or to latch onto some funding. The net result was unimpressive. Several papers surveyed the results.

Laure Wynants et al. (BMJ 2020, Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal) surveyed 232 models that variously promised to help diagnose COVID, predict patients outcomes etc. and assessed them for clinical suitability. In short, could these systems actually deliver any clinical benefit.

Of the 232 models, only two could be argued to hold any promise.

The reasons were multitudinous and varied. The system was opaque. It used data that was difficult or impractical to gather. It targeted the wrong or inappropriate patients. It was developed based on inappropriate data. It was never validated.

Mike Roberts et al. (Nature Machine Intelligence 2021) did a couple of similar studies, looking at using ML over medical images to diagnose or prognosticate COVID. They accounted much the same issues, plus the added issue of some datasets including duplicated images, due to the authors just combining datasets haphazardly:

Of 62 studies that could be adequately assessed, none of the models identified were of use due to methodological flaws and/or underlying biases.

And this is a sad statement of the quality of much code that underlies AI models. It’s poorly written. PoCs are often just literally that, thrown together in Jupyter notebooks with no consideration for production deployment. You could understand this problem, looking at academic software - a lot of the critical software in science is built by people who aren’t programmers and aren’t primarily interested in delivering robust, software to be used by other people. But even informatics professionals can write janky, barely-functioning code. The drive for interesting results and flashy demos overrides good software engineering or data science skills. There are few incentives to do things right. At a recent industry roundtable, a speaker sighed, “It’s no one’s job to make systems run right. You don’t get rewarded for being careful.”

At AstraZeneca, we used to talk about “university quality code” - runs on the author’s local machine in their local account, kinda works, kinda shows something, not really reproducible. In the real-world mess of legacy systems, data pipelines, and workflows, will you be able to get the data you need and make a system that runs? Unclear.

This is important. There’s no zero-cost for bad biomedical AI models. Every bad one consumes space and attention from useful models.

Is this just an IT problem?

Similar complaints can be found about IT and software development projects, going back years if not decades. But it's not just an IT problem. AI-centric projects fail at twice the rate of non-AI projects, as they inherit a lot of problems of IT initiatives and layer on complications. But AI-driven systems have their own unique challenges:

• Model outputs may be probabilistic, which is fundamentally alien to work functions that expect answers to be definitive.

• Model outputs need to be explained or justified, especially in regulated or patient-centric spaces, which can be difficult with black box systems. This is even keener

• AI systems have a greater and more singular dependency on data. In a very real way, the data makes the system, changes what it does. Data can be biased or wrong in many subtle ways. Or it can be right but in the wrong way, based on irrelevant characteristics (the “wolf-husky problem”). Furthermore, as the data or population changes, this can result in performance decay over time.

• Arguably, AIML systems are used on more complex and sensitive issues (e.g. patient selection, treatment choice, diagnosis), which makes the potential impact of biased or incorrect models catastrophic.

We may be running a massive multiple-hypothesis test

Consider:

• Maybe millions of researchers working on similar problems

• Using different approaches and assumptions

• Using different data, processed differently

• Using different software stacks

• Using different tunings & hyper-parameters on these models

• Throwing out models that “don’t work”

How many of our results, our “good models” are due to simple, dumb chance?4

The solution …?

If the solutions were easy or obvious, everyone would be doing them. But that's a topic for another time.

It’s slightly ironic that it took me 18 months to read Deep Work.

I won’t comment on the level of irony involved in me taking years to write down my thoughts about it.

Cal Newport’s books (So Good They Can’t Ignore You, Digital Minimalism, A World Without Email, et al.) have a strong following and for good reason. Newport has identified a lot of the malaise and troubles of the modern knowledge worker, which I would crudely summarise as:

There is always too much to do and modern life and the modern work environment seem to conspire against us actually getting any of it done.

These are very familiar ideas to me and, I’d guess, to most of my readers. Development projects that consist mostly of meetings. Calendars stuffed with team meetings, town halls, project standups and executive “fireside chats”. Spending most of the week just trying to get on top of emails. It’s a mess, and it’s not just me. I see my colleagues and teammates struggling to get to their actual work.

(I once joined a software development project, and 8 hours of weekly meetings blossomed in my calendar, 20% of my working week. For some of these meetings, I was unsure why I’d been invited and asked the other attendees. They also were unsure why they were there …)

(A talented, hard-working scientist once confessed to me that he often didn’t get to his “work” before Thursday. He sighed, “I need Monday to Wednesday just to keep up with the emails …”)

(In another workplace, everything was done the night before it was needed. Slide decks, analyses, proposals & formal documents, everything. I tried to change it but it was such a part of the culture, they actively resisted doing anything in advance.)

Deep Work is Newport’s most complete statement on these issues. While there are some caveats, let me first summarise the points that leapt out to me and how we might use Newport’s diagnosis and advice.

Deep Work is a superpower

Newport’s premise is simple but brutal: in a world of constant distraction, the ability to focus deeply is becoming increasingly rare and increasingly valuable. There is more and more to do and more and more to distract us. But we cannot solve hard problems with a collection of minutes scattered throughout the day. We need sustained effort over long periods of time. That’s a sobering truth in a world hooked on multitasking and 24/7 availability.

“You tend to think with enough motivation you can become great... but this ignores the difficulty of focus.”

We need to rewire our brains for depth

What hit hardest for me was how much damage constant attention-switching does to your cognitive machinery. The insight that “you don’t take breaks from distraction, you take breaks from focus” flips everything I’ve seen in startup and corporate culture. Most people live in a reactive state, bouncing from Slack to email to Zoom, never really thinking. This is the almost unquestioned default state of most people I work with. But in biotech and AI, deep work isn’t just ideal, it’s necessary. You can’t do machine learning model development or drug response analysis in 10-minute bursts between meetings.

Tactics

Some things Newport suggests:

• Schedule every minute: this is brutally hard at first. Others try to disrupt your calendar. You try to default from your calendar or make exceptions. But you have to guard your time fiercely and protect pockets of depth.

• If it’s important and tough, schedule long, uninterrupted stretches of time to attend to it: this is so central to Newport’s thesis and so obvious that it feels ridiculous to say it. But don’t we try and work against this truth, cramming difficult work into short breaks between meetings, hoping that this will be enough.

• Scheduled hours for internet use, both at home and work: this is also tough - we are just not used to being “away” from the Internet. But even 60 minutes of truly offline time, once you get past the initial discomfort, can feel like an oasis. During these breaks, resist temptation due to boredom. Put your phone away. If you get stuck, switch to another offline task, rather than going online, looking for a solution.

• Productive meditation: solving problems while doing something else (walking, cleaning, showering, exercising) can be surprisingly effective.

• Only adopt tools that make your life simpler and better: in fact, fewer and better of everything is a generally wise approach.

• Fixed-schedule productivity: the complement of scheduling your day is to draw a hard line at the end of the day to be done with everything and recover.

Becoming hard to reach

One of Newport’s most radical suggestions is rethinking email entirely: fewer messages, more clarity. Don’t reply to everyone or everything immediately. Make people do more work to get a reply out of you. Shift to a mode where not replying is the default. This can make you feel a jerk, but haven’t we all been on email chains that trickle on without any additional information?

Are you interested in this?

Just checking back in to see if you’re interested in this.

I’m not but thanks.

Thanks!

Maybe next time.

Okay, thanks! By the way …

This, I suggest, can also apply to LinkedIn messages, Whatsapp, phone calls … Also, it’s amazing how often “urgent” things fix themselves if you give them a day.

The limits of Deep Work in practice

Here’s where I challenge Newport: there are people and environments where these tactics will be difficult, if not impossible. He’s talking primarily of his own life and work, that of a tenured professor at a university. That’s a kingdom of rugged individualists, people in one place working by themselves, with an incredible amount of autonomy. That’s not a universal situation:

• Not everyone can simply ignore email and the like. Some people are in roles that are inherently interrupt-driven—like managers, customer support, or cross-functional leads. The book assumes you have a lot of control over your time, which isn't true for everyone.

• The book doesn’t address the value of relational, emotional, or team-based work and downplays the need for collaboration and working together. It’s easy to control your schedule when you don’t have to worry about anyone but yourself.

• Actually, the book focuses almost entirely on individual behaviour, ignoring that organisations and cultures often create and foster these reactive environments. It's fine to give personal tips when massive cultural shifts might be needed.

Closing Thought

In a field where innovation and noise are constant, deep work is how you find the signal - not just professionally but personally, too. Deep work isn’t a hack; it’s a way of life. There are difficulties in implementing it, certainly, but to do great things, we have to resist distraction and focus on the truly valuable. The biggest struggle may be breaking the “busyness” habits of both ourselves and our organisations.

For over a decade, Git has been the de facto standard for version control. It’s fast, flexible, and ubiquitous—but also famously opaque, with a steep learning curve and workflows that often confuse even experienced developers. But there’s a new competitor on the block: Jujutsu1.

Aside - why use version control

I'm continually surprised at how many analysts or analysis-adjacent people don't know any version control system (VCS). It was once a non-negotiable skill for bioinformatics and data science (along with SQL), but it seems to have dropped off the map. It's especially absent in fields like biostatistics, where many practitioners "collaborate" by emailing files to each other. Perhaps it's because there's too much else to know; perhaps it's because version control "looks" like software engineering. But actually, it's a basic and critical tool for anyone who writes code:

1. It saves your work locally and remotely. You won't have to worry about losing a file or accidentally deleting anything.

2. It allows teams to work on the same project in an organised and consistent way. No one's work will be overwritten.

3. It allows you to track changes and progress.

4. You can fork off different versions for experimentation without any fear of corrupting the original version or losing work.

5. You can integrate things like testing, deployment, or code reviews, to just happen automatically and seamlessly.

In short, if you don't use version control, start doing it now.

But what's wrong with Git?

There's a long history of version control systems, and many (most?) of the earlier choices were opaque, difficult and cumbersome. (Remember Arch, Bazaar, CVS, Subversion? And those were some of the better options.) But with the arrival of Git - and Mercurial to some extent - it felt like the problem had been solved. Git was so much better than the competition and a lot of default infrastructure (e.g. Github) sprang up around it.

But, actually, Git could still be much better.

1. Git wasn’t built with usability in mind. Some commands are arcane, inconsistent, and often don’t mean what you think they mean.

2. There are lots of ways that Git makes it too easy to do the wrong thing, to shoot yourself in the foot. To quote, "Git assumes you know exactly what you're doing. Most people don’t and shouldn’t have to."

3. Cleaning up Git repo history is fragile and feels risky.

4. Collaboration is a bit fragile. Once again, it's easy to shoot yourself in the foot.

5. The mental model of how Git works is very complicated and exposes a lot of internal logic.

6. As a result, most developers don't have a strong idea of how Git works. They just have a handful of commands they’ve learned over the years, enough to get by.

You shouldn’t just adopt another tool because of marginal gains. But the pain points with Git are such that it makes sense to look for alternatives.

So what Is Jujutsu?

There are several answers to this, coming from several different directions.

• Jujutsu (command-line tool: jj) is a new version control system, that aims to combine the power of Git with a significantly improved developer experience.

• Jujutsu is a user-friendly front-end that uses other version control systems on the backend, as storage.

• Rather than building practices on top of a version control system, Jujutsu tries to incorporate workflow and best practices into simple commands.

There are some caveats, but we'll get to those.

Comparing Jujutsu & Git

An easy point to start is with a straight-up, head-to-head comparison at the command level. I've deleted some entries where the tools are more or less equivalent.

Jujutsu's working model

Much of the above is a big "so what". The syntax is similar if simpler. But the ideas behind it are different.

• Jujutsu leads with the idea is that version control should prioritize safe, understandable history. Every change you make - committing, rebasing, amending - is tracked immutably. You never lose information.

• The Working Copy is a First-Class Citizen. Jujutsu uses a real commit to represent the working copy. Any edit you make on disk is immediately reflected in the current commit. You don't have to tangle with `git stash` to manage multiple working copies. In most version control systems, as soon as you edit a file locally, that new change is in a kind of limbo state outside of the system and has to be managed separately.

• Actually, everything is a commit and "bookmarks" are pointers to specific commits. This is a broader and simpler concept than Git.

• Merge conflicts are a common source of pain in Git. Jujutsu tries to make collaboration less error-prone by using conflict-free branching and a cleaner approach to synchronization. If a merge results in conflicts, information about those conflicts will be recorded in the commit(s). The operation will succeed and you can resolve the conflicts later. No more being stuck having to fix a horde of conflicts before you can actually finish a merge.

If there's a single broad idea, it's that Jujutsu tries to match how developers work and their intuitions, making the common, safe behaviours easy and the risky behaviours hard if not impossible.

The caveats

• While Jujutsu advertises itself as a version control system - and has it's own commands and infrastructure and potential to operate as a standalone VCS - it is currently practically functioning just as a user-friendly face to other VCSs. That's still a great advantage.

• And, again in reality, it's practically a frontend just to Git. That's where most of the development effort has gone.

• The idea of compatibility with Git needs some careful wording. From my experience, Jujutsu is compatible with a remote Git repo (e.g. use Jujutsu locally and push to Git repo). Using both Git and Jujutsu on a local working copy is a far more complicated issue, and there are some uncertainties.

• Jujutsu is stable and largely feature-complete, but some more marginal cases (e.g. submodules, git-lfs, email-based workflows) aren't supported as yet.

• Realistically, everyone else in the world is still using Git. So for the near future at least, Jujutsu is going to play nice with Git, which erodes some of its usefulness. For example, there's support in Jujutsu for branches, even though "branch" isn't a distinction in Jujutsu.

Pointers

There's a growing body of instructional material for Jujutsu out there.Some recommendations:

• https://github.com/jj-vcs/jj: The official repo. Instructions dive straight in immediately without a lot of background

• https://jj-vcs.github.io/jj/latest/tutorial/: The official tutorial. A bit better but still starts in the middle.

• https://neugierig.org/software/blog/2024/12/jujutsu.html: A short, but more from-first-principles introduction. A good place to start.

• https://steveklabnik.github.io/jujutsu-tutorial/: A lengthier tutorial, in progress.

• https://v5.chriskrycho.com/essays/jj-init/: A thought piece about mental models and the motivations behind Jujutsu, followed by some instructions.

Originally published back in 2020, now updated with extra anger.

While mentoring some juniors, I started to think about the rules of thumb for analysing data that I've built up over the years. I'm certainly not the world's greatest data scientist (or bioinformatician, epi-informatician, statistician, AI engineer, or anything else), but I do have a lot of hard-earned, battle-worn experience and it’s worthwhile capturing them here. These are obviously shaped by my experience in analysing healthcare and biomedical data. In no particular order:

Analysis should be code

Non-trivial1 data analysis should be done with code, proper code, in a script—not Excel, JMP, a plotting package, or macros. Why?

• It's reproducible

• It's documentable

• It's versionable

• It can be worked on by multiple people

• It's debuggable

• It's easier to revise rather than just repeating it wholesale

• You can re-run it easily on other datasets or the same dataset

• It might seem like the long (hard) way. But for non-trivial tasks, in the long run, it’s actually faster.

I remember with horror a colleague who insisted on splitting and filtering a 20K-strong patient cohort on a huge list of complex criteria, by using a single massive Excel spreadsheet. It involved a stupendous number of sub-sheets, macros and scripting, linked in a thick mesh of cross-references. It took months to put together. Did it work? There was no way of knowing - it just didn't obviously not work.

That’s not a way to good results. Which leads me to …

Not obviously wrong does not necessarily mean right

There’s an old programming quip along the lines of “If it compiles, ship it”. But a lot of data analysis code implicitly works like that - if there’s not an obvious problem or error, we quietly assume that it works.

Reality isn’t so kind.

• Treat your analysis like proper code, like a program that has to run robustly and independently from you, that lives in an unreliable and variable world. Sprinkle it with asserts, validations, dummy checks, preconditions & postconditions (e.g. how many records should we be processing in this stage, what range should the values be in, are missing values possible or allowed, are the IDs well-formatted and clean). Trust but verify but with less trust2. Why?

• For many problems, our ability to spot a bad answer is severely limited.

• You might accidentally load the wrong dataset - the one that hasn't been cleaned or is from a previous analysis - and silently get the wrong answers.

• Your code might work flawlessly with good data, but with bad data, it might silently hand you the wrong answer. This is true of all too many R and bioinformatic scripts. Try this: get a gene expression dataframe, transpose the axes (swap the gene names with the record ids), feed it through an analytic pipeline and see if it fails.

• It's better to fail fast, to stop immediately upon an error occurring rather than having to check the end results and repeat the whole analysis. Fail early and fail often.

Don't invent your own data formats

This is an old adage in bioinformatics that applies elsewhere. Why?

• You're probably not as good at designing formats as you think

• How will anyone else know how to parse your data? Will they have to write a custom parser?

• You'll probably never get around to formally documenting your format, just making things hard for other people (see previous point).

If there a published or popular format that's 90% right for you? Maybe you should use that. Meta-formats, or formats built on top of other formats like storing data in JSON or YAML, are acceptable. At least then, there are tools to read and write them, and the base format will stop a lot of bad design decisions.

As an example lesson, look at the long history of ambiguities, extensions and inconsistencies in the NEXUS phylogenetic format or even something as simple as the NEWICK trees. Another university-created software program for phylogenetics used to have to evolve and correct its own proprietary format so often that saved analyses were essentially locked to the version they were created with.

Store data in humane common ways

Store data in CSV. If you can't do that, store in JSON or YAML. If you can't do that, SQLite. Why?

• It's human-readable & thus debuggable

• (Except for SQLite) It's plain text, thus in a pinch you can look or edit it with a text editor

• We can deal with different text encodings easily

• It can be read and written by a huge variety of software

• The behaviour of and the definition of those formats are well-defined and understood.

Obviously, there are datasets that don't fit easily into this schema because of being too big or inherently binary or just weird. But much of the data we use can be readily handled by the above tools. If you have to go big, think about using something common like HDF5.

I dislike storing data in Excel because it tends to think it knows what I want better than I do and start changing data3. But it’s so prevalent and widely used in the industry (a huge number of biotechs store their lab data in Excel 😭) that it might be easier to go with the flow. You also get something that your stakeholders understand and can use, and there are some nice toolkits in Python for writing and manipulating Excel tables.

XML has its place, but as a data format for computers, not people. As a rule of thumb, most systems that insist on using XML upfront are deeply pathological.

Reports are generated straight from analysis

Summaries and results should be auto-generated from analysis rather than cut and pasted into Word or PowerPoint. Why?

• It’s faster, once it's set up, and faster to repeat

• It’s reproducible

• It prevents mistakes and omissions when you update

• Copy-and-pasting into Powerpoint tends to encourage a style with excess verbiage and showmanship, with discretely massaged results being shamefully tacked on the end, and questions discouraged4

This admittedly can take a while to set up the first time, but in the long term, it pays off. I once worked on a detailed monthly report of clinical trials that I tried to help by automating. Unfortunately, every trial and every page needed something different, some additional text, some modifications, commentary, adjustments … so that monthly report took 2 weeks to prepare 🙄

No closed shop, proprietary tools

Why?

• You'll want to study them, see how they do their work, check that they do their work correctly

• You'll want to share your work, and you can't do that with people who don't have a license for

• It avoids lock-in. The two critical questions for any analysis platform or tool are: What can't it do? When that happens, can I get my data out to use it somewhere else?

Make sure that debug and dev modifications are blindingly obvious

Make any debug code blindingly obvious, strongly labelled, easy to find and easy to switch off in a single step. Why? You will forget to switch off or delete the debug code and so will accidentally ship analyses that only look at the first 100 records, use dummy datasets, short-cut algorithms or make unsuitable assumptions.

We once had a web-tool that ran observed epi data against a historical database. Every once in a while, it would crash for no obvious reason. The development version ran flawlessly. One day, I got frustrated and sat down to puzzle it out. I sprinkled diagnostic print statements through the code of the production version, repeatedly entering data into the tool until it crashed, narrowing down the problem. Eventually, I found the culprit: a single line of code that referred to the debug version of the database, clearly placed there to work out some bug. In development, this worked fine because the only database the code saw was the debug one. In production, most of the data was being pulled from the production db, of course, except in this one place. And the two database versions were similar but not identical. This web-tool had been running for a year, delivering potentially incorrect results.

If there's more than one substantial processing or analysis step, and this analysis will be run more than once, use a pipeline or workflow

Why?

• It'll save you time in the long term

• It documents your work

• It makes your work reproducible

• It makes your work easier to revise

The actual choice of workflow software is a Religious Issue, but I'm partial to Snakemake, and many people like Nextflow and Drake. Just use something.

Perhaps you can't prove this is true, but is it useful?

Often, you can't treat the result of an analysis as being true in a rigorous, statistical sense. Why?

• Data is biased, or cherry-picked

• A la Brian Wansink5, you've tried so many different approaches and slices of the data that one of them was bound to show something

• The algorithms you're using have only the veneer of a statistical basis, and / or obscure assumptions that may or may not be satisfied. Biomedical data, in particular, is heterogeneous, biased, and incomplete. Who knows if it meets the necessary assumptions?

This is not to say that statistics is unimportant, but it is sometimes difficult and can be unreliable or easily misused. You can easily lie with statistics, and maybe it’s impossible to place a statistical measure on your results. So, if your results aren't necessarily true, what can you do? You can ask, "This is better than what? And how can I show that?" Look for ways of independently validating or reproducing your findings. Or, if your answers aren't guaranteed to be true, do they show that something is not true? Look for ways for answers to be useful, to be advisory on other approaches.

Another way to say this, is to quote Bill Clinton’s favorite response to any statement: Compared to what? This lead binds the target well. Compared to what? This trial design is fast and efficient. Compared to what? This accurately describes the stratification of the patient population. Compared to what?

(Alice Wong rightly noted this was a bit of a drive-by statement: stats, p-hacking, multiplicity, etc. are vast subjects. But I can't do justice to them in this space, other than to underline their complexity. Sometimes, we have to work out how to move forward when the maths can’t be applied.)

Validate

The result of any model is a hypothesis. It’s something that is cooked up from a particular set of data, that relies on a set of assumptions, that may involve confounders or inconvenient correlations, that may contain mistakes. It’s not right, it’s a pointer at what might be right. Find a way to externally validate it.

Maybe you should use less SQL

Author’s note: this hails from the original version of this article, at a point when I was encountering a lot of solutions that consisted of massive and impenetrable SQL one-liners. The times have changed, and now, a lot of analysts don’t even SQL. Which is a loss. I’m leaving this here because the basic advice is solid. Nowadays, I would instead say that a little SQL can be a powerful thing.

Why?

• Complex subsetting and massaging of data can be difficult to express in SQL, because it's depauperate as a programming language, and often those operations are cumbersome and verbose.

• Perhaps because of this, there's a tradition of enormous multi-line SQL statements to get work done. It is difficult to understand what these do or debug them. You end up taking the code on faith.

• SQL puts the internal representation of domain objects is right in your face, when you should be thinking of their external representation, what the objects represent.

• Asserts, checks and defensive programming? What's that?

• And there are dialects of SQL so forget porting your analysis.

• If you're pulling data live from a database, how do you version that?

• In summary, excessive use of SQL moves towards the code being a black box.

Obviously (gestures at 50 years of use), SQL is a tremendously useful tool. But it's not a great tool for communicating intent or writing good, complex code. Fortunately, you could just use a small amount of SQL to do a broad extract of data and then do the heavy lifting in a real programming language. Or you could use an abstraction layer (like SQAlchemy) so that you're manipulating objects, not rows scattered across tables.

Based on some consulting work I recently completed and some talks I gave.

Introduction

The key to cancer is neoantigens — novel malformed proteins generated by tumor mutations. Being absent in normal cells, neoantigens present a specific and promising target for cancer therapies: make something that hits a neoantigen and you can precisely direct therapies at the cancer and avoid the collateral damage of off-target effects.

• Biomarkers: being able to classify patients based on their antigens and therefore select effective therapies

• Cancer Vaccines: using the neoantigen to effectively train the immune system to recognise and attack the tumours. This includes the promise of making individualised vaccines, created by analysis

• Enhancing other neonantigen-targetting therapies: like T cell therapy or checkpoint inhibitors.

Despite this promise, identifying effective neoantigens targets remains difficult. The overwhelming majority of any “identifications” are false positives. This is a huge problem.

Challenges

The singular throughline in finding actionable neoantigens is that biology is absurdly complicated, there’s a long chain of causality from genomic mutations through protein expression to an immune response, and attempts to solve for any part of the system can easily be thwarted by issues elsewhere. As I am fond of quoting:

Immunology is where intuition goes to die

(Ed Yong)

So let’s walk down that problematic chain:

Sequencing

• How do you find a neo-antigen in a tumor sample? You sequence it and look for divergent coding regions that could give rise to an abnormal protein. However, the usual sequencing methods can miss or misinterpret putative neoantigens, especially those in complex or repetitive DNA regions (GC-rich, etc.)

• Just because it exists in the genome, doesn’t mean it is expressed as a protein on the cell surface.

• And, because of tumour heterogeneity, we might be sampling the “wrong part” of a tumour that isn’t immunologically accessible or only represents a subpopulation of the total tumour and is thus not a useful target.

Immunology

• Just because a protein is expressed, doesn’t mean that the immune system will process it usefully. A neoantigen has to be bound by immune system proteins (the MHC) for display on the cell surface. Predicting which mutated peptides will be bound effectively is still inexact.

• And, even if it is bound by the MHC, this doesn’t mean it will provoke an immune response. (It’s this step that sinks a lot of neoantigen identification. Binding is measurable and can be done computationally, but the link to immunogenicity just takes us into the dark.)

• Variation between patients, especially on the immune system / MHC level, can lead to a highly variable response due to variable binding - the same set of tumour mutations might be immunogenic in one patient and not another.

• Tumors can, of course, evolve or produce molecules that suppress the immune response locally.

Validation

Ensuring that a putative neoantigen actually exists, is expressed, and can raise an immune response in the real world is obviously the gold standard. This leads us to:

• Directly identifying neoantigenic peptides presented by MHC molecules through mass spec would be ideal. However, many neoantigens are expressed at a low level and are difficult to detect in standard MS workflows, when swamped by so many abundant proteins. Sample prep and even the vagaries of protein processing can mean that it’s difficult to know what fragment of a protein you’re seeing. At the best of times, interpreting mass spec results is difficult and fraught with potential biases. Here, you have all the problems in one place.

• Why not validate biologically with in vitro binding assays or T-cell activation assays? Absolutely. But it’s a wet lab experiment which means that it’s slow, fiddly and subject to noise variation.

Operational challenges

It’s one thing to be able to identify a neo-antigen. It’s another thing to do that at scale, reliably and consistently. This, I think, might be the greatest barrier.

• At a very fundamental level, it’s not clear yet what the best practices are in this area. There’s a real lack of comparative studies or even mechanisms that allow comparison. Much of the work in the field exists in their own isolated kingdoms, with methods and data difficult to share.

• Can you get consistent tumour samples from patients for sequencing? Biopsies tend to be very variable, packaging a problem, and there is little infrastructure for industrialising this.

• How do you get a sample from the patient to the sequencing facility fast enough? Actually, how do you do any of this fast enough?

• How do you execute the sequencing - computational analysis - immunological analysis at scale? Validation (massSpec and T-cell activation assays) will always be very slow.

• What’s the regulatory framework for a therapy individualised on a set of neoantigens? Every “dose” could be different from every other dose.

• Fundamentally, this is a different model to the “conveyor belt” one that pharma is used to operating in. It needs a tight loop from the patient through the healthcare provider to the manufacturer and back. That really doesn’t exist at the moment.

Solutions?

• Long-read sequencing can identify complex mutations (insertions/deletions, fusion genes, and large structural variants) and alternative splicing events that traditional sequencing methods miss. They can also deal better with difficult-to-sequence regions.

• LRS can also be used to more accurately sequence and type the immune (HLA) type of a patient, allowing that to be incorporated into the pipeline. In my eye, this is one of the big ways forward - methods that account for patient variation outperform those that don’t.

• Multiomics: using not just genomics but transcriptomics and proteomics will provide multiple layers of evidence, increasing confidence that any putative neoantigen is actually real.

• More AI: (perhaps inevitably) improved AIML would help us better predict neoantigen binding and immunogenicity. This, of course, relies upon getting more and better data, which is a challenge in itself.

Conclusion

Neoantigen identification is critical to the efficacious development and use of many emerging therapies. However, it is definitely in its “wild west” phase: no one is completely sure what the way forward is, everyone is going their own way, building their own tools and setting their own goals. The field is ripe for someone to step in, start setting standards, make benchmarks and shake the field up.

A pharma company is a machine for consuming and producing documentation

I was happy to give a talk on Faculty.AI the other day on the possible uses for AI in biotech and pharma, and intrigued by some of the work they have going on. Thanks to my gracious hosts Karland King and Alistair Stuart. The slides can be found here below but following some good questions from the audience, here’s a potted summary and set of thoughts:

Large models have saved us from the bad, hard days of tediously parsing symbols from clinical narratives for mediocre results. But the implications are wider - we can deploy AI models for automatic and instant translation of clinical data (no more waiting for the programmers to get an STDM conversion together) and search mounds of documents for the right information, saving us from "information attrition"

AIML is a universal acid for clinical trials. So much could be improved with better modelling and forecasting: how does patient burden & trial complexity
affect trial performance, predictive analytics on patients / sites / trials, are there patient subgroups or treatment heterogeneity ...

Clinical trial simulation is a no-brainer. In an age of increasing trial complexity, it's a flexible strategy that can be extended as we need and capture vital dependencies in time. However, there may be some operational challenges (i.e. how it is packaged up for clinical teams).

Causal AI is going to have its moment. So many of the questions within biomedicine are about mechanisms, but patients are "complex longitudinal objects" and spurious correlations about. (See vitamin D & COVID outcomes, HRT and heart attacks.) Causal inference will be valuable not just for mechanistic understanding but also deciphering complex real-world populations and indications where we can't run the necessary experiments (e.g. rare diseases).

If you're trying to sell to a pharmaceutical or biotech company or exert change within one, you need to pay attention to the culture and social dynamics within. Early development is different from clinical development, which is different from medical affairs. Understand that most people don't have "AI" as their job but instead are concerned with screening leads, developing a drug asset, or finding a patient population. Talk to them in that language.

Asking which opportunity is the "best" underlines a tricky issue. There are opportunities that have huge potential for impact but sprawl across the company, touching many people and existing systems (e.g. clinical trials). There are others that are more contained, almost a drop-and-drag replacement (e.g. generating regulatory documentation). If you were being strategic, the latter might be a better place to start.

Thanks to Samit Kundu, Stephanie Mou and Bayezian for valuable suggestions and insights.

This is another set of loose thoughts about how we can best use AI in pharma, biotech and healthcare, inspired by conversations at the Festival of Genomics & Biodata. See here for part one. A part 3 is likely.

Thanks once again to the Festival, including Richard Lumb and Celine Hashweh, and to Stephanie Mou.

A disclaimer: some assume that complaints like this mean that I’m an AI sceptic. Far from it, I very keen on the promise of AI for developing better drugs, faster. I believe it so much, I want to see these problems solved …

The data foundations for AI are rotten to the core

“So, what ELN or dataflows do you guys use?” I asked a friend, who is running a bioinformatics team in a midsize startup.

He laughed. “Excel and Sharepoint. You should see our monthly bills.”

I hear the same stories again and again. Web labs manually stuffing analyses into spreadsheets. Cantankerous data pipelines that barely work and regularly broke down. Companies say they want to be “data-driven” but don’t want to invest any time or effort, or change at all. Data is saved anywhere and everywhere. One scientist complained about how important analyses for drugs in trials - that regulators could ask for at any time - were just being tossed into random cloud archives, regularly threatened with deletion “to save space”.

You don’t get to make AI models if you don’t have data, data in the right state. You don’t get to trust the results of those models if you can’t trust your data. The plumbing has to be taken care of. The longer you wait, the worse the situation gets. As has been said:

Bad data is an existential threat to AI

There are two angles to solving this. The first in the technical, using approaches like robust data flows, observability and quality monitoring. Plenty of work has been done already. (As a first pointer, follow Barr Moses on LinkedIn who posts a lot on this area.) The other half of the problem is cultural. As abused as the term “data literacy” is, it is a very real thing. You need people who understand that data isn’t just a product but fuel, the lifeblood of the business. If we are to benefit from AI fully and (like any tool) have it used correctly (and sceptically), it can’t just be imposed upon an organisation. You need bi-directional buy-in and a change in behaviour. And culture transformation is hard1.

Who “does” AI?

The issue of recruitment looms large in everyone’s minds. Everyone says that it’s hard to find good people. However, what constitutes “good people” differs wildly. For some, it’s the question of how they can afford to pay FAANG “10x” engineers, after convincing them that biology is an interesting field. I disagree that this is a major issue. We’ll never be able to match FAANG salaries, but most of our biggest problems don’t require superhuman engineering skills but rather alignment with business needs and an understanding of biology. We absolutely need engineers, but relatively few super-engineers. Those few need to be supported and guided by people with different skill sets.

However, the opposite view - that we need people who are great biologists and great AI scientists - is hardly more realistic. It’s a quest for unicorns. People have specialities and talents and limited time to grow and exercise them. I’m reminded of the suggested syllabus for a bioinformatics degree some years ago that somehow expected students to absorb up to three degrees worth of genetics, medicine, statistics and computer science. You can’t be good at everything.

So what’s the solution? I don’t have one. Some of the most impactful biomedical AI I’ve seen have come about through great teams rather than great people. Some of the most effective people I’ve met had a diverse set of skills that they were not great at but good at. They were also good communicators, good team mates and adaptable.

Unfortunately, there’s no university course for “hard-working, humble and easy to talk to”.

Where does AI get done?

Data science and bioinformatics went (and continue to go) through this issue. In many ways, the adoption of AI in biomedicine is recapitulating that course in other ways —for example, the question of where AI expertise lives in an organisation. A once popular solution was the idea of a central “core” of experts, to which people could bring their (AI/data science/bioinformatics…) problems. Most everyone I know who has experienced such an arrangement, now rejects it. Core staff are isolated and treated like a service facility, much like glass-washing and supplies. Non-core staff regard the experts as being “over there”, and tend not to understand or appreciate what they do. Projects get designed in pieces and “thrown over the fence”. As the head of one core facility said to me:

We try not to have casual conversations with (external teams). It just creates more work for us.

Various organisations have instead created an AI division or Centre of Excellence. This feels to me like it might be the same idea, just dressing it up as a high priesthood that commands exclusive use of the technology: “AI is done over there”. It also feels like it’s confusing means with ends, like having a Division of Pencils or Centre of Using Spreadsheets. Once again, I don’t have a solution, but AI technology should be for the whole organisation.

At the recent Festival of Genomics and Biodata, AI was on everyone’s lips. After listening to talks, catching up with the gossip and taking part in several closed-door discussions, here are some thoughts about where we are and what challenges need to be overcome. There’s a lot to talk about, so a second instalment is forthcoming.

All quotes from actual conversations with participants, all opinions my own. Many thanks to the Festival, Richard Lumb and in particular Celine Hashweh.

2025: the year that AI ate everything

It seemed that every second talk at the Festival was about AI. How can AI help with healthcare? Is AI shifting the needle in drug development? Are large AI models unravelling the complexity of genomics? How can AI be incorporated into clinical practice? There was even a competition for innovators to pitch AI-driven solutions for transforming healthcare to the NHS Genomic AI Network. Many attendees spoke of how they were constantly using ChatGPT or one of its rivals, even paying for high tier access plans.

I wouldn’t say it was a healthy level of activity - we’ve long gone past that - but it was busy to overwhelming. AI is here, everyone is paying attention, and there’s too much to keep track of.

2025: the year that hype ate AI

This focus on AI (and GenAI in particular) was almost oppressive at times. A presentation from one company (a household name) blithely asserted that while we used to analyze data with statistical models, and then machine learning, and then deep learning, GenAI was the way “we did it now”. As if large models had abolished a century of statistical theory, the need to understand confounders or understand mechanisms. Elsewhere, several attendees confided that while the AI-based talks were impressive and interesting, many had an abstract, bragging air. The speakers seemed less interested in educating and informing the audience and more in carrying out publicity for themselves.

This is unfortunate but perhaps inevitable - the oxygen of the startup ecosystem is hype. Even in larger organisations, there’s a push to “do AI”, and people might avoid being insufficiently enthusiastic. However, there is a widespread feeling amongst practitioners that things have gotten way out of hand. There’s a lot of promise to be delivered upon.

As an example, last month I gave a lecture on the use of AI in drug development at the University of Westminster, what I thought was a positive view, tempered with some scepticism and appreciation for the barriers to be overcome. In the questions afterwards, a student asked how confident I was in the promise of AI, rated “from 1 to 10”.

I answered “7”. Which, to me, seems a optimistic outcome.

Nonetheless, the student was shocked at my negativity. “That low?”

This is the stage of the hype cycle we’re in. Any scepticism is seen as almost irrational.

Getting it done

There’s a massive gap between the sea of Proofs-of-Concept and (the relatively much smaller) number of systems in actual business-as-usual use. (Vibhor Gupta and I presented a workshop on this very issue back in 2021. Nothing changes.) How can we get business value out of AI? How should we use AI operationally?

A keen point: How can we show what AI is contributing, what ROI & KPIs can we use? Do we even need to measure contribution? This echoes a debate in the software engineering field about whether and how programmer productivity should be measured. The general feeling is that productivity absolutely should be measured - why should AI-centric teams be sacred in not having to justify their existence? However, the actual measurement is difficult. Drug development has a poor correlation between input and output, often with years between the two. There’s no great solution here, but this can’t stop us. The least-worst solution is a mix of quantitative and qualitative KPIs.

There’s a lot of platforms being built. Nearly every hot, new biotech bragged about how their “proprietary platform” would bring about breakthroughs. Similarly, there was a lot of talk about constructing in-house pipelines and tools, leading some to wonder exactly how much of this was necessary. It’s the old “buy vs build” argument, except no one gets favourable press if they reuse established technology and code. And there’s the added challenge that results and systems are impossible to compare or reproduce given an absence of standards and compatibility. For example, look up the literature on neo-antigen detection and see a field where everyone insists on working past each other.

Conversely, as someone pointed out, there are standards, but it’s no one’s job to use them.

Assorted thoughts

• If large models are non-deterministic and, when rerun, can return different results, is this encouraging people to simply rerun models until they get an answer they like? We already have a massive multiple-hypothesis and false-discovery problem. What’s the solution?

• There’s a widespread and largely unexamined belief in scaling and that any problem can be solved by just throwing more data and bigger computers at it.

• Disappointingly, there was little talk of causality, which is where I think the next big advances are going to come from.

That’s enough for now; part 2 will follow.