To restrict the means of committing suicide, we must know how they are committed. The most common suicide method varies across countries. Firearms account for 46% of all suicides in the US. In East Asia, suffocation from helium or carbon monoxide produced by charcoal burning is common.2

Unfortunately, national-level data on the suicide methods are limited. Only 76 of the 194 WHO Member States report data on methods of suicide in the WHO mortality database. Consequently, how 72% of global suicides were committed is unclear.3

A systematic review estimated that around 30% of global suicides are due to pesticide self-poisoning, most of which occur in low-income countries.4 If true, pesticide poisoning is among the most common methods of suicide. However, because of the lack of data from each country, it is difficult to implement policies to prevent these types of suicide.

If this data is so important, why are we lacking it?

There are several reasons:

1. Suicide is taboo, so it is often deliberately not registered.

2. You cannot always be certain whether someone died from an accident or a suicide. It can also be difficult to distinguish a suicide attempt from self-harm.

3. Many countries do not have the resources to collect the data.

4. Many methods of suicide do not exist in the International Classification of Diseases (ICD-10). Imagine having to select the cause of death from a drop-down menu, but the option you need is missing. You cannot register suicide by helium or carbon monoxide poisoning, so you are forced to code something nonspecific, such as X67: Intentional self-poisoning by and exposure to other gases and vapors.

In this blog post, I will focus on the last issue: how the flaws of ICD-10 damages research. But, before we start bashing ICD-10, let us travel back in time to its origin.

The International Classification of Diseases

In 1837, William Farr, England’s first medical statistician, faced a fundamental problem: how do you count instances of tuberculosis when it is recorded under various names, such as consumption, phthisis, and lung disease? The same disease was frequently recorded under multiple different names, while identical terms were applied to entirely different conditions. This lack of standardization made it nearly impossible to track disease patterns, compare mortality data across regions, or conduct meaningful statistical analysis of public health trends.

Farr developed a classification system that standardized disease terminology across medical institutions. His system organized diseases by medical categories and incorporated synonyms and local terms that captured the various regional names used for the same conditions. This systematic approach enabled consistent recording and comparison of disease data across hospitals and geographic areas.

Using his new system, Farr showed that after age 35, mortality among miners was much higher than average. From his data, Farr concluded that pulmonary diseases were the main cause of the high mortality. Nothing was done to improve the miners' condition — it was just interesting knowing why they died.

John Snow (no, not him)

Farr and many other scientists at that time believed in the miasma theory. They believed that diseases such as cholera and the bubonic plague were caused by bad air. Farr had even found a correlation in his data between cholera instances and the altitude at which people lived.

John Snow, an English physician, discounted this theory. By talking to local residents, he identified the source of a cholera outbreak as the public water pump on Broad Street. While his evidence was persuasive enough for the local council to disable the well pump by removing its handle, it was not enough to disprove the miasma theory.

A few years later, using Farr’s system, Snow showed that homes supplied by the Southwark and Vauxhall Waterworks Company had a cholera rate fourteen times that of those supplied by Lambeth Waterworks Company. The former obtained water from sewage-polluted sections of the Thames, while the latter obtained water from cleaner upriver sections. This evidence persuaded Farr and other researchers to believe that water was the source of Cholera.

Farr’s system today

William Farr's classification of diseases is widely regarded as the forerunner of the modern International Classification of Diseases (ICD). The newest version is ICD-11, but almost every nation still uses ICD-10. ICD’s original purpose was statistical: to record causes of death consistently so that patterns in mortality could be studied. That research-driven aim has faded in ICD-10. While ICD-10 still supports research, particularly mortality statistics, its design increasingly serves billing5. The system is no longer built solely to capture accurate health data; it is shaped by the need to price clinical encounters, and this shift has several unfortunate consequences.

The problems with ICD-10

Dumb rules

Several ICD-10 guidelines6 introduce systematic gaps and biases into data used for medical research. In inpatient care, you must code all conditions that are uncertain or suspected. This means that if a patient is labeled with an ICD-10 code representing an infection, you do not know whether the patient actually had that infection or whether it was a hypothesis that was later ruled out. I suspect that worldwide healthcare statistics (e.g., how many people worldwide suffer from influenza type A annually) have included suspected or uncertain cases in their numbers. To make it more complicated, uncertain cases are not coded in outpatient cases.

This rule exists to justify a physician's treatment of an unestablished diagnosis. The insurance company must know why the treatment was necessary (I.e., antibiotics before having established a bacterial infection).7

Including an uncertain diagnosis would be great if it was somehow indicated that it was uncertain. Many electronic health records (EHRs) store ICD-10 codes this way. However, many EHRs do not and a worldwide comparison of research data becomes difficult if we must account for the different ways each EHR stores ICD-10 codes.

Another dumb rule is when to code symptoms. In the American modification, you should not code symptoms that are integral to one of the established diagnoses. So, you should not code R50.9: Fever, unspecified if the patient has influenza, but you should if they do not have any conditions that typically cause fever. In the International version of ICD-10, you should only code symptoms that affect patient care. Many symptoms will not affect patient care, despite the knowledge of their presence being valuable to medical research. The unreliability of symptom codes means you cannot use ICD-10 codes to research symptom prevalence across conditions. That is a shame. Such information could provide us with the probability of a patient having disease A given symptoms X, Y, and Z, which could help us improve evidence-based medicine. Why not code all symptoms and let algorithms determine whether they affect the encounter cost?

Similarly, secondary conditions should be coded only if they affect patient care. This is sometimes a subjective decision. Did the patient's obesity affect patient care during the MRI scan or during the blood test?8 Did the patient’s epilepsy affect the physician’s decision-making? This rule means that patients have conditions that do not get coded. Yes, many of these conditions are coded in previous encounters, but it is not guaranteed. I doubt that clinicians checks that a condition has been coded in previous encounters before they decide not to code it. Also, researchers may not have access to data from previous encounters. This means that you cannot trust the co-occurrences of diseases in health data. Does obesity cause diabetes, or does obesity more often affect the care for patients with diabetes?9

In ICD-10’s defense, some of these rules were probably introduced to save clinicians’ time. Coding every symptom and condition takes time, and certain rules may be introduced to code only the most essential. However, hopefully, with the rise of AI for medical coding, it may be possible to code everything using less time.

ICD 10 is too limited

ICD-10 comprises +14,000 codes. That may sound like a lot, but there are actually too few codes. ICD-10 does not cover suicide from helium or charcoal burning. Many rare diseases and manifestations are forgotten.

The lack of codes would have been less of an issue if ICD-10 was designed better. ICD-10 represents every possible manifestation of a disease with a single code, rather than combining them. Every manifestation of a disease has its own unique code; for example, M84.375P: Stress fracture, left foot, subsequent encounter for fracture with malunion. In other code systems, such as ICD-11 and SNOMED CT, you combine codes. You combine the code for stress fracture with a code for the left foot, and a code for a subsequent encounter. This means that you can express more with less.10

Each country uses its own modification

As a consequence of ICD-10’s limited expressiveness, each nation has developed its own modification to suit its needs. This makes sharing health data across borders more difficult. While the international version of ICD-10 organizes the codes into a five-level hierarchy, the American modification expands it to seven levels and +70 000 codes. The Danish modification uses extension codes to encode information such as laterality. The German modification introduces codes with an exclamation mark that are used only optionally with a primary diagnosis. In addition, each modification adds and removes codes and has different rules for what to code and when. Chaos.

The ICD 10 hierarchy

As mentioned, the ICD-10 codes are organized in a hierarchy. At the top of the hierarchy are the chapters such as infections, neoplasms, and blood diseases. Moving down the hierarchy, the conditions become more specific. At the bottom of the hierarchy are the codes, which are the specific manifestations of conditions.

Unfortunately, medicine is too complex to be expressed in a single hierarchy. Which chapter should “Sepsis following a procedure” be in? Certain infectious and parasitic diseases or Injury, poisoning and certain other consequences of external causes? Both are valid, and a single hierarchy cannot reflect that.

If you are researching sepsis, you must locate all sepsis-related codes. You cannot rely on the hierarchy for that. Sepsis is in one chapter if the patient is pregnant, another if the patient is newborn, and another if the sepsis is caused by a procedure. What is the point of a hierarchy if you cannot rely on it?

In other systems such as SNOMED CT and ICD-11, each code is in multiple hierarchies. You can find the same code multiple places. In SNOMED CT, “Sepsis following a procedure” is both a member of “Sepsis” and “Disorder following clinical procedure”. If you need all sepsis variants, you simply retrieve the codes with “Sepsis” as the parent node.

Will ICD-11 solve the issues?

ICD-11 replaced ICD-10 in 2022; however, almost no nation has adopted it yet. It solves many of the issues with ICD-10. It introduces extension codes, meaning most manifestations of conditions can be coded by combining codes. Each code can be part of multiple hierarchies, making it easier than in ICD-10 to find all codes related to a condition. It comprises more codes overall and now includes most rare diseases. It is an excellent upgrade.

The rules for coding have improved in ICD-11. All coexisting conditions should be coded, rather than only those that affect patient care.11

That said, ICD-11 still has complex rules for when something should be coded. When a diagnosis is unconfirmed, one must instead code the symptoms, abnormal findings, or problems. Same for ruled-out conditions. The ICD-11 reference (2.23.5.4) states:

The health care practitioner should document as main condition a “ruled out” condition when the episode of care involves a person who presents some symptoms or evidence of an abnormal condition which requires study, but who, after examination and observation, show no need for further treatment, follow-up or other medical care.

Also, conditions related to an earlier episode that have no bearing on the current episode should not be coded.

Why not code all symptoms, abnormal findings, problems, and diagnosis, and add an “unconfirmed,” “ruled-out,” or “past condition” extension code to the diagnosis when appropriate? Why exclude such conditions when the ICD-11 can precisely express them with extension codes? I am a strong opponent of rules that add complexity, subjectivity, and ambiguity.

The perfect system

ICD-11 is a substantial improvement over ICD-10, but its rules for excluding codes remain a problem.

In my view, ICD-11 would be close to a perfect system if every condition were coded, with extension codes added to express context such as uncertainty, temporality, and negation. Using this information, algorithms could automatically estimate the cost of an encounter (e.g., by filtering past or negated conditions). We would then have codes valuable for both research and billing.12 We would have a system that would ensure that hospitals are reimbursed while properly record suicide methods, so that we can prevent them.

Coding this way would take more time. However, I believe AI-assisted medical coding will make such comprehensiveness possible without adding to that time. Rather than replacing medical coders, I hope AI will allow them to code more comprehensively.

Wrapping up

Healthcare systems are diverse and complex, so some of my points here are probably inaccurate or mistaken. If you spot one, feel free to leave a comment; I always appreciate counter-arguments and constructive criticism.

The fever was back, and this time accompanied by a deep pain behind his eyes, as though two thumbs were squeezing them out from the inside of his skull. His skin took on a yellowish cast. Then came the vomiting. It was dark, almost black, like wet coffee grounds. The Spanish called the disease Vómito Negro. Soon, he was bleeding from his nose and eyes. He lost consciousness, and then he died. In that, Jacques was fortunate. Other men had to be strapped to their cots, frantic and struggling against nothing, fully awake until the end.

Of the 33,000 soldiers Napoleon sent to Haiti, fewer than 4,000 came home.1

Napoleon had dreamed of an American empire, and that dream rested on Haiti, the most profitable colony in the Caribbean. In 1803, the year after the loss of Haiti, he sold 828,000 square miles of North American territory to the United States in the Louisiana Purchase. For fifteen million dollars, the young nation doubled in size. Many historians attribute this historical event to yellow fever.

Yellow fever is a terrifying disease. While most people experience only influenza-like symptoms — headache, chills, fever — 15% move on to the second phase after a brief, deceptive recovery. The liver begins to fail, staining the skin and eyes a deep yellow. Then the bleeding starts. Blood seeps from the nose, the eyes, and the gums. Inside the stomach, internal bleeding produces a dark, viscous vomit the color of coffee grounds. The kidneys follow the liver into failure. In the final hours, patients become delirious and frantic until they lose consciousness. Half of those who reach the second stage do not survive.

The disease is caused by a virus transmitted by the same mosquitoes that spread dengue fever, chikungunya, and Zika virus. There is no cure, but a vaccine exists. A single dose provides lifelong immunity. Despite this, yellow fever kills an estimated 78,000 people in Africa every year.2

So if the vaccine gives lifelong immunity, why haven’t we eradicated yellow fever the way we did with smallpox? To answer that question, we must travel to São Paulo.

In 2016, no one in Brazil’s cities had contracted yellow fever in almost 80 years. Most people assumed it was gone. But yellow fever had never truly disappeared. It had simply retreated into the jungle, circulating quietly among monkeys and forest mosquitoes, beyond the reach of vaccination campaigns. Then, without warning, it moved. The first sign was the deaths of 501 monkeys. Within weeks, the virus had spread to the humans of São Paulo, killing 578 people. Many more would have died were it not for the swift vaccination campaign.

This is what makes yellow fever so difficult to eradicate. Monkeys.

Strategies for eradicating yellow fever

I wouldn’t be an engineer in tech if I didn’t try to find solutions to problems I have no competence in. I will therefore discuss strategies for eradicating the disease.

Vaccinating humans

The most obvious solution is to vaccinate humans. We only need one vaccination, and then we are almost certainly safe for life. However, there are three challenges.

1. Delivering vaccines to everyone is difficult, particularly in developing countries and war zones, and a growing number of people refuse vaccination altogether.

2. Eradication is effectively impossible. Unlike smallpox, which infected only humans, yellow fever persists in wild monkey populations in the jungles of Africa and South America. Even if every human on earth were vaccinated today, the virus would survive. The moment vaccination rates dropped, it would return.

3. The vaccine itself carries a very small risk. Roughly 3 people per million develop a serious yellow fever-like illness from the vaccine, and about 4 in 10 of those cases are fatal. You are more likely to be struck by lightning. But when you vaccinate billions of people, it will cost lives.

Targeting mosquitoes

Yellow fever can only be transmitted by certain species of mosquitoes. In cities, the culprit is Aedes aegypti, a small dark mosquito native to West Africa that was carried around the world by the slave trade. It is the same species responsible for dengue fever, chikungunya, and Zika virus. Eliminating it seems like an obvious solution.

But even if Aedes aegypti were wiped out entirely, yellow fever would not disappear. In the jungle, separate species of Haemagogus and Sabethes mosquitoes keep the virus circulating among monkeys. People who enter jungle areas could still be infected. What elimination would prevent is the urban chain reaction — the process by which a single jungle infection becomes a city-wide epidemic. As a bonus, it would also eliminate dengue fever, chikungunya, and the Zika virus. The question is how to do it and whether the ecological cost is worth it.

1. Remove breeding grounds: Aedes aegypti breeds in still water: flower pots, old tires, buckets, bird baths, even a bottle cap left in the rain. Eliminating standing water can drastically reduce the adult mosquito population. It is one of the oldest and most widely used control strategies, requires no technology, and costs almost nothing. The problem is that eliminating all or even most standing water is nearly impossible, especially in dense cities with poor infrastructure.

2. Gene editing: A British biotechnology company, Oxitec, has developed genetically modified male Aedes aegypti mosquitoes whose female offspring die before reaching maturity. Only males survive to continue the cycle. Since male mosquitoes do not bite, releasing them poses no risk to humans, but over successive generations, the absence of females collapses the local population. Field trials in Brazil reduced Aedes aegypti populations by up to 96%. The catch is that the effect is not permanent. Genetically modified males must be reintroduced continuously, as the population recovers once releases stop.

3. Mosquito nets: Sleeping under insecticide-treated nets prevents mosquitoes from biting during the night. It is a cheap, widely used, and effective measure. However, Aedes aegypti is primarily a daytime biter, feeding just after sunrise and around sunset. Mosquito nets offer limited protection against a mosquito that strikes while you are awake.

4. Insecticides: Before the DDT campaign in the 1940s, between one and six million Americans contracted malaria annually. Within a few years, DDT spraying had eradicated malaria from Europe and the United States. In fact, Aedes aegypti used to live in Southern Europe, but was eradicated in this campaign, even though it was not the main target. Can we do the same in Africa and South America? DDT proved catastrophic for the broader environment: it killed other insects (such as bees), accumulated in the food chain, thinned the eggshells of predatory birds, and persisted in soil and water for decades. Furthermore, the mosquitoes evolved resistance to the toxin, much like bacteria become antibiotic-resistant. DDT was banned for agricultural use worldwide in 2004.

Vaccinating monkeys

Vaccinating humans and eliminating Aedes aegypti mosquitoes will severely reduce the spread of yellow fever, but not eradicate it. To eradicate it, we must remove it from the monkeys. The current approach is laborious and expensive. First, you shoot a monkey with a tranquilizer dart. You must catch it with a net to prevent it from dying from the fall. While ensuring the monkey is breathing properly, you vaccinate it, tag it to prevent vaccinating the same animal twice, and then release it. One study estimated these costs at around 200 USD per monkey. Yet even at that price, vaccinating the thousands of monkeys near urban areas may be more cost-effective than vaccinating the millions of humans living in the cities. Monkeys also tend to be less susceptible to conspiracy theories about vaccines altering their DNA.

However, to truly eradicate yellow fever, we must also vaccinate the monkeys living in the forests far from humans. This will be more expensive. Here are some potential solutions:

1. Vaccine darts: It would be substantially cheaper if we could vaccinate monkeys with darts without tranquilising them first. This technology would need to be developed. We would also need a way to track which monkeys have already been vaccinated, perhaps through monkey facial recognition. One could even create an app that offers bounties to people who successfully vaccinate an unvaccinated monkey.

2. Oral vaccines: In Europe, rabies in foxes is controlled using oral vaccines hidden in food. This would be a far simpler approach. But the yellow fever vaccine cannot survive the low PH in the stomach, so new delivery technology would need to be developed first. This is expensive, and we do not want to distract biotech companies from humanity’s true challenges: hair loss and erectile dysfunction.

3. A vaccine trap: It may be possible to construct a trap that automatically injects a vaccine into an animal. But vaccines must be kept cold, require mixing before injection, and the trap must attract only monkeys while tracking which individuals have already been treated. These problems may simply be too hard to solve within a reasonable budget.

4. Drones: Okay, this idea is a little stupid. But imagine a swarm of drones targeting monkeys in the forest canopy. They would use facial recognition to identify unvaccinated individuals and shoot vaccine darts at them. The monkeys will almost certainly flee when they hear the loud hum of approaching drones, and we face all the same challenges as with the dart approach, plus several new ones. This idea will probably not work.

Of all the approaches discussed in this piece, I think a combination of vaccinating humans, reducing mosquito populations, and developing an oral vaccine is the most promising path toward eradicating yellow fever.

Fortunately, the world is not relying on my drone swarm idea. The WHO’s Eliminate Yellow Fever Epidemics initiative, launched in 2017, aims to end yellow fever epidemics by 2026 and has committed to protecting over a billion people. Progress has been real. But the deadline is approaching, and the disease is not retreating. Yellow fever cases have resurged in several countries in Latin America in 2024 and 2025, including in areas previously unaffected. To make matters worse, on his first day in office, Trump signed an order to withdraw the US from the WHO, removing roughly 18% of its funding overnight. Beyond being a dick move, it may prove to be a self-defeating one. As the planet warms, Aedes aegypti is moving north, and the US is directly in its path.

We have a vaccine. We have strategies. And yet tens of thousands of people will die of a preventable disease this year. Most of us will never hear about it.

Wouldn’t it be great if people stopped puking black vomit?

In 1843, Phineas Gage was using an iron rod to pack explosive powder into a hole. It wasn’t a hobby of his; he was working on a railway construction project. Suddenly, the powder detonated, driving the iron rod up through his head and exiting his skull before landing 25 meters away from him. Miraculously, Gage survived, but the rod had destroyed much of his frontal lobe. Before the accident, Gage was a capable and shrewd businessman with a well-balanced mind. After his recovery period, his new personality became apparent. He had become vulgar, impatient, and fitful. He was “no longer Gage,” according to his acquaintances. He was able to keep neither friends nor jobs and died alone 12 years after his accident.

Gage’s story served as one of the first sources of evidence that the frontal lobe is involved in personality. Because of his and others’ tragic stories, we now know that the frontal cortex contributes to higher-order functions such as reasoning, language, and social cognition. Similarly, from patients with damage to the hippocampal regions, we understand how we form memories, and from patients with strokes affecting Broca’s and Wernicke’s areas, how we process language.

Similar to how neuroscientists aim to understand the mechanisms of the brain’s neural network, AI interpretability researchers seek to understand the mechanisms of artificial neural networks. While there are many differences between biological and artificial neural networks, we can employ similar techniques to understand both. When working with artificial neural networks, we can systematically damage, manipulate, and restore any groups of neurons we choose, then observe the results instantly—luxuries unavailable when studying living brains. These advantages make artificial neural networks a promising testbed for understanding intelligence, yet leveraging them effectively remains an open challenge.

Circuit discovery

One emerging approach is circuit discovery, which involves identifying the artificial neurons that contribute to specific behaviors. Think of an artificial neural network as a graph, where each node is an artificial neuron and each edge is a synapse (a connection where one neuron can send signals to another)1 . A circuit is a sub-graph that is sufficient for a task. In other words, if we remove all the nodes and edges that are not in the circuit, the model can still perform the task. The goal of circuit discovery is to identify the circuit for a specific task. For example, when studying how models do arithmetic, researchers might ask: when given the input 34 + 28 =, which artificial neurons contribute to the model’s output: 62? After identifying the circuit, researchers can study how the neurons communicate with each other to solve the task. Circuit discovery is the first step in understanding the inner mechanisms of artificial neural networks.

So how do researchers actually find these circuits? Most circuit discovery work has focused on large language models (LLMs). Most LLMs comprise billions of parameters, which means billions of edges in our graph. To simplify the circuit discovery task, we often group the neurons into components and define an edge as the communication from one component to another. In large language models, natural components in which to group neurons are attention heads and feed-forward layers.

The traditional approach to finding circuits is hypothesis-driven and manual. A researcher might observe that a model successfully performs arithmetic (say, computing 34 + 28 = 62) and form a hypothesis about which components might be responsible. To test this hypothesis, they ablate (disable) those specific components and observe whether the model can still correctly solve arithmetic problems. If the model’s accuracy drops significantly, those components are likely part of the circuit. If not, the researcher refines their hypothesis and tests different components.

This iterative process—hypothesize, ablate, observe, refine—can take weeks or months for a single circuit. It’s painstaking detective work that requires both technical expertise and creativity. What if we could do this automatically?

Automated circuit discovery

The first automated circuit discovery method uses a brute-force approach to identify circuits. It sequentially measures the importance of each edge in the graph. To measure an edge’s importance, the model is run with that edge ablated (disabled), and the edge’s importance is determined as the resulting change in the model’s output. For our arithmetic example, if disabling an edge causes 34 + 28 to produce an incorrect answer, that edge is part of the arithmetic circuit. If ablating the edge causes a negligible change in the model’s output, the edge is deemed unimportant and excluded from the circuit. This process is repeated for every edge in the network, resulting in a complete circuit.

This brute-force approach is slow. In GPT-2 Small (120 M parameters), there are hundreds of thousands of edges, meaning the approach requires running the model hundreds of thousands of times (forward passes). For larger models with millions or billions of edges, this method becomes prohibitively slow. To enable circuit discovery for larger models, researchers are developing new, faster methods.

Gradient-based methods

Edge attribution patching (EAP) is a fast approach for automated circuit discovery. Instead of sequentially ablating edges and measuring the resulting change in the output, EAP computes the importance of all edges simultaneously using gradients. So how can gradients estimate the importance of edges?

Imagine wiggling each edge by a tiny amount (infinitesimally small) and measuring how much the output wiggles in response. The gradient tells us exactly this: the rate at which output changes as we apply an infinitesimally small change to each edge. Through back-propagation, we can compute these gradients for all edges simultaneously in a single pass—far faster than testing edges one by one.

The speed-up comes at the cost of accuracy. As I argued in my blog post “Gradients are not explanations”, in non-linear neural networks, applying an infinitesimally small change to a component is different from applying a large change. Furthermore, changing two components separately is not necessarily equivalent to changing them simultaneously. This is because neural networks are full of interactions: components that behave differently depending on what else is active or inactive in the network. One such interaction that is known to occur in LLMs is self-repair.

Self-repair

Imagine if the human brain contained a region that functioned as a personality backup. If something were to happen to the frontal lobe, this region would take over its function, leaving our personality unchanged. In this scenario, Phineas Gage would have kept his personality after the accident, and the frontal lobe’s role in our personalities would have remained a hidden mystery. Unfortunately, in artificial neural networks, such backup regions exist and constitute a significant challenge to automated circuit discovery.

There are several mechanisms in artificial neural networks that cause self-repair. The Hydra effect is when ablating one attention head causes another attention head to increase its signal, thereby compensating for the ablation. Self-repair can also be caused by LayerNorm normalization. When you add two similar embeddings and then normalize them by their standard deviation, you get roughly the same magnitude as if you had normalized just one embedding alone. In other words, LayerNorm automatically rescales its inputs to maintain a consistent magnitude. So removing one input causes the normalization to adjust, reducing the impact of the ablation and hiding the removed component’s contribution.

We recently discovered a new self-repair mechanism occurring in attention heads. Two conditions must be met for self-repair in the attention head to occur:

1. There must be two or more large attention weights in an attention vector.

2. All the value vectors must be similar (as in Key, Query, and Value) at the positions with large attention weights.

When both conditions are satisfied, ablating a single attention score has minimal effect on the output because the softmax activation function compensates by increasing the weights at the other positions. The figure below illustrates this mechanism for an attention head with three input embeddings and an embedding size of one.

These self-repair mechanisms deceive automated circuit discovery methods—both ablation-based and gradient-based. All these methods assume that each component is independent. But as we just learned, this is not the case. When self-repair occurs, the circuit discovery methods will underestimate the importance of components. When self-repair occurs in attention heads, the gradients of the attention scores will be zero, despite their importance.

How can we improve gradient-based circuit discovery methods?

In our paper, GIM: Improved Interpretability for Large Language Models, we modified the computation of gradients (backpropagation) to deal with self-repair. As you will see later in the blog post, this worked remarkably well! Our method currently tops the leadboard for the Mechanistic Interpretability Benchmark. So how did we do it? We implemented three modifications:

1. Temperature-adjusted Softmax Gradients (TSG): This modification aims at handling self-repair in attention heads.

2. Layernorm freeze: This modification aims to handle self-repair in LayerNorm normalization.

3. Grad norm: This aims at normalizing the gradients. Without it, the two modifications above did not work.

Temperature-adjusted Softmax Gradients

TSG modifies backpropagation through attention to address the attention self-repair problem. The motivation behind TSG is well explained through the “firing squad” analogy: two soldiers fire at a prisoner, and we want to measure the causal importance of each soldier for the prisoner’s death. If we measure this by separately preventing each soldier from firing, we would incorrectly conclude that both are innocent, since either shot alone is fatal. To measure their true responsibility, we must simultaneously prevent both shooters from firing.

Attention self-repair exhibits similar OR-gate behavior to the firing squad example. The gradient of the output with respect to the attention scores measures the effect of perturbing each attention score separately. Instead, we designed TSG to approximate the impact of perturbing multiple attention scores simultaneously. We developed TSG empirically by evaluating which techniques led to the fastest and most accurate approximation of jointly ablating multiple attention scores, while also producing faithful explanations.

So how does it work? Each time we back-propagate through the softmax activation, we recompute the softmax with a temperature larger than 1. We then back-propagate as usual. That’s it.

It is a bit of a hack, and there are probably better approaches out there, but it worked pretty well. In the figure below, we compare ablating two attention scores simultaneously with a) ablating each attention score separately, b) gradients, and c) TSG. As you can see, TSG better approximates ablating both attention scores simultaneously.

Layernorm freeze

As previously mentioned, LayerNorm normalization can cause self-repair. The fix here is simple, and others have suggested it before us: don’t backpropagate through the normalization factor. In pytorch, your detach the gradient of the normalization factor during the forward pass:

 y = x / std.detach()

Grad norm

The final modification, Grad norm, was crucial for the two other modifications to work. The implementation is simple. When two variables are multiplied (A*V, Q*K, or MLPgate*MLPin), we divide the gradient by 2. I initially believed I had discovered this method, but my heart sank when I found that another paper from 2024 had already proposed it. Anyway, I think that the 2024 paper doesn’t provide a strong intuition for this work, so here is my attempt at explaining it:

When multiplying two variables, you must use the product rule to compute the gradient. Let’s look at an example. In the attention mechanism, we have the query (Q) and key (K).

Where X is the input embeddings, and W_Q and W_K are weight matrices. We compute the attention matrix by multiplying the query and the key.

Using the product rule, we compute the derivative of the attention matrix given the input as follows:

Let’s say we have an embedding size of 1, and that our variables have the following values:

Using these values, we get:

Which gives us:

When computing EAP (the gradient-based circuit discovery method), we estimate a variable's importance by multiplying it by its gradient2. This gives us:

Do you see the issue? If we were to change X from 3 to 0, the output would change from 180 to 0. Yet, EAP estimates that the change would be 360. The gradient is doubled because of the product rule. This is why we divide by 2.

Here is another way to understand why gradients overestimate the importance of variables multiplied together. First-order gradients estimate the impact of varying a single variable while holding the others constant. When two variables are multiplied, varying one changes the product, but the gradient computation assumes the other variable remains fixed. However, when both variables depend on the same underlying parameter (X in our example), changing that parameter varies both variables simultaneously. The gradient captures each variable’s contribution separately through the product rule, effectively counting the impact twice. This is why multiplied variables systematically overestimate importance by a factor of 2, and why dividing by 2 corrects for this double-counting artifact.

However, in our example, we looked at embeddings with only one dimension. When we have embeddings with more dimensions, it becomes more complex.

Gradient Interaction Modifications

We named the combination of all three modifications as Gradient Interaction Modifications (GIM). We evaluated GIM on the circuit discovery track in the Mechanistic Interpretability Benchmark, where it currently sits at the top of the public leaderboard!

The mechanistic interpretability benchmark uses two metrics: circuit performance ratios (CPR) and circuit-model distance (CMD). In essence, these metrics measure how well we can maintain the model’s task-specific behavior while removing edges deemed unimportant by the circuit discovery method. For CPR, higher is better, while for CMD, lower is better. Check out the paper if you want a deeper understanding of these metrics.

Below, we compare GIM with other circuit discovery methods. EActP is the first circuit discovery method that ablates edges sequentially to estimate their importance. EAP is the edge attribution patching method, which uses gradients to estimate the importance of all edges simultaneously. EAP-IG is similar to EAP, but uses integrated gradients. NAP uses gradients to estimate the importance of nodes instead of edges. The two other methods, IFR (Information Flow Routes) and UGS (Uniform Gradient Sampling), are two approaches that I’m not covering in this blog post.

GIM achieves substantially better CPR scores than other approaches while achieving similar CMD scores to EAP-IG. Since CMD ignores the sign of attribution scores (using high-magnitude scoring) while CPR preserves it (using high-value scoring), this difference suggests that GIM excels specifically at correctly determining whether edges contribute positively or negatively to the output.

We released a Python package that you can easily use for most transformer-based language models.

Wrapping up

Phineas Gage’s accident gave us crucial insights into the frontal lobe, but came at a terrible human cost. With artificial neural networks, we can conduct the equivalent of thousands of Phineas Gage experiments every hour—systematically disabling components, observing changes, and mapping the circuits responsible for specific behaviors. Methods like GIM are making these experiments more accurate by accounting for the backup systems that would have kept Gage’s accident a mystery. As circuit discovery continues to improve, we’re building the tools to finally understand the mechanisms of intelligence—artificial and perhaps eventually biological.

One would assume that X-rays are universal; if a model can diagnose pneumonia in one hospital, it should be able to do so in another. But the model failed spectacularly.

It turned out the model had learned to identify specific hospital systems with near-perfect accuracy by detecting a hospital-specific metal token placed in the corner of the images. It combined this “token detection” with the hospital’s specific pneumonia prevalence rate to make a guess. It achieved high accuracy without learning much about pneumonia. When images from new hospitals lacked these tokens, the model struggled.

This is the textbook definition of shortcut learning.

In machine learning, a shortcut is a decision rule that performs well on in-domain data but fails to transfer to out-of-domain data. The model finds a cheat code: a feature that happens to correlate with the label in the training set but has no causal link to the object itself.

For instance, a model might learn that “sand” equals “camel.” Consequently, if it sees a cow on a beach, it predicts “camel” — not because it recognizes the animal, but because it relies on the sandy background as a proxy for the class.

The hidden shortcut: Class priors

The cow example relies on a spurious correlation with the environment. But there is an even simpler, more pervasive shortcut that doesn’t rely on the background pixels at all, but rather on the raw statistics of the label itself: The class prior.

In most datasets, we have class imbalance: one class is far more frequent than another.

If 90% of the patients in a training dataset have a certain disease, the model is heavily incentivized to predict that disease to minimize its error rate. This is what I call the class prior.

While deep learning models don’t explicitly calculate probabilities like a Bayesian statistician, they effectively approximate the same logic. During training, the model learns to optimize the relationship between what it sees and what it expects:

The Bayesian breakdown

To understand why this breaks models, we need to understand the three components of that equation:

1. Likelihood (evidence): This is the visual information. It asks: “Given that this patient definitely has pneumonia, what is the probability that their X-ray looks like this?” Ideally, this is what we want our model to learn—spotting lung shadows.

2. Prior (class prevalence): This is the context. It asks: “Before looking at any X-rays, how common is pneumonia?” The model will learn the prior based on the training data. If the data is from an infectious disease ward, the prior probability of an infection is high. If the data is from a grocery store, the prior is low.

3. Posterior (prediction): This is the final answer. It combines the likelihood and the prior probability to determine: “Given this X-ray, what is the probability the patient has pneumonia?”

The model learns to weigh the Likelihood against the Prior. If the prior is overwhelming (e.g., 90% prevalence), the model learns a massive bias term for that class. Even if the visual evidence (Likelihood) is ambiguous, this learned bias dominates the calculation, forcing the final prediction (Posterior) toward the majority class. This is why researchers often try to balance the classes when training models.

Priors are weak features

From a robust machine learning perspective, class priors are just shortcuts in disguise.

We can think of the prior as a “feature” the model uses to make decisions. But unlike causal features, the prior is a weak feature because it is not intrinsic to the subject; it is a property of the environment.

• Intrinsic feature (strong): “Fluid in the lungs.” This is a strong predictor of pneumonia regardless of where the patient is located.

• Environmental feature (weak): “Being in a dataset where 90% of people are sick.” This is a weak feature that is specific only to the training environment.

The danger of treating the prior as a feature becomes obvious when we swap domains. This is where the shortcut breaks.

Consider a model trained in a Specialist Hospital (Training Domain) and deployed to a General Practitioner’s (GP) Office (Deployment Domain).

1. Training (high prior): In the specialist hospital, the disease is common (e.g., 50% prevalence). The model learns a large bias term for this class.

2. Deployment (low prior): At the GP’s office, the disease is rare (e.g., 1% prevalence).

When the model is deployed at the GP’s office, it carries that large bias term with it. It sees a healthy patient with vague symptoms (low likelihood), but when it multiplies that evidence by its massive internal prior, the result is a false positive.

The model applies the hospital's statistics to the population of a local clinic. By relying on the prior, the model becomes brittle and fails the moment the environment changes. We want our models to be causal reasoners: predicting a class only when there is evidence in the input, not guessing based on prevalences.

Should we decouple the prior?

Dismissing priors entirely is too hasty. Input data is often noisy, incomplete, or ambiguous. In these situations, the “likelihood” signal is weak. If a radiologist sees a blurry shadow that could be a tumor or a glitch, ignoring the base rate might lead to worse performance if the prior is well calibrated

Perhaps, we want the model to output the likelihood independent of the priors. If we train models to output the pure likelihood (the strength of the visual evidence), we gain control. We can then manually inject the appropriate prior for the specific deployment environment.

If the model is in a GP office, we multiply the likelihood by a low prior. If it is in a specialist ward, we multiply by a high prior. By decoupling the evidence from the prevalence, we turn a brittle shortcut into a flexible tool.

Automated medical coding models predict medical codes from clinical notes. These codes are alphanumeric representations of diagnoses or procedures used for documentation, billing, and statistics.

Progress in automated medical coding has stagnated. There has been virtually no innovation since the introduction of label-wise attention in 2018 — only variations of the same ideas. It is time for something new.

This is a bold claim, but we believe our new framework, Code Like Humans (CLH), is what the field desperately needs. Our paper presenting the framework was recently accepted for EMNLP 2025 findings. In this post, we will cover how it works and what comes next. However, to motivate our framework, we will first explain why existing approaches fail and how professional medical coders assign codes.

Why current approaches fail

Current state-of-the-art medical coding models require large datasets of clinical notes annotated with medical codes. In effect, these models memorize codes present in the training data. This approach has several limitations:

1. Acquiring clinical notes annotated with medical codes is challenging. Most health care providers are unable or unwilling to share such valuable private data.

2. A code must appear frequently for models to predict it reliably. Thus, models are unable to predict codes that do not occur in the training data and struggle with rare codes. In MIMIC-IV, only 6,000 out of the 70,000 ICD-10-CM codes appear in the training data.

3. Coding systems, such as ICD-10-CM and CPT, are updated annually. Adapting the models to predict the new versions requires re-annotating the datasets and re-training the models.

4. Models trained on codes from one department or specialty may not generalize well to another.

Several studies address these challenges by training models to use external resources describing the codes rather than memorizing them. External resources include the code hierarchies, code descriptions, code synonyms, and code co-occurrences. Although using external resources is a logical solution to the problem, previous studies on ICD coding have misused the resources and neglected essential resources such as the alphabetic index and the official guidelines (see this blog post for a more detailed criticism of these approaches).

Recent studies also explore large language models (without fine-tuning) for code extraction, data augmentation for rare codes, and few-shot learning. Most show poor results, except for those that deviously evaluate on artificially generated data, despite real-world data being publicly available.

The greatest weakness of current approaches is that they overlook how humans assign medical codes. Here is how humans do it.

How humans assign medical codes

Many coding systems are used in healthcare, but we will focus on ICD-10: the International Classification of Diseases, 10th Revision, which is the primary coding system used for documenting diagnoses in most countries worldwide.

Understanding ICD-10 Code Structure

ICD-10 codes follow an alphanumeric format that reflects a hierarchy. Each code begins with a letter (A-Z) followed by two digits, creating three-character categories (e.g., J44: Other chronic obstructive pulmonary disease). Codes can be extended with a decimal point and additional characters, up to two in the international version and up to four in the American modification, ICD-10-CM.

The hierarchy is structured as follows:

• ICD Chapters: The initial letter groups codes into 22 broad chapters covering different body systems or types of conditions (e.g., Chapter 10, "Diseases of the respiratory system," uses codes J00-J99)

• Categories: Three characters representing general condition types within each chapter (e.g., J44: Chronic obstructive pulmonary disease)

• Subcategories: Four characters (four to six in ICD-10-CM) providing additional information (e.g., J44.0 = COPD with acute lower respiratory infection)

• Codes: Leaf nodes in the ICD hierarchy. Can have between three and five characters (or up to seven in ICD-10-CM). A category is a code if it has no child nodes.

With approximately 14,000 codes in the international ICD-10 (and about 70,000 in the American ICD-10-CM modification), navigating this vast classification system requires systematic tools and resources.

The Three Essential Coding Resources

For ICD coding, human medical coders rely on three resources to navigate this extensive hierarchy:

The tabular list contains the hierarchy. It further contains instructional notes that specify when to exclude or add supplementary codes. For example, at code I50.9 (Heart failure, unspecified), the tabular list includes the instruction "code first," directing coders to also assign a code for the underlying condition that caused the heart failure, such as hypertension.

The alphabetic index serves as the entry point into the hierarchy, with an extensive cross-reference of medical terms, conditions, and synonyms that direct coders to the appropriate categories, subcategories, or codes in the tabular list. Rather than searching through thousands of codes sequentially, coders can quickly jump to the relevant part of the hierarchy.

The guidelines provide rules for code selection within the hierarchy. The guidelines include general coding principles that apply across all chapters, plus chapter-specific instructions that address unique coding considerations for different body systems and condition types.

The Coding Process

Medical coding is translation, converting the messy language of clinical practice into ICD-10's rigid hierarchy. The process mirrors how we naturally move from general to specific, uncertainty to precision.

Step 1: Extract signal from noise

The medical record contains everything. First, one must extract what matters for coding. Identify the conditions that influenced care delivered in the current admission. Everything else is noise.

Step 2: Navigate the alphabetic index

For each identified condition, look up the main term (usually the condition's noun) in the alphabetic index. Each main term has modifier terms, which you must use to specify the condition further. The final term points one to a code or sub-category. The alphabetic index suggests codes but gives no final answers. It's like a search engine.

Step 3: Verify each code in the tabular list

Use the tabular list to verify the code suggested by the alphabetic index or to traverse down the hierarchy when a sub-category is suggested. As mentioned, the tabular list also provides instructional notes that offer more information for selecting the appropriate codes and when to add or exclude codes.

Step 4: Verify the final list

While the two previous steps have focused on each condition separately, this step considers the entire list of code candidates identified so far. Using the instructional notes and guidelines, remove codes that should not be reported together. For example, mutually exclusive ones or a symptom code when a definitive diagnosis is coded. Then sequence the codes.

Code like humans

The idea behind Code Like Humans is simple. We designed four agents following the steps above, as shown in the figure:

1. Evidence extractor. Identifies codeable conditions in clinical notes.

2. Index navigator. Iterating through each codeable condition, the agent locates the term in the alphabetical index. The output is a list of candidate codes.

3. Tabular validator. Verifies and refines the candidate codes from the Index navigator by using the tabular list and chapter-specific guidelines.

4. Code reconciler. Considers all the code candidates identified by the Tabular validator. Using the guidelines and instructional notes, it removes codes and sequences the final list.

Our implementation was primitive; it was intended as a proof-of-concept. Our agents were large language models without any fine-tuning. The agents don’t need to be large language models. For instance, the index navigator can be a search engine.

Results

We compared Code Like Humans with PLM-ICD, a state-of-the-art model trained on clinical notes annotated with ICD-10 codes (MIMIC-IV). We evaluated the methods on MDACE, which consists of 300 clinical notes from MIMIC reannotated by professional medical coders.

Despite not being fine-tuned for the task, Code Like Humans achieved a higher F1 macro score than PLM-ICD (28% vs 25%). However, PLM-ICD achieved a superior F1 micro score (48% vs 43%). These results suggest that Code Like Humans is better at predicting rare codes, while PLM-ICD is better at predicting frequent codes. So which is better?

We trained and evaluated PLM-ICD on data from the same department. As a result, the codes that were frequent in the training data were also frequent in the test data. This gave PLM-ICD an unfair advantage. If we had tested on a different department in a different specialty, other codes would have been frequent, which would remove PLM-ICD’s advantage.

On the other hand, PLM-ICD is only trained on discharge summaries, while we evaluated the methods on all clinical note types. This was a disadvantage for PLM-ICD; however, it is something it ideally should be capable of handling.

Next steps

Whether this version of Code Like Humans is superior to PLM-ICD is inconsequential; what matters is its potential. PLM-ICD trained on MIMIC can only predict the 6,000 codes that appear in the training set; Code Like Humans can predict all 70,000 ICD-10-CM codes. PLM-ICD must be retrained to support a new ICD-10 version; Code Like Humans simply needs access to the latest guidelines, tabular list, and alphabetic index. PLM-ICD sucks at predicting rare codes; our basic implementation of Code Like Humans sucks less.

There are many low-hanging fruit for the proceeding studies. Each agent can be evaluated independently:

1. Evidence extractor: Compare the output with human-annotated evidence spans from MDACE or Douglas et al. (2025).

2. Index navigator: Compare the output with ground-truth terms. One can find the ground-truth terms by back-tracing from the annotated ICD-10 codes. To control the input of the agent, provide it with ground truth evidence that spans from MDACE.

3. Tabular validator: Provide the agent with a set of candidate codes/sub-categories, where the human-annotated code from MDACE is among them.

4. Code reconciler: Provide the agent with a list of codes where some of them are incorrect. Measure how well the model performs in removing incorrect codes from the list. You can also measure how well the agent orders the list. We did not do the latter in the paper.

Since each agent can be evaluated individually, one study can focus on improving one agent. The field can work in parallel on different agents, and by combining the improved agents, we may collectively achieve a new state-of-the-art method. If you are interested in joining this effort, check out the error analysis of each agent in the paper. We show, for instance, that the evidence extractor struggles in identifying codeable terms that are not related to diseases, such as smoking habits and instructions about resuscitation.

Why is ICD-10 insufficient?

ICD-10 lacks expressiveness. ICD-10 “only” comprises 14,000 codes. Although this may sound like a lot, it is insufficient to document all clinical scenarios, which has consequences for monitoring and policy making. For example:

• Using helium to commit suicide has become more frequent in the past decades1. However, there is no ICD-10 code for this specific suicide method. Consequently, implementing policies to prevent this form of suicide and monitoring their effectiveness is difficult.

• Respiratory acidosis, lactic acidosis, and metabolic acidosis all have the ICD-10 code: E87.2 Acidosis. The lack of specificity hinders research and finding ideal participants for clinical trials.

Due to the lack of expressiveness of ICD-10, most countries have designed national modifications. The American modification, ICD-10-CM, comprises 70,000 codes, encompassing conditions such as being struck by a turtle and burns resulting from water-skis on fire. Yet, it does not include enough conditions (for example, the variations of acidosis above). Furthermore, national modifications are undesirable because comparing data across countries becomes challenging. We want everyone to use the same system!

Moreover, ICD-10’s single enormous tree-like hierarchy is awkward. Which chapter should “Sepsis following a procedure” be in? A00–B99 Certain infectious and parasitic diseases or S00–T98 Injury, poisoning and certain other consequences of external causes? Both are valid, and a single hierarchy cannot reflect that.

SNOMED CT

SNOMED CT is a clinical terminology that comprises 350,000+ concepts structured in multiple hierarchies. In nerdy language, SNOMED CT is a graph, where each concept is a node, and the nodes are connected with different types of one-directional edges. Each concept has a unique identifier, a preferred name, and a list of synonyms.

The power of SNOMED CT lies in its ability to represent clinical concepts with incredible granularity. Where ICD-10 gives you "E87.2 Acidosis," SNOMED CT offers distinct concepts for respiratory acidosis (12326000), lactic acidosis (91273001), and metabolic acidosis (59455009). This specificity enables precise documentation, better clinical decision support, and more targeted research.

SNOMED CT's graph structure solves the hierarchy problem. Take "Sepsis due to and following procedure" (122041110001119105) as an example. We can see how SNOMED CT represents this complex concept through different types of relationships with other concepts (notice that concepts are not necessarily diseases):

• Is a: Organ dysfunction syndrome (capturing the systemic nature) 238147009

• Due to: Infectious disease (the underlying cause) 40733004

• Due to/After: Procedure (the temporal and causal relationship) 71388002

• Finding site: Body organ structure 113343008

• Associated morphology: Inflammatory morphology 409774005

• Pathological process: Dysregulated host response 769256002

This graph structure elegantly captures what ICD-10 struggles to categorize: “sepsis due to and following procedure” is simultaneously an infection, a procedural complication, and an organ dysfunction syndrome. The graph structure allows each relationship to be explicitly defined rather than forcing the concept into a single hierarchy.

The system also supports post-coordination, meaning clinicians can combine existing concepts to create new, more specific descriptions on the fly2. For instance, you can represent suicide using helium by combining the two concepts: “Suicide (event) 44301001” due to “Helium (substance) 90317004.“ With post-coordination, you can precisely describe conditions without pre-existing codes for every possible combination. SNOMED CT uses a specific grammar for post-coordination. An overview of this grammar is shown in the figure below. You can read more about this grammar here.

Here is an example of how to use the grammar to represent a family history of a particular disease.

A really cool feature of SNOMED CT is that you can generate specific queries using the different attributes (read relationship). So if you have electronic health records annotated with SNOMED CT, you get a powerful search engine. Below are two examples of queries.

We still need ICD-10

Despite SNOMED CT's superiority for clinical documentation, ICD-10 isn't going anywhere. Here's why:

Billing and reimbursement: Healthcare payment systems worldwide are built on ICD-10. Insurance companies, government programs, and hospital finance departments have decades of infrastructure designed around these codes. Rebuilding these systems would be monumentally expensive and disruptive.

Statistical reporting: WHO requires ICD-10 for international health statistics. Mortality reports, disease surveillance, and epidemiological studies all depend on ICD-10's standardized, albeit limited, classification system.

The learning curve: ICD-10's 14,000 codes are already challenging for many clinicians to master. SNOMED CT's 350,000+ concepts represent a steep learning curve that many healthcare systems aren't prepared to climb.

The path forward: Mapping, not replacing

The future isn't about choosing between SNOMED CT and ICD-10; it's about using them together. SNOMED CT can capture the clinical detail at the point of care, while automated mapping tools translate these detailed concepts into ICD-10 codes for billing and reporting. However, I’m uncertain of the accuracy of existing automated mapping tools.

If the EU mandate rumours are true, we're looking at a dual-system future: SNOMED CT for clinical documentation and decision support, ICD-10 for administrative functions. The challenge will be building robust mapping tools and training healthcare workers to navigate both systems effectively. These challenges may be alleviated by using machine-learning models to predict SNOMED CT codes. However, this may prove difficult to develop as most countries do not have clinical notes annotated with SNOMED CT codes available. There exist NLP approaches for predicting SNOMED CT, but they rely on old NLP techniques, so I’m unsure whether they are sufficiently accurate. Another possibility is using large language models. Perhaps it is possible to develop a multi-agent approach that leverages the structure and information in SNOMED CT to select the appropriate concepts, even without fine-tuning.

The transition won't be easy or cheap. Still, the potential benefits of better clinical documentation, improved patient safety, enhanced research capabilities, and eventually, true interoperability across European healthcare systems might make implementing SNOMED CT worth the effort.

Resources to learn more

• The SNOMED CT starter guide is a great resource for learning about SNOMED CT. Things are clearly explained with a lot of figures.

• You can play around with SNOMED CT in this browser.

• You can learn more about the differences between SNOMED CT and ICD-10 in this excellent PowerPoint presentation

Transluce, a non-profit start-up, recently released a really cool demo to investigate and change unwanted behaviors of large language models. In the demo, they dive into why a large language model (Llama-3.1 8B Instruct) struggles with comparing numbers.

They demonstrate that neurons representing biblical verses and calendar dates activate when comparing 9.9 with 9.11. In these two settings, 9.11 is indeed bigger than 9.9. They then suppress the neurons representing biblical verses and calendar dates, and the model correctly answers that 9.9 is bigger than 9.11. I encourage you to check out the demo!

There are still improvements to be made. In the demo, they only investigate neurons. However, in most cases, a feature is represented by multiple neurons, not one. If they considered the directions in representation space instead of neurons, they might have detected more features, such as software package versions. That being said, extracting features represented by multiple neurons is an unsolved researched problem, so it’s understandable that they started with individual neurons.

We hypothesize that this decline is because current LLMs cannot perform genuine logical reasoning; they replicate reasoning steps from their training data. Adding a single clause that seems relevant to the question causes significant performance drops (up to 65%) across all state-of-the-art models, even though the clause doesn't contribute to the reasoning chain needed for the final answer.

While I find their findings interesting, they failed to consider alternative hypotheses. Instead of showing that LLMs can’t reason, they may have shown that their logical reasoning is sometimes flawed.

So, what were their findings, and why do I disagree with their conclusions?

GSM-symbolic

GSM8K is a popular benchmark for evaluating LLMs’ mathematical reasoning capabilities. The dataset comprises 8000 grade-school math questions and answers. However, many researchers fear that LLMs are trained on GSM8K, meaning that we have shown the math students the exam questions and answers before the exam.

Mirzadeh et al. (the Apple researchers) evaluate what happens if you make small changes to the GSM8K questions. They create question-answer templates that allow them to change the variables in the questions effortlessly.

LLMs are sensitive to meaningless changes.

Below are the results when the names (e.g., Sophie, Jack, etc.) and numbers are varied. Each of the six plots shows the result for a specific LLM. The dotted black vertical line indicates an LLM’s performance on the original GSM8K benchmark.

Changing the names impacts the performance, and changing the numbers changes it even more. Changing the names of the characters in a math question should naturally not affect the answer. But, is being affected by irrelevant changes evidence that the models are incapable of reason?

It could, but it could also indicate that their logical reasoning is flawed. For example, LLMs are known for preferring frequent tokens over in-frequent. If any of the names or numbers in the template are more common, that may impact the performance. Also, how each model tokenized the numbers may impact the performance. Is it stupid? Yes. Does it mean no reasoning? No.

LLMs are worse at harder questions.

Next, they evaluated the effect of changing the questions’ difficulty. They created three additional templates with increasing difficulty: M1, P1, and P2.

They found that as the difficulty increases, the performance decreases, and the variance increases. Again, is this proof that LLMs can’t reason? Nope. You can perform logical reasoning but struggle with difficult questions. Also, the more tokens you have to output to answer a question, the more probable are errors. It’s quite simple. The more times you try to predict sometimes, the more likely you are to make a mistake.

Trick-questions trick LLMs

They create a template to test how inserting “irrelevant” sentences affects LLMs. In the example below, they add, “but five of them were a bit smaller than average,” which the LLMs infer that the kiwis should be subtracted.

The figure above depicts the performance drop when inserting these irrelevant sentences. Mirzadeh et al. claim this is evidence of a lack of logical reasoning.

I completely disagree. LLMs are trained on a vast amount of text. Most of this text is not produced by mathematicians. Most people are not as precise as mathematicians. They will often imply things despite not stating them explicitly. Understanding imperfectly formulated questions is the desired LLM behavior; otherwise, LLMs will misunderstand too many questions posed by non-mathematicians.

So, no, this is not evidence that LLMs can’t reason. They are just trained to infer information from imprecise questions.

Summary

While the experiments in this paper were excellent, they failed to consider alternative hypotheses in analyzing their findings. It seems they had decided on their conclusion beforehand, i.e., confirmation bias. Don’t get me wrong. Their hypothesis could be correct. I just wish they did a better job considering other explanations.

This blog post will explore the paper's core ideas and findings. Specifically, we'll examine how we developed a new method for generating explanations for automated medical coding models that rival supervised approaches' performance—without requiring expensive annotations.

The Power of Explanations

Explainability is crucial for making automated medical coding useful in the real world. Consider the following example:

Validating these codes is time-consuming, even for this relatively simple example. You must locate the evidence of each medical code and then verify if that evidence is sufficient. Imagine when the clinical note comprises thousands of words!

Now, look how much easier it becomes with explanations:

You can hover your mouse over the code, and it will show you the evidence—much faster! Not only can it help us validate errors faster, but it can also be an excellent tool for debugging the model.

Unexpected Model Behavior

For example, in one case, our model predicted the code Z72.51: High-risk heterosexual behavior for a patient involved in a motorcycle accident. This was surprising because Z72.51 represents irresponsible sexual behavior, and there was no mention of sexual activity in the text. The explanation showed that the model predicted the code because the patient rode a motorcycle. The model had learned some incorrect correlation between motorcyclists and irresponsible sex!

Explanations are obviously helpful; however, producing high-quality explanations is a challenging, unsolved problem.

The State of Explainable Medical Coding

Multiple studies propose explanation methods for automated medical coding models. Most of them use the attention weights from the final layer as the explanation. A study from 3M provides the state-of-the-art explanation method. They hired medical coders to annotate medical codes and the relevant evidence in the clinical notes. When training their automated medical coding model, they also trained the attention weights to focus on the annotated evidence spans, improving the attention weights’ explainability.

However, evidence-span annotations are expensive. We once were offered to buy a dataset comprising 12,000 examples annotated with medical codes and evidence spans. It cost $420,000. We would need such a dataset for each code system and language—not scalable!

Our Approach: Explanations Without Expensive Annotations

The beauty of medical coding is that almost every medical document in the world is annotated with medical codes because they are required for statistics and billing. Consequently, most hospitals have datasets that can be used to train automated medical coding models. Therefore, we asked the question: Can we produce high-quality explanations only using medical documents and codes, i.e., no evidence-span annotations?

Spoiler alert—yes, we can. In our EMNLP paper, we demonstrated that we could produce explanations of similar quality as 3M but without using evidence span annotations. We achieved this feat with two contributions:

1. We improved the explanation method.

2. We improved the model's robustness.

Let's dive into the details!

Model Architecture: PLM-CA

To understand our experiments, it's helpful to know the model architecture we used. We employed a variant of PLM-ICD, which we named PLM-CA. Here's a simplified overview of its architecture:

Components:

• Input: Clinical text (e.g., "Patient has heart failure") and potential medical codes

• RoBERTa Encoder: Transforms words into rich vector representations

• Cross-attention: Compares medical codes with word meanings

• Output: Probability scores for each medical code

How It Works:

1. Text Encoding: The clinical text is fed into RoBERTa, which encodes each word (token) into a vector that captures its meaning and context.

2. Label Representation: Each potential medical code (label) also gets its own vector representation.

3. Cross-attention Magic: This is where the real matching happens. The cross-attention layer compares each medical code with every word in the text.

4. Final Prediction: A linear layer takes each updated code representation and calculates a probability score.

Now that we understand the model architecture let's move on to how we improved the explanations.

Improved Explanation Method: AttInGrad

We aimed to improve the explanation method. Since previous studies only evaluated attention-based explanation methods, we tested other approaches, such as gradient-based and perturbation-based methods. Surprisingly, these methods produced worse explanations than attention.

We noticed that the errors from attention-based and gradient-based methods rarely overlapped. By multiplying the feature attribution scores, we created a new method called AttInGrad, which significantly improved the explanations.

So, how much better is it, exactly? We evaluated plausibility, which is how convincing the explanations are for humans. We estimated plausibility by comparing the predicted important words with the annotated evidence spans. These are the F1 scores:

• InputXGrad (gradient-based method): 31.6%

• Attention: 36.5%

• AttInGrad: 41.5%

Quite an improvement! AttInGrad also improved the faithfulness of the explanations, i.e., it more accurately reflected the model’s inner workings.

I will explain why AttInGrad improved the results later.

Enhancing Model Robustness

An explanation can be entirely faithful—accurately reflecting the model's internal mechanisms—yet still be difficult or impossible for humans to understand. If the model's underlying processes are flawed, a faithful explanation of those processes will inherit those flaws, making it challenging to understand or interpret correctly. For instance, if our model thinks that a person has irresponsible sex because he rides a motorcycle, our explanation will look weird, no matter how accurate it is.

Adversarial examples reveal flaws in a model’s inner mechanisms. Basically, you can add invisible noise to the input that tricks machine-learning models. For example, you can fool an image classifier into thinking an image of a cow is an image of an airplane. There is a paper called "Adversarial Examples Are Not Bugs, They Are Features" which shows that image classifiers learn to use tiny pixel changes as features. Consequently, faithful explanations for such sensitive models look weird, and they should! If a tiny change to a pixel changes the output, the explanation should highlight it as important. The issue isn't the explanation method; it is the model!

Several papers show that we can make explanations more plausible by training the models to be robust against adversarial examples. The results from the paper Robustness May Be at Odds with Accuracy by Tsipras et al. are particularly striking. The top row shows the image, the second row shows the explanation from a standard model, and the last two rows show explanations produced by two different adversarially robust models. They all use the same explanation method, yet the difference is striking.

We evaluated whether improving our medical coding model's robustness towards adversarial examples also improved the plausibility of explanations. This has been demonstrated in image classification studies but not in text classification.

We tested three adversarial training approaches and found that token masking was most effective. Token masking identifies the least important tokens during training and replaces them with a <mask> token, teaching the model not to rely on irrelevant tokens.

Token masking improved the plausibility of most explanation methods:

• InputXGrad (gradient-based method): 31.6% → 33.1%

• Attention: 36.5% → 37.0%

• AttInGrad: 41.5% → 41.9%

In my previous blog post, I explain the relationship between the model and the explanation method in more depth.

Combining the Improvements

By combining our improved explanation method (AttInGrad) with the enhanced model robustness (using token masking), we produced explanations with plausibility similar to that of the supervised state-of-the-art method but without requiring expensive annotations.

Here's how our approach compares to other methods:

As you can see, our unsupervised method (AttInGrad+TM) performs comparably to the supervised state-of-the-art method from 3M (Attention+Bs), while significantly outperforming previous unsupervised approaches (Attention+Bu).

Unmasking the Mystery: Why AttInGrad Outperforms Attention

While both AttInGrad and Attention produced better explanations than other methods, they showed a much higher variance in their results across different runs of the model training. This means that training the same model multiple times with different random starting points could lead to very different explanations, even if the model's accuracy remained consistent.

Upon closer examination, we discovered that Attention often highlighted "special tokens" as highly important for its predictions. These special tokens include things like punctuation marks, spaces, and artifacts from the text encoding process. These tokens don't carry significant meaning for humans trying to understand the reasoning behind a medical code assignment. In fact, we showed that special tokens accounted for only 5.8% of the tokens within the human-annotated evidence spans.

This reliance on special tokens by Attention explains the high inter-seed variance. The figure below shows the relationship between the proportion of special tokens among the top five most important tokens and the F1 score (representing plausibility). Each point on the plot represents the average statistic for a single run of the model training.

For both Attention and AttInGrad, there is a strong negative correlation between the proportion of special tokens and both F1 score and Comprehensiveness. This means that as the explanations rely more heavily on special tokens, their quality decreases.

InputXGrad, in contrast to Attention, demonstrates much lower variance across runs and focuses more consistently on meaningful words. By incorporating InputXGrad, AttInGrad benefits from this more stable behavior and reduces the influence of these misleading special tokens. AttInGrad is essentially using InputXGrad to reduce its reliance on special tokens, thereby improving its performance.

Findings after the paper submission

It’s been 3.5 months since we submitted the paper. Since then, we have learned a lot. Here are our main realizations:

1. Gradient-based methods are terrible. I cover this in this blog post.

2. Cross-attention (also called label-wise attention) was key to the good performance of Attention and AttInGrad. We later found that these methods produced poor explanations for architectures with multi-head attention and skip connections in the final layer. I will discuss this in a later blog post.

3. DecompX achieved impressive results in our recent paper on the sentiment classification class. I evaluated it for automated medical coding and found it to perform similarly to Attention.

Conclusion

Our research demonstrates that it's possible to produce high-quality explanations for automated medical coding without relying on expensive annotated datasets. By improving both the explanation method and the model's robustness, we've taken a significant step toward making automated medical coding more transparent and trustworthy.

I will present the paper at EMNLP 2024 in Miami; I hope to see you there!

Traditionally, the focus has been on refining explanation methods to improve the quality of the explanations. However, the model being explained is a critical and often overlooked factor in this equation. The interplay between the model and the explanation method significantly impacts the resulting explanations, yet this relationship remains largely unexplored.

In this blog post, we delve into this relationship, examining how a model's characteristics influence the effectiveness of explanation methods and, consequently, the quality and accuracy of the insights we can derive from them. Before we examine how the model influences the quality of an explanation, we must define what makes an explanation good.

What makes an explanation good?

Usually, researchers use two key criteria:

1. Plausibility: The explanations must be understandable and convincing to humans.

2. Faithfulness: The explanations must accurately reflect the model's internal mechanisms. Essentially, the explanation shouldn't be misleading.

An explanation must be faithful; otherwise, it will be misleading. An explanation must be plausible; otherwise, it will be useless to show humans. Therefore, we want explanations to be both faithful and plausible. Makes sense? Great! Let's explore how the model can influence each of these criteria.

How the model impacts plausibility

An explanation can be entirely faithful—accurately reflecting the model's internal mechanisms—yet still be difficult or impossible for humans to understand. This disconnect can occur for two main reasons:

1. Flawed Internal Mechanisms: If the model's underlying processes are flawed, a faithful explanation of those processes will inherit those flaws, making it challenging to understand or interpret correctly.

2. Excessive Complexity: Even if the model's internal mechanisms are sound, they may be so complex that a faithful explanation becomes too intricate for easy human comprehension.

This phenomenon isn't unique to machine learning. In academic research, we often encounter papers that are difficult to understand, not only because of poor writing but also because the underlying concepts are either flawed or exceedingly complex. The same principle applies to explanations of machine learning models. Here are some examples of how a model's internal mechanisms can impact plausibility.

Shortcut learning

Machine learning models often use shortcuts. It's like students who learn how to do well on the exam instead of understanding the subject. For example, an image classifier can classify a cow by only looking at the green background—it fails to predict a cow on a beach. A faithful explanation would highlight the background, not the cow. This explanation would confuse users without the background (🤣) information that machine learning models learn such shortcuts. To learn more about shortcut learning in deep neural networks, check out the paper Shortcut Learning in Deep Neural Networks by Geirhos et al. It is one of my favorite papers!

Adversarial examples

Image classifiers that are sensitive to adversarial examples also produce less plausible explanations. Basically, you can add invisible noise to images that trick the models into thinking an image of a cow is an image of an airplane. Weird, right? There is a paper called "Adversarial Examples Are Not Bugs, They Are Features" which shows that the models learn to use tiny pixel changes as features. Consequently, faithful explanations for such sensitive models look weird, and they should! If a tiny change to a pixel changes the output, the explanation should highlight it as important. The issue isn't the explanation method; it is the model!

So, can we improve plausibility by improving the model?

Improving plausibility

Several papers show that we can make explanations more plausible by training the models to be robust against adversarial examples. The results from the paper Robustness May Be at Odds with Accuracy by Tsipras et al. are particularly striking. The top row shows the image, the second row shows the explanation from a standard model, and the last two rows show explanations produced by two different adversarially robust models. They all use the same explanation method, yet the difference is striking.

In our paper, An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Records, we showed that the same is true for text classifiers. However, the improvements were less drastic than those for image classification. I think it is because image classifiers are more sensitive to adversarial examples than text classifiers.

Making models robust towards adversarial examples is relatively straightforward, but how do we prevent the cow classifier from relying on the background? This is far more difficult and unsolved at the current moment. Preventing models from relying on spuriously correlated features is one of the main objectives of causal machine learning. The paper Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond by Feder et al. introduces the topic well.

How the model impacts faithfulness

Some models cause certain explanation methods to produce less faithful explanations than other models. Our paper, Normalized AOPC: Fixing Misleading Faithfulness Metrics for Feature Attribution Explainability, showed that BERT trained on Yelp produced far more truthful explanations than RoBERTa trained on IMDB when using the same explanation method. Why does that happen?

Explanation methods make assumptions about the models' internal mechanisms. For instance, most explanation methods assume feature independence (a quite severe assumption). Faithfulness measures how accurately the methods' assumed mechanisms reflect the model's actual mechanisms. If an explanation method assumes feature independence, but the model relies on feature interactions, the explanations will be less faithful.

Improving faithfulness

There are two approaches to improving faithfulness:

1. Train the model to fit the explanation method's assumptions better. For example, train the model to not rely on feature interactions.

2. Develop explanation methods that make fewer assumptions. For example, develop explanation methods that don't assume feature independence.

Multiple studies have tried the first approach. However, we demonstrated in Normalized AOPC: Fixing Misleading Faithfulness Metrics for Feature Attribution Explainability that the evaluation metric they used can’t be compared across different models, which all of them did. Therefore, the conclusions of these studies may be misguided. Furthermore, most of them use gradient-based explanation methods, which I argued are not explanations in my previous blog post.

Even if you could train the model to produce a more faithful explanation, I'm skeptical about whether it is the best approach. I think we should try to improve the explanation methods before we limit the models' internal processes. For a starter, we should not assume feature independence—feature interactions are the whole point of using attention.

Wrapping up

Models can impact the explanations' plausibility or faithfulness in different ways. They can improve plausibility by relying on features and patterns that align with human intuition and domain knowledge. Models can improve faithfulness by aligning more closely with the assumptions made by explanation methods, though this approach has its limitations.

However, I believe we should divide the responsibilities between the model and explanation methods:

1. The model's responsibility is to learn meaningful and robust features that generalize well, which often leads to more interpretable decision-making processes.

2. The explanation methods' responsibility is to accurately reflect the model's internal mechanisms, regardless of their complexity.

This division of responsibilities allows us to optimize both aspects without unnecessarily constraining either the model's performance or the explanation's accuracy.

I hope you enjoyed this blog post. I would love to hear if you disagree or have interesting insights on this topic. See you next time!

However, my recent research has led me to question this assumption. In An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Records, we compared a wide range of explanation methods. To my surprise, gradient-based explanation methods produced the worst explanations—even worse than attention. In this blog post, I will try to explain why.

By the end, I hope to convince you that gradient-based explanation methods, such as InputXGradients and Integrated Gradients, are far from being a gold standard—they barely qualify as a rusty iron standard. Let's dive in!

Gradient-based explanations 101

Gradient-based explanation methods produce explanations using the output gradients with respect to the input features. For deep neural networks, they are typically calculated using backpropagation. While gradients can be challenging to grasp intuitively, especially in the context of deep neural networks, I find the following explanation helpful:

Think of the output gradient with respect to an input feature as how much the output wiggles if you slightly wiggle that input feature. We're talking about an infinitesimally small input wiggling—so small it's almost undetectable.

These methods measure the importance of each input feature by calculating how much the output changes when each feature is wiggled separately. This results in a score per feature representing its importance to the output, a form of explanation known as feature attribution.

Let's look at a concrete example to better understand how gradient-based explanations work. Consider the following linear model:

For this model, we calculate the input gradient as follows:

As you can see, the input gradients of a linear model are simply its weights. At first glance, this might seem like a good estimation of each feature's contribution. However, there's a significant issue: this explanation is unaffected by the actual input values.

For instance, given an input (0.3, 0, 0.6, 0.1), the contribution of the second feature should be 0, not 0.5. This is where a method called InputXGradient comes in. It addresses this limitation by multiplying the input and the input gradients:

Using our example input (0.3, 0, 0.6, 0.1), we get (0.03, 0.00, 0.12, 0.02). This explanation more accurately reflects each feature's contribution to the output, taking into account both the model's weights and the specific input values.

This example demonstrates how gradient-based methods attempt to provide meaningful explanations for model predictions. However, as we'll explore in the next section, these methods have significant weaknesses that limit their effectiveness as explanatory tools.

Their weaknesses

Non-linear functions

Above, I showed you that InputXGradient works well for linear models. However, deep neural networks are rarely linear; that’s the whole point of using them. Unfortunately, gradient-based explanation methods often fail when dealing with non-linearity. Let's look at a simple demonstration.

Take the following model:

This model takes two input features. It uses a sigmoid function (σ) that takes the second feature as input. A sigmoid function is a non-linear function and looks like this:

Now, let’s say we provide the model with input (10, 10). We then get the following output:

In blue, you can see the contributions from each feature. You can clearly see that the second feature is far more influential than the first. So, which features are important according to InputXGrad?

According to InputXGrad, the second feature is unimportant, which is clearly wrong. So why does this happen? If you look at the plot for large values, wiggling the input of a sigmoid function has little impact on the output—the line is flat.

This example illustrates how gradient-based explanation methods struggle with non-linearities, which are fundamental to the activation functions in deep neural networks. However, while basic gradient methods struggle with non-linearity, more advanced techniques have been developed to address this issue. One such method is Integrated Gradients. Let's explore how it works.

How Integrated Gradients deals with non-linearity

Integrated Gradients requires the user to define a baseline value representing an uninformative input. In image classification, this can be an all-black image or an image of white noise. Mask tokens are commonly used in text classification.

Imagine a straight line between the input and the baseline value. Integrated Gradients takes small steps along this line. At each step, it calculates the gradients. It then averages the gradients across all steps. Finally, it multiplies this average by the difference between the input and baseline values.

Let’s consider the previous example. We use a baseline value of (0,0) and a step size of 50. Imagine a straight line between (0,0) and (10,10). We take 50 steps along this line and calculate the gradient at each step. We average the gradients and get (0.01, 0.98). We then multiply it with the input minus the baseline values.

So (0.1, 9.8) is our final explanation. This is much better! It correctly identifies that the second feature is more important than the first, aligning with our intuitive understanding of the model.

It is currently unclear to me whether Integrated Gradients solves the non-linearity issue and exactly why. I can see that it works for toy examples, but I am unsure how it works for real data. In my two explainability papers, Integrated Gradients perform similarly to InputXGradients, which could suggest it doesn’t solve the issue in all scenarios.

Even if Integrated Gradients solves the non-linearity problem, it's important to note that they don't solve all the problems associated with gradient-based explanations. There are still other issues to consider, which we'll explore in the next section.

What does wiggling the input even mean?

While it's easy to imagine wiggling input features in our simple numerical examples, the concept becomes much more complex and abstract for real-world tasks, especially in text classification.

The weirdness of word gradients

Consider an email spam classification task. The model takes an email as input and outputs a score representing the probability of the email being spam. In this context, what does wiggling the input mean? How do we wiggle words?

Most text classification models represent each word or sub-word as a vector. You can think of these vectors as points in a high-dimensional space. For example:

When we calculate gradients for these word vectors, we get a vector per word, not just a single number. This gradient vector points in the direction of maximum impact on the output, and its size indicates the magnitude of that impact. Researchers typically use this size to estimate a word's importance.

But here's where things get weird:

1. Interpretation Challenges: How do we make sense of directions in this word space? What does it mean to move a word towards "computer" or away from "fruit"?

2. Semantic Discontinuity: Small changes in this continuous space might lead to nonsensical "words.” What meaning does the midpoint between "computer" and "fruit" have?

These issues reveal a fundamental mismatch: We're using continuous mathematics to explain discrete, semantic concepts. The result? Gradient-based explanations often lack clear linguistic meaning.

Let’s be clear: they accurately measure how tiny changes to the input change the output; I just don’t think it’s aligned with how people interpret the explanations.

Can Integrated Gradients save the day?

Spoiler alert: No, they can't. Let's see why.

Consider this visualization:

This simplified 2D space shows words as points, with the red line representing the path between a baseline (<mask>) and the word "cat" in Integrated Gradients.

This method introduces new problems:

1. Meaningless Intermediates: The path between <mask> and "cat" passes through non-existent words. What do these points mean?

2. Arbitrary Baselines: Why choose <mask> as the baseline? How would using "animal" or "dog" change our explanation?

Edit

I’ve realized that passing through nonsensical representations may not be an issue. However, I’ve found another issue with integrated gradients that causes it to struggle with architectures such as transformers. I haven’t decided whether to publish my new findings as a blog post or paper.

Wrapping up

Throughout this discussion, we've focused on two primary weaknesses of gradient-based explanation methods:

1. Struggle with Non-linearity: Basic gradient methods often fail when dealing with non-linear functions, which are ubiquitous in modern machine learning models.

2. Difficult to interpret: In the realm of text classification, gradients - whether basic or integrated - prove to be unintuitive and challenging to interpret. I don’t think “how much the output wiggles when wiggling an input feature” is how most people interpret an explanation.

While we focused primarily on text classification, these interpretability issues likely extend to other domains as well. Even in image classification, where gradients might seem more intuitive, the fundamental challenge of translating mathematical operations into human-understandable explanations persists. For example, what does it mean if the output changes because a pixel becomes slightly darker?

So, what are the alternatives to gradient-based methods? I think for inputs with few features, perturbation-based explanation methods, such as LIME or SHAP, are excellent choices. Attention sometimes works well, and sometimes don’t. It depends on your model. If you use transformers, transformer-specific explanation methods, such as DecompX or ALTI, are good choices. I’ve found DecompX to work better than ALTI. However, DecompX uses a lot of memory for long inputs, so that may be a deal-breaker. I’m currently working on a way to fix this, so hopefully, I will have a good explanation method in a short time!

I’ll wrap up here. Do you agree with my points or think I’m off base? I’d love to hear your thoughts!

1. Leveraging relationships between symptoms, medications, and diseases to predict medical codes more accurately.

2. Using medical code co-occurrences to improve prediction accuracy.

3. Exploiting the ICD hierarchy to teach models about code similarities and differences.

These ideas might sound reasonable at first glance—I certainly thought so until recently. But I've come to realize they have some weaknesses. Let me break it down for you, but first, here is a quick primer on medical coding for those new to the field.

Medical coding 101

After you’ve been discharged from a hospital or seen your physician, someone (physician, secretary, or professional medical coder) must sift through your medical documentation and assign a set of medical codes. These medical codes are machine-readable identifiers for diagnoses or procedures used for documentation, statistics, and (importantly) billing.

Getting these codes right is crucial, especially for healthcare providers who rely on insurance reimbursements. If a code is assigned but not backed up by the documentation, the insurance company won't pay. On the flip side, if codes are missing, the insurance company will happily underpay.

WHO created ICD-10, a medical code system comprising ~14,000 diagnosis codes. The American healthcare system didn’t think there were enough codes, so they created an American version called ICD-10-CM, which comprises more than 70,000 diagnosis codes. We now have codes for such vital medical conditions as W59.22: Struck by a turtle and V91.07: Burn due to water skis on fire.

For procedures, American hospitals typically use two systems: ICD-10-PCS (70,000+ codes) for inpatient care and CPT (11,000+ codes) for outpatient. There are other code systems too, but these are the major players.

If you're new to this field, you absolutely must read the ICD-10-CM coding guidelines (at least the first 20 pages) and the ICD-10-PCS guidelines. They are crucial for understanding the problem we are trying to solve.

Two key takeaways from these guidelines:

1. Use both the Alphabetic Index and the Tabular List. The Alphabetic Index maps terms to codes or subcategories, while the Tabular List is the tree structure most people associate with ICD.

2. Code only diagnoses explicitly described in the medical documentation.

Let's dive a bit deeper into how medical coders actually use the Alphabetic Index and Tabular List. When assigning codes, a medical coder doesn't just jump straight to the Tabular List (the tree structure we all think of when we picture ICD codes). Instead, they start with the Alphabetic Index.

Here's the process:

1. The coder looks up a term (let's say "sepsis") in the Alphabetic Index.

2. They navigate through the index to find the type of sepsis that best matches the condition described in the medical documentation.

3. This leads them to either a specific code or a subcategory in the Tabular List.

4. The coder then goes to that section of the Tabular List, where they'll find more detailed instructions to help them pinpoint the exact right code.

This two-step process is crucial because it helps ensure accuracy and consistency in code assignment. It's also why simply relying on the hierarchy in the Tabular List for understanding code relationships can be misleading—but more on that later.

Now, let's dive into why I think the three popular research ideas above are problematic.

The good, the bad, the ugly

1. The ICD hierarchy: not as helpful as you might think

Many studies use the Tabular List (the ICD tree) to teach models about code similarities. The assumption is that codes close to each other in the tree are similar, while distant codes are different.

Here's the problem: the Tabular List isn't always structured based on code similarity. Take sepsis, for example. Most sepsis codes are in the A-chapter, but for newborns, you use the P36 category. And if a procedure caused the sepsis? That's T81.44. Same condition, different circumstances, but a model trained on the Tabular List would think they're completely unrelated.

That said, this idea might work better if we use both the Alphabetic Index and the Tabular List together, as the guidelines instruct. There's a great study waiting to be done here (hint, hint)!

2. Medical coding ≠ diagnosing

Some studies try to leverage relationships between medical codes, symptoms, medications, and conditions to infer codes. But here's the thing: medical coding isn't diagnosing; it's documenting established diagnoses and procedures. Coders are prohibited from assigning codes for conditions not mentioned in the documentation.

So what's the harm? Well, models that infer codes not explicitly mentioned are less useful in real life:

• For medical coders, these models will suggest codes they are prohibited from assigning, which is just annoying.

• For physicians, these models might encourage assigning codes without proper documentation, leading to incomplete records and potential insurance denials.

Don't get me wrong—a separate model for inferring diagnoses could be super helpful. Combined with a solid medical coding model, it could flag diagnoses the physician forgot to document. But a medical coding model shouldn't try to be a diagnostic tool at the same time.

3. Why relying on code co-occurrences falls short

Many studies propose using medical code co-occurrences for automated coding. The logic goes: if code A often co-occurs with code B, we can infer code A when we're sure about code B. Also, if two codes never co-occur, we should avoid predicting both.

This approach has three major issues:

1. Again, we shouldn't be inferring codes. The guidelines are clear: diagnoses must be explicitly mentioned.

2. There are over 70,000 diagnosis codes, and most are rare. In MIMIC-IV (a popular dataset), more than 50% of code co-occurrences in the validation set never appear in the training set.

3. Code co-occurrences vary between hospitals and specialties. Relying on them will result in overfitted models that don't generalize well.

Why do these flawed ideas persist?

You might be wondering why these problematic approaches keep popping up. Well, they do seem to cause small performance improvements on popular benchmark datasets like MIMIC-III and MIMIC-IV. But here's the issue: I think we're seeing these improvements because of noisy annotations in these datasets.

Many clinical notes in these datasets are annotated with codes that aren't actually mentioned in the text. Sometimes the relevant text was removed during anonymization (like the social history in MIMIC-IV), or the information is simply elsewhere. When the mention of a diagnosis is missing but the code is still there, models that infer codes from other factors will score better. But this isn't because they're actually better—it's because of flawed evaluation.

For example, smoking habits are often mentioned in the social history section of discharge summaries. In MIMIC-IV, this section is removed. A model that infers smoking status from other conditions (like hypertension or COPD) will perform better on such a dataset. But on a clean dataset, this model would produce more false positives when guessing about smoking habits that aren't explicitly mentioned.

Wrapping Up

So, there you have it—my thoughts on why some popular approaches in automated medical coding research might be leading us down the wrong path. What do you think? Do you agree with my arguments, or do you see things differently? Let's start a conversation and push our field in a more productive direction!

John Oliver started the latest episode of Last Week Tonight with John Oliver with the following quote:

A 2016 analysis included that more than 250,000 deaths per year are due to medical error in the US, making it the third leading cause of death.

Medical errors are common and do cause a lot of pain and deaths. Pointing out the problem can help us make healthcare better. However, the 2016 analysis John Oliver refers to contains heavily inflated numbers, which has had negative consequences. In this video, the National Rifle Association uses the claim to argue that medical errors kill 500 times more people than guns. Supporters of alternative medicine use the claim to paint a picture of how dangerous conventional medicine is: “more Americans are killed in U.S. hospitals every six months than died in the entire Vietnam War.”

The 2016 analysis is called Medical error — the third leading cause of death in the US. The authors estimate that medical errors cause 251,454 hospital deaths a year in the US, making it the third biggest killer behind cancer and heart disease. The article is published in a prestigious medical peer-reviewed journal (BMJ) and has over 4,000 citations. That is more citations than most researchers accumulate during their entire careers. It is understandable that people would use the claim from a paper with such impressive credentials. I have myself done so. However, the article has received heavy criticism, and as you will learn in this blog post, rightfully so.

Kaveh G. Shojania and Mary Dixon-Woods pointed out that there are 700,000 annual hospital deaths in the US. If medical errors cause 250,000 hospital deaths, that would mean that errors cause 1 out of 3 hospital deaths. This is far from estimates in other studies in the 1%–4% range. How can the estimates be so different?

Not everyone has kidney failure.

The authors of the 2016 analysis estimated that 0.71% of hospital admissions result in a lethal medical error by averaging the estimates of four studies. They multiplied the estimate by the total number of hospital admissions in the US (35,416,020), which resulted in 251,454 hospital deaths.

Three of the studies were small, each reporting fewer than 14 deaths. The patients in the studies were often very sick. For instance, one of the studies was conducted on Medicare patients who were 65 years or older, with disabilities or end-stage renal disease. The authors of the paper counted the number of medical errors that they believed contributed to their deaths, and the 2016 analysis used this number to estimate the number of deaths in the entire population. Not everyone has end-stage renal failure; one out of ten hospital admissions are patients giving birth. A medical error is more likely to be fatal if you are already dying. You cannot extrapolate death statistics on a small group of dying patients to the entire population. How can we even know that the medical error killed the patient if they would have died anyway?

Correlation is Not Causation.

The studies included in the 2016 analysis reported the number of deaths following a medical error. The 2016 analysis assumed that if a death followed a medical error, the medical error caused the death. However, an event followed by another event doesn’t necessarily mean that the first one caused the second. The sun doesn’t rise because the rooster crowed, the prisoner wasn’t executed because he had his last meal, and the Make-A-Wish Foundation doesn’t kill children. Most medical errors result in minor consequences. A patient can develop a rash because the physician prescribed the wrong medication and then die of another cause (e.g., kidney failure). The studies used in the 2016 analysis did not analyze whether the patients who died after medical errors were killed by it. Given the flaws of the 2016 analysis, does it mean you cannot trust John Oliver?

Who’s to blame?

We cannot blame people for referring to the 2016 analysis. Everyone cannot be expected to fact-check every peer-reviewed article they quote, especially if they are not researchers. It is the author’s responsibility to be thorough in their analyses and the journals’ responsibility to reject and retract weak articles. If we can’t trust peer-reviewed articles, what can we trust? I’m not the first to point out the flaws in the 2016 analysis, and yet, despite the criticism and negative consequences, the authors and BMJ have not retracted the article.

Does this mean that medical error is not an issue?

Medical error is a significant issue despite not being the third leading cause of death. Other studies estimate that errors cause 1–4% of all hospital deaths. That’s still a lot of lives. Eliminating medical errors would prevent thousands of deaths annually, both inside and outside the hospitals, and reduce unnecessary suffering. Reporting and preventing medical errors is essential, but that doesn't mean we should overestimate its consequences.