On the morning of October 8th, 2005, just as children reached school at 8:50 in the morning, a powerful earthquake struck the Himalayan region. Measuring 7.6 on the Richter scale it left more than 80,000 people dead and virtually all physical infrastructure in the affected regions destroyed. Winter came shortly afterwards, and although relief providers and the government made heroic efforts, many were left to fight the elements in makeshift tents and broken houses jury-rigged with tin-sheet roofs, Pictures 1 and 2.

Over the next few years, people made a heroic effort to rebuild their lives. Schools and roads were rebuilt, health clinics and markets came back up, and critically, every family who had suffered destruction during the earthquake received substantial compensation from the government.

In 2009, four years after the earthquake, I returned to the region, after having been actively involved in the relief efforts in the initial days. Over 6 months, with Tahir Andrabi and the RCONS team, we visited 125 villages scattered throughout the region, and interviewed more than 120,000 people with a detailed follow-up for 5,000. In the detailed follow-up, along with specifically tailored modules that documented the experience of the earthquake and the household’s subsequent recovery from it, we measured consumption poverty and mental health, using the 12-item PHQ-12.

Our previous work had established that, four years after the earthquake, there were no differences in household and adult outcomes between villages that had been struck by the earthquake and equivalent villages that had escaped the catastrophe. However, children’s educational outcomes and health (as measured by height) continued to show significant deficits. We now examined the question of heterogeneity, asking whether a mental health-poverty gradient emerges in the villages that were struck by the earthquake.

Pictures 1 and 2: Makeshift shelters after the Pakistan Earthquake

Two key features of the earthquake are important for our analysis of heterogeneity.

First, we confirmed through our survey that the households distance from the fault line was a key marker of the impact of the earthquake. At 40 kilometers from the fault line, mortality rates were around 0.1% and 20% of homes were destroyed, but right next to the fault line, the mortality rate of 4.5% was 45 times as high and home destruction, at 95%, was virtually universal.

Second, using additional existing data and “fixed characteristics” such as the height and education of adults, we confirmed that prior to the earthquake, households living close to the fault-line were no different from those living farther away along dimensions that could have directly affected their mental health. They were just as educated, seemed to be just as healthy as measured by adult heights and adult weights and seemed to have enjoyed very similar access to all kinds of infrastructure.

With these two elements in place, we could assess if the earthquake, which surely led to significant societal and individual upheaval, and struck some villages worse than others, led to sustained differences in the gradient between mental health and poverty.

This idea, of identifying societal differences in how poverty translates into mental health, has key parallels to Durkheim’s study of suicide:

• We work with averages, rather than individuals. This is Durkheim’s first fundamental insight: “Since suicide is the act of an individual that concerns only that individual, it appears that it should depend entirely on individual factors and consequently pertain solely to psychology.(…) Among individual conditions, there are surely many that are not general enough to affect the ratio of the total number of voluntary deaths to the population(…) Just as these do not relate to a particular state of social organization, so they do not have any social repercussions. As a result, they concern the psychologist, not the sociologist. What the latter investigates are causes through which one may act not on individuals in isolation, but on the group.” (I added the underline)

• We work with variation in this average. This is Durkheim’s second fundamental insight; the societal causes of suicide can be determined by examining differences across time and space in the suicide-mortality rate at the level of the society. In our case, the variation is across space and differentiates among households who lived close to, versus far from the affected fault-line.

Modern empirical methods have also evolved. Durkheim analyzed his data by starting with variation in suicide rates (the “Y” variable) and inferred the social forces that might explain it (the “X” variables). Empirical economics reverses this sequence. We start with variation in exposure to the earthquake (the “X” variable) and then ask whether mental health, our “Y” variable, changes with that exposure. We assess, and argue against, any systematic pre-earthquake differences between more and less affected communities that could have directly led to a mental health-poverty gradient in more affected communities even in the absence of the earthquake.

Once the method and analysis allow us to model the earthquake-shock as exogenous to (pre-earthquake) mental health, the results are straightforward.

Figure 2 tallies the specific questions in our survey and the responses against the overall score. These questions were all “well-behaved” in the sense that each question shows lower frequencies for those whose overall mental health score is higher (and therefore better) in our survey. The histogram shows the overall distribution; for instance, those below the score of -2 are thought to be in potential severe distress, and, for most of the survey questions, they indicate extreme distress.

Figure 3 then plots the relationship between mental health scores and the consumption measure of poverty, along with the histogram of consumption.

There are three noteworthy patterns in this figure. The first is the lack of a poverty-mental health gradient among households far from the fault-line, as shown by the solid black line. This exactly mimics my previous finding—in general populations, the relationship is zero or weak.

The second, is that in villages that are close to the fault-line, we do not see sustained mental health deficits. In fact, mental health scores are higher (better) than villages farther away. The finding that there is no lasting effect of the earthquake on mental health is different from what has been found before—both after the Armenian earthquake of XXXX and the Tsunami of 2004, mental health has remained a concern for many years after.

The third is that a poverty-mental health gradient has emerged—the rich are in better mental health than the poor in the villages that were most affected by the earthquake. The paper rules-out several possible explanations for why the mental health-poverty gradient emerged in villages worst hit by the earthquake. One issue in particular is that the consumption metric of poverty is measured after the earthquake, and the gradient may reflect the fact that households with poor mental health fell into poverty, perhaps because they were unable to access aid. Our detailed work on the receipt of aid does not find such adverse targeting, and we conclude that, taken together, exposure to the earthquake altered the relationship between economic position and mental health, producing a gradient where none is visible in less-affected communities.

Even so, the coefficient is quantitatively small and it would require, again, a massive impact of better mental health on income to generate a poverty trap. As we highlight: “Our results provide a starting point to think about why there is heterogeneity in the poverty-mental health gradient across populations, but our estimated elasticity still falls short of the coefficients that we would need to generate poverty traps in its classic form.”

The idea that we should take heterogeneity across societies in the mental health-poverty gradient is interesting and perhaps valuable, and the methods proposed in my paper with Andrabi and Daniels provide a structured way to take this idea to other data. But thus far, the project does not yield an immediate path for incorporating economics into mental health policy through the entry point of the poverty-mental health gradient.

References

Andrabi, T, B Daniels, and J Das (2023), “Human capital accumulation and disasters: evidence from the Pakistan earthquake of 2005,” Journal of Human Resources, 58(4): 1057–1096.

Andrabi, T, B Daniels, and J Das (2026), “The emergence of the poverty-mental health gradient: evidence from the great Pakistan earthquake of 2005,” Journal of Development Economics, 182: 103795.

Durkheim, Emile, and John A. Spaulding. “Suicide: A study in sociology.” New York (1951).

And two top-notch studies on disasters and mental health (the second, unfortunately, I uncovered only after publishing our paper...)

Frankenberg, E, J Friedman, T Gillespie, N Ingwersen, R Pynoos, I U Rifai, and B Sikoki et al. (2008), “Mental health in Sumatra after the tsunami,” American Journal of Public Health, 98(9): 1671–1677.

Goenjian, A K, A M Steinberg, L M Najarian, L A Fairbanks, M Tashjian, and R S Pynoos (2000), “Prospective study of posttraumatic stress, anxiety, and depressive reactions after earthquake and political violence,” American Journal of Psychiatry, 157(6): 911–916.

1. Education (and more broadly, human capital) is a big part of development.

2. Unfortunately, low- and middle-income countries are way behind richer countries in educating their citizens.

3. The reason why this is so is because of ____ and the way to fix it is ____. (Options for the first blank include “apathetic governments/elite control,” “low demand,” “lack of know-how”, and for the second “find champions,” “conditional cash transfers,” and “impact evaluations.”

Recent analysis recognizes that the deficits in LMIC were the result of decades of deliberate neglect and destruction from colonial governments, and that the real surprise is not that education levels are low, but just how fast they have bounced back. India constructed a new college every 8 hours for more than a decade from the early 2000s; tertiary enrollment among the youth in LMIC is hitting 40% in many countries; Vietnam has some of the highest reading and Math scores in its schools anywhere in the world; Gen Z women in rural Pakistan have 8 years more education than their parents....the list goes on.

But what I had never realized till I read Jonathan Kozol’s “Illiterate America” was that even as Western countries were being held up as models to emulate, education outcomes for many people in America--usually the poor--were dismal, at least in 1985. I had always imagined that 45 years later, things would be much better.

Recently, I teamed up with Yash Dhuldhoya at Gui2de and our amazing former student, Ethan Sager to see how things looked now, not only in the U.S., but in multiple countries in the OECD (mostly Europe and North America, but also Chile, Japan and Korea). With tons of assistance from the fantastic editors at SciencePolitics (thanks Emily Mendenhall, Diana Rayes and Anay Shah among others!) our piece looking at adult literacy in these, mostly Western countries, is now out.

Here is the shorter summary in SciencePolitics: https://sciencepolitics.org/2026/08/21/adult-literacy-is-mysteriously-declining-across-many-high-income-countries/

Here is the longer piece on Digital Georgetown: https://repository.digital.georgetown.edu/handle/10822/1111939

And here is a summary.

Today, 57 million American adults are functionally illiterate. That number represents 28% of the adult population, a significant surge from 18% just over a decade ago. Recent articles are talking about the decline in skills among the young and the lack of preparedness of undergraduates the U.S., but that is only the tip of the problem.

The key facts that we highlight (taken directly from the article and lightly edited!):

• Functional illiteracy is not a problem of a single country or a small group of countries; it is affecting many rich countries, from Chile to Japan. In Italy and Spain, 35 percent and 31 percent of adults, are functionally illiterate. In England, 18% of adults have difficulty processing and integrating instructions in English.

• Functional illiteracy is increasing. In the United States, the number of functionally illiterate adults increased from 35 million to 57 million, representing a 63% increase since 2012–2014. Across the 26 countries that participated in the 2012–17 and 2023 PIAAC cycles, the number of functionally illiterate adults increased from 111 million to 159 million, or from 18% to 25%. The number of functionally innumerate adults increased from 149 to 170 million over the same period. The figure below shows the change in mean literacy scores across multiple countries.

  

• Adult skills have declined across ages — often the most among older adults. This is a key point that is being missed in the articles on the performance of the young. In general, we are seeing larger declines among the older 55 to 65 year old population compared to the young. In Chile, close to three-quarters of the elderly are functionally illiterate; in Spain it is close to 50%.

• The root causes of these declines are heavily debated, and range from smartphones to Covid-19 to declining returns to cognitive skills in the workplace. But here is the problem: If these universal explanations are the main drivers of the declines, and they are immune to policy responses, we should expect very similar patterns around the world. But we do not--the patterns across different countries are very different. As we write:

“Why did literacy decline so sharply among young adults in New Zealand, while France largely protected the young but experienced large losses among older adults? Why were the losses among older Koreans so extreme?

Our sense is that focusing only on the U.S., as many studies have done, misses valuable opportunities to learn from a wider group of countries and potential explanations.

Our extended analysis shows, instead, that an age-period-cohort framework provides a useful way to organize these patterns. As people age, cognitive decline sets in. But it is also the case that different birth cohorts have had very different educational and labor market experiences, and these may affect their functional literacy scores. For instance, younger people in Chile are much more likely to have attended college than their elders. Finally, comparisons across rounds show that similarly aged adults performed worse in 2023, consistent with period effects.

These forces cannot be cleanly separated from two repeated cross-sections because age, survey year, and birth cohort are mechanically linked. But the framework disciplines explanations: Any proposed cause must fit the patterns across ages, birth cohorts, survey rounds, and countries.

Why these age, cohort, and period effects play out differently across countries is a key question we need to address; universal explanations in which smartphones or COVID-19 affect all groups and countries in the same way are insufficient.

Common shocks such as COVID-19 or smartphones may still matter, but only through mechanisms that produce very different effects across countries and age groups. Comparative research across multiple countries can help us understand which institutions and policies have either protected adult skills or accelerated these declines across different age groups.“

Bottomline: We do not know what the reason for these declines are, but understanding them better will require us to dive into the specific experience of different countries across entire age range, rather than focusing only on the U.S. and only on the young.

This research agenda is critical: With demographic change and the decline in literacy, we are looking at precipitous falls in the number of adults who can read, write and perform basic arithmetic operations in most Western Countries. A country like Chile may end up with fewer than a million adults who can read and write within our lifetimes.

As we write: “In a world where we desperately need young people to drive innovation and change and where the demographics of decreasing populations are now working against us, unless we can massively increase global human capital, our shared humanity is in trouble.”

Share

There are two excellent review articles, one by Ridley et al. and one by Haushofer and Fehr, that summarize the findings from these types of studies. Both reviews argue, going beyond correlations, that poverty causes poor mental health. For instance, Ridley et al. make the categorical statement that: “We now know that loss of income causes mental illness.”

I am less certain.

Below, I have copied Figure 4 from Ridley et al., where they have shown multiple studies conducted by economists (mostly after 2016) that have either given respondents cash or used a multifaceted anti-poverty intervention, both in the context of a randomized controlled trial. Their conclusion from this set of studies, based on what is called a `meta-analysis’ is that the pooled effect is positive. They write:

“The most compelling causal evidence that poverty causes mental illness comes from RCTs that evaluate antipoverty programs. Several studies that evaluated cash transfer and broader antipoverty programs have found substantial positive impacts on mental health, including over long time horizons, after the effects of any initial celebratory reactions among recipients have worn off. A meta-analysis of these studies is provided in Fig. 4. For example, cash transfers to Kenyan households worth $400 to $1500 at purchasing power parity (about 3 to 12 months of household income) increased consumption and happiness while reducing depression, stress, and worries (36, 37). Scores on a depression scale were 0.12 standard deviations (SD; closely related to Cohen’s d) lower 4 months after completion and 0.16 SD lower after almost 3 years, with larger transfers causing substantially larger effects.”

Yet, among the cash transfers, 11/16 had effects that were statistically insignificant. The numbers are slightly better for the larger multi-faceted anti-poverty programs, but even there, 4/10 studies were insignificant. Especially in the case of cash transfers, the statement that the pooled effect is positive is also simultaneously consistent with a statement that a majority of studies were unable to reject the null of no effect.

Even within the cash transfer programs, the pooled effect sizes themselves may be affected by the very large impacts of the program in Afghanistan led by Bedoya and colleagues in 2019. As Bedoya et al. note in their paper, 80% of the ultra-poor households in their study lived below the poverty line, per-capita consumption hovered at $30 per month, 90% were illiterate and 50% of men and women were depressed and traumatized. These are very difficult circumstances and this kind of extreme poverty and depression is, thankfully, extremely rare in the populations we usually work with. In the particular context of this very deprived population, a program that distributed almost a year’s worth of income was probably life changing in all kinds of ways. Perhaps it is not surprising that it also had a large impact on mental health.

The mean effect of 0.07 standard deviations could arise from small improvements for many people or from much larger improvements for a small group. Those possibilities have very different theoretical and policy implications. If most recipients experience only small gains, the resulting effect on income would have to be implausibly large to sustain a poverty trap. If a small group experiences life-changing gains, the average conceals precisely the heterogeneity we most need to understand.

Rather than an unequivocal assessment that “more income = better mental health” I took away a more qualified conclusion from these careful reviews. The average effect is positive but modest, with substantial heterogeneity and many studies finding effects that are statistically indistinguishable from zero.

But what causes this “substantial heterogeneity?” This is what I turn to next: I want to ask whether there are settings or environments where a link between mental health and poverty emerges. This is a question that takes us back to one of the first approaches to mental health and data, with Durkheim’s writing about the link between societal circumstances and suicide in 1897.

References

Bedoya, Guadalupe, Aidan Coville, Johannes Haushofer, Mohammad Isaqzadeh, and Jeremy P. Shapiro. No household left behind: Afghanistan targeting the ultra poor impact evaluation. No. w25981. National Bureau of Economic Research, 2019.

Durkheim, É. (2006). On Suicide. Translated by Robin Buss, with an introduction by Richard Sennett and notes by Alexander Riley. London: Penguin Books. [1, 2, 3, 4, 5]

Ridley, Matthew, Gautam Rao, Frank Schilbach, and Vikram Patel. “Poverty, depression, and anxiety: Causal evidence and mechanisms.” Science 370, no. 6522 (2020): eaay0214.

Haushofer, Johannes, and Ernst Fehr. “On the psychology of poverty.” science 344, no. 6186 (2014): 862-867.

Share

Public policy usually requires governments to spend more money. For an economist, this request raises two questions. First, are the benefits large enough to justify the expenditure, especially when public funds have many other morally compelling uses? Second, if the private benefits are so large, why are people not already undertaking the action themselves?

The first question asks about the returns to public expenditures. If the returns are monetary (“more income”) the comparison with expenditures is easier; you can compare how much was spent versus the monetary return to the policy.^[1] Economists have also developed ingenious ways to monetize intrinsically non-monetary returns, from time savings to greater happiness or better health. People are familiar with some of this literature under its many avatars, ranging from `cost-benefit analysis,’ and `cost-effectiveness’ to `marginal value of public funds,’ and `internal rates of returns.’

What people are less familiar with is that, typically, a favorable benefit-cost computation is not sufficient. Economists also typically look for an explanation for why, if the benefits exceed the cost, people were not undertaking the action themselves.

That is, we look for a market failure.

At first, the insistence on finding a market failure can seem puzzling. After all, if a program has a large benefit-cost ratio why do we need to understand whether there is market failure?

But if we trust that it is not driven by an ideological bias, the question of `where is the market failure’ is worth considering carefully. What is at stake here is that that the measure of improvement we have chosen to examine may not reflect a full accounting of welfare for the individuals we are concerned about. For instance, we may have measured that a drug will help someone with a mental health condition earn a higher income, thus yielding a positive benefit-cost ratio, but if the drug deadens the person’s range of emotional responses, they might reasonably choose not to take it. The law recognizes that right: there are very limited circumstances under which, ethically and legally, we are allowed to force the person to take the drug. With mental health conditions, where questions of sovereignty and informed choice become particularly tricky, interrogating why a seemingly positive intervention is not `taken-up’ is a reasonable ask.

It is in this role that poverty traps offer a powerful explanation for why individuals do not undertake an action by themselves even though the private benefits of the expenditure exceed its costs. In a poverty trap, small improvements are not self-sustaining. The vicious cycle linking poor mental health to poverty and poverty to poor mental health implies that people cannot escape the trap by improving their mental health “a little bit,” which then leads to a “little increase in income,” which then feeds back into a “little better mental health” in a virtuous cycle. The problem is that even if mental health improves a bit and raises incomes a bit, that small increase will fade out very quickly.

A sufficiently large intervention will take people out of the trap to a higher level of income and better mental health but people cannot undertake these actions themselves because the small interventions that are affordable to the poor cannot `sum up’ to realize the larger returns. We need the government to step in with a big push.

It is within this framework of poverty traps that my previous posts highlighted the problem that I ran into when I tried to look at the link between mental health and poverty. Across five countries, with Do, Friedman, McKenzie and Scott, I found no evidence of a correlation between mental health and poverty.

There were correlations with broader concepts of deprivation, such as low levels of education, but the correlation between mental health and a monetary concept of poverty remains critical. Why? Because I can link income today to income tomorrow to generate a poverty trap, but I cannot link `education today to education tomorrow,’ both because we think that education is fixed over the short-term for an adult and because we do not have well-developed ideas of how people may lose education.

The poverty-trap account therefore rested on an empirical relationship that, in our five-country data, appeared not to exist.

^[1] Understanding `how much is spent’ is not at all easy; some of the substantial complications comes from the fact that a government policy may induce further private expenditures and all of that spending needs to be accounted for. This is primarily a question of data and accounting—difficult, but conceptually straightforward.

Subscribe now

During the pandemic, I would drop by George's house every now and then for a walk. One day, George looked noticeably tired. Was he OK, I asked. "Oh, I am fine, its just the teaching." I ask why he was teaching so much and he says "No, no, its not the volume, its just that I teach a class at midnight." It *was* Covid, but this was going too far; I asked why his class had been scheduled at midnight, and he says "No, no, it's not scheduled at midnight. But, I have 3 students in China who did not get a visa (the consular offices had been shut down) so I am teaching the class for them at midnight." That's right: he was giving the same 2.5 hour lecture twice, every time.

When a Nobel Prize winner is up at midnight teaching a class for 3 students he has never met and he has not told a single person, every time you feel a complaint coming on, you just keep very, very quiet.

My second story.

In our swanky new building, offices were going to be allocated by seniority and I was pretty high on the list! That is, until someone decided we should form a "cluster" of senior and junior faculty who all get offices in the same vicinity.

Still fine. By the time we got to our cluster, some of those lovely corner offices overlooking the Capitol were still available. But George came before me.

And George chose Room 778.

You have to understand: Room 778 is fine, but it a nondescript office in a row of nondescriptive offices. If a Nobel Prize winner chooses a standard office before you, it is *very very* difficult to then choose a corner office for yourself.

With little option, I chose 772; James, who came after me chose 774. And that is how we ended up in a cluster with delighted junior faculty sitting in corner offices while we were staring at the wall.

This is the George I came to know, the George who showed up at every faculty meeting; the George who read all his colleagues' papers; the George who had time for the most ill-formed ideas; the George who would spend hours teaching you how to think; the George who showed us all how to become a faculty that brings dignity and grace to every engagement.

George did not want a farewell, or gathering.

But we can all raise a toast to the retirement of this remarkable person.

Thank you, George.

This pattern is much broader than our five-country study, and psychologists typically find that for `internalizing’ disorders such as depression and anxiety, prevalence is higher among women. If you are looking for a cite, I particularly liked the paper by Seedat et al. (2009), which they published in the Archives of General Psychiatry and you can find in the references.

We wanted to understand why this pattern arises, and we had the following idea: In the ISERDD sample from Delhi, which I have talked about previously, we interviewed both men and women in the same household. Because we had interviews from one adult man and one adult woman in most households, we could compare gender differences within households and ask which household-level experiences predict a larger female-male gap in distress.

Share

Our idea was that if we could understand what was `special’ about these households, we would make some progress—whatever stressor emerged would be a candidate for something that either affected women more, or was expressed by women in ways better captured by the instrument. (There is a technical point here: Statistically, you can use a household fixed-effect and a household-level stressor to identify the difference between men and women in the same household, although you cannot identify the impact of the stressor on men or women separately)

The story we eventually found can be told quite simply in two pictures. The first picture shows the Global Symptom Index or GSI—the measure of mental distress in the SCL-90-R, which is the 90-question survey we used to screen for mental distress—and how it varies with age for men and women. As we wrote:

“For women, the relationship between GSI and age is an inverse U-shape, increasing till age 40 followed by a slight decline; for men, GSI decreases sharply till age 35 and then increases equally sharply. Increasing GSI among women through the reproductive cycle indicates a link between adverse reproductive outcomes and mental health. Equally, that men do not follow the same pattern. suggests that over the reproductive cycle of the family, processes that affect the mental health status of men and women may be very different.

For each woman, we measured adverse reproductive events as the share of pregnancies that did not result in a child who was alive at the time of the survey. In the ISERDD sample, adverse reproductive events were common among both poor and rich women: 27 percent of pregnancies did not result in a living child in both groups, although the reasons were different. Among the poor, women reported 4.95 pregnancies and 3.45 living children, with the difference evenly spread among children who had died (0.51), miscarriages (0.43) and abortions and stillbirths (0.55). Among the rich, pregnancies dropped to 3.79 and living children, to 2.66. The difference was accounted for by abortions and stillbirths (0.53), miscarriages (0.34) and children who died (0.27).

You can see how adverse reproductive events affects mental health for men and women separately in the figure below. Here, we showed the GSI score for men and women separately, divided by households in which there were very few adverse reproductive events (0-20%), some adverse reproductive events (21-40%) and many adverse reproductive events (41-100%). Since older women will have more pregnancies, the easiest way to account for age was just to bin the ages and show two different age-groups.

The result is quite striking. For men, there is no difference in the GSI score by the level of adversity in reproductive events. For women, the GSI scores are much higher in the households where adverse reproductive events are also higher. We confirmed all of this in a regression framework so that we could correctly account for age and other covariates, and the results were perhaps even more striking. In this specification, once we included adverse reproductive events, the estimated gender difference essentially disappeared (the coefficient was +0.15 with a standard error of 0.028 in a simple regression and it becomes -.017 with a standard error of 0.047 in a specification with household fixed-effects and an additional control for adverse reproductive events—the coefficient on the latter was 0.541 for women relative to men, with a standard error of 0.134). In other words, once we compared men and women within the same household and allowed adverse reproductive events to matter differently by gender, the average female–male gap was no longer visible as a standalone gender coefficient. The remaining gender difference was concentrated in households with more adverse reproductive histories. This was not necessarily a causal result. Adverse reproductive events could have captured other household stressors that affected mental health more for women than men; abuse in the household is one such confounder. Nevertheless, it pointed us toward a domain of social life where reproductive histories, gender, and distress seemed to be intertwined.

Around this time, I had just returned from Madhya Pradesh and one searing memory from my fieldwork was showing up at a PHC sometime in the morning (around 10?). My notes on that visit say:

“The PHC is a low-lying brick building, single floor and perhaps two rooms and there is a motorcycle outside the PHC. There is a young man sitting on the motorcycle and he has obviously been crying.

A little while later he is joined by a young woman, and she is carrying a small body, their child, wrapped in a white shroud. I asked what happened. They had come for the birth of their baby. When the baby was born, he was not breathing and was blue in the face. Panicked, they kept asking the doctor to do something, but he just says that they should take the baby to the bigger hospital. Again and again, the husband repeats “Usne kuch parwah hi nahin kiya, usne kuch parwah hi nahin kiya.” The wife cannot speak, the trauma is too recent, and the cloudy sky is about to break open. They get on the motorcycle, a 100 CC Hero Honda, she sits side-saddle, and she is holding the baby in her arms. They will drive back now.”

That memory has always made the quantitative result difficult for me to interpret in a simple way. I understand a literature that shows that children’s well-being affects women more than men, but I have also seen men run through the night carrying a child sick with diarrhea, seen them devastated when a child is not getting better and I have seen children run to their fathers as they come back to the village from the fields. There may be households where husbands remain largely unaffected when a child dies. But I found it very difficult to understand how it could be that the deaths of multiple children would leave no trace in men’s mental distress.

Here is how Veena Das, the anthropologist in our group, summarized what we learnt from the ethnographic interviews with men and women.

“Ethnographic interviews with men and women suggest, first, that the cultural expressions used by women and men for communicating emotions of sadness, anxiety and psycho-somatic distress are particular to the local worlds in which these adverse events occur. We find considerable variation in the manner in which women and men portray the sense of loss. Women take greater recourse to reporting somatic symptoms as well as expressed emotion while men use languages of flight or of the cultivation of detachment.”

But it may be better to simply read her own words.

How do we explain the fact that adverse reproductive outcomes in the same families affect upon women and men in different ways? The feeling of grief at the death of a child seemed to be expressed in a much more intensified form in those cases by women where the narrative linked it to other events such as a husband’s drunkenness or the abandonment by the natal family. Rodrigues, Patel, Jaswal, and de Souza (2003) have suggested that women who received very low levels of physical help and moral support after childbirth were more likely to suffer from post natal depression than others. This is why we suggest that it was the aggregation of adverse conditions, including cumulative reproductive failures that affected women most deeply. In other cases the death of a small infant could be mentioned briefly or sometimes as something that was a “normal” risk of childbirth though we should be cautious in interpreting this form of talk as evidence that women did not grieve such losses.

The death of children figured in the narratives of men but they did not use emotional expressions suggesting tearfulness or sadness. Instead, the rhetoric expression of distress was in terms of a language of flight. For instance, a Nepali unskilled worker who had lived in Delhi for the last thirty years described the reasons for migrating to Delhi from his village in the following terms: “Two of my eldest children died, one after another. There was no hospital, no doctor. People said that some jealous relative or neighbor had done something (a euphemism to refer to magic or sorcery). So I said to myself—run, Bahadur, run—run from here. First I came away without telling my parents—then I got one of my cousins to bring my wife here.”

In another case an upper middle-class man was describing the loss of his two sons soon after their birth. Though he had one surviving son, he felt that he could not bring himself to love anything in his home any more. He cited examples from the Bhagvad Gita, the religious text that expounds on the virtues of detachment. He also cited examples from Hindu mythology in which famous heroes had laid down their weapons when faced with the death of a son in battle, losing all interest in the game of honor and heroic fame. He seemed to imply that he too had metaphorically laid down his weapons. His wife, in her own interview, commented on the saintly nature of her husband but did not express any criticism of the manner in which he seemed to have withdrawn all affect from the family. He was a dutiful husband and father, she said, simply not given to expressing his love. While such complex emotions are not easily captured in any survey, they point to a different way of expressing grief and should be treated as integral to the social lives of the respondents (Kleinman & Byron, 1985).”

If you read the paper, you will find two case-studies of Chanchal and Lakshmi. I still find them both difficult to read, but always necessary to understand how mental distress plays out in complex ways within the milieu of the slums in Delhi.

Reading the paper again after so many years, I had the following thoughts:

1. This pattern, that women tend to report more internalizing mental disorders and men more externalizing mental disorders, such as substance abuse, is now quite well established. It does raise the question of whether the questionnaires that we use are able to capture distress equally across men and women.

2. With issues related to fertility and adverse reproductive events, I find it difficult to make sense of both the cross-section and the changes over a longer period. Fertility has declined so much, as has child mortality, that if the results in our paper were correct, we should expect a narrowing of the female penalty over time. The paper by Seedat and others does suggest that this is the case, but I am not sure that in the Delhi slums and neighborhoods, the female penalty has disappeared.

3. I still find the paper valuable because the quantitative and ethnographic evidence speak to each other. I think of papers as building a narrative; unravelling a story with the reader. And here, the narratives prevent us from flattening that pattern into a simple claim that women grieve and men do not. What they suggest instead is that grief enters social life through different routes: through the body, through blame, through abandonment, through withdrawal, through flight, through silence.

References

Seedat S, Scott KM, Angermeyer MC, et al. Cross-National Associations Between Gender and Mental Disorders in the World Health Organization World Mental Health Surveys. Arch Gen Psychiatry. 2009;66(7):785–795. doi:10.1001/archgenpsychiatry.2009.36

Das, Jishnu, Ranendra Kumar Das, and Veena Das. “The mental health gender-gap in urban India: patterns and narratives.” Social Science & Medicine 75, no. 9 (2012): 1660-1672.

Das, Jishnu, Quy-Toan Do, Jed Friedman, David McKenzie, and Kinnon Scott. “Mental health and poverty in developing countries: Revisiting the relationship.” Social science & medicine 65, no. 3 (2007): 467-480.

When I first started thinking about mental health as an economist, poverty seemed like the natural point of entry. Both an efficiency rationale — poverty traps — and an equity rationale — the possibility that mental distress is concentrated among the poor — suggested a clear role for economic analysis.

The story turned out to be much more complicated although the central message is simple.

I teamed up with Quy-Toan Do, Jed Friedman, David McKenzie and Kinnon Scott, all of whom worked with me at the World Bank’s research department and had completed community surveys of mental health. Two sentences from the abstract lay out our findings:

First, the study:

We revisit the relationship between poverty and mental health using data from nationally representative household surveys in Bosnia and Herzegovina, Indonesia and Mexico, along with special surveys from India and Tonga.

Then, the result:

“(…) there is no consistent association between consumption poverty and mental health – in two countries mental health measures are marginally worse for the poor; in two countries there is no association; and in one country mental health measures are better for the poor compared to the non-poor. While the lack of an association between consumption poverty and mental health implies that poor mental health is not a ‘‘disease of affluence,” neither is it a disease of poverty.”

Here is the picture.

The article was not well received by public health folk working on mental health, and we responded to a subsequent comment in the journal; you can follow these in the references. I want to highlight one theme from the comment and our response that is important and commonly missed in the policy literature.

One critique in the comment was that we focused on poverty (and other X’s), but instead, we should have used an index that combined multiple measures. The particular suggestion was to use the Human Development Index.

An index that combines several variables is not a neutral summary. It is a weighted average, and the weights embody assumptions. There are certainly cases where indices are useful: for example, when several observed measures are noisy proxies for the same underlying latent variable, and aggregation improves precision. Test scores are weighted indices of responses to individual items; asset indices combine ownership of multiple assets, often using principal components analysis. But each of these choices comes with assumptions.

The problem arises when the constituent variables are available separately and we nevertheless collapse them into a single index. Suppose, for simplicity, that I combine income and education into an index: SES = I + E. This index assumes that income and education receive equal weight and are perfect substitutes. One additional unit of income can offset one fewer unit of education, leaving SES unchanged.

If I am interested in the relationship between mental health, income, and education, I can estimate these associations separately: MH = α + β₁I + β₂E + ε. Or I can regress mental health on the SES index: MH = α + γSES + ε. Further, I can move between my `separate’ regression and my `index’ regression by opening up the SES formula and writing: MH = γ (I + E) = γI + γE.

It is now clear that the separate regression and the index regression are the same if and only if β₁ = β₂ (=γ). The index regression is a restricted regression where we are forcing I and E to act as perfect substitutes in their association with mental health. It is never clear to me why we should force that restriction when we have the underlying data to estimate a and b separately, and this is what we wrote:

Second, Corrigall et al. (2008) claim we use a narrow metric of welfare. In fact we use multiple measures which are not aggregated into one scalar value. There is no statistical or theoretical justification for using indices, such as the Human Development Index (HDI) advocated by Corrigall et al that combine the multiple dimensions of welfare into a single scalar quantity. These indicators are linear combinations of the underlying variables—in the case of the HDI with equal weights on life-expectancy, income and education. That is, the HDI assumes that increasing life-expectancy by (log) 1 has an equal impact on welfare as increasing education by one year (!). When the constituent parts of the index (consumption, health, and education in this case) are available to the researcher, using measures such as the HDI instead of the separate components of the index imposes an added restriction on the (unrestricted) specification—specifically, that the coefficients on consumption, education and health are equal. This is clearly rejected in our case.

I conclude with two final thoughts.

The first is about why the poverty measure matters. One possible response to our findings is: why focus so much on consumption poverty? Why not use a broader notion of deprivation — education, assets, housing, social exclusion, or some composite index? My answer is that broader deprivation certainly matters. If mental health is consistently worse among those with less education, or worse housing, or greater social exclusion, that is an important fact that could generate a certain type of policy response. But the specific relationship between mental health and money-metric measures of deprivation generates prioritizing mental health spending as a policy in the following specific manner:

In the “poverty trap” diagram, income today is on the X-axis and income tomorrow on the Y-axis—this is the fundamental dynamic of interest, showing that people can fall into a vicious cycle or trap. Money allows people to buy healthcare. A poverty trap implies that spending today on mental health can generate more than sufficient resources tomorrow to cover the initial spend, if it gets people out of the cycle. Now think about education. Across generations, you could have “education today” and “education tomorrow”, but for an adult, we really don’t think about education in that dynamic sense. If mental health is associated with education, that may be an important descriptive fact, and it may strengthen the case for investing in education. But it does not provide the same rationale for investing in mental health. For that argument, the relevant link is between mental health and money-metric deprivation: the possibility that poor mental health reduces income today and tomorrow, trapping people in a cycle that public spending could help break.

The second is that, although our study showed no systematic association between mental health and poverty, we consistently found that all kinds of changes in circumstances, from sickness to economic conditions, adversely affected mental health. We wrote:

“Changes in life circumstances brought on, for instance, by illness may have a greater impact on mental health than levels of poverty. Effective public health policy for mental health should focus on protecting individuals and households from adverse events and on targeted interventions following such adverse changes.”

This idea drove my further work on mental health. I will write two more pieces to complete the series: one on gender and mental health, and one on the four-year impact of a large earthquake in Northern Pakistan.

Thanks again for reading—and for your continued thoughts!

References:

Das, Jishnu, Quy-Toan Do, Jed Friedman, David McKenzie, and Kinnon Scott. “Mental health and poverty in developing countries: Revisiting the relationship.” Social science & medicine 65, no. 3 (2007): 467-480.

Das, Jishnu, Quy-Toan Do, Jed Friedman, and David McKenzie. “Mental health patterns and consequences: results from survey data in five developing countries.” The World Bank Economic Review 23, no. 1 (2009): 31-55.

Das, Jishnu, Quy-Toan Do, Jed Friedman, David McKenzie, and Kinnon Scott. “Revisiting the relationship between mental health and poverty in developing countries: a response to Corrigall.” Social Science & Medicine 66, no. 9 (2008): 2064-2066.

Lund, Crick, Mary De Silva, Sophie Plagerson, Sara Cooper, Dan Chisholm, Jishnu Das, Martin Knapp, and Vikram Patel. “Poverty and mental disorders: breaking the cycle in low-income and middle-income countries.” The lancet 378, no. 9801 (2011): 1502-1514.

Corrigall and others published a response to our paper:

Corrigall, J., Lund, C., Patel, V., Plagerson, S., & Funk, M. K. (2008). Poverty and mental illness: fact or fiction? A commentary on Das, Do, Friedman, McKenzie & Scott (65:3, 2007, 467e480).Social Science & Medicine, 66(9), 2061e2063.

They then published a follow-up systematic review, where they reported very similar findings to ours. Varied and consistent impacts with broader measures of poverty, but no consistent association between poverty and mental health.

Lund, Crick, Alison Breen, Alan J. Flisher, Ritsuko Kakuma, Joanne Corrigall, John A. Joska, Leslie Swartz, and Vikram Patel. “Poverty and common mental disorders in low and middle income countries: A systematic review.” Social science & medicine 71, no. 3 (2010): 517-528.

At the time they introduced this module, mental health, especially in low-income countries, was poorly studied. Vikram Patel had done influential work from his clinical setting in Goa and started to publicize the importance of mental health in low-income countries. Renu Addlakha had worked on the indigenization of psychiatry in India. However, community studies, which the ISERDD sample would provide, were few and far between. A canonical study by Patel and Kleinman in the WHO Bulletin found only 6 studies that they could include in a review of poverty and mental health. Further, as they reported, they could not really look at income measures of poverty because:

“The studies reviewed do not permit an analysis of correlation of absolute levels of income and common mental disorders, because average indicators of income for countries do not indicate the true levels in the populations being studied.”

Veena and Ranen Das introduced two key innovations in the mental health component.

First, they worked with a module of 80 questions, known as the Symptom Checklist-90 or SCL- 90-R, developed by the psychiatrist Leonard Derogatis. Over several months, they worked with ISERDD staff to translate, pilot, and validate the survey in the Indian context.

Second, they had the brilliant idea of comparing responses on the questionnaire against outpatients receiving care at a psychiatric clinic. This type of construct validity, called criterion validity, allowed them to assess the performance of the survey in a very different context from where it had been designed. Their argument was that if the questionnaire was actually capturing mental distress, it should be the case that on average, outpatients would score higher on the scale compared to the average population. But it is also the case that outpatients were being treated. Therefore, if there were untreated people with severe mental disorders in the community sample, we may actually expect the tail of the survey sample to score even higher than the average score among outpatients.

ISERDD implemented the survey over a six-month period in 2003. That process itself was eye-opening. Along with the ISERDD staff, we had all been visiting these families for two years and had developed a strong rapport and connection with many of the family members. Several would welcome the surveyors as part of the family and wait for the time of the week when they would come to talk about how their week had been.

Despite the great familiarity that we had developed, the mental health questionnaires seemed completely new. These surveys took an average of 45 minutes to one hour to complete, and the longest survey lasted 4 hours. People we spoke to would sometimes cry, and even through the tears, the dominant response we received was that nobody had ever asked them how they felt, or how their lives were going. For many of the ISERDD staff, it felt like a dam had broken as the individuals they talked to spoke about their past tragedies, their anxieties and the various indignities of daily life. I realize that there was a possibility of triggering past trauma with this questionnaire, but overall, it seemed that rather than reigniting past trauma, just the act of completing the surveys may have brought a measure of relief to many households in our study. I do not know whether the great familiarity that we had developed over the 2 years of previous work helped in this regard; I invite Veena to remark on that in the comments (if she has the time…).

The strategy of comparing mental health scores with outpatients worked very well. I have copied the figure from one of our papers below, where we showed the (smoothed) histograms of the mental health index, where higher scores denote greater distress. As is clear in the figure, the data aligned quite closely to the hypothesis that Veena and Ranen Das had posited: Average scores were higher among patients, but there was a long tail of severe distress, presumably untreated mental disorders among respondents in the household survey.

The patterns in these data also mimicked what had been found in previous surveys. Women reported higher mental distress than men, an issue I discuss in a future write-up. More educated people reported lower mental distress. Both in terms of construct validity and the basic correlations lining up with what had already been reported in the literature, it looked like the survey worked well.

But when I started analyzing the relationship between consumption expenditures, which are the standard money-metric measure of poverty in low-income countries, and mental health, I ran into a problem: There wasn’t any. The lack of any relationship between poverty and mental health was surprising to me, and led me to doubt whether we had actually captured the right construct, or whether it was even possible to capture mental distress accurately in community settings with staff who were very well trained, but not professional psychologists or psychiatrists.

I think that researchers implementing mental health modules today have a lot more information and studies to work with. At the time we were trying to do this, there was not that much material to work with. You must also remember that as an economist, a constant critique I faced was that questionnaires such as these are self-reported and therefore not entirely believable. We know today that, like with pain, the difference between what people think and what the body experiences is not that sharp. But at that time, and perhaps even today, economists continue to make a sharp distinction between self-reports and observed behavior.

These issues created considerable doubt about whether I was proceeding in the right direction. My solution was to try and work with others doing similar community surveys to see if the results we were finding were an anomaly. Something that opened up this possibility of combining results from multiple studies was the factor structure in the ISERDD data: It seemed that while the 80-question module added a lot of information on 9 different domains, 80 percent of the variation was captured solely by the anxiety and depression domains. These are similar to the domains covered in shorter 10 to 12 question surveys, such as the GHQ 12 (which focuses on general psychological distress) or the PSS-10 (which focuses on perceived stress), which is what other nascent studies were relying on.

Ultimately, I was fortunate to find a stellar group of researchers at the Development Research Group within The World Bank (where I was at that time) to work with. I discuss this work with Quy-Toan Do, Jed Friedman, David McKenzie and Kinnon Scott in my next entry. Thanks for reading through!

References

Addlakha, R., 2010. ‘Indigenisation’ not ‘Indianisation’ of psychiatry: An anthropological perspective. Sociological Bulletin, 59(1), pp. 46-68.

Patel, Vikram, and Arthur Kleinman. “Poverty and common mental disorders in developing countries.” Bulletin of the World Health Organization 81 (2003): 609-615.

Patel, Vikram, Jerson Pereira, Livia Countinho, Romaldina Fernandes, John Fernandes, and Antony Mann. “Poverty, psychological disorder and disability in primary care attenders in Goa, India.” The British Journal of Psychiatry 172, no. 6 (1998): 533-536.

Derogatis, Leonard R., and Kathryn L. Savitz. “The SCL-90-R, Brief Symptom Inventory, and Matching Clinical Rating Scales.” (1999).

I always associated mental disorders with treatment that required some degree of individualized attention with a commitment to therapy and a serious engagement. Therapists need to work closely with their patients, both to unravel the sources of distress and its potential remedies. Indeed, from Sigmund Freud to Oliver Sacks, the idiographic approach to mental health has been a mainstay of psychiatry and a key lesson from the more recent literature on trauma maintains an unrelenting focus on uncovering the source of distress and a continued openness to experimenting with treatments. Even in this age of chemicalized depression, any one particular drug, any one particular method, may work for some but not for others.

Our methods as economists are, however, unsuitable for this task of discovering the particularities of causes and treatments. At the end (and the beginning), we need to aggregate individuals in some way to make progress, whether in our theories or in our empirical applications.

My attempts to do so focus on the relationship between poverty and mental health in South Asia. This choice, in part, reflects a frame chosen by public health researchers arguing for greater attention to mental health in low-income countries. But it also reflects two key issues that are of great importance for economists.

The first is the idea of poverty traps. Scholars such as Haushofer and Salicath, who have worked extensively on behavioral economics, believe that mental distress, including common mental disorders may lead to situations where people are unable to escape poverty. This is because if people are depressed, they make poor decisions that worsen their financial position. But if a poor financial position further exacerbates their depression, they are stuck in a vicious cycle from which they cannot escape.

Certain parts of this hypothesis have been carefully tested; for instance, researchers have shown that financial decision making becomes worse under conditions of scarcity. I must stress that this research is not equivalent to the idea that `poor people are responsible for their poverty.’ Instead, the research shows that you can take anyone, put them under conditions of financial stress, and they will make poor decisions. That distinction is very important: Poor people remain poor, not because of their character, but because poverty creates the conditions under which decision-making acumen declines.

Yet, for a poverty trap to occur, a key technical requirement needs to be satisfied, as first demonstrated by Dasgupta and Ray. The canonical Figure 1 below clarifies this requirement. Suppose I plot your income today versus your income tomorrow. Multiple dynamics are possible—you may, for example, invest some of money today and become richer tomorrow. Or, you might have too little money, in which case you cannot eat enough and this lowers your income tomorrow.

It is reasonable to assume that there are two points N and Q that must lie on this curve. N is a “subsistence equilibrium”. If you are at N, you remain at N because your income today equals your income tomorrow (it intersects the 45-degree line). If you are Q, you remain at Q. You are rich, but presumably your lifestyle demands expenditures that keep your wealth constant. A poverty trap then occurs if, starting from N, small increases in income do not take you back to Q—they take you back to N. But a sufficiently large injection of income can propel you upwards.

Dasgupta and Ray showed that for a poverty trap to occur, it must be that the curve connecting N and Q must look like the one below, and showed how such a curve could arise. The key point is that there is one unstable equilibrium, P. If your income today is below P, the dynamics take you back to N: You can see that by tracking the income path through the arrows. But if you start above P, you will eventually converge to Q. The technical requirement then is that at some point in its journey, the curve mapping future to current income must cut the 45-degree line from below.

Subramanian and Deaton tested the empirical counterpart to statement, which is that the product of the elasticity of income to mental health and of mental health to income must exceed 1 at some point. In other words, for a poverty trap to emerge, we must have “sufficiently strong” relationships both from poor mental health to reduced income and from reduced income to poor mental health. In a world of mental health poverty traps, people will find it difficult to lift themselves out of poverty because incremental improvements are immediately washed away.

But poverty traps are not the only reason why economists are interested in the links between poverty and mental health. The second reason to study the link between poverty and mental health follows from an equity rationale, potentially to disprove the idea that mental health disorders affect only the rich. To the extent that more public spending on healthcare should go to the poor compared to the rich, especially in situations where the rich may be able to fund their own care, focusing government efforts on those conditions that affect the poor is likely to be more progressive.

Even here, showing that mental disorders are not just a condition of affluence is insufficient. An equity focus in public health suggests that we should focus on the relative rather than absolute incidence of different conditions. The poor, for instance, are 3x to 5x more likely to be affected by Tuberculosis in Indian than the rich. If the poor are equally likely to be affected by mental disorders as the rich, taking money away from Tuberculosis and spending it on mental health will decrease the progressivity of public health spending, not increase it. Of course, that conclusion must be modified, if, for instance, the efficacy of public health spending on mental health is much higher than that for Tuberculosis, but in all these cases, understanding the burden of mental health disorders across the income distribution can help adjudicate spending decisions among equally morally compelling causes.

These two observations provided me with one starting point for my investigations. If I could look at the link between poverty and mental health, it could unlock important ways for me to bring tools from economics to the discussion.

References

Dasgupta, P., & Ray, D. (1986). Inequality as a determinant of malnutrition and unemployment: Theory. The Economic Journal, 96(384), 1011-1034.

Haushofer, Johannes, and Daniel Salicath. “The psychology of poverty: where do we stand?.” Social Philosophy and Policy 40, no. 1 (2023): 150-184.

Mani, Anandi, Sendhil Mullainathan, Eldar Shafir, and Jiaying Zhao. “Poverty impedes cognitive function.” science 341, no. 6149 (2013): 976-980.

Sacks, Oliver. 1985. The Man Who Mistook His Wife for a Hat and Other Clinical Tales. New York: Summit Books

Subramanian, S., & Deaton, A. (1996). The demand for food and calories. Journal of political economy, 104(1), 133-162.

Van der Kolk, Bessel. “The body keeps the score: Brain, mind, and body in the healing of trauma.” New York 3 (2014): 14-211.

1. Yes, the U.S. is looking really bad at 28% functionally illiterate. But 22.5% of Germans, 31% of Spaniards and a staggering 53.4% of Chileans are functionally illiterate. The Nordics are doing much better, but still Finland and Sweden are running at around 12%.

2. The U.S. has participated in 3 cycles--2012/14, 2017 and 2023--and the first and the third are 10 years apart. Conveniently, the age-splits are shown in 10 year bands, so (say) 25-34 year olds in 2013 will be 35-44 year olds in 2023, if net migration can be assumed away. I put together the picture for the U.S. and it is not good, with declines of 10-20 points in every 10 year block. For context, 225 is the upper end of PIACC Level 1, which is functionally illiterate and around 250 puts you squarely at PIACC Level 2, which is just about able to read and comprehend basic instructions. At this rate, by the time the current cohort hits their 40s, the average person will be close to functionally illiterate in the U.S.

Customarily I start my education talks describing the state of learning in low-income countries, where fewer than 10% of children graduate middle school knowing how to solve a fraction or read a paragraph.

Share

Lately, though, I start with facts about a country where 59 million adults cannot figure out when the library closes if they see a sign that says “The library will close on July 3rd at 5pm for independence day.” If you tell them how many calories a cookie has, they cannot figure out the total calories in a package of 3 cookies.

Then I tell them that, of course, I am talking about the United States, where 28% of adults are functionally illiterate. I tell them how debilitating it is to be an illiterate adult in a country that assumes literacy at every corner, and for every task.

I start with the U.S. because I am trying to correct three misperceptions.

1. First, that the U.S. and OECD countries are doing much better than low-income countries. Yes they are doing better, but no, they are not doing well. Whatever is happening in the U.S., it is leaving 28% of adults functionally illiterate. In a restaurant with a random selection of American adults, one in four cannot read the menu.

2. The second is that we can improve scores, but looking for advice and guidance from the OECD countries is not going to cut it. Look at the time trend for reading test scores of 13-year old children in the United States from 1971 to now--there has been no change at all. In fact, the parents of today's 13-year old children reported higher reading scores when they were tested at age 13 (probably around 1994). Whatever the U.S. (and many OECD countries) are doing, it is just not working.

3. The third is to, please, temper our tech enthusiasm. Between 1971 and 2022, the period of the graph, the cost of Floating Point Operations per Seconds (FLOPS--3.14+4.17 is a FLOP), a measure of computing power, fell by (roughly) 4.2 million x. If what is sold as a key ingredient in the learning recipe used to cost $4.2 million but now costs $1, and *learning does not improve*, that ingredient must be useless. Yet, even as the U.S. finally realizes that putting all kinds of tech in the classroom was like taking a forklift to the gym, low-income countries are placing all their bets on the latest App. And now with Einstein.ai, which will do all your homework, discussions etc. etc. we are bringing bulldozers into the classroom to lift weights.

Truly, this was the one time that Ghalib got it wrong; App ko chahiye KAI umr bhar asar hone tak. (Cue the groans; irresistible, but inexcusable. I apologize, but shall do it again).

On the anniversary of the stop-work order for USAID, it’s safe to say that aid for global health is not coming back in any substantial way. Although the U.S. may have been particularly cruel and unusually swift in its withdrawal, it is not alone. Countries around the world are pulling back from global health.

Share

In the short-run the loss of foreign aid is a tragedy. But in the longer run the reduction of foreign aid in global health also implies that countries now have greater freedom to determine their own trajectories.

As economist Angus Deaton has argued, foreign aid has often allowed leaders to access funds without acquiring the legitimacy of the local population necessary to raise taxes. This “aid Dutch disease” has permitted corrupt and authoritarian governments to continue long past their due dates.

In the longer run, the reduction of foreign aid in global health implies that countries now have greater freedom to determine their own trajectories.

With greater accountability to their local populations, governments now have to deliver health programs that local populations must be willing to fund. This shift implies that health programs need to better match the needs of the local population while at the same time working to improve access and quality with less money.

In the best of worlds delivering health programs that meet people’s expectations is a difficult task because healthcare is one of those curious cases where wants and efficiency often clash — people may want antibiotics but the efficiency of healthcare does not increase by giving antibiotics where they are not needed.

In an environment ridden by crises and a need for immediacy, the task of delivering tailored, affordable health programs becomes doubly difficult. At the minimum, it will require a fundamental change in the way that research and knowledge is generated in health, with far more attention to the local needs and context.

Rushing to action only risks entrenching one poorly functioning system of knowledge acquisition with another.

Unfortunately, rushing to action only risks entrenching one poorly functioning system of knowledge acquisition with another. Developing systems of knowledge and research that are locally grounded and sensitive to the local context requires us to take a breath, step back, and deliberate.

Ultimately, we must ask the question: Can foreign aid increase the production of locally relevant research and knowledge on health systems? And if so, how?

Invitation to a Discussion

To be honest, I do not know.

Some people believe the way forward is through the decolonization of knowledge and greater funding for researchers working in the countries where the research is produced. See, for instance, Seye Abimbola’s discussion of decolonization.

But initiatives like decolonization, mandated local authorship, and mandated funding to local researchers rarely invite inquiry into whether these initiatives lead to greater equity or better knowledge. Instead, they rely on a heavy dose of moral persuasion that must, to be effective, remain steadfast in its certitude.

Such moral persuasion leaves little room for ambiguity, little room for nuance. And, in this particular case, it divides the world into North and South, sweeping aside the complex collaborations that have emerged over two centuries and disregarding the wide differences within each.

I worry that the knowledge and research systems now being put into place are insufficient for the massive task that is at hand. Instead of spurring better, more locally relevant research, I am now concerned that these systems will end up cementing an elite cabal that controls all the funds and research.

My own impression is that moral persuasion rather than empirical argumentation has silenced many top researchers who have a lot to contribute to this debate. At this critical time, we cannot afford that silence and we must work hard to spur an honest discussion of how to produce locally relevant research and knowledge, especially as authoritarian tendencies slowly but surely gain ground in many of the countries we work in.

Instead of spurring better, more locally relevant research, these systems may end up cementing an elite cabal that controls all the funds and research.

In the spirit of starting that discussion (and obviously leading with my chin….) here are five topics with my own answers that will hopefully establish a starting point for greater debate about the role of foreign aid in the production of knowledge in global health.

Should foreign organizations mandate funds to local organizations for research?

The Tinbergen principle in economics says that you cannot efficiently achieve two policy goals with one policy lever. If funders want to build local researchers, it is best not to enmesh that goal with the second goal of funding research — unless that is shown to be the best way to do so.

For instance, if the idea is to build local capacity, the best option may be to fund scholarships for students. Or, to provide unrestricted funds to institutions with a proven track record.

Take for instance, the case of the Center for Economic Research in Pakistan, a global South organization that revolutionized economics research in Pakistan. The center did so by working with local collaborators only when these collaborators were the best for the project, and by hiring local research assistants, each of whom worked for two to three years.

Some research assistants went on to do PhDs, almost entirely in the U.S. Then, some stayed on in academia. My guess is that 1 in 50 research assistants eventually became a researcher — and that was more than enough to build Pakistan into a powerhouse in development economics.

At no point was funding mandated. At no point were local authors put on a team other than for the reason that they were the best. I can think of no better way to spur locally relevant research than unrestricted funding to organizations such as this.

Local populations deserve the best possible research, for which we need the best possible team. I have always found that the best researchers from within the local context are eager to collaborate and learn from those with complementary skills, in a process of co-creation, collaboration and learning.

But that assumes that there aren’t already local researchers who would be just as good — if only they had the funds.

No, it does not.

For me, decolonization means that resource opportunities are equalized for foreign and local researchers. Local researchers who are just as good will go through a usual proposal process, and as long as the adjudication process is fair — which is admittedly quite difficult to do — they will do just fine.

We are not asking for handouts. We are asking for fairness — both in the funds, and in our access to conferences, visas, journal editors and the whole infrastructure of knowledge production.

But wait — doesn’t the proposal process privilege a foreign epistemic gaze?

The first lung of the problem.

What I call “horizontally differentiated” systems of knowledge require adjudication by a different set of peers without a dilution of quality.

So far, no model has done it well.

We are either in systems that privilege foreign adjudication (points for publishing in international journals) or in systems that are devolving into discussions about how our ancient texts held the secrets of all science.

Appropriate adjudication is a very difficult problem to solve, but mandated funding is just as likely to make it worse.

Again, we are not asking for handouts. We are asking for time to evolve our own systems of adjudication. The process will be messy and adversarial. Do not put your finger on the scale by funding elite cabals.

But what about impact?

The second lung of the problem.

Foreign aid for knowledge seems to be driven by a search for impact, and in many cases, the idea of impact is morphing into the notion that research should directly influence policy, instead of contributing to a broader democratic conversation.

Academia relies on adjudication by our peers, with every generation of judging peers appointed by the current judges. This system is far from perfect. But rewarding impact just replaces peers with another set of adjudicators, the policy makers, with their own set of biases and agendas on top of the existing problems.

For instance, the temptation to create impact by finding a so-called policy champion is enormous. But all a policy champion does is replace the judgment of our peers with the preferences and biases of one single person.

This narrowing of the adjudication process was never a good idea, and with increasingly authoritarian governments it is a grave mistake because it is precisely in authoritarian governments that the value of knowledge is at its highest. As researchers, we should strive for impact by changing the societal conversation, not by undermining democracy through foreign-funded influence over local officials.

How will we know if foreign funding is having a positive effect on the production of knowledge?

The effect of foreign funding on locally relevant knowledge production is very difficult to assess. One reason is because contextual high-quality research is very difficult to publish in the top journals because it will not satisfy the foreign gaze.

The idea that therefore we should look to systems with weak peer review like Reddit channels, predatory journals, or WhatsApp messages is equally worrisome precisely because we do not know how to curate knowledge without some expert judgment.

The difficulty of assessment is why I increasingly think that instead of focusing on knowledge directly, one way to gauge the effect of foreign funding is to think about it in terms of the availability of data and local resources for knowledge production.

If we find that over time the availability of public information is decreasing, that the diversity of local researchers is going down, that young people are not joining the field and local funding is not emerging, the elite cabal is probably winning — if it hasn’t won already.

Local Context Matters

Look, I am not arguing that things should just go on as before without greater attention to the local context. Quite the opposite. I have vociferously and publicly argued that too much work in development economics, my chosen field, is insufficiently grounded in local context and therefore abuses its epistemic authority.

What I am arguing, though, is that there is no simple solution to this problem.

Differentiated systems of knowledge require years of contentious debate (and bloodshed) to emerge. Rushing to right the wrongs of the past, spurred by a need for action with the reduction in global aid, is surely the correct thing to do for life-saving drugs and vaccinations, and just as surely the wrong thing to do for the production of knowledge.

Why? Because if foreign funds, driven by a need for impact, inadvertently cement an elite local cabal that controls the access to and production of knowledge, it will take years, if not decades, to undo.

Take my discussion, then, as an invitation to deliberate on how to create the institutions of adjudication that we will need in order to produce the locally relevant, high-quality knowledge and research to build our health systems for tomorrow.

Read the other articles in this series by Mara Evans (Georgetown University), James Pfeiffer (UW), Susan Erikson (SFU), Tom Frieden (RSL), Steve Radelet (Georgetown University), Pete Epanchin (USAID) and Matt Kavanagh (Georgetown University).

Last year we worked a lot with students who reached out to help navigate the market. We learned some stuff from doing that. So, these posts are a kind-of summary. Some technical, some process, some random thoughts. Take what you will.

This one is technical: It relates to discussions I have had over the last 4 years on trying to understand the link between how we analyze data and how we do policy. That link is subtle and I am still learning how to best think about it.

In my course on causal inference, I spend 6 lectures on inference and 8 on the Big 4 causal methods. That choice was driven by the fact that (a) most of our students will have a big role as consumers of research and (b) consuming research in a world that produces upwards of 80K papers a day is not trivial. Many policy questions that the course helps students think through arose again and again in job interviews for those thinking of going into data/analysis/policy fields.

So, here is a summary of big ideas from the inference part of that class. I pulled out multiple discussions over the year and asked ChatGPT to do the summary. If there are parts of that summary you don’t understand, let me know, and I can post some stuff.

Subscribe now

If the writing is more formal than usual, blame the GPT, not me. Seriously. In response to my prompt, the GPT has the gall to tell me:

“Got it. Let me strip this down into clean, careful teaching notes, written for students, with zero email drama, no side rants, and no assumed sophistication beyond a standard econometrics course. Think of this as something you could hand out after a lecture and say: “This is the conceptual map.””

I now know what Trurl and Klapaucius felt when they started kicking that infernal machine...

What Do Experiments, Estimates, and Confidence Intervals Actually Tell Us?

A short guide to avoiding common statistical mistakes

1. What Question Are We Answering?

Statistical methods answer different kinds of questions, and confusion happens when we mix them up.

Frequentist inference asks:

If the true effect were X, how likely is it that I would observe data like this?

Policy and decision-making ask:

Given the data I observed, what should I believe about the size of the effect, and what should I do?

These are not the same question.

2. What a Hypothesis Test Does (and Does Not Do)

In a randomized experiment, a hypothesis test evaluates a null hypothesis (often “no effect”).

• A p-value tells us how surprising the data would be if the null were true

• A small p-value allows us to reject the null

That is all.

A hypothesis test does not tell us:

• the probability that the true effect equals the estimate

• the probability that the effect lies in any interval

• how large the effect “really is”

3. What a Confidence Interval Really Means

A 95% confidence interval is often misinterpreted.

Correct interpretation (before the experiment):

If we were to repeat this experiment many times and compute a confidence interval each time, 95% of those intervals would contain the true effect.

Important implications:

• The interval is random before the experiment

• After the experiment:

Incorrect interpretation:

“There is a 95% probability that the true effect lies in this interval.”

That statement is not meaningful in frequentist statistics.

A Note: This is the slide that leads to the *most* confusion. The correct interpretation sounds circular—shouldn’t it be “compute an “effect” each time? The answer is “No”—it is truly “compute a confidence interval” each time. This is a key idea—the Confidence Interval is a random variable (interval) that changes each time a sample is taken. What then is “the confidence interval’? Well, there is no single confidence interval, but there is a procedure for generating a confidence interval based on every sample. What the interpretation tells us is that if our procedure is correct, then the CI that I have computed for each sample will have contained the true effect in 95% of the samples. The Wikipedia article on Confidence Intervals is actually really good if you are still having trouble!

A Second Note: You can invoke a Bayesian perspective to justify the `bad’ claim that there is a 95% probability that the correct confidence interval contains the true effect from a single sample. This will happen if the prior is uninformative over its support, in that, it attributes each probability to every possible outcome. Then, the 95% frequentist CI and the 95% Bayesian credible interval will coincide. That’s fine, but my point is that if we are going to use a Bayesian workaround to justify a frequentist procedure, why not then account for the fact that in many cases, I should *not* use an uninformative prior. What you can’t do is claim a Bayesian antecedent for your policy analysis, but then cry foul when people don’t agree with the implicit assumption of an uninformative prior.

4. Why the Point Estimate Is Not “the True Effect”

The estimated coefficient (β̂) is:

• a random variable

• one realization from a sampling process

The true effect (β) is:

• fixed

• unknown

Frequentist theory shows that β̂ has good properties under repeated experiments (e.g., consistency), but:

Once a single experiment is complete, frequentist methods do not assign probabilities to different values of β.

5. Repetition Is the Key to Frequentist Logic

All frequentist guarantees rely on repetition:

• repeated samples

• repeated estimates

• repeated confidence intervals

Policy decisions are not repeated experiments. You observe:

• one study

• one estimate

• one confidence interval

Frequentist inference alone does not tell you how to convert that single result into a belief about the true effect size.

6. Why This Matters for Policy Analysis (CEA / CBA)

Cost-effectiveness and cost-benefit analysis require:

• beliefs about effect sizes

• expected outcomes

• trade-offs and losses

Frequentist outputs (p-values, confidence intervals) do not provide these directly.

When economists use estimated effects for policy:

• they are making additional assumptions

• often implicitly

• often without acknowledging them

7. The Bayesian Alternative (Briefly)

Bayesian analysis allows statements like:

“There is a 95% probability the effect lies in this range.”

But only because:

• effects are treated as random variables

• beliefs depend on a prior

• different priors lead to different conclusions

There is no “prior-free” way to do this.

8. The Bottom Line

• Experiments are powerful tools for rejecting hypotheses

• Confidence intervals describe procedures, not beliefs

• Point estimates are not probabilistic statements about truth

• Using evidence for policy requires:

A rule of thumb

Frequentist statistics tells us when data are inconsistent with a hypothesis. It does not tell us what to believe instead.

Thanks for reading The Cranky Corner: Putting Complexity Back in Policy! Subscribe for free to receive new posts and support my work.

At that point, I asked:

“What are people concerned about when it comes to private schools and impact investing?”

Many of you were kind enough to engage and send in questions. With apologies for the delay, here they are. I have collated them into three groups that capture the variety of concerns that were voiced. Even though it looks like I have included the questions from only three people, this is because the captured most of the issues that were raised. In subsequent posts, I will put together what we know about these questions and where the evidence is sparse.

Group 1

There were many questions about the overall impact of private schooling on learning and equity. Paul Skidmore covered most of them, so I reproduce his (edited) message.

Paul Says:

The more empirically-based concerns one hears most often fall into 5 categories:

1. Exacerbating learning inequalities. The critique is that fee-paying schools inherently worsen learning inequality, and that supporting them makes that problem worse.

2. Regulatory compliance. Virtually every big investment program in low-cost private schools has tried to reverse the vicious circle of bad regulation that is input-focused, leading to informality and therefore schools where it is hard and risky to provide capital. Is there experimental evidence on the impact of more progressive/outcomes-based regulatory reforms?

3. Child protection risk. What do we know about how child protection risks vary across schools and school types. You could also imagine how better evidence on specific approaches to reducing in-school harm could inform technical assistance packages to be bundled with funding.

4. Negative spillovers from ‘cream-skimming’. When private schools enter, do better off families leave, ‘depriving’ public schools of a potential source of improvement and teachers.

5. Disconnect between quality and commercial performance. Because ‘true’ quality signals are noisy and parents are often choosing on the basis of other proxies, schools are not rewarded for (and will therefore not invest in) quality enhancing measures. Is that a concern

?

Group 2.

Is it possible to fund private schools in a way that spurs competition?

Q2. Aqsa Zaidi raises fascinating questions about how to fund private schools in a way that increases competition. She says:

1. First, there might be a general misconception on how funding private schooling differs from funding schools unless someone reads through your research. Using Varthana as an example, which gives loans explicitly to individual schools, could also be contrary to your explanations.

2. There is an increasing focus on funding public schooling because only that is the only route to sustainability; government involvement/ ownership is the best exit option for funders. What are potential exit options in case of funding private schools?

3. If vouchers for parents are out, what is the best method of putting money into private schooling? Public schooling in this way is much easier from funders point-of-view.

Group 3.

There is a giant question about demand versus financing. This was best illustrated in a back and forth I had with @PaddyCarter.

Paddy Carter says:

“I often find myself thinking about “chicken and egg” problems in development. We want to invest in productive large firms to deliver growth and poverty reduction, but they cannot sell their products if people are too poor to buy them. We have “big push” models where complementarities could deliver a big leap forward, but that always leaves me wondering what would actually happen if you turned up in Sierra Leone, borrowed a ton of money and built everything all at once – would local incomes rise quickly enough to recoup your investment, or would you bankrupt the country? How fast it is possible to increase the quality of schooling, even if that should pay-off in high future productivity, if that requires investments and consequent price increases, and how close is the sector to hitting the buffers of demand?”

Let me clarify my thinking on that question.

Suppose I invest in a private school by giving it loans that it has to pay back with interest. School revenues are the only source of cash, which means that either enrollment has to increase or fees have to increase. That is possible if either the school (a) poaches from other schools without parents paying more money or (b) parents pay more money to the school, relative to what they were paying earlier (either in the same school or a different school).

If schools do (a) and poach students from other schools, the overall impacts are unclear since it depends on why the children moved. For instance, if they moved for higher test scores, that will boost human capital, but if they moved because of better amenities, that may increase welfare in the short run without increasing human capital in the long-run.

That then leaves us with (b) and the question that Paddy Carter asks is why parents would be willing to pay more money. Even if they recognize that the school is now higher quality and will therefore allow their children to earn more as adults, it still leaves open the question of whether parents will be able to borrow against those future earnings in the credit market. If not, it could be that we have solved a credit market failure for schools, but this does not help because we have not solved the credit market failure for parents.

This is the key question that Paddy Carter raises—can impact investing work without boosting demand among parents through other means?

I will respond to these concerns in subsequent posts!

Share

I am in interested in our discipline, both what is working and what is not. I know that at our best, we are working to solve some of the world’s biggest problems, but I am concerned about our collective public silence on authoritarian governments and atrocities around the world.

I am also interested in what I call `The Modernity Project.” As colonialism came to an end, many ideas emerged of who we wanted to be and these were further refined in the consensus building exercises around our constitutions. That national myth-making seems to have collapsed. Do we need to revive it, and what kinds of ideas are emerging from countries around the world that may point to a different way forward?

I have tried to order the posts in those buckets and will update this as I write more.

Development Economics

1. What is going right with development economics (Products): Many people think that development economists are tinkering at the margin, have little to say about “big” problems and that this is a direct result of the increase in RCTS. They could not be farther from the truth. I show how development economics has made fundamental progress on giant problems, changing minds and narratives and bringing in crucial change. [LinkedIn, Substack]

2. What is going right with development economics (Process): I talk about the huge strides that development economics has made in bringing more women into the field, encouraging fieldwork, and learning across sub-fields, all the while maintaining very high standards around peer review. [LinkedIn, Substack]

3. Moral compass: Have we lost our moral compass in development economics? A striking feature of our discipline is our collective public silence on atrocities around the world. I just do not subscribe to the notion that we should be dispassionate observers and experimenters, far from the lives of the people on whose behalf we work. [LinkedIn, Substack]

4. What is wrong with development economics: Resource Poor = Research poor. Resource Poor = Representationally Poor. My reflections on a panel at the World Bank. [LinkedIn, Substack]

5. Abuse of epistemic authority: Statistics and statistical methods are regulatory requirements. Like all well-formulated regulations, they are one ingredient in a search for meaning, but they are not--and should not be used--as the only input in understanding the world. The (ab)use of epistemic authority occurs when we use our statistical methods to inspire silence, instead of provoking discussion and debate. [LinkedIn, Substack]

6. Should we change the way we teach: Development economics has traditionally been taught through the lens of “convergence”—why don’t poor countries grow fast when the theory suggests they should? That question needs to be flipped to “how did post-colonial countries manage to accomplish so much in so little time, despite 200 years of complete destruction of their societies?” [LinkedIn, Substack]

The Modernity Project

1. A question for these times: Who do we want to be? An old national project of modernity collapsed as the devastation wrought by communism in Russia became evident. Should it be revived, and what kinds of new thinking is emerging from South Africa, Namibia, India, and Argentina? [LinkedIn, Substack]

2. Further thoughts on the modernity project: I elaborate further on the big ideas around environment, cash grants and positive rights emerging from South Africa, Namibia, India, and Argentina. [LinkedIn, Substack]

My goal here is to use a specific example to explain how and why this happens.

Part 1: A quick primer on where we are

The number of articles indexed by Scopus now approaches 3 million papers a year or 8200 a day, which itself is a likely undercount. If the supply of papers has increased but the prestigious outlets remain fixed (the `Top-5’ in economics), it must necessarily strain the architecture of peer review.

Different disciplines have approached this problem in different ways. Some journals have broadened peer review and editorial boards. This has expanded access but at the cost of severe quality reductions. Reviews are mediocre at best and drivel at worst. See the public peer reviews for BMC public health, for instance.

Economics, instead, appears to have doubled down on “serious” peer review. Reviews run 25-30 pages and responses can run to 150 pages for a single article in a Top-5 journal. Selectivity has sharply increased, as has the powers of the curators. A common measure of concentration shows that economics is now more concentrated than the giant monopolies of the gilded age. The legitimacy of what is to be researched, the adjudication over its quality and the subsequent debate over its implications is now entirely concentrated within a few institutions.

Knowing that our processes around the adjudication of knowledge have become so concentrated should lead us to exercise ever more caution in our deliberations and a desire to seek and understand diverse voices that question our methods and our conclusions. Yet, the opposite seems to have happened: We now seem to be using our monopoly to delegitimize local voices and opinions, even as we speak of decolonization and greater input from local communities.

This is a grave mistake. To see why, I use the example of a fictional researcher who adheres to the regulatory requirements and adjudication procedures of our gatekeeping journals. I borrow two frames from the literature.

The first is Paul Starr’s idea of authority as a substitute for argumentation:

“Authority incorporates two sources of effective control: legitimacy and dependence. The former rests on the subordinates’ acceptance of the claim that they should obey; the latter on their estimate of the foul consequences that will befall them if they do not (…) Authority, then, also refers to the probability that particular definitions of reality and judgments of meaning and value will prevail as valid and true. I will call this form of authority cultural authority to distinguish it from the social authority that Weber had in mind.”

Starr’s fundamental point is that the medical profession becomes powerful once it gains authority. That authority may be grounded in scientific knowledge, but it need not be constrained by it—the key is that it can perceived as legitimate in the absence of argumentation.

The second is a technical definition of disguised nonsense, a statement the appears to be meaningful but actually has no meaning, a statement without cognitive content. The claim that our findings are disguised nonsense in a particular case is therefore a technical statement, not a value judgment.

With these two frames, using an example from the economics of education, I argue that local knowledge is critical to our aim of producing meaningful work. My stronger argument is that several statements frequently endorsed in the economics of education (including my own) might well be disguised nonsense from a technical standpoint. This means two things. First, that those of us who are steeped in the literature fully realize that the statements we make might be meaningless within the framework in which we function. Second, we also realize that our statements can be given meaning through debate and discussion. But when we replace argumentation with epistemic authority instead chose not to understand.

Part 2: The Example

A researcher has conducted a large randomized controlled trial. The use of an RCT here allows me to ignore the additional problems that may arise with causal statements in other study designs. The trial is well-powered, with power = 0.8, and a critical value of 0.05.

Correctly interpreted, this implies two different IF-THEN statements.

· Power = 0.8 means that IF the alternative hypothesis is true, THEN the probability the trial will correctly reject the null hypothesis (presumably that `The treatment had no effect’) in favor of the posited alternative is 80%.

· A critical value of 0.05 means that IF the null hypothesis of no effect is true, THEN in 5% of cases the experiment will wrongly reject the null.

We further assume that the trial is implemented perfectly. Results “show” that the intervention increased test scores by 0.4 standard deviations. The authors report this treatment effect. They compare the size of this effect to how much children learnt in the control group. They find that children in the control group also gained 0.4 standard deviations over the same time. They therefore report that children in the treatment group increased their test scores by an additional year of learning compared to the business as usual in the control group. This is known as the “Learning Adjusted Years of Schooling.”

{They may also try to compute the `cost-effectiveness’ of this intervention by comparing the test score gains and costs to other interventions, but the additional problems with doing so is a discussion for another day.}

Part 3: Deliberation

I now describe two exchanges, the first on the U.S. seminar circuit and the second in a university where the study was conducted, and where researchers are more familiar with the local context.

Exchange 1: In multiple conferences and seminars in U.S. universities, participants enthusiastically kick the tires on the study. Was there a Pre-Analysis Plan? What was the attrition? What about ex ante balance? Have they computed the costs correctly? And what about the clustering? Are we absolutely sure about the standard errors? Don’t we need multiple hypotheses tests—and which test should be used?

Ultimately, they overcome the obstacle course. Using Ben Recht’s argument, they have passed multiple regulatory requirements. In addition, they have also engaged in argumentation that is not part of their ultimate results or framework (again, explanations for another day). They emerge a bit dazed, a bit bruised, a bit battered, but still standing. They are now ready to disseminate their results in the country where the experiment was conducted expecting many accolades, and perhaps also the much vaunted `government buy-in’ to `scale up’ their experiment.

Exchange 2: However, their presentations in local universities run into unexpected hurdles. In addition to the usual technical questions, they are bombarded with questions about the context. People think that their experiment cannot have worked because their AI tutor, meant for smart phones, is just not used in the way they think. It becomes increasingly clear that the authors conception of the world is completely different from what it is for the seminar participants.

Our researchers are flummoxed. Their results passed all the regulatory hurdles and the gauntlet of academic seminars in the U.S. Obviously, this audience just does not understand; their grasp of statistics and econometrics, is, well, a bit lacking. Our researchers, steeped in cultural sensitivity, politely, and perhaps unknowingly, invoke their epistemic authority:

“I am not sure how much I should be diving into the statistics, but it is quite well known that if we find a large significant effect, the treatment must have worked. Perhaps we can discuss this offline?”

Perhaps they bolster their exercise of authority with the use of pithy phrases that clearly demonstrate their verbal agility: “Extraordinary claims deserve extraordinary evidence,” “absence of evidence is not evidence of its absence,” “are you talking about external validity?”

The researchers think that they will have to explain basic statistics to their doubters. They also know that very little rides on the results of that back-and-forth. They have gone to disseminate their findings; there is nothing more to learn. And, it does not matter for their publication chances since the regulatory requirements do not require a knowledge of the context, and no one from the audience will be a reviewer. Ultimately, their adroit use of epistemic authority has cowed the audience to silence. Little do they realize that it has fundamentally come in the way of their learning about their own experiment.

Vencer no es convencer.

Part 4: My claim

Every bit of the argumentation used by the researchers may be disguised nonsense. I make three observations, none of which should be controversial or new for people working in this space.

1. The fact that an experiment rejects the null is uninformative of its p-value in a future replication. The likelihood of a false positive depends on what people think of the likelihood of success—there is nothing in the experimental setup that allows you to get at this. But, even worse, the rejection of the null does not even tell us about the likelihood of the null hypothesis being true!

2. The structure of the experiment is uninformative of the size of an estimate. Again, local knowledge can help.

3. Even if we thought that 0.2sd could emerge as a valid statistical conclusion, that number itself is meaningless without further context. Attempts to use ancillary statistics, such as Learning Adjusted School Years, just makes matters worse.

Claim 1. The rejection of the null is an IF-THEN statement. When you reject the null, the only statement you can make is that IF the null was true, THEN the probability that you would have seen this data are lower than the critical value you have chosen. Therefore, when you have a significant effect, the correct statement is that “IF there had been no effect, THEN the chances that I would have seen these data by chance through the sampling process is lower than 5%.”

Like all conditional statements, this tells us nothing about the overall likelihood that your experiment produces a non-zero impact. One way to think about the problems that this causes is to instead follow Neyman-Pearson’s formulation and allow for 2 worlds, and (at the risk of further antagonizing Fisher, Neyman and Pearson), mash it up with Bayes theorem to show that the probability of a false positive, or the likelihood that the null is true given that it has been rejected is

where pi is the prior probability that the null is true, alpha the critical value of the test and (1-Beta) is the power of the test. Under the assumptions of our experiment

If we know nothing about the likelihood of success prior to the treatment, our priors are uninformative. Perhaps , in which case, the likelihood of a false positive is 5.8%. But if we have good reasons to believe that the treatment is unlikely to work perhaps our prior that the null is true is 90%. Then the false positive rate jumps to 38% and there is no good reason to think that your intervention is producing the results you thought it did. Beliefs over the efficacy of the experiment are critical to what we can learn from it—and it is very likely that local expert knowledge will play a large role in helping us understand our eventual results.

Claim 2. The point estimate is not the population parameter. Economics follows an indefensible two-step routine when it comes to empirical results. We first lean heavily on the frequentist paradigm and present p-values and tests of statistical significance. Then, if we demonstrate statistical significance, we magically forget inference, invoke the law of large numbers and claim that the coefficient we have estimated is actually the size of the effect.

As has been pointed out ad nauseum, this is wrong.

Suppose my experiment tosses a coin 100 times, with H₀: My coin is fair. Under the frequentist paradigm, prior to tossing the coin I have created a rejection region prior to the experiment. Perhaps I used the Central Limit Theorem to see that the mean number of Tails in 100 coin-tosses under H₀ will be normally distributed with a mean of 0.5 and a standard error = .05. A rejection region with significance level of 5% will therefore reject if we see (roughly) fewer than 40 or more than 60 tails (0.5 ±1.96*0.05).

Suppose that I now toss the coin and observe 80 Tails. I can correctly conclude that these observations were extremely unlikely to have arisen under my null hypothesis of a fair coin. I could also use the principle of maximum likelihood to claim that IF my coin came up with Tails with 90% probability, the outcomes that I observe would be the most likely outcome among all other prob(Tails) I could assign to the coin.

I certainly cannot claim that my coin will produce tails 80% of the time! In fact, even a fair coin will produce 80 Tails or more every 1 in 7.8 million (!) experiments of 100 tosses. And if my coin instead produced tails 70% of the time in reality, it would produce 80 tails every 1 in 50 experiments.

The frequentist paradigm we use allows us to make an IF-THEN statement: IF my null was correct, THEN these data are unlikely. It tells us nothing about the world that the data actually came from. But while our hypothesis tests are all about Pr(Data|Truth), once we demonstrate statistical significance, we blithely ignore all statistics/econometrics and move quickly to interpreting our results as Pr(Truth|Data).

Claim 3. Specific to test scores is the interpretation of test scores and effect sizes such as 0.2 standard deviations. It is well known that test scores have an ordinal interpretation, so that, for a well-constructed test, we can claim that a child who has scored 80 has learnt more than a child who has scored 60. But the further assumption that 80 > 60 > 40 also implies that “the difference between 80 and 60 is the same as the difference between 60 and 40.” That additional statement of cardinality is true only under very stringent conditions (double cancellation and the Archimedean axioms if you must know) and is almost certainly not comparable across tests without further anchors, such as common questions across tests.

That we need extremely stringent assumptions to hold test scores are to have a cardinal interpretation is well understood from the theory of conjoint measurement. What is less well recognized is that it is difficult to even justify test scores as a measure in the sense of Steven’s definitions of interval or ratio scales. Joel Mitchell has argued this point repeatedly, but his arguments have yet to permeate the economics of education. The point that Mitchell makes and must resonate with those of us who work in this field is that we cannot mindlessly invoke cardinality or force comparability when we find a result. Instead, we must try and interrogate what our results mean by interacting further with the community and the children. It is only through this interaction that our results can be given meaning.

Instead of leading to the kind of deliberation implicit in Mitchell’s ask, we have instead chosen to try ad hoc fixes for the lack of standardization and cardinality in test score measures. Yet these measures are incorrect, and could indeed make the problem worse. For instance, one proposal is to use something like Learning Adjusted Years of Schooling, which divides gains in one group by gains in another. But when you subtract two variables, the variance adds up, it does not reduce. Atheoretical approaches like LAYS divide one measure by another potentially making the problem worse; not surprisingly, simulations we are doing suggest that with the same intervention, different question subsets can lead to LAYS estimates from 0.1 all the way to 8.

Part 5: Conclusion

I conclude with three observations.

1. My aim here is not to take a wrecking ball to the education literature. Quite to the contrary, I believe that the empirical economics paradigm has generated many useful insights, and as the evidence has accumulated, some genuine wins have emerged. I would even go further to argue that it is the combination of economic approaches and experimentation that has led to critical wins in this space.

2. I also understand counterarguments to my use of the frequentist paradigm. We could invoke Bayesian credible intervals instead of confidence intervals, report Bayesian computations, incorporate meta-analytic insights from across multiple sites etc. etc. But each of these rescues requires further data and knowledge. We simply cannot rescue the results from a single experiment (or even multiple experiments) and treat them as divine knowledge without grounding them in the local context through debate and deliberation. This is precisely the process that I think of as making sense from disguised nonsense.

3. Finally, I also understand that deliberation is hard. Many people (including statisticians and economists genuinely do not understand the statistics. Equally, there are `interested’ parties whose motives are unclear. There will also be those who will expand the boundaries of the study beyond what they were designed to address. But every one of those problems is ALSO true for our conferences and seminars in U.S. universities and forums. We have invested massively to ensure that our deliberations through seminars move our work forward, instead of paralyzing us. Perhaps we should be spending a fraction of that funding on learning how to deliberate better in the countries we are working in and then learning from the deliberations to understand our own experiments.

The results from an experiment are the beginning, not the end of a story. Along the journey we have much to learn from every possible source. And when we invoke our epistemic authority to avoid much needed argumentation in unfamiliar terrain, we end up delegitimizing local knowledge and impeding our creation of much needed knowledge.

We can—and must—do better.

References

On the two frames I use for this post:

Starr, P. (1982). The social transformation of American medicine: The rise of a sovereign profession and the making of a vast industry. New York: Basic Books.

Ian Darden on nonsense: https://philarchive.org/archive/DEADPT-2 (I have *not* read the book yet)

On concentration in economics: Freeman, R. B., Xie, D., Zhang, H., & Zhou, H. (2024, June). High and Rising Institutional Concentration of Award-Winning Economists. NBER Conference Paper, https://conference. nber. org/conf_papers/f204525. pdf.

There are many papers on the problems with Null Hypothesis Significance Testing or NHST. One of my favorites is Gigerenzer, Gerd. “Mindless statistics.” The Journal of socio-economics 33, no. 5 (2004): 587-606.

You can also see Mayo, Deborah G. Statistical inference as severe testing: How to get beyond the statistics wars. Cambridge University Press, 2018.

There is a *lot* of discussion on NHST and how to go beyond it. Yet, most of this debate is about additional statistical methods that can address the problems of NHST. I find Ben Recht’s formulation that a good paper is about constructing a good narrative particularly compelling. The statistics serves as a regulatory device, but it should not be thought to be more than that. For me, talking to people in the communities we work in is just as much a part of constructing the narrative as the statistics that are required to cross a regulatory hurdle.

Recht, Benjamin. “A Bureaucratic Theory of Statistics.” Observational studies 11, no. 1 (2025): 77.

On educational testing:

Michell, Joel. “Is psychometrics pathological science?.” Measurement 6, no. 1-2 (2008): 7-24.

Michell, Joel. Measurement in psychology: A critical history of a methodological concept. Vol. 53. Cambridge University Press, 1999.

Angrist, Noam, David K. Evans, Deon Filmer, Rachel Glennerster, Halsey Rogers, and Shwetlena Sabarwal. “How to improve education outcomes most efficiently? A review of the evidence using a unified metric.” Journal of Development Economics 172 (2025): 103382.

Share

A course I teach develops two ideas:

1. Poverty is a lack of money. Giving people money relieves them from poverty.

2. However, in many cases people face specific constraints that prevent them from doing the best they can.

In (2), the constraints I deal with are “static market failures” that arise because of a problem with the way society is distributing resources. For instance, perhaps one group is super entrepreneurial and would develop fantastic technologies if they only had the money, but the money is going to an entrenched, unproductive elite. Or, people would be willing to pay for health insurance, but a market for health insurance does not exist.

A key lesson I learnt from Jeff Hammer was that if we could only (a) agree to give people cash and (b) resolve, or at least alleviate, some of these market failures, life would be much better for the poor. Incredibly, and despite more than 100 years of consensus among economists, alleviating market failures has proven incredibly hard. Monopolies (a market failure that restricts trade) are roaring back and climate change (a market failure we call externalities) is devastating our planet.

So, generating the evidence that we can improve the lives of the poor by solving market failures and distributing cash is one of the most important topics we can tackle today. It is also an incredibly difficult job because the evidence has to be strong enough to battle an adversarial and entrenched elite. For instance, how would you counter these questions:

(a) If you give poor people money, they would just drink it away? (a pure economist would respect people’s preferences, but you can see where that leads…)

OR

(b) It is correct that the private market for education or health may not work well because consumers do not have information, but will solving this constraint really be enough? Isn’t the real problem that schools and households need our direct support?

OR

(c) If governments could alleviate these constraints, things would be better. But we have to balance market failures against government failures and we all know that the government is corrupt and inefficient, so why invest in the government?

Perhaps surprisingly, development economics has made fundamental contributions to each of these problems over the last 20 years. There has been *a lot* of work on such GIANT problems, not just tweaks around the edges. More strikingly, progress on these problems owes a lot to the use of RCTs. RCTs have allowed us to say that the outcomes we see are we see are NOT a reflection of the different preferences of the poor, but the constraints they face. Why? Because the people for whom we alleviated the constraint were exactly the same as the group where we did nothing. The RCT vs. non-RCT debate is entirely the wrong one—the question is one of the contributions of the work, not the methods used.

I will give three examples, all using RCTs:

Example 1: What happens when you give people cash.

Example 2: Does solving market failures improve outcomes, even in non-traditional sectors like education?

Example 3: Is the government corrupt and inefficient and can nothing be done to fix that?

Example 1: If governments gave poor people cash, would they just drink it away?

It is worth highlighting, once again, that an economist takes no view on what people do with the cash they receive. A preoccupation with how the poor use cash grants requires us to assume that the person giving the money (say, the government) has a different welfare function from those of the people it is giving the money to. It is unclear why this should be so, but that argument never holds water….

So, a development economist, Paul Niehaus, set up GiveDirectly, which has become a powerhouse in the philanthropy community with critical expertise on a single, focused task—giving people money while keeping administrative costs low. They are now doing some of the most fundamental experiments in the world around the role of cash and the alleviation of poverty. Many of these RCTs allow us to understand precisely what happens when you give cash to the poor.

There are four main results that have now been shown to hold across multiple studies.

RESULT 1: When you give people cash, their consumption of all kinds of things improve—from spending on education and health to decreasing debt.

RESULT 2: When you give people cash, they do not spend more on “temptation goods” like tobacco and alcohol.

RESULT 3: When you give people cash, they do not become lazy welfare recipients. In fact, they tend to work more.

RESULT 4: When you give cash to a lot of people in a community, the entire community benefits. This result, known as the “government multiplier” shows that giving $100 increases local activity so that total “GDP” increases to $250, give or take.

These results have completely changed the discussion around the form of social protection programs.

Example 2: Solving market failures improves outcomes—even in non-traditional sectors like education.

The dominant `problem’ of education in the 1990s was insufficient demand. Apparently, people did not want to send their children to school—or so it seemed. But, around the turn of the century, my co-author Tahir Andrabi noticed that villages in rural Pakistan were seeing tremendous growth in private schooling. Instead of insufficient demand, we thought that the question would quickly turn to the role of the government in (what we call) “education markets,” with people demanding higher quality schools and private schools arising to address that demand. Taking an economist’s viewpoint, we set out to see if education outcomes could be improved just by solving market failures.

To demonstrate this approach, we introduced the new idea of `market-level randomizations.’ Under a market-level randomization, instead of working with children or schools, the researcher works with an entire market (a village in our case) with multiple schools. The researcher then introduces a new innovation in some markets, randomly chosen, but not others. We then assess what happens, which could include a whole range of complex reactions from schools improving, children switching schools, private schools shutting down or new private schools entering.

Our first market-level randomization took on the idea that the lack of information was a market failure in the education sector. We tested children in Grade 3, and then in randomly selected villages, we provided `report cards’ detailing the child’s performance as well as that of the school, compared to other schools in the village.

After 2 years we found significant improvements in the test scores of children in the report card villages (compared to the control) and a significant decline in the fees of the private schools. These results were precisely in line with an economic model of asymmetric information, first articulated by George Akerlof in his famous `lemons’ paper. And they showed something quite remarkable—you could improve the quality of education while simultaneously decreasing its price. That is exactly what solving a market failure should do, since these are the equivalent of $-bills lying on the pavement.

We have not publicized these results outside the academic community because we did not know what the longer-term outcomes would be. Instead, have been intensively tracking the children who received the report cards and the markets where the report cards were given for the next 15 years, with a final round in 2017-2018.

Our new papers on the long-term effects of report cards will become available over the next 6 months, but here is a sneak preview:

RESULT 1: The children in 2003 who received the report cards were more likely to go on to high school, graduate high school and go to college.

RESULT 2: The children in 2003 who received the report cards were also more likely to earn higher wages in the labor market.

Putting these two results together alone generates incredibly high returns to the intervention. But a third result raises the returns even more:

RESULT 3: The villages where we gave report cards in 2003, continued to have lower prices and costs and higher test scores eight years later in 2011. That is, we managed to reverse a common finding of declining productivity in education—we are now producing better learning at lower costs than before.

These results are incredibly gratifying. We did nothing to support either schools or parents—we only lifted a constraint that they faced in the schooling markets. But exactly as an economic model of information would suggest, lifting these constraints led to large gains, suggesting both that solving market failures can increase the productivity of education and that, once we lift these constraints, people in villages can figure it out for themselves.

Example 3: Yes, the government often does not work well—but we can fix that

At the turn of the century, many of us were very pessimistic about the ability of our governments to fix problems. You have to understand where we were coming from. I grew up in a very well-to-do neighborhood in Delhi. Without fail from the time I was 7 till I was 18 the electricity would go in the middle of the night and after tossing and turning for 2 hours, my father and I would drive to the electricity office, find a repairman and make it back home after they fixed the problem. Funeral processions would be carried out in cities for phones that had been `dead’ for a long time with no hopes of resurrection. Not surprisingly, then, many of the papers we wrote at that time highlighted the severe problems with government functioning and services in all kinds of sectors.

But this does not mean that state capacity would not develop over time, or that it would not improve, both as our nascent governments figured out how to work better and newer technologies became available.

This is where Karthik Muralidharan’s work has been so important. Muralidharan has been an incredible force working with the Indian government on very large programs to see if these programs can generate better returns for poor people. In many of his studies, he works with the government to randomize programs across entire Indian districts, some as large as small countries. His main result is that you can improve the efficiency of governments by 10X through these programs. Notable examples include:

RESULT 1: Biometric payments in NREGA increase what workers receive—and their wages in the private market.

RESULT 2: Informing officials that farmers would be called to see if they received money from a government program reduced the fraction of farmers who did not receive money by 7.8%.

RESULT 3: Adding a half-time worker to government-run preschools massively increased test scores among children.

Muralidharan has also been clear about programs that did not work. For instance, biometric ID requirements in a social protection program reduced payments to legitimate beneficiaries, and a large school management program had zero impact on the functioning of the schools.

Conclusion

My point here is not to claim that all development economics is great or that we are always looking at the giant, contextually relevant questions or that RCTs are the way forward. Those who have read my previous posts know that I am quite concerned about many aspects of the discipline, and therefore feel that I should be working hard to advocate for change.

But to understand the flaws in a discipline, you have to understand the strengths. And the examples I have illustrated hopefully go to show that development economists ARE working on some of the biggest and hardest problems in the world today and ARE making significant progress. AND this progress is being made using RCTs in highly imaginative ways.

The flaws in the discipline seem to be a more recent turn towards work that is not contextually grounded, but bolstered through an unearned epistemic authority.

But that is a topic for another day.

Read More on cash transfers

Egger, Dennis, Johannes Haushofer, Edward Miguel, Paul Niehaus, and Michael Walker. "General equilibrium effects of cash transfers: experimental evidence from Kenya." Econometrica 90, no. 6 (2022): 2603-2643.

Crosta, Tommaso, Dean Karlan, Finley Ong, Julius Rüschenpöhler, and Christopher R. Udry. Unconditional cash transfers: A Bayesian meta-analysis of randomized evaluations in low and middle income countries. No. w32779. National Bureau of Economic Research, 2024.

Evans, David K., and Anna Popova. "Cash transfers and temptation goods." Economic Development and Cultural Change 65, no. 2 (2017): 189-221.

Read more on market-level experiments in education

Andrabi, Tahir, Jishnu Das, and Asim Ijaz Khwaja. "Report cards: The impact of providing school and child test scores on educational markets." American Economic Review 107, no. 6 (2017): 1535-1563.

Long-term results from this experiment coming soon!

Andrabi, Tahir, Jishnu Das, Asim I. Khwaja, Selcuk Ozyurt, and Niharika Singh. "Upping the ante: The equilibrium effects of unconditional grants to private schools." American Economic Review 110, no. 10 (2020): 3315-3349.

Andrabi, Tahir, Natalie Bau, Jishnu Das, Naureen Karachiwalla, and Asim Ijaz Khwaja. "Crowding in private quality: The equilibrium effects of public spending in education." The Quarterly Journal of Economics 139, no. 4 (2024): 2525-2577.

Read more on improving public sector efficiency

Muralidharan, Karthik, Paul Niehaus, and Sandip Sukhtankar. "General equilibrium effects of (improving) public employment programs: Experimental evidence from India." Econometrica 91, no. 4 (2023): 1261-1295.

Muralidharan, Karthik, Paul Niehaus, Sandip Sukhtankar, and Jeffrey Weaver. "Improving last-mile service delivery using phone-based monitoring." American Economic Journal: Applied Economics 13, no. 2 (2021): 52-82.

Muralidharan, Karthik, Paul Niehaus, and Sandip Sukhtankar. "Identity verification standards in welfare programs: Experimental evidence from India." Review of Economics and Statistics 107, no. 2 (2025): 372-392.

Ganimian, Alejandro J., Karthik Muralidharan, and Christopher R. Walters. "Augmenting state capacity for child development: Experimental evidence from India." Journal of Political Economy 132, no. 5 (2024): 1565-1602.

Share

My posts on “what’s wrong with my discipline” are a plea for further improvement rather than a condemnation of the status quo. A student’s passing remark (that my post was really depressing) and a couple of recent talks reminded me that I should really correct the record.

That is,

· I believe that there are (serious) problems with our discipline.

· I believe that there have been many positive changes during my professional career.

This post is about the second of those two points. This post is on process and the next one on products.

What’s going right on process

Greater representation among women. Economics is notorious for its leaky pipeline and under-representation among women, even compared to other STEM subjects that were traditionally biased towards males. In development economics men are not over-represented (labor/health is the other). In fact, there is greater representation among women compared to men in their doctoral studies and potentially in faulty positions (I cannot find the numbers for faculty, so any data would be helpful!)

More fieldwork than before. Little known story: When Asim Khwaja and I were doing our PhD’s, in our third year, both of us decided to go and spend a year `in the field.’ Asim went off to Skardu, and I went off to the Indian mountains, where I spent a year setting up a clinic to understand how people in the small village and town where I was staying accessed healthcare. Many of the insights from that year drove my later thinking on healthcare, but at the time that we did this, no PhD students were “going to the field.” Abhijit Banerjee, one of our supervisors had pretty much given up on us at that time (I think), but we conveniently took his silence as acquiescence (“well, he didn’t actually say don’t go”). Remarkably his support for what we choose to do and our work has never wavered and continues to this day (thank you, Abhijit!).

From what I can tell, taking a full year off to spend time in the field is still kind of not done, but students are spending a *lot* more time in the field compared to before, and many people have pre-doc and internship opportunities where they go and spend months and sometimes a year or more living in a country. So, yes, the work is still falling short on context in a lot of cases, but the opportunity and ability to learn from where your work is situated is infinitely better now.

Learning across fields. Banerjee, Duflo, Foster, Kremer, Ray, Rosenzweig….. The founding parents our field have innovated across multiple fields in economics and we are luck that, to this day, the single moniker “development economics” encompasses multiple subfields within economics. This means that, unlike the conferences on health/education/labor where only papers from the subfield are presented, we get to see papers that span the spectrum from political economy to health to firms to industrial organization to trade. Yes, this does mean that sometimes you are quite lost, but it also means that we are exposed to the latest thinking in a lot of areas, and that cross-percolation vastly improves the quality of our work.

And finally….peer review. I review many papers every year. And I am often frustrated by my fellow reviewers’ bias towards evaluating the technical details rather than whether the question and the execution of the paper is sensible given the context. I have reviewed papers where totally implausible spending patterns are deep in appendix tables or the fact that a place is buried in snow for half the year is never mentioned in a paper on facility usage, but comments from other reviewers focus on clustering for standard errors rather than the glaring contextual flaws.

But here is the thing. Peer review was never really meant to deal with the unprecedented surge in the production of academic knowledge that we are now seeing. Ugo Gentilini points out that there are 1.4 million papers on cash grants; just in the sciences 3 million papers are published a year (almost certainly an undercount) . And the quality of peer review has completely collapsed. I have reviewed papers for Science where the 2^nd reviewer writes literally one paragraph. If you want to see for yourself or show your students, take a hike over to BMC Public Health, click on an article and look at the peer reviews for that article, which the journal has courageously chosen to make public. This is not a predatory journal, in fact it is a respectable journal in the field—yet, finding a single substantial review in the morass of atrocious comments will be a chore.

So, the fact of the matter is that we are all struggling. But development economics (and, I think, the field as a whole) has been quite remarkable at keeping the quality of peer reviewing at a very high standard. In fact, our primary complaint is that the reviews are too long, and it will often require a 100-page response to satisfy our reviewers. The cost of keeping our reviewing process at such a high quality may be that we have ended up monopolizing the discipline and pushing entry costs way too high, but the honest truth is that this is a trade-off not a pareto decreasing decision.

Nobody foresaw the huge increase in the production of knowledge that started with the turn of the century, and no one knows how to rejig a system built around scarcity in the production of knowledge to a system built around an abundance of knowledge production but a scarcity of knowledge curation. Honestly, I do not think we have thought sufficiently about this problem or its fundamental parameters. See for instance Ben Recht on substack. The increased monopolization of epistemic authority reflects the fact that the production of knowledge has increased while we have chosen to keep the supply of curation constant, and that has been bad for our discipline. But in science and medicine, where the supply of curation seems to have increased are now facing the opposite problem: Authority is dispersed and massively contested from all directions.

There is no “right” way here—we are just going to have to fight our way out of this.

Read more:

On the Leaky pipeline: Buckles, Kasey. 2019. "Fixing the Leaky Pipeline: Strategies for Making Economics Work for Women at Every Stage." Journal of Economic Perspectives 33 (1): 43–60.

On under-representation among women in economics: https://theconversation.com/the-gender-gap-in-economics-is-huge-its-even-worse-than-tech-156275

On the representation of women across sub-fields in economics: Sierminska, Eva, and Ronald L. Oaxaca. "Gender differences in economics PhD field specializations with correlated choices." Labour Economics 79 (2022): 102289.

The only change is that the coefficient on logGDP halves when we include English, which is not surprising given the dominance of the U.S. in economics, but it does not change for the broader set of journals they consider—-which I can’t explain completely unless the correlation between GDP and English is 0, given the weighting of articles in their corpus…..

Here is a fact: Resource poor countries are research poor and representationally poor. But why?

The other day I participated in a panel on what’s wrong with development economics, with Markus Goldstein at CGD, Leonard Wantchekon (Princeton) and Orianna Bandiera (UCL). Markus and Juan Menendez have written a blog on the panel, which highlights problems with the 3Rs: Rates (too little work on development), Relevance (very little concordance between what policymakers say they need and what is actually done) and Representation (most of the work is done by people in institutions outside the country).

I want to bolster the rates and representation arguments with a couple of pictures and tables.

Resource Poor is Research Poor

Figure 1 is based on data from Das and others (2013). It shows just how few papers are published outside the U.S. (and some parts of Europe) in the top economics journals. For example, over the 20-year period from 1985 to 2005, there were 34 papers on any Sub-Saharan African countries. Numerous authors have extended our work, but the results do not change. Markus pointed out that if you squint in just the right way, you can see some improvement (38 papers over the last 10 years from SSA in the top journals), but in the journal with the greatest representation—the American Economic Review—papers from Sub-Saharan Africa still account for just 1% of all papers. Resource poor places continue to be research poor—and not much has changed since I last worked on this.

Figure 1: Resource Poor is Research Poor

Poor resources = poor representation

Figure 2: Editors of major journals who have a substantial connection to a low-income country

American Economic Review: NONE

American Economic Journal: Applied Economics: NONE

Journal of Development Economics: THREE

I did not make a figure because there is no complex information to convey, but you get the idea. I had expected very little representation in the AER and AEJ, which are both journals of the American Economic Association, still very much a U.S. focused institution. And that is indeed the case, without a single editor with a substantial connection to a low-income country.

But it is very troubling that the premier journal in development, JDE, has three editors with a substantial connection to a low-income country, not a single one from Africa or the Middle-East and not a single one based in India or China? (People may/will disagree about how I classify `substantial connection’ so I invite them to look at the full list of editors on the journal website and DM me if I got it wrong; I defined substantial connection as someone who is either located in or studied in an institution outside U.S./Europe.

An answer then to the question of what’s wrong with development economics is that the field has not managed to increase representation from the very countries where the work is done.

Why?

One possibility is that this reflects bias and discrimination. Indeed, Leonard Wantchekon provides a compelling example in the panel of his attempts to increase African representation in the Econometric Society, which is very low despite the fact that there are candidates with better records than those who are already members. I do not disagree that bias and discrimination are part of the problem, and indeed there are multiple papers that demonstrate racial bias in academia.

But what does not sit right with me is that the editors I know are all just as worried about the lack of representation as the panel. These are editors who have spent months/years in low-income countries with strong connections to academicians in the countries they work in, and they have been forced to conclude that the pipeline is just not strong enough to guarantee a steady stream of editors from low-income countries. Too see what I mean, if you look at articles published in the JDE, a point that Markus makes in his blog, very few are written by people with substantial connections to the country they are writing on. The fact is that, even as development economics has been singularly successful at solving one of the hardest problems in economics—increasing representation among women--it has failed at increasing representation from low-income countries.

Rather than a story of bias or discrimination (which conditions on the academic standing of the researcher), my worry is that the more pernicious problem is a bias in epistemic authority. By which I mean that we are OK with wider representation only as long as the research methods agree with our own epistemological approach.

1. The bias in epistemic authority undercuts the whole point of wider representation, which is to bring on board people who approach problems in an entirely different way.

2. The bias in epistemic authority is not `OK because our approach is better.’ There is no questioning that the epistemic approaches favored over the last 20 years have yielded substantial gains in our understanding of low-income countries across a host of topics. But for many problems the research has become increasingly technically sophisticated and increasingly less contextually aware. Fundamental mistakes are being made because researchers do not know enough about the context they are working in. Worse, there is often no knowledge-based reason for why our approach is preferable: Much of our epistemic authority has necessarily flimsy foundations.

3. The bias in epistemic authority seems to come from a place of some arrogance and some forgetting. We believe our approach is superior. We forget that whatever approach we take is cobbled together from a set of tacit agreements arrived at over years of research. Agreements that allowed us to make progress by walling off a set of problems `to deal with later.’ But by now, many of those tacit agreements have been forgotten and the conversations that wider representation surface often reflects the fragility of those agreements. We would be fools if we did not foster those conversations and if we did not listen carefully.

Look, I am not claiming that the regulatory and epistemic authority that is implicitly played out in the journals has not brought us substantial gains in knowledge over the last two decades. It has, and I am not arguing that we should go back to the way development economics was practiced, say, in the 1980s.

What I am saying is that even though the very best in the field always combine a deep contextual understanding with technical mastery, we are moving/have moved to a situation where deep contextual understanding can be sacrificed on the altar of technical mastery, but not the other way around. I feel that is a huge mistake that we are making.

The bias in epistemic authority implies that editors who are `different’ and who approach questions in a fundamentally different way will not pass muster with our peers. We, and our editors, are then stuck with the problem of adjudicating epistemic authority in a manner that must be acceptable to the majority of the discipline, in a situation where the peer composition has clearly shifted towards very little representation among those from low-income countries.

That, I fear, is a structure that can lead us into indefinitely long periods of increasing irrelevance.

Subscribe now