I enjoyed making a presentation titled “Questioning Some Metascience Assumptions” to the Association for Interdisciplinary Meta-Research and Open Science (AIMOS) at Victoria University of Wellington, New Zealand on 13th August. There’s a recording of my presentation below and a copy of my slides here. (The recording appears to cut out towards the end, but bear with it and it will get back on track.)

Many thanks to for the invitation and to the audience for their interesting questions.

Mindless Transparency

Jenni Adams and colleagues published an article that “considers openness, transparency and rigour across different subject areas and research types.” As they explained:

Applying a unitary model of openness that aligns primarily with quantitative and positivist research paradigms risks perpetuating the devaluation of AHSS fields [arts, humanities and social sciences] (Leonelli 2023; Ross-Hellauer 2022), homogenising research approaches (Bazzoli 2022; Field 2025; Guzzo et al 2022), and sustaining AHSS researchers’ disengagement from open research. In this context, it is crucial for researchers across a range of methodologies and epistemologies to be given the space to work through what openness can meaningfully look like in their research.

Relatedly, it was good to see Grant et al. (2026) reflecting on the concern about the applicability of open science practices in their recent update to the Transparency and Openness Promotion Guidelines (TOP 2015). Specifically, they acknowledged that TOP2015

was developed primarily (though not exclusively) by social and behavioral scientists concerned about misuse of null-hypothesis significance testing…, selective reporting, and computational irreproducibility in published experimental research…. As a result, the full set of TOP standards is most directly applicable to quantitative studies and other designs that test questions or hypotheses about potentially generalizable knowledge. Some standards may be less applicable or less important for other epistemologies, study designs, or phenomena of inquiry. (p. 6)

Returning to Adams et al.’s (2026) study, there’s lots of interesting findings here that highlight the heterogeneity of perspectives on openness, transparency, and rigor. However, I was interested in participants’ perceived practical constraints of transparency. In particular, the finding that caught my eye is reported in this single sentence:

Some participants also expressed concern that sharing too much information may obscure rather than clarify.

The concern about transparency obscuring rather than clarifying has been discussed previously. For example, Elliot (2020) argued that “transparency can also be dangerous. It has the potential to confuse people by burying them under a flood of information that they cannot easily digest” (p. 349).

This issue also relates to my own concerns about “mindless transparency.” It’s not possible to be transparent about everything in a research study. We need to make a selection of information on the grounds that we believe it to be relevant to our research claims. Nosek and Bar-Anan (2012) made the same point:

Published articles do not report everything that was done or found. They cannot do so. Reports of a study’s methodology reflect the researchers’ best understanding of what is crucial in the design. However, this almost always reflects a qualitative assessment of the key factors of the sample, setting, procedures, measures, and context. In other words, the reported methodology describes what the researcher thinks is important, not necessarily what is actually important. (p. 225)

Consequently, in my view, it’s important to distinguish between relevant and irrelevant transparency because it’s only useful to ensure transparency about relevant information, not irrelevant information (Rubin, 2020, p. 383; see also Fiedler and Schwarz, 2016, p. 46; Leonelli, 2023; Popper, 1962, p. 230). As Popper (1962, p. 230) put it, the principle of reporting “the truth, the whole truth, and nothing but the truth” only pertains to the relevant truth. Indeed, mindless transparency that fills research reports with irrelevant information may impede the efficient communication and understanding of research claims. Hence, mindless transparency can be detrimental to effective open science, and one of the most important open science practices is to think carefully about what we’re being transparent about and why.

Of course, researchers may be biased when determining what information is “relevant” and “irrelevant,” and different epistemic communities may have different norms about what they consider to be “relevant” (e.g., depending on different philosophies of science and statistics). However, where there’s disagreement, people can request transparency about the information they think is relevant, and if that transparency is not forthcoming, they can downgrade their evaluation of research claims. Hence, the potential for bias and disagreement about relevance does not warrant indiscriminate transparency that has the potential “to obscure rather than clarify,” to quote Adams et al.’s (2026) participants.

Publication Bias

Mark Ramos published an article in the statistics magazine Significance in which he comments on the recent Systematizing Confidence in Open Research and Evidence (SCORE) project. News reports of the SCORE project’s 55% replication rate (Tyner et al., 2026) took a “glass half empty” view, claiming that:

• Only about half of social science results can be replicated, finds new study

• Across the social sciences, half of research doesn’t replicate

• Half of social-science studies fail replication test in years-long project

I’ve recently queried this “glass half empty” interpretation here, and Ilka Gleibs has provided a “glass half full” perspective here. Ramos has a similarly optimistic view. He concludes, that

reasonably speaking, having a 50% replication rate in social and behavioural studies is much better than it has been made to sound. The results from the SCORE project should be considered a reassuring sign of good science prevailing.

A summary of Ramos’ (2026) argument is as follows:

A balanced perspective on positive publication bias recognises that although publishers’ preferential treatment of statistically significant studies when selecting among studies of otherwise comparable quality inflates the false positive rate in the published literature beyond nominal per study levels, it also increases the rate of true discoveries. Moreover, the nature of statistical significance testing is such that rejections of the null hypothesis carry more actionable information than failure to do so. As such, evaluating this phenomenon requires more nuance than focusing solely on its effects on false positive rates. Under reasonable hypothetical scenarios, such as one where selecting a null hypothesis to study that was actually false is rare, positive publication bias does not severely inflate false positive rates in published literature as long as all researchers are acting in good faith – that is, their nominally reported significance levels are consistent with all design and analysis decisions made in their study.

Ramos posted the following cartoon on BlueSky, which I think works well to illustrate his position:

See also Ramos’ related work in this area:

Ramos, M. L. F. (2026). Balanced examination of positive publication bias impact. Accountability in Research, 33(4). https://doi.org/10.1080/08989621.2025.2538066

Top 10 Posts

For your edification, here are the top 10 posts from the Critical Metascience Substack in descending order of number of views:

1. What’s Special About Metascience?

2. What is Critical Metascience and Why is it Important?

3. Is a 55% Replication Rate Too Low, Too High, or Just Right?

4. (Re)Conceptualizing Open Science’s “Openness”

5. Preregistration, Severity, and Deviations

6. A Brief Review of Research that Questions the Impact of Questionable Research Practices

7. Psychology’s Recurring Crises

8. Q&A with Berna Devezer

9. Sabina Leonelli’s “Philosophy of Open Science”

10. Let’s Talk about Science Reform! Introducing a Workshop on Theoretical and Methodological Approaches to Investigating the Open Science Movement

Subscribe now

Share

Looks like a wealth of excellent work here, but, for now, I want to discuss Carlos Cornejo and Pablo Rojas’ chapter on theory aversion, methodolatry, and scientism in psychology. In particular, I want to consider how these issues might relate to what I call phenomenon-first accounts of psychological science (e.g., Lavelle, 2024; Nak et al., 2026) as well as Lakatos’ (1978) concept of naïve methodological falsificationism. I’ll also consider how these issues might affect not only psychology but also metascience.

Theory Aversion

Cornejo and Rojas (2026) argued that contemporary psychology suffers from an aversion to theory, and they set out to understand why.

The view that theoretical analysis is often met with disdain or even aversion has persisted since the inception of scientific psychology (Danziger, 1983; Dilthey, 1883/1989; Koch, 1992a). Jaan Valsiner has referred to this attitude as ‘theory-phobic’ (Valsiner, 1991, p. 481), and Gerd Gigerenzer as a ‘theory aversion’ (Gigerenzer, 2024, p. 298). (p. 179)

This chapter inquires the reasons behind psychology’s theory aversion. (p. 180)

According to the authors, one reason for theory aversion in psychology is that psychologists have not fully embraced Popper’s anti-verificationist philosophy of science. As they explained:

Popper (1935/2002) pointed out the impossibility of verifying a hypothesis by empirical evidence, for singular empirical data can never suffice to uphold a theoretical statement of a general nature. Nonetheless, scientific theories can be falsified by empirical evidence. As a consequence, scientific theories end up being conjectural. What distinguishes scientific knowledge from any other form of knowledge is its permanent proneness to be falsified. These arguments led to the methodological prescription to actively look for evidence that potentially contradicts a held hypothesis. (p. 182)

However, as Cornejo and Rojas (2026) explained:

In psychology, Popper’s philosophy was assimilated only at a methodological level. Psychologists omitted his arguments against verification but adopted his proposal of falsifiability as part of the methodological prescriptions of doing science correctly. Thus, Popper’s scheme of ‘falsification instead of verification’ was replaced by ‘verification + falsification.’ (p. 183)

The Phenomenon-First Approach

In my view, the hybrid “verification + falsification” philosophy may have also influenced a two-stage, phenomenon-first approach in which psychologists are encouraged to: (1) first “establish” (verify?) that a phenomenon (sometimes called an “effect”) exists before, (2) testing and potentially falsify theories that explain that phenomenon.1 Lavelle (2024), Nak et al. (2026), and Gal et al. (2026) justify and explain this approach.

Lavelle (2024) argued that:

Psychology needs a greater number of well-established phenomena, and that this step is prerequisite to having well-established theories, formalised or otherwise. These phenomena would come in the form of theory-lite observations of people in their everyday environments. (p. 645)

Similarly, Nak et al. (2026, pp. 5-6) argued that:

Establishing empirical phenomena to explain is an essential step in the scientific process; a step that typically precedes the development of explanatory theories, and can therefore not be cast in the theory-testing mold.

Research methodology for detecting and consolidating empirical phenomena should have a more prominent role in psychological research.

Finally, Gal et al. (2026) proposed a three-stage approach in which

a typical trajectory begins with a striking effect, proceeds to a theory-of-the-effect account of that phenomenon, and only later (if at all) extracts a framework that transcends the original effect paradigm. (pp. 12-13)

Prima facie, the phenomenon-first approach appears to represent theory aversion because it excludes theory when establishing phenomenon and only considers it during a subsequent theory-testing stage. However, this characterization needs some qualification. The phenomenon-first approach does not relegate all theory to the theory-testing stage; it only relegates explanatory theory to this stage. For example, Lavelle’s (2024, p. 645) concept of “theory-lite observation” respects Popper’s view that “observations can never be theory-free” (e.g., Popper, 2002, p. 90). Similarly, Nak et al. (2026, p. 17) accept that theories are inherent in the auxiliary hypotheses that underpin methods as well as the concepts and categories that are used to interpret phenomena.

Hence, the phenomenon-first approach accepts that interpretative or observational theories (Lakatos, 1978, p. 44) are ubiquitous, because they are required to both establish phenomena and test theories. In contrast, it argues that explanatory theories only become involved when testing theories and not when establishing phenomena. Consequently, the phenomenon-first approach can only be said to be “theory-averse” in the sense that it excludes the testing of explanatory theories when establishing phenomena.

Is Explanatory Theory-Testing Absent When Establishing Phenomena?

But is it really the case that tests of explanatory theory are absent when establishing phenomena? Nak et al. (2026) argue that establishing a phenomenon consists of (1) “detecting” (discovering) the phenomenon and then (2) “consolidating” the robustness of the phenomenon vis-à-vis its reliability, validity, and generalizability. In my view, explanatory theory-testing can enter into several of these activities.

First, a phenomenon may be conceived as an explanatory theory. As Popper (1962) explained, “the ‘empirical basis’ consists largely of a mixture of theories of lower degree of universality (of ‘reproducible effects’)” (p. 41). From this perspective, establishing a phenomenon amounts to testing a low-level theory that explains observations that constitute a reproducible effect. In turn, this low-level theory is explained by (deduced from) higher level theories (Popper, 2002, pp. 38, 107).

Second, detecting a phenomenon involves determining whether it is new, interesting, unexpected, and surprising. Consequently, it should not already be adequately explained by extant explanatory theories. After all, if a current explanatory theory provides an adequate explanation of an observed phenomenon, then that phenomenon’s status as “new,” “surprising,” and “unexpected” becomes questionable. Hence, checking whether a phenomenon is genuinely “new” may require determining whether it is already explained by extant explanatory theories. Importantly, this initial step may require testing extant explanatory theories in order to rule them out as potential explanations of the phenomenon.

Third, ruling out issues that might threaten the validity of a phenomenon can also contribute to its explanation. For example, if we rule out demand characteristics as an explanation for a phenomenon (i.e., by testing and tentatively refuting this explanatory theory), then we will have narrowed down the list of potential causal explanations for that phenomenon.

Fourth, the identification of boundary conditions of a phenomenon is not simply a descriptive exercise; it can help to explain the phenomenon. For example, if we discover that a phenomenon occurs in Western cultures but not in nonWestern cultures (via hypothesis testing), then we have tentatively ruled out causal mechanisms that are not moderated by cultural factors.

Fifth, a critical discussion of a phenomenon’s potential explanatory theories may help to estimate the pursuit worthiness of those theories and, consequently, the amount of time and effort that should be dedicated to establishing the phenomenon. If a putative phenomenon has relatively uninteresting explanatory theories, then researchers may not wish to invest resources in its establishment.

Finally, explanatory theory may guide the process of establishing a phenomenon. For example, a tentative explanatory theory of a gender difference in self-esteem may imply that men only have higher self-esteem than women in situations in which they have a higher social status than women. Hence, explanatory theory can help to refine the conceptualization of a phenomenon by generating hypotheses about its boundary conditions. Contrary to this view, Nak et al. (2026) argued that “trying to explain something before knowing whether it actually occurs seems to be a futile exercise” (p. 17). In my view, however, trying to explain a phenomenon before we know whether it exists can help to determine what to look for and where to look for it.

Conversely, excluding explanatory theory when establishing phenomena can contribute to naïve empiricism (Strong, 1991). As van Rooij and Baggio (2021) explained:

Psychological scientists may believe in the need to build large collections of robust, replicable, uncontested effects before even thinking about starting to build theories. The hope is that, by collecting many reliable effects, the empirical foundations are laid on which to build theories... As reasonable as this seems, without a prior theoretical framework to guide the way, collected effects are unlikely to add up and contribute to the growth of knowledge (Anderson, 1990; Cummins, 2000; Newell, 1973). (p. 683)

Hence, without the guiding influence of explanatory theory, attempts to establish phenomena may become directionless, inefficient, and tend towards naïve empiricism (Popper, 2002, p. 88; Strong, 1991).

Phenomenon-first advocates may rightly object that they are not ignoring explanatory theories, they are only delaying their consideration until the theory-testing stage. However, as described above, tests of explanatory theories appear to play a key role in establishing phenomena, and so actively delaying their consideration may lead to a suboptimal establishment process.

Naïve Methodological Falsificationism

Excluding explanatory theories when establishing phenomena represents one form of theory aversion. A second form may involve shielding interpretative theories from criticism when testing explanatory theories. This latter form of theory aversion is symptomatic of what Lakatos (1978) described as naïve methodological falsificationism (p. 42).

Naïve falsificationism prevents criticism of interpretative theories by treating them as “unproblematic” during tests of explanatory theories (Lakatos, 1978, p. 23). This approach allows the explanatory theory to be forced into logical isolation and logical refutation during a theory test (Lakatos, 1978, p. 32). Hence, naïve falsificationism represents a form of theory aversion because it treats interpretative theories “not as theories under test but as unproblematic background knowledge” (Lakatos, 1978, p. 23, italics in original). In contrast, sophisticated methodological falsificationism acknowledges the perpetual fallibility of interpretative theories and, consequently, only results in “anomalies” rather than full-blown logical “refutations” of explanatory theories (Lakatos, 1978, p. 4).

Importantly, unlike naïve falsificationism, sophisticated falsificationism encourages an “appeal procedure” against putative refutations of explanatory theories (Lakatos, 1978, p. 42). During this procedure, a test’s interpretative theory may be specified, criticized, falsified, and replaced. Hence, what may be initially presented as the refutation of an explanatory theory can be challenged and reinterpreted as the refutation of the test’s interpretative theory. As Lakatos (1978) explained:

The theoretician may demand that the experimentalist specify [their] ‘interpretative theory’ and [they] may then replace it - to the experimentalist’s annoyance - by a better one in the light of which [their] originally ‘refuted’ theory may receive positive appraisal. (p. 42)

To illustrate, when faced with a failed direct replication, sophisticated falsificationism allows the consideration of an “independently testable” hidden moderator in the test’s methodology (i.e., generating a new interpretative theory) rather than assuming that the original explanatory theory has been logically refuted (Popper, 1974, p. 986; Rubin, 2025, p. 12). In other words, a failed replication may be provisionally explained by hypothesizing important methodological differences between the original and replication study. In contrast, naïve falsificationism rejects the consideration of hidden moderators on the grounds that they “can be used by critics to question the entire enterprise of direct replication” (Zwaan et al., 2018, p. 46).

As a further illustration, sophisticated falsificationism does not treat a preregistered method as a default position against which to identify and evaluate “deviations” from that method (Rubin, 2020, p. 379). Instead, a preregistered method is regarded as being based on an interpretative theory that may be appealed against, tentatively falsified, and replaced with a better theory. From this perspective, it is the falsified interpretative theory that should be considered a “deviation” from the actual experimental situation.

In summary, psychologists may be theory averse in at least two ways. They may exclude explanatory theory when establishing phenomena, and they may temporarily treat interpretative theory as infallible when testing explanatory theories. In both cases, the problem is that researchers fail to treat relevant theories as fallible objects that are vulnerable to criticism.

Methodolatry

Cornejo and Rojas (2026) also considered the problem of methodolatry in psychology:

The peculiar way in which psychology teaches, applies, and evaluates its methodology has been diagnosed in terms of a “methodolatry” (May, 1969), “method-fetishism” (Koch, 1992a), and a “cult of empiricism” (Toulmin & Leary, 1992).

As Chamberlain (2000) explained, “methodolatry … works to prevent us looking at the assumptions behind our research” (p. 293). Hence, methodolatry deters criticism of a test’s interpretative theory during theory-testing and supports the naïve falsificationist view that a method is “unproblematic” and an appeal procedure is unnecessary.

Of course, all things being equal, rigorous methods are better than non-rigorous methods. In particular, methodological rigor can tentatively reduce what Gustafson and Rice (2020) described as technical uncertainty based on measurement error, modeling approximations, statistical assumptions, and so on. In turn, reductions in technical uncertainty may contribute to reductions in consensus uncertainty (Gustafson & Rice, 2020) about the status of a theoretical explanation. Methodological rigor may even take account of deficient uncertainty or “known unknowns”—the things we know we don’t yet know (Gustafson & Rice, 2020) or what Merton (1987) called “specified ignorance” (i.e., specified but as yet untested hypotheses).

However, no amount of methodological rigor can protect against scientific uncertainty or “unknown unknowns” (Gustafson & Rice, 2020)—the things that we don’t yet know we don’t know (i.e., hypotheses that have yet to be proposed). By definition, this “unanticipated ignorance” (Rheinberger, 2026) cannot be taken into account by our current methods or theories because it can only be specified in the future when it becomes a known unknown. The danger of methodolatry is that it distracts from the fallibility of our methods and the fact that today’s rigorous method may well become tomorrow’s invalid approach.

Moreover, because unknown unknowns are only revealed as known unknowns in the future, a critique of a study’s interpretative theory is likely to be more effective in retrospect, when we have more information to hand, including any unexpected results (Rubin & Donkin, 2024, p. 2031). Hence, in my view, we should not discourage peer reviewers from “critiquing after the results are known” or CARKing (Nosek & Lakens, 2014, p. 138), despite their potentially motivated reasoning, because unexpected results may lead them to generate new, “independently testable,” auxiliary hypotheses (known unknowns) that provide a legitimate challenge to a study’s interpretative theory (Popper, 1974, p. 986; Rheinberger, 2026).

Scientism

Cornejo and Rojas (2026) argued that psychologists are motivated to engage in methodolatry in order to portray psychology as a “real” and “legitimate” science. However, this approach backfires because it results in scientism rather than science (Hayek, 1952). Haack (2012) described six signs of scientism:

1. Using the words “science,” “scientific,” “scientifically,” “scientist,” etc., honorifically, as generic terms of epistemic praise.

2. Adopting the manners, the trappings, the technical terminology, etc., of the sciences, irrespective of their real usefulness.

3. A preoccupation with demarcation, i.e., with drawing a sharp line between genuine science, the real thing, and “pseudo-scientific” imposters.

4. A corresponding preoccupation with identifying the “scientific method,” presumed to explain how the sciences have been so successful.

5. Looking to the sciences for answers to questions beyond their scope.

6. Denying or denigrating the legitimacy or the worth of other kinds of inquiry besides the scientific, or the value of human activities other than inquiry, such as poetry or art.

As Cornejo and Rojas (2026) explained:

One finds similarities between the way psychology has implemented the scientific method and the ritualized form of science called scientism (Hayek, 1952). …. Much of contemporary psychology may be not scientific, but rather scientistic. (p. 188, italics in original)

Methodolatry plays a key role in scientism by reducing the opportunity for critique and appeal:

Science is an activity that supposes and prompts the art of questioning. Yet scientism denies the fallibility of science. …. In this framework the careful replication of inherited methods is held higher than reflecting on their aptness to study the issue at hand. (Cornejo & Rojas, 2026, p. 189)

Again, Cornejo and Rojas’ (2026) depiction of scientism reminds me of naïve methodological falsificationism and its aversion to treating methods as fallible interpretative theories that are vulnerable to appeal.

From Psychology to Metascience

In my view, theory aversion, methodolatry, and scientism may have also moved from psychology into mainstream metascience given that psychologists have had a disproportionate influence in this area (Flis, 2019; Rubin, 2026).

For example, scientism may underlie the demarcationist tendencies that can be found in some methodological reforms. Hence, replicability is “seen as a bright line of demarcation that separates science from non-science” (Zwaan et al., 2018, p. 1), and “in principle, preregistration can establish a bright line between prediction and postdiction …. and preserve accurate calibration of evidence” (Nosek et al., 2018, pp. 2601-2602).

Methodolatry may also be encouraged by preregistration. In particular, preregistration may exacerbate a “researcher commitment bias” (Rubin & Donkin, 2024, p. 2027) that deters the critique of a preregistered method’s underlying interpretative theory. As Rubin and Donkin (2024) explained,

the act of preregistering a research plan may reduce researchers’ potential to deviate from that plan and increase their preference for and commitment to the plan. In turn, these commitment biases may deter researchers from testing alternative post hoc predictions and hypotheses that may provide better explanations of their observed results. (p. 2027)

Finally, some of metascience’s methodological reforms are more aligned with establishing effects than with testing theories (for an alternative view, see Lavelle, 2024, p. 657). As van Rooij and Baggio (2021, p. 683) explained:

Methodological reform so far seems to follow the tradition of focusing on establishing statistical effects, and, arguably, the reform has even been entrenching this bias.

Gal et al. (2026) agree that:

A focus on establishing effects is reflected in recommendations that call for conducting more independent replications of an effect, using large sample sizes, and requiring preregistration.

Finally, a focus on direct replications rather than conceptual replications is more consistent with establishing effects rather than testing theories (see also Crandall & Sherman, 2016).2 Consistent with this view, Nak et al. (2026) observed that:

Few of the replication studies that we know—single-study projects, many-labs, and many-studies projects—speak of reconfirming or falsifying theories based on their result (a notable exception is Coles et al., 2022). Instead, they speak of replicating or failing to replicate effects. In that sense, the turn towards open science and reproducibility already embodies a significant shift towards phenomena-oriented research, and many of the methodological innovations that have been implemented in the wake of the reproducibility crisis are best cast in that way. (p. 9)

More generally, Cornejo and Rojas’ (2026) concerns about psychology’s (a) “methodolatry,” (b) “use of statistics as a fix-all,” (c) “ontological belief that there is a ‘world of facts’ susceptible to be objectively and neutrally described,” and (d) “cult of empiricism” match some of my own concerns about mainstream metascience’s theory of science. For example, in Rubin (2023), I argued that some aspects of mainstream metascience contain elements of:

1. predictivism: the view that a priori predictions are superior to post hoc inferences (Oberauer & Lewandowsky, 2019, p. 1605; Rubin, 2017b, 2022);

2. methodolatory/methodologism: the prioritizing of methodological rigor over other research concerns, such as theory (Chamberlain, 2000; Danziger, 1990, p. 5; Gao, 2014);

3. statisticism/mathematistry: an overemphasis on statistics as both a problem and a solution in science (Boring, 1919; Brower, 1949; Fiedler, 2018; Proulx & Morey, 2021);

4. naïve empiricism (Strong, 1991): the view that science progresses through the accumulation of replicable effects (Flis, 2022; Proulx & Morey, 2021; van Rooij & Baggio, 2021);

5. naïve objectivism: the view that it is possible for scientists to adopt unbiased and objective perspectives (Field & Derksen, 2021; Penders, 2022; Wiggins & Christopherson, 2019)

Of course, not all of these issues are applicable to all of metascience, given the diversity of approaches that exist in this field. However, each issue has been raised as a potential concern in the general case (which is itself a fallible theory of modern mainstream metascience).

Conclusion

Theory aversion may pose problems in both psychology and metascience. In particular, the exclusion of explanatory theory when establishing phenomena may lead to naïve empiricism, and the shielding of interpretative theory from criticism when testing theories may lead to naïve falsificationism.

In my view, we should acknowledge the primacy of fallible theory throughout all stages of scientific investigation. Contra scientism and methodolatry, no scientific method, phenomenon, or theory can ever be sufficiently “established” to evade criticism, even temporarily.

References

Burgos, J. E. (2024). Getting ontologically serious about the replication crisis in psychology. Journal of Theoretical and Philosophical Psychology, 45(2), 79–100. https://doi.org/10.1037/teo0000281

Chamberlain, K. (2000). Methodolatry and qualitative health research. Journal of Health Psychology, 5(3), 285-296. https://doi.org/10.1177/135910530000500306

Cornejo, C., & Rojas, P. (2026). Scientism in contemporary psychology, and its underlying mythic order. In F. Hutmacher & A. N. Wendt (Eds), Theory and model building in psychology. Palgrave Macmillan. https://doi.org/10.1007/978-3-032-01453-5_9

Crandall, C. S., & Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93-99. https://doi.org/10.1016/j.jesp.2015.10.002

Flis, I. (2019). Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology, 29(2), 158-181. https://doi.org/10.1177/0959354319835322

Gal, D., Calder, B., & Sternthal, B. (2026). Three strategies for claiming research progress. Journal of Consumer Psychology, 00, 1–18. https://doi.org/10.1002/jcpy.70033

Gustafson, A., & Rice, R. E. (2020). A review of the effects of uncertainty in public science communication. Public Understanding of Science, 29(6), 614-633. https://doi.org/10.1177/0963662520942122

Haack, S. (2012). Six signs of scientism. Logos & Episteme, 3(1), 75-95. https://doi.org/10.5840/logos-episteme20123151

Haig, B. D. (2022). Understanding replication in a way that is true to science. Review of General Psychology, 26(2), 224-240. https://doi.org/10.1177/10892680211046514

Hayek, E. A. (1952). The counter-revolution of science. Free Press.

Lakatos, I. (1978). The methodology of scientific research programmes (Philosophical Papers, Volume I). Cambridge University Press.

Lavelle, J. S. (2024). Less theory, more observation: A response to psychology’s ‘theory crisis’. Philosophy, 99(4):639-665. https://doi.org/10.1017/S0031819124000202

Merton, R. K. (1987). Three fragments from a sociologist's notebooks: Establishing the phenomenon, specified ignorance, and strategic research materials. Annual Review of Sociology, 13(1), 1-29. https://doi.org/10.1146/annurev.so.13.080187.000245

Nak, J., van Dongen, N. N. N., Woodward, J. F., Scheel, A. M., Haig, B., van Bork, R., … Borsboom, D. (2026). Establishing psychological phenomena: An alternative methodological perspective on scientific practice. PsyArXiv. https://doi.org/10.31234/osf.io/rh9cu_v1

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115, 2600-2606. http://dx.doi.org/10.1073/pnas.1708274114

Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results [Editorial]. Social Psychology, 45(3), 137–141. https://doi.org/10.1027/1864-9335/a000192

Popper, K. R. (1974). Reply to my critics. In P. A. Schilpp (Ed.), The philosophy of Karl Popper (Book II) (pp. 960-1197). Open Court.

Popper, K. R. (2002). The logic of scientific discovery. Routledge.

Rao, K. R. (1981). On the question of replication. Journal of Parapsychology, 45(4), 311. https://www.proquest.com/scholarly-journals/on-question-replication/docview/1292177160/se-2

Rheinberger, H. J. (2026). Ignorance and error are at the heart of scientific knowledge production. In A. Karakas & A. T. Tuboly (Eds), Scientific mistakes and mistaken science. Springer. https://doi.org/10.1007/978-3-032-25925-7_6

Rubin, M. (2020). Does preregistration improve the credibility of research findings? The Quantitative Methods for Psychology, 16(4), 376–390. https://doi.org/10.20982/tqmp.16.4.p376

Rubin, M. (2023). Questionable metascience practices. Journal of Trial and Error, 4(1), 5–20. https://doi.org/10.36850/mr4

Rubin, M. (2025). The replication crisis is less of a “crisis” in Lakatos’ philosophy of science than it is in Popper’s. European Journal for Philosophy of Science, 15(5). https://doi.org/10.1007/s13194-024-00629-x

Rubin, M. (2026). What is critical metascience and why is it important? Frontiers in Psychology, 17:1860740. https://doi.org/10.3389/fpsyg.2026.1860740

Rubin, M., & Donkin, C. (2024). Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests. Philosophical Psychology, 37(8), 2019−2047. https://doi.org/10.1080/09515089.2022.2113771

Strong, S. R. (1991). Theory-driven science and naïve empiricism in counseling psychology. Journal of Counseling Psychology, 38(2), 204–210. https://doi.org/10.1037/0022-0167.38.2.204

van Rooij, I., & Baggio, G. (2021). Theory before the test: How to build high-verisimilitude explanatory theories in psychological science. Perspectives on Psychological Science, 16(4), 682-697. https://doi.org/10.1177/1745691620970604

Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, e120. https://doi.org/10.1017/S0140525X17001972

Citation

Subscribe now

Share

Bartosz Maćkiewicz and colleagues conducted a survey of 50 experimental philosophers in order to investigate the prevalence, causes, and potential solutions to publication bias and the file-drawer problem:

Contrary to the common assumption that journals reject null results, we found that most projects halt during early data analysis or presentation stages. The primary drivers for this attrition are resource constraints and competing priorities rather than statistical non-significance or rejection during peer review.

In their analysis, Maćkiewicz et al. found that

relatively few projects were discontinued at the submission or peer-review stages; when they were, the leading reasons concerned fit with a journal’s scope/profile and manuscript-related considerations, as well as judgments that results were unimportant. Notably, “null results” was among the least frequently selected reasons overall and was not selected at the submission or peer-review stages.

The researchers concluded that:

Overall, the results fit a multi-cause account of the file drawer problem and point to a diverse range of solutions.

Because a recurring theme is that studies remain unpublished for resource and prioritization reasons, remedies should minimize barriers.

The Cooptation of Science Reform

In my last post, I discussed a special issue on feminist metascience. One of the articles in that special issue is by Molly King and Jeff Lockhart. It looked at women’s representation and participation in open access (OA) publishing. Based on a survey of 19,462 social scientists, they found “a distinct underrepresentation of women in green OA and on SocArXiv specifically.”

In a follow-up blog post, Lockhart discussed the cooptation of science reform ideas by anti-science actors and the incentives to sensationalize metascientific work in order to attract funding:

I don’t have a solution to the cooptation of science reform by anti-science actors. More sober language than “crisis” from the reformers would be good, but it would not prevent motivated actions like we have seen in the last few years. The language of ‘crises’ and brave ‘heterodox’ outsiders challenging consensus is an effective way to draw in some philanthropists and politicians. But as any good metascientist should recognise: there are incentives here. To attract ‘metascience venture capitalist’ money or excite the current federal science leadership, it pays to make bold, sensational claims and discredit past work.

“10 Years Ago…” - The Weaponization of Transparency

Karen Levy and David Merritt Johns published a paper in 2016 titled “When open data is a Trojan Horse: The weaponization of transparency in science and governance.” As they explained:

Legislative efforts that invoke the language of data transparency can sometimes function as ‘Trojan Horses’ designed to advance goals that have little to do with good science or good governance.

Though transparency is often framed as an unalloyed good (provided that privacy interests can be adequately protected), in practice it provides a means through which diverse stakeholders attempt to achieve diverse political goals. Politically motivated proponents of transparency, in some cases, may exploit the epistemological and institutional realities that accompany the production of science and science-based policy.

What are these epistemological realities? They’re the inherent uncertainty of scientific conclusions:

Cacophony and contention are core elements of the scientific enterprise. By invoking a “narrow, idealized portrayal of science” in which research reliably produces clear and reproducible facts, the special interests behind the DAA, DQA, and SSRA mischaracterize science’s “inevitably incomplete, uncertain, contested, and … often unreliable” nature in their efforts to stymie regulatory activities with which they disagree.

Hence, bad actors may (a) mischaracterize science as producing clear and reproducible facts and then (b) weaponize transparency by using it strategically to manufacture doubt about undesirable “facts.”

Similarly, and in the same year, and warned that calls for open data may be motivated by “the aim of finding an analysis method that makes undesirable results go away,” and that “calls for more data can also be used to create the false impression that data are being withheld.” They strongly support open data but recommend protective action against the abuse of scientific transparency.

Subscribe now

Share

She is interested in advancing models and theories of the science of science. Her research brings together statistical theory, experimental methodologies, mathematical modeling, philosophy and history of science to advance an interdisciplinary understanding of the scientific process.

(1) What first got you interested in this area?

My own experiences in experimental consumer psychology during grad school and beyond. I ran so many experiments that did not work, so many replications of published experiments that failed. And I experienced so much frustration with the content of the graduate methods courses I had to take, so much self-doubt about my own perception of the quality of research getting published in our top journals. When the initial conversations about replications, p-hacking, and research quality started to pick up speed, I was immediately on board because I had so many questions of my own. Turned out my vision did not quite align with that of the science reform movement, and my research program diverged, but the fact that we were having these conversations gave me the initial push I needed pre-tenure to start formally asking the questions that were keeping me awake at night.

(2) What’s your biggest concern about mainstream metascience?

It’s difficult to rank because I have many, but my major concerns could be boiled down to one thing: let’s call it a “cart before the horse” attitude. Many metascientific claims and prescriptions about science have preceded scholarly research, and these unexamined conclusions continue to guide most of metaresearch. This is also where my own path diverged from the mainstream, because I was curious and wanted to understand the what, how, and why of scientific progress (or lack thereof) rather than believing pre-packaged answers science reform pioneers provided. Mainstream metascience has completely bought into its own hype and lost its ability to ask consequential questions and genuinely seek answers.

(3) What’s your take on the replication crisis?

It’s past time we stopped using the phrase altogether. It’s not only misguided but has started to become actively harmful to the scientific endeavor and community. The first part (misguidedness) of my sentiment is based on my own research. After a decade of studying replications and reproducibility, it would be self-defeating if I still believed that there was a replication crisis or that the label itself made any sense. For the specifics, I can refer the blog readers to my research program, starting with our 2019 PLOS ONE paper.

The second part (harms) is due to the ongoing assault on science and higher education, and external interventions in both, in the US and beyond. In the current political environment, the crisis rhetoric is being routinely weaponized as evidence that science is broken and scientists can’t be trusted. I recommend this recent post by Jeff Lockhart as an insightful read.

(4) How do you view open science?

It depends on what is included under the umbrella. There are many definitions and even more interpretations. In general, openness, transparency, and honesty are important and uncontroversial scientific virtues that we should cultivate and strive to uphold. But a virtue is not a standard. I think every field needs to determine the boundaries of what open science means within that context, carefully evaluate who bears the costs and what potential harms need to be mitigated. Coming up with universal rules, checklists, or impositions that apply to all of science is futile, and essentially authoritarian. I am also not a fan of treating arbitrary open science criteria as a purity test. Quick and intentionally broad answer because I don’t think it’s mine to settle: open science is an open discussion about how we can open the boundaries and gates of science to internal and external audiences.

(5) What’s one thing you’d change about the way we do research?

Not quite about how we do research but something more systemic: take the prestige economy and citation metrics out of the equation altogether. Perhaps I’d want to create more resources, distribute them more equitably, take publication pressures away, and provide the support people need to do the best research they can. But I fear all of that might require a magic wand, and if I had it, I would hesitate to use it. It’s not that I don’t believe we can change the current system, but in all likelihood, my well-intentioned attempt would end up replacing it with something problematic in different ways; perhaps a new kind of hierarchy that’s just as bad if not worse. And I am not being cynical. I just do not trust that external, top-down interventions in complex social systems have straightforward, positive consequences. I’d rather change how I do research (as I have).

(6) Which one of your papers should we read first and why?

Our first metascience paper in PLOS ONE (mentioned earlier) Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity because I genuinely believe it is fantastic. It also exemplifies how my path diverged from mainstream metascience. The modeling approach is very novel and thorough. It’s exploratory, interdisciplinary, ambitious yet humble, insightful, and rigorous. If we had a certain type of street/career smarts, I’m sure we could have turned it into five different papers or sent it to a glamor journal. We were all ECRs, simply extremely excited about the research and the collaboration, had open minds and no assumptions about what we should find. We did not treat modeling as a vehicle to get to the answers we needed but as a challenge that compelled us. So the paper is curiosity-driven and is based on years of hard work. I love what we did every time I reread it.

(7) What’s a common misconception about your work you’d like to correct?

I don’t have a good understanding of how my work is perceived to be able to identify misconceptions. I do know how it gets cited, however, and it looks like most colleagues don’t read past the abstract or the conclusion. Just wish more readers were interested in the modeling and the theoretical contributions as ends in themselves instead of citing (or criticizing) the work for soundbites. Most reviews and comments we get critique the implications of our conclusions as if that’s something we should have planned for before doing the research. It’s all backwards. Some readers may disagree with the conclusions that follow from our work, but that’s what we found, and pushing back is not going to change the evidence. Let’s focus on what we can learn from it.

(8) What are you working on at the moment?

Currently finalizing our latest work The difference between “replicable” and “not replicable” is not itself scientifically replicable before submission. Luckily, we got some of the good (genuinely useful) kind of informal feedback on the preprint, raising valid questions about our model. It’s helped us clarify some misunderstandings, improve a few weaknesses, and hopefully present the theory more clearly. After that, I hope to move away from replications and reproducibility for a while since we have done quite a lot in this domain and made a lot of theoretical progress. Hope to revisit our original model in the PLOS ONE paper mentioned earlier and expand it to study other aspects of the scientific process. I have a lot of theoretical questions, all of which require translating theory and philosophy to math, which requires a lot of brain juice.

(9) What advice do you have for researchers starting out in your area?

I’m not good with advice. Since I don’t like taking it, it’s only fair that I don’t offer it either. Not that I have an area to speak of. What do I call it? Theoretical metascience? Still a long way from becoming an area.

(10) What question should I have asked you, and what’s your answer?

Q: In your research program, what has benefited you most?

A: Reading outside of my area (such as it is) and collaborating with colleagues from diverse backgrounds and disciplines. In the last decade, I have read anything I found interesting, setting all pragmatic considerations and disciplinary boundaries aside. I find myself reading books and papers in statistics, mathematics, philosophy of science, history of science, science and technology studies, psychology, sociology, computer science, economics, political science — you name it. Qualitative, quantitative, empirical, theoretical… I was also lucky enough to interact and collaborate with scholars from many fields. I cannot imagine doing the work of the past decade without such cross-pollination. Allowing myself to feel professionally homeless (without a proper base discipline) has set me free to do the kind of work that compels and excites me, and that I find internally rewarding.

Subscribe now

Share

In their Introduction to the special issue, Christin Munsch and Daisy Verduzco Reyes define feminist metascience:

Gender organizes social life, is historically and socially produced, and operates in relation to other axes of domination like race, class, sexuality, nationality, coloniality, and disability. Broadly speaking, then, we might define feminist metascience as an approach to the study of science that brings this orientation to bear on how knowledge is produced, evaluated, circulated, and corrected.

They also define feminist open science:

If open science is committed to transparency, sharing, and reproducibility, a feminist open science necessitates asking how those commitments can be pursued in ways that also attend to power, inequality, care, and the unequal conditions under which knowledge is produced. In this sense, feminist open science might be defined as an approach to open science practice that seeks to advance transparency and accountability while remaining attentive to the ethical, political, and epistemic consequences of openness itself. Such an approach treats openness not as an absolute good or fixed checklist, but as a situated set of practices whose value and risks must be assessed in relation to specific communities, methods, and institutional conditions.

They then guide us through the articles in the special issue. As they explain:

We open the issue with Madeleine Pownall’s article, “Women as the First Open Scientists.” Much like Morris’s (2015) landmark account of how Du Bois’s foundational contributions to sociology were obscured by racism, Pownall recovers five pioneering women in the field of psychology whose work laid important foundations for what are now framed as contemporary open science reforms. By documenting women’s early contributions to practices such as replication, analytical precision, and conflict-of-interest disclosure, she challenges narratives that cast openness and transparency as recent innovations.

The next two articles examine specific professional practices, drawing attention to the ways everyday scholarly activities are themselves gendered. For example, Jina Lee’s article (“Claiming Novelty, Claiming Authority”) shows that scientific authority is performed through routine rhetorical practices. Specifically, Lee finds that the use of novelty claims—rhetorical assertions that frame research as groundbreaking—vary by gender and disciplinary context. In science, technology, engineering, and mathematics (STEM) fields, which possess a high level of paradigmatic consensus, professional success is narrowly tied to assertive, masculine-coded self-promotion, creating a double bind for women that results in fewer novelty claims and, in turn, fewer downstream citations. Conversely, the humanities and social sciences offer more varied pathways to recognition, which appear to mitigate these constraints.

Likewise, in “Sharing Knowledge Openly,” Molly King and Jeffrey Lockhart examine open access and repository participation not as straightforward indicators of scholarly virtue, but as practices embedded in unequal institutional and interpersonal conditions. They find that women and scholars of color are underrepresented in “green” open access publishing—the practice of self-posting alternate versions of papers online for free—and situate these patterns within “bropen science” culture, differential exposure to harassment, unequal access to resources, and uneven distributions of time and labor.

The final set of articles concern the spread of ideas both in and outside of the academy. Laura K. Nelson and colleagues (“What Gets Lost in Translation?”) develop a feminist-informed metascience framework showing that the success of academic ideas depends not only on who produces them, but also on how those ideas are translated. Using publications tied to the NSF ADVANCE program, the authors propose a creation-diffusion model demonstrating that context, author expertise, and the substantive content of ideas shape whether ideas are taken up and spread. Ideas framed in more institutionally palatable ways diffuse more widely, whereas sharper feminist critiques of structure and inequality are more likely to be resisted, softened, or left behind.

Similarly, Jennifer Lee (“Who Receives Scientific Deference?”) examines what happens when scholarly ideas enter public discourse. Through an examination of media representations of women and men social scientists, she shows that the public reception of expertise is filtered through gendered cultural schemas. Women’s work is often trivialized, politicized, or recoded as popular writing, whereas men’s work is more readily framed as rigorous, objective, and properly scientific. In positioning these articles at the end of the issue, we emphasize that the fate of ideas depends not only on their creation and dissemination, but also on their visibility and diffusion. For this reason, we call for a broader conception of metascience that attends to the processes through which scientific findings are legitimated, undermined, or distorted.

The editors conclude that:

Science requires more than technical reform. It requires metascientific work and open science initiatives capable of reckoning with the gendered and intersectional conditions that shape every aspect of knowledge production itself.

Subscribe now

Share

In her new paper, Sabina Leonelli discusses three forms of structural injustice associated with disparities in scientific research: (1) resource injustice, (2) focus injustice, and (3) labour injustice.

(1) resource injustice [is] grounded on inequity in resourcing research in terms of both human labour and access to relevant tools and infrastructures;

(2) focus injustice [is] grounded on a misalignment between resourcing and the goals that researchers would like to pursue, leading to a redirection on the focus of the research; and

(3) labour injustice [is] grounded on a misalignment between scientific goals and labour conditions, again with repercussions on the choices and methods used for research.

Leonelli (2026) links all three types of structural injustice to open science and argues that they may result in:

ineffective openness, with open science policies and related tools being implemented unsystematically and in ways that discriminate against low-resourced environments, and thus without necessarily increasing the visibility and findability of research carried out in the majority world.

Similar issues have been raised in recent work by Batool Almarzouq (2026), Helen Longino (2026), and Raffaela Kunz (2026), which I covered a few weeks ago.

(Re)Conceptualizing the “Openness” of Open Science

Leonelli (2026) also unpacks her conceptualization of “openness,” which has implications for the ways in which we understand and implement open science. She has previously conceived openness as “judicious connections” with other researchers, rather than as merely sharing research objects such as data and materials (for a summary, see my 2023 post here). As she explained in her 2023 book:

The idea of openness is quintessentially linked to that of learning as going beyond one’s boundaries. This arguably applies to systems of research practice as much as to research groups and individual learners: establishing new connections often means expanding one’s learning, challenging existing assumptions around what is considered external or irrelevant to a given system, and considering whether new boundaries need to be established that incorporate the novel relations.

Establishing new connections with relevant individuals, groups, and systems allow us to see things from different perspectives and identify and interrogate previously unrecognized assumptions. This approach aligns with the view that there is always room for improvement in our conceptual and methodological approach and that others may spot issues we’ve missed.

Leonelli’s (2026) most recent piece reinforces this view by arguing that an “open” science must be a “vulnerable” and incomplete science:

Science does not progress unless researchers look for new ways of knowing, which implies an acknowledgement that current knowledge, methods and instrument are not infallible and can be improved to achieve better results in the future. This is what I take as the most significant meaning of ‘openness’ in research: the awareness that knowledge is never complete and definitive, but rather always exposed to improvement and failure, as well as the challenge to comprehend and interact with an ever-changing world. In this processual understanding of science, scientific methods, norms and institutions cannot ensure 100% safety, trustworthiness and reliability; and fallibility, need and weaknesses (in conceptual as well as material and social resources) are part and parcel of the research effort. Research thus needs to start from an admission of vulnerability, defined in a relational sense as being subjected to one’s environment in ways that limit one’s control.

According to this conceptualization of openness, a genuinely “open” science is one that is “never complete” and always searching for (“open to”) new perspectives and approaches. In social psychological terms, open scientists should have a low need for closure in order to prevent them from seizing and freezing on particular approaches and conclusions (Kruglanski & Webster, 1996).

Scientific “vulnerability” means that we need to concede our limited and incomplete perspectives and accept that we need to seek assistance from others in different environments in order to improve our approach. Hence, open scientists must be open minded to others’ views, approaches, and ways of knowing, and social “connections” are means of accessing these epistemic resources. As Leonelli (2026) explains:

This in turn requires placing social relations and social agency at the heart of scientific practice, emphasising the significance of forging meaningful connections among the humans involved in research efforts – thus facilitating dialogue and exercising judgement about whose needs to prioritise, for which reasons and according to which values – over the quest for efficient technical means to exchange information (Leonelli 2023).

Leonelli’s (2023, 2026) call to forge “meaningful connections” reminds me of Michela Massimi’s (2022) work on perspectival realism:

Making the most of epistemological pluralism in practice involves more than just ensuring that a plurality of epistemologies coexist but self-conscious efforts at dialogue and collaboration across divides.

From this perspective, an attitude of scientific vulnerability and exposure to epistemic diversity should encourage meaningful social connections that lead to higher quality and more transparent research, rather than vice versa.

In particular, connecting with researchers who have different approaches should make us question our own approach and motivate a sort of “robustness analysis” across different approaches (see also Massimi, 2022). In this sense, scientific “rigor” and “credibility” come from not only adhering to the methodological norms within our own epistemic communities but also from doubting those norms and exploring beyond our scientific ingroup to seek out the views of other epistemic communities.

Special Issue on Open Science in Social Epistemology

Leonelli’s (2026) article is part of a special issue of the journal Social Epistemology that she edited with Rena Alcalay and Paola Castano as a key output of her European Research Council project on the Philosophy of Open Science for Diverse Research Environments. The special issue will be out as an integrated collection in September.

The full list of articles is as follows:

• Ación, L., Ascenzi, L., Calero, A. D., Castaño, P., Formoso, J., Loto, P. A., … Xhardez, V. (2026). More than a NASA badge: Metadocencia and capacity building for open science communities in Latin America. Social Epistemology. https://doi.org/10.1080/02691728.2026.2669558

• Alcalay, R., & Vandekerckhove, J. (2026). Epistemic disadvantage in open science practices. Social Epistemology. https://doi.org/10.1080/02691728.2026.2669557

• Ankeny, R. A. (2026). How has ‘opening up’ science changed scientific practices? Social Epistemology. https://doi.org/10.1080/02691728.2026.2669568

• Lee, C. J. (2026). Critically engaged pragmatism: Scientific norm and social, pragmatist epistemology for AI science evaluation tools. Social Epistemology. https://doi.org/10.1080/02691728.2026.2669578

• Leonelli, S. (2026): Not all research environments are created equal: Structural injustice in low-resourced scientific settings and its epistemic implications. Social Epistemology. https://doi.org/10.1080/02691728.2026.2682149

• Longino, H. E. (2026): Knowledge and politics: Reflections on challenges to open science. Social Epistemology. https://doi.org/10.1080/02691728.2026.2669576

• Sheehan, N. (2026). The sequence and the standard: An account of participatory informational format exclusion (PIFE). Social Epistemology. https://doi.org/10.1080/02691728.2026.2682153

• Trappes, R. (2026). Global citizen science as a source of epistemic diversity: The case of eBird India. Social Epistemology. https://doi.org/10.1080/02691728.2026.2669567

• Wieber, F., & Hocquet, A. (2026). A portrait of the scientist as a user: Openness and temporality of scientific software. Social Epistemology. https://doi.org/10.1080/02691728.2026.2669573

  

The Munich Manifesto for Equitable Open Research

Leonelli and others’ work in this area has also recently inspired the Munich Manifesto for Equitable Open Research. A few extracts from the Manifesto follow:

National and international policies tend to implement OR [open research] guidelines, tools and principles in a top-down manner and across domains, with scarce consideration of the diverse capacities, motivations and methods characterising different research communities. Typically, OR tools are produced by the best-resourced research environments, which thereby set the goals that those tools are meant to serve, and define what counts as relevant standards and methods. In consequence, already advantaged contributors end up benefiting disproportionally from OR, which may in turn widen disparities in the research landscape, making some forms of expertise more visible than others regardless of their scientific and societal value.

This has significant consequences for the content and quality of research. Well-documented forms of discrimination and disparities in resources affect which voices are heard, which findings are pursued and recognised, and which methods are considered as best practice. Such divides are compounded by for-profit exploitation of OR initiatives that negatively affect the trustworthiness and sustainability of OR (such as the monetisation of open access publishing or the AI-driven scraping of data shared online), and the failure to reward commercial initiatives committed to enhancing equity within the global research community.

The Manifesto is intended to spur research and policy action on Open Science, with the goal of supporting reliable, responsible, and equitable research worldwide.

You can read and sign the Manifesto here.

I am grateful to Sabina Leonelli for her feedback and suggestions on a draft of this post.

Subscribe now

Share

Rena Alcalay and Joachim Vandekerckhove discuss “epistemic disadvantage in open science practices.” Following Davis-Stober et al. (2025), they distinguish between two attitudes to the scientific literature: the Book of Truths (BoT) and the Book of Conversations (BoC):

Those holding a BoT attitude see the primary purpose of the academic literature as creating a definitive, reliable repository of established knowledge, a canon that can be assessed and trusted at face value by expert and lay readers alike. By contrast, those adopting a BoC attitude view academic literature primarily as an ongoing exchange of ideas and communication among researchers, valuing provisional findings, preliminary data, and emergent questions as inherently worthwhile contributions to cumulative science, regardless of whether they achieve formal publication or definitive verification.

These divergent attitudes imply fundamentally different thresholds for what counts as acceptable epistemic risk in data sharing. Where a BoC attitude treats intermediate, unvetted data as intrinsically valuable, a BoT attitude worries that sharing raw findings risks misleading the scientific community or distorting the epistemic record. What appears to one researcher as reckless exposure to another appears as essential democratisation. This attitudinal divide, we argue, underlies many tensions in contemporary OS [open science] practice.

Metascience’s Directionality

Andy Stirling wrote a Substack post on the way in which metascience addresses uncertainty, directionality, and democracy. I’ll pull out three paragraphs on directionality here, but the whole post is worth a read:

We often treat key foci of metascience as if they are simply about magnitudes, whose main relevance is whether they grow or shrink in scale – more or less. This is true, for instance, of ‘excellence’, ‘novelty’, ‘impact’, ‘productivity’, ‘pace’ – or ‘winning’ or ‘losing’ in some particular research or innovation ‘race’. All are treated as if they are simple self-evident magnitudes – independent of any judgement about how to measure, by what scales, aggregated how and with what relative weightings?

In other words, these major subjects of metascience are treated as ‘scalars’ (simple single numbers) rather than as what they really are: vectors (with multiple constituting dimensions that give them directions as well as scale). All the above values, for example, rest not just on ‘how much?’, ‘how fast?’ or ‘who wins?’ but on fundamentally qualitative alternative directions for possible change. Inconvenient to interests that wish to conceal awkward resulting questions, a crucial truth in metascience is that the most defining properties of any given field of research or innovation are given not simply by associated magnitudes, but by their directions.

This suggests a subtle but profound shift for metascience. Evaluation should become less concerned with demonstrating what one particular pathway can achieve, and more concerned with comparing alternative pathways. Objectivity lies not in presenting a single notionally authoritative picture, but in exploring systematically how different assumptions, values and priorities generate different pictures.

Women as the First Open Scientists

Madeleine Pownall published an article celebrating the historical contributions of women as champions of open science principles:

Contemporary narratives often frame Open Science and metascience as novel frontiers of transparency and rigor, obscuring their deep roots in feminist methodologies and critical social science traditions. Rather than asking how feminist and qualitative research can conform to Open Science standards, we should recenter the values of reflexivity, care, inclusivity, and accountability long championed by feminist scholars (Siegel et al. 2021). The methodological critiques and calls for accountability articulated by earlier women scholars anticipated many reforms now gathered under Open Science; had these insights been integrated, the severity of today’s “crisis” might have been reduced.

“10 Years Ago…” - A Comment on Confirmatory vs. Exploratory Research

Back in 2016, Hans Phaf addressed the distinction between exploratory and confirmatory research offered by Wagenmakers et al. (2012):

The primary aim of this comment is to juxtapose the statistically oriented approach and a more theoretically oriented approach. The statistical approach of Wagenmakers et al. (2012) entails a two-way classification in either exploratory or hypothesis-confirmatory research. The latter type can only have a binary outcome with respect to the decision being made, the hypothesis is either confirmed or not. To arrive at such an outcome, a replication attempt must rely on the original research having uncovered and made explicit all relevant processes (i.e., an exhaustive theoretical analysis). All other types of research fall in the exploratory category, even when they further develop the theory starting from quite specific hypotheses. Merely confirming preregistered hypotheses has, however, never yielded new hypotheses, whereas unexpected findings stimulating further investigations do have that capacity and may even be the royal road to scientific innovation (e.g., Lehrer, 2009). Calling it undirected exploratory research, moreover, also does not do justice to the gradual progress-by-adjustment type of research (cf, Lakatos, 1970). The latter type of research is often guided by well-specified and concrete process hypotheses, which may be far superior above merely expecting a difference.

Subscribe now

Share

Hartmut Blank and colleagues ask whether preregistration protects against the hindsight bias, in which researchers perceive their results as more predictable after they are known.

As Blank et al. (2026) explained:

The idea that there is hindsight bias for one’s own research predictions and that preregistration is needed to protect researchers from it has been taken for granted and has become part of the Open Science folklore.

For example, in their article on “the preregistration revolution,” Brian Nosek et al. (2018) explained that:

Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration.

Contrary to this reasoning, Blank et al. found that preregistration was unrelated to self-reported post hoc expectedness of research outcomes among 351 conference poster presenters (r = −.01). They concluded that:

It will come as a surprise to many (including ourselves, initially) that the first empirical test of this idea failed to show any support for it. In hindsight, and upon a closer look at the hindsight bias literature, it seems that the generality of the hindsight bias phenomenon has been overestimated (a typical research bias in itself; see, e.g., Simons et al., 2017).

The lack of support in our study for a preregistration-hindsight bias link remains intriguing, and this adds to recent discussions around benefits but also costs of preregistration (e.g., Lakens, 2019; Navarro, 2020; Nosek et al., 2019; Rubin, 2022; Shiffrin et al., 2018; Szollosi et al., 2020).

Postdoc Position on Science Reform

There’s a postdoc position available working with Prof Jesper Wiborg Schneider at Aarhus University, Demark, as part of the research project “Reforming science: Investigating the reflexivity and reflectivity of (non)academic actors advocating for scientific reforms.”

The project examines contemporary science reform movements and communities associated with, for example, Open Science, metascience, reproducibility, research integrity, research assessment and related reform agendas….The position will involve a combination of quantitative and qualitative research tasks, including bibliometric analysis, social network analysis, survey design and analysis, citation-context analysis, and historically informed comparative analysis of science reforms.

The postdoc is expected to contribute to:

• Mapping the social and intellectual structures of science reform communities

• Designing and analysing a survey of actors involved in science reform communities

• Conducting citation-context analyses to examine how key science reform publications are cited and used across research fields

• Contributing to a historical comparison of current science reforms with earlier episodes of reform in science

Applications are due by 10th August 2026. For further information, please see here.

Open Science’s Marginalization of Qualitative Epistemologies

In an editorial in Qualitative Psychology, Heidi Levitt considers the “practice of assigning badges for articles in journals that adhere to the values of open science, irrespective of their methods or epistemological perspectives.”

Although many virtues of open science exist, I question practices of open science that may support some epistemological approaches to science while devaluing or compromising others. Specifically, I am concerned that unreflective practices marginalize the historical practices endemic to qualitative methods that already serve open science goals and strengthen methodological rigor.

For example, Levitt addresses the issue of preregistering qualitative research:

Constructivist and critical epistemic perspectives dominate among qualitative psychologists (Grzanka & Moradi, 2021; Kovács et al., 2019; Rizo et al., 2025) and these researchers tend to view their analytic processes as locally bound and interpretively based.

Just because it is now possible to register qualitative studies does not mean that this process holds value for qualitative researchers or serves their goals then.

Following Andrea Bazzoli’s (2022) commentary on “open science and epistemic pluralism,” I’ve raised similar concerns about the epistemic encroachment of open science’s postpositivist principles on qualitative researchers here:

“10 Years Ago…” - Replication Failures are Good for Science

is the author of the 2012 book “Ignorance: How it drives science.” In 2016, he wrote an inspiring op ed in the LA Times about the importance of replication failures to scientific discovery. It ends as follows:

Science would be in a crisis if it weren’t failing most of the time. Science is full of wrong turns, unconsidered outcomes, omissions and, of course, occasional facts. Replication is part of that process, as open to failure as any other step. The mistake is to think that any published paper or journal article is the end of the story and a statement of incontrovertible truth. It is a progress report. Don’t be fooled.

Subscribe now

Share

Jenni Adams, and colleagues put together a zine on feminist open science, including some quotes from Elizabeth Bennett’s (2021) article: “Open Science From a Qualitative, Feminist Perspective.”

My favourite quote from Bennett (2021) is:

From which position(s) is this [open science] framework speaking? Where is it located in a hierarchy of research power and epistemological dominance?

Open Science in an Unequal World

Batool Almarzouq (2026) considered how the open science movement’s promise to return science to the public is not being realised with respect to researchers from the Global South.

The movement behind Open Science emerged with noble aims and genuine intentions but has evolved into a flawed reality. The goal of open science is not openness itself; it’s about redistributing power in knowledge production—challenging the extractive systems that enrich the Global North at the expense of the Global South. Real change requires actively divesting from and delinking with the economic engines that Open Science systematically ignores or even relies upon. To fail in this is to betray everything the movement claims to stand for.

Helen Longino (2026) recently made a similar point:

In spite of conceptual commitment to the openness of science, disadvantaged regions have less access (as contributors to or readers of) international publications, less access to networks of communication, fewer material resources with which to conduct research, and less input to the international scientific research agenda. Furthermore, their economic and political weakness leaves them vulnerable to extractive practices, whether of natural resources or of local knowledge.

These international imbalances between the highly resourced (dare we say over-resourced?) and the under-resourced are the context in which openness of science would have to be implemented.

The Instrumentalization of Open Science

Raffaela Kunz (2026)

traces the evolution of open science from a grassroots movement for knowledge democratization into a policy-driven and increasingly contested governance framework. [She] then examines how constitutional law should respond to the diminishing autonomy of science, arguing that traditional legal protections centered on individual academic freedom are inadequate to address systemic encroachments.

Paralleling some of Almarzouq’s (2026) points about open science in an unequal world, Kunz argues that:

What began as a bottom-up movement rooted in the ideals of knowledge democratization has evolved into a policy-driven framework increasingly shaped by broader political and economic agendas. In this process, open science—more broadly understood as open scholarship—has not only lost some of its critical edge, but has also come to reflect, and at times reinforce, the very dynamics it initially sought to resist.

Open science is increasingly losing its liberating and democratizing potential, as it becomes entangled with the interconnected and mutually reinforcing logics of commercialization, political instrumentalization, and media-driven visibility of science. Rather than challenging the dominance of commercial publishers and reinforcing the public nature of knowledge, open science policies have often sustained or even deepened existing market structures, increasing academia’s dependence on private infrastructures.

The original promise of open science—to enhance accessibility, foster collaboration, and strengthen scientific autonomy—risks being sidelined, diluted, or co-opted by powerful institutional and corporate actors.

Metascience Heuristics in AI Review

In a new preprint, (2026) argues that we should “stop treating metascientific heuristics as quality filters in AI review”:

AI-implemented checks for reproducibility, robustness, preregistration, claim scope, and other intended proxies for scientific credibility can extend human reviewers’ capabilities. However, treating metascientific heuristics–whose theoretical grounding remains contested or incomplete–as necessary and sufficient signals for filtering out bad science is counterproductive to scientific progress. The emerging literature blurs the line between integrity filtering, based on necessary but insufficient signals of validity like reproducibility of stated results or lack of fake citations, and epistemic filtering, which uses machine-detectable signals to judge scientific quality. Drawing on critical metascience, we show that commonly proposed signals of research quality are insufficiently justified as general indicators of scientific value.

Hullman wrote a Substack post about her new preprint here:

Critical Metascience - EASP Symposium

We had a great symposium at the General Meeting of the European Association of Social Psychology yesterday titled: “Who critiques the critique? Toward a reflexive metascience.”

I began by asking “what is critical metascience and why is it important?” I argued that critical metascience’s “external criticism” can help to highlight collective biases that occur within mainstream metascience. A copy of my slides can be found here.

Rizqy Zein and Mario Gollwitzer then considered “the role of context in explaining effect size heterogeneity in replication studies.” They distinguished between “conceptually relevant” and “conceptually irrelevant” variables and found evidence that “conceptually relevant context characteristics explain effect heterogeneity in the effects found by Shnabel and Nadler (2008).” Hence, “context does matter!” A copy of their slides can be found here.

Finally, Fabrice Gabarrot asked “Why do we need critical meta-science? From normalizing to problematizing the replication crisis.” He argued that the three crises in psychology (early 1900s, 1960s-70s, & 2010s) produced the same “immune response” of redefining “rigor” and excluding non-congruent work. He distinguished between “normative problematization” (identify a malfunction, propose a procedural fix, reconduct the frame) and “reflexive problematization” (interrogate the frame itself). He argued that critical metascience can assist with the reflexive approach. A copy of his slides can be found here.

Critical Metascience - New Article

Related to my EASP presentation, I published a new article today that asks: “What is critical metascience and why is it important?”

I define critical metascience as a multidisciplinary research area that takes a step back to question some of metascience’s commonly accepted assumptions, methods, problems, and solutions …. In the current article, I aim to raise the profile of this research area and offer a rationale for viewing it as a distinct field of multidisciplinary inquiry.

In my article, I:

• define critical metascience;

• explain why it’s important;

• address some potential concerns about it;

• consider some emerging themes in the area; and

• evaluate its potential relationships with metascience.

I conclude by reflecting on Popper’s (1976) point that “the objectivity of science is … [the result] of the friendly-hostile division of labour among scientists” (p. 95). I argue that:

Critical metascience can play an important part in Popper’s “friendly-hostile division of labour” by offering a special type of external criticism that exposes and challenges potential biases and dogma in metascience in order to improve its objectivity and approach.

“10 Years Ago…” - 2015 Replication Rates

and colleagues (2016) published a comment on the Open Science Collaboration’s (OSC; 2015) “Estimating the reproducibility of psychological science.” They concluded that:

Metascience is not exempt from the rules of science. OSC used a benchmark that did not take into account the multiple sources of error in their data, used a relatively low-powered design that demonstrably underestimates the true rate of replication, and permitted considerable infidelities that almost certainly biased their replication studies toward failure. As a result, OSC seriously underestimated the reproducibility of psychological science.

Consistent with Gilbert et al.’s (2016) view, reviews of more recent well-powered, high quality replication studies have found much higher replication rates than the OSC’s original estimate of 36%. For example, Nosek et el.’s (2022) review of 307 replication studies found a replication rate of 64%, Cobey et al.’s (2023) review of 116 replication studies found a replication rate of 71%, and Tyner et al.’s (2026) test of 274 claims found a replication rate of 55%. Based on these more recent estimates, the observed replication rate appears to be around 63% rather than 36%.

Is a 63% replication rate grounds for a “crisis”? As I’ve discussed recently, it depends on what you expect the replication rate to be. Glibert et al. also raised concerns about the “correct benchmark” in their follow-up article, arguing against a purely statistical interpretation based on random sampling error per se:

The correct benchmark must recognize all genuine sources of variability and cannot be limited to the fiction that the only way in which the replications differed from the originals was that they drew different subjects from the identical population.

A key challenge for future work which claims to have found “low” replication rates is to explain and justify an “acceptable,” “optimal,” or “expected” replication rate (see also Rubin, 2023).

Subscribe now

Share

Over on Never Met a Science, discusses his new preprint on “Contained and Uncontained Experiments.”

Contained experiments are those for which it is plausible to “perform the same procedure” and expect “the same result.” Uncontained experiments are those for which practitioners expect intrinsic variation to dominate; even when performing the same procedure, the goal is to understand how results differ across time and context. I propose three conceptions of containedness: pragmatic (community expectations about replication), complexity (information required to specify the procedure), and temporal (speed of experimentation relative to drift in the phenomenon).

I was particularly interested in the pragmatic conception.

The pragmatic conception depends on the extent to which the conductors of an experiment and the scientific community with whom they interact expect the experiment to be able to be replicated. If a “contained” experiment is replicated and different outcome results, the community takes this as strong evidence for a problem. If the same occurs for an “uncontained” experiment, this is taken as evidence for some important variation in the context or implementation of the two experiments.

As Munger (2026) noted, Feest (2024) has made a related distinction between “effect seekers” and “context mongers.” This distinction is important because it predicts

different flavors of solution to the replication crisis. The former are at least nominally interested in behavioralist stimulus-response patterns: these are the “effects” that they seek to produce and, then, re-produce when replicating a procedure. Their prescriptions tend to be methodological, premised on the idea that failures to re-produce these effects are caused by some combination of statistical and operational procedures. The “context mongers,” in contrast, argue that this psychology research is undertheorized and that replication failures stem from an incomplete account of all of the dimensions of complexity involved in conducting those experiments.

Munger asks “why do different scientific communities have different pragmatic conceptions of containedness?” It’s an important question that deserves more attention!

Reproducibility Crisis?

, Elizabeth Campolongo, and Phillip Popovich took part in a panel discussion at the Ohio State Center for Ethics and Human Values in which they responded to the question: “Is There a Reproducibility Crisis in Science?”, focusing in particular on ethical and practical issues.

Popovich: “Let’s face it: Science papers are stories, and the story rarely occurs in the order in which it is presented.”

Devezer: On Bem’s (2011) precognition study: “We know that is not a good study. Why run it again and again and again?”

Six Types of Metascience

and discuss “The Six Camps of Metascience” over on Macroscience.

If policymakers and agency leaders are going to change how they spend billions of dollars of taxpayer money based on metascience advice, they should have a clear picture of the landscape of metascience and its various goals, perspectives, and terms.

They identify six “camps” in metascience:

There have been other approaches to illustrating the diversity of metascience. In particular, Peterson and Panofsky (2023) located metascience at the intersection of the science of science, open science, and methodological activism.

My own, admittedly biased and incomplete, taxonomy would:

• characterize the promulgation of open science as “Activist Metascience” (Peterson & Panofsky, 2021);

• describe work investigating research integrity issues as “Forensic Metascience” (e.g., Heathers, 2025);

• merge Watney and Gustetic’s categories of “innovation economics,” “R&D management,” and “institutional entrepreneurs” into a single category, perhaps called “Institutional Metascience”; and

• acknowledge four other lesser-known but nonetheless important areas:

  • “Theoretical Metascience” (e.g., Devezer, 2026; Munger, 2025)

  • “Feminist Metascience” (e.g., Cole & Gopalakrishna, 2025; Pownall, 2026)

  • “Critical Metascience” (e.g., Rubin, 2026; Ulpts & Bartscherer, 2025)

  • “Developmental Metaresearch” (e.g., Forscher & Schmidt, 2024; Schmidt, 2024)

So, I’d end up with the following…

“10 Years Ago…”

Mark Schaller published a paper on “the empirical benefits of conceptual rigor” in 2016 in which he argued that the replication crisis is partly due to researchers conceptualizing research hypotheses as personal predictions rather than theoretical predictions. Echoing Popper, he argued that:

Although scientists may subjectively experience hypotheses as their own personal predictions, a more formal approach to scientific inquiry treats scientific hypotheses as statements that have an independent logical status—independent of the scientists who articulate them and test them, and independent of the extent to which scientists personally believe them to be true or false.

Any truly rigorous approach to psychological science requires that scientific hypotheses cannot be equated to personal predictions; hypotheses must instead be articulated as depersonalized products of some systematic analysis, and appraised accordingly.

Schaller argued that conceiving hypotheses as personal predictions is likely to produce unrealistic expectations about replication rates because personal predictions are invested with self-serving biases and motivated reasoning that bias researchers towards viewing effects as less likely to be false positives and more likely to be larger and more generalizable than they really are. Hence:

If scientists can deliberately avoid thinking about hypotheses as personal predictions, then they are less likely to optimistically assume hypothesized effects to be truer, bigger, and broader than they actually are.

Subscribe now

Share

The Journal of Research on Research has recently been launched, with as the Editor-in-Chief, and Serge Horbach, Bart Penders, and as Editors.

Why call it “research on research”? In their Editorial, Derrick et al. explain that:

We have purposefully chosen a term which lays aside questions of epistemological inclusion or exclusion of contributions and instead embraces all ways of doing, knowing and understanding how research is practiced, produced and governed. The term, as opposed to adopting the more politically popular ‘meta-science’, includes perspectives from fields with long-standing traditions in the study of research: scientometrics; science and technology studies; computer science; library sciences; higher education; evaluation studies; the philosophy, sociology, anthropology; history of science, as well as, and more importantly, entire communities of metascience and meta-research.

The Politics of Open Infrastructures

Katja Mayer, Astrid Mager, and Renée Ridgway have edited a new open access book titled “The Politics of Open Infrastructures: Power, Governance, and Justice in Digital Knowledge Practices.” Substack authors include , and .

Bringing together scholars from science and technology studies, critical data studies, media studies, organisation studies, arts-based research and political sociology, this edited volume explores openness as an ongoing socio-technical process rather than a fixed ideal.

The book is associated with a companion blog here.

Over the coming months, we will invite our authors to reflect on their chapters, share additional materials and discuss the questions that emerged during the making of the book. We also want to bring in other scholars, practitioners and experts working on open knowledge infrastructures, digital commons, open source, public data systems, citizen science, cultural infrastructures and related fields.

Adaptive Preregistration

has an excellent blog post that provides a critical analysis of “adaptive preregistration” (Gould et al., 2026):

Gould et al. … say that, once you make a decision that you anticipated needing to make, or after you deviate from your preregistered plan in some way, you can file an updated preregistration. This is the ‘adaptive’ part of ‘adaptive preregistration.’ But I confess I don’t see what’s accomplished by labeling all the deviations from your preregistered plan as ‘adaptations’ and then repeatedly updating your preregistration. How is that any different from having a single preregistered plan and then describing all the deviations from the plan in text boxes in the resulting paper? Or indeed, how is it any different from not having a preregistered plan at all?

Fox also considers the evidence for preregistration more generally:

When it comes to preregistration specifically, it feels to me like the time is right for its advocates–in psychology, in ecology, and in other fields–to take a step back and pause to reflect. We now have a lot of evidence from psychology as to whether preregistration will actually take off, in a way that actually improves scientific practice rather than merely being an empty box-ticking exercise (or worse, a way to mislead readers). To my eyes at least that evidence looks quite negative.

Science Reformers

will present her work on the reflexivity and reflectivity of science reformers at 2.00pm on Wednesday 17th June as part of a Metascience Research Seminar at the University of Sussex Business School. Register for free online attendance here.

I will present my newly started research consortium “Reforming Science”. The project is looking to investigate the different actors who are currently, but also historically advocating for science reforms, how they discuss these matters publicly, how critiques are interacted with (reflexivity / reflectivity), and how they organise themselves (e.g., as a social movement / research speciality such as Metascience).

Questioning Some Metascience Assumptions

I’m grateful to for the invitation to present some of my work to the Association for Interdisciplinary Metaresearch & Open Science (AIMOS). My presentation is titled “Questioning Some Metascience Assumptions,” and it will take place online on Thursday 13th August at 7.00am British Standard Time (but also be recorded!). You can register for free here.

Metascience uses a scientific approach to understand and improve the scientific approach! In this presentation, I question some common assumptions in modern metascience. For example, I question whether exploratory research is more “tentative” than confirmatory research and whether questionable research practices such as HARKing (hypothesizing after the results are known) and undisclosed multiple testing are problematic. I discuss these and other metascience assumptions and argue that they may not be as problematic as commonly assumed.

“10 Years Ago…”

In their highly cited article (470+ citations), and Jeff Sherman considered the distinction between direct (“exact”) and conceptual replications.

In direct replications, the unit of analysis is the previously observed effect. The question to be addressed is to what extent can that effect be repeated? Such a replication cannot tell us any more about the meaning of the effect or any more about why the effect is obtained.

By contrast, ideas are the unit of analysis in conceptual replication. The question becomes not whether a specific finding may hold, but whether a theory can be retained in the face of multiple and variable tests of its hypotheses.

The authors concluded with a call for a pluralist approach that supports both direct and conceptual replications:

Some scientists want—and need—exact replications to generate confidence in their experiments. These scientists should conduct them, they should recommend publication of them in the review process, and they should read, cite, and enjoy them. But these same scientists should not require the same behaviors of others, make a narrow type of scientific practice necessary for publication, nor expect all others to share their values. We recommend the same tolerance for those who prefer conceptual replications. A lack of dissent and diversity in a scientific community is a sure prescription for lethargic progress.

Subscribe now

Share

Madeleine Pownall writes about Carolyn Wood Sherif, a pioneer of feminist psychology.

Pownall connects Wood Sherif’s ideas with her own work in the area of feminist metascience:

But while psychology is now in an era of science reform, there are two parallel conversations going on – by those who continue to insist upon reproducibility to strengthen psychological research, and those trying to reform the science as communal, compassionate and open to issues of bias.

For more work in the area of feminist metascience and its critical approach of “questioning the questions,” see:

• Pownall, M. (2025). Teaching feminist psychology epistemology, ontology, and methodology through conceptual replication research proposals. Scholarship of Teaching and Learning in Psychology, 11(3), 499–506. https://doi.org/10.1037/stl0000383

• Bennett, E. A. (2021). Open science from a qualitative, feminist perspective: Epistemological dogmas and a call for critical examination. Psychology of Women Quarterly, 45(4), 448-456. https://doi.org/10.1177/03616843211036460

• Matsick, J. L., Kruk, M., Oswald, F., & Palmer, L. (2021). Bridging feminist psychology and open science: Feminist tools and shared values inform best practices for science reform. Psychology of Women Quarterly, 45(4), 412-429. https://doi.org/10.1177/03616843211026564

• Sabik, N. J., Matsick, J. L., McCormick-Huhn, K., & Cole, E. R. (2021). Bringing an intersectional lens to “open” science: An analysis of representation in the reproducibility project. Psychology of Women Quarterly, 45(4), 475-492. https://doi.org/10.1177/03616843211035678

Risk-Averse Peer Review

Rebekka Brandt offers a nuanced view on the vexed problems of peer review, including a tilt at its reinforcement of methodologism:

Once a methodological template becomes standard, peer review tends to enforce conformity to that template rather than question its scope (Lamont, 2009; Ziman, 2000; Latour & Woolgar, 1979). Reviewers ask whether the study did the right things correctly, not whether it did the right things at all.

This produces a peculiar form of blindness. Entire research programs can become insulated from fundamental critique precisely because they are methodologically well-behaved. Over time, the distinction between “well-executed” and “well-conceived” collapses (Lamont, 2009).

In such environments, peer review functions as a disciplinary reinforcement mechanism. It polices boundaries rather than interrogating assumptions. Novelty is tolerated only if it arrives in familiar form (Ziman, 2000; Latour & Woolgar, 1979).

Brandt concludes that:

The deepest risk facing peer review today is not that it occasionally gets things wrong. It is that it mistakes procedural rigor for epistemic adequacy, and stability for truth. When that happens, science becomes excellent at reproducing what it already believes—and hesitant to confront what it does not yet understand.

Can Systems Change Theory Strengthen Metascience?

Systems change theorists use tools such as concept mapping, social network analysis, and system dynamics modelling. These tools may be useful for addressing the interconnectedness and complexity of the research ecosystem. Hence, in her recent article, Alice Gibson argues that:

Metascience aims to attend to the complex issues that arise in the research ecosystem. [However,] it has yet to adequately heed the potential of utilizing the resources available from the field of systems theory research, which arose out of a recognition of the need to attend to the challenge of complexity.

“10 Years Ago…”

In 2012, John et al. published an influential study on questionable research practices. They found that “the percentage of respondents who have engaged in questionable practices was surprisingly high” (p. 524). In response, 10 years ago, Fiedler and Schwarz (2016) published a follow-up study disputing John et al.’s conclusions. As they explained:

Claims about violations of the standards of good science deserve to be held to the high standards they endorse, not the least in light of the damage that misleading inferences can cause. Unfortunately, one of the more widely cited publications on QRPs in psychological research (John et al., 2012) suffers from ambiguities that prohibit the damning conclusions drawn.

Klaus Fiedler recently passed away at the age of 74. The European Association of Social Psychology’s In Memoriam can be found here.

Subscribe now

Share

• Average replicability rates

• Theories don’t die!

• Is the replication crisis overblown?

• Postdoctoral Fellowships

• Partial replications

Average Replicability Rates

and Erkan Buzbas published a preprint disputing the idea that a high average replication rate indicates a healthy literature and a low rate indicates a crisis. They argue that:

Aggregating replicability rates across heterogeneous literatures produces averages that conflate incommensurable experimental regimes and lack a stable scientific interpretation.

The common metascientific practice of referring to an average replicability rate for a diverse population of results (e.g., Camerer et al., 2016; Collaboration, 2015), sometimes even across disciplines (e.g., Tyner et al., 2026), is statistically fraught and misleading.

Devezer, B., & Buzbas, E. O. (2026). The difference between “replicable” and “not replicable” is not itself scientifically replicable. ArXiv. https://doi.org/10.48550/arXiv.2604.26268

Theories Don’t Die!

wrote a Substack post on the futility of trying to kill (falsify) scientific theories. He argues that, despite our best efforts, theories about power posing, ego depletion, growth mindset, stereotype threat, etc. are hard to falsify and tend to linger on in some form or another.

It might seem like the way to resolve these disputes is to weigh up all the evidence, meta-analyze all the data, deploy your p-curves, your moderator analyses, and your tests for heterogeneity, maybe even run a big, multi-lab, preregistered replication. Do all that and then we’ll finally know whether these effects are real or not!

This is a trap. We have spent the past decade doing exactly those things, and yet here we are. Clearly, no amount of data-collecting, number-crunching, or bias-correcting is going to lay these theories to rest, nor will it return them to the land of the living.

Experimental History

Nothing ever dies. It merely becomes embarrassing.

Here’s a reasonable thought: as the replication crisis has unfolded over the past 10-15 years, a bunch of psychological phenomena have been debunked and discarded forever. Power posing, ego depletion, growth mindset, stereotype threat, walking slower after reading the word “Florida…

Read more

5 months ago · 344 likes · 54 comments · Adam Mastroianni

Is the Replication Crisis Overblown?

wrote a Substack post on the dangers of making sweeping claims about a universal replication crisis:

He concludes that:

If we want to preserve confidence in science, we must speak precisely. Not all findings are equally unstable. Not all literatures are equally fragile.

Plagues, Pollution & Poverty

Is The Replication Crisis in Science Overblown?

In the early 2010s, a tremor ran through psychology. Findings that had once felt sturdy—clever experiments about priming, willpower, subtle shifts in judgment—began to wobble when other laboratories tried to reproduce them. A landmark effort by the Open Science Collaboration attempted to replicate 100 prominent studies. Many of the effects shrank. Some …

Read more

4 months ago · 25 likes · 11 comments · Bruce Lanphear

Postdoctoral Fellowships

Bart Penders posted on BlueSky that “we are joining forces with our metaresearch centre and are offering ourselves as hosts for potential MSCA [Marie Skłodowska-Curie Actions] postdoctoral fellows.”

We are especially keen on candidates and projects that span qualitative and quantitative #methods, pioneer and reflect on new dimensions of #openness and #openscience and target various conceptualisations of improvement and reform in science.

For more information about MSCA postdoctoral fellowships: https://marie-sklodowska-curie-actions.ec.europa.eu/actions/postdoctoral-fellowships

For enquiries, please contact Bart at b.penders@maastrichtuniversity.nl

Partial Replications

will present his interview research on “partial replications” at the European Association for the Study of Science and Technology’s 2026 conference.

During the interview reflections researchers state they employ “partial replications” in which they take parts of a previous study and incorporate it into their own research. They state that direct or exact replication is not worth the effort while “partial replication” allows them to partially build on previous research, while still exploring something novel and following their own interests.

Abstract of Sven’s presentation: https://nomadit.co.uk/conference/easst2026/paper/102612

Part of the Open Panel “Making Order in Science through Reform: The Politics of Replication and Research Information Infrastructures”: https://nomadit.co.uk/conference/easst2026/p/20333

More info about the EASST2026 conference: https://easst.net/conferences/easst2026/

List of Critical Metascience Articles

For an up-to-date list of research articles in this area, please visit: https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/list-of-critical-metascience-articles

Subscribe now

Share

• 20:20 hindsight on scientific progress

• Open research in the humanities

• The role of scientific sleuths

• Individual bias and collective criticism

20:20 Hindsight on Scientific Progress

Whitney Barlow Robles looks back over the history of scientific progress and notes that a lot of it would look “unscientific” by today’s standards . . . which will look “unscientific” in the future!

People sometimes look back at past scientists and ask:

“Could they really have been this dumb? Thank goodness we know better.”

In response, Robles argues that science sometimes “progresses” in terrible ways, and that:

Brushing off these practices as unscientific implies that “true” science is, and always has been, unimpeachable. It denies us a chance to examine what flaws may fuel our own ways of thinking. And it absolves us from reckoning with the wreckage of the scientific past.

Robles, W. B. (2026, April 16). They thought what? Contingent Magazine. https://contingentmagazine.org/2026/04/16/they-thought-what/

Open Research in the Humanities

Beatriz Ferreira’s recent article considers open research in the humanities:

[It] contributes to the broader discussion on the need to reframe openness not as a set of procedural norms or policy mandates, but as a relational and epistemically situated practice.

Ferreira, B. B. (2026). Open for debate: Situating open research for the humanities in a neoliberal setting. The Journal of Electronic Publishing 29(1). https://doi.org/10.3998/jep.7850

This article is part of a special issue on “Open Research for the Humanities and Social Sciences,” edited by Jenni Adams, , and Samuel Moore: https://journals.publishing.umich.edu/jep/issue/456/info/

The Role of Scientific Sleuths

Serge Horbach and Auste Valinciute interrogate the emerging and contested role of the “scientific sleuth” in forensic metascience.

[Sleuths] form a distinct class of actors, most importantly because of the absence of formal accountability structures. This has triggered comparisons of scientific sleuthing to vigilantism.

Serge, P. M. J., & Horbach, A. V. (2026). Contextualizing the emerging and contested role of sleuths in the science monitoring system. Science and Public Policy, scag044, https://doi.org/10.1093/scipol/scag044

Individual Bias and Collective Criticism

Commenting on a recent paper by Shiffrin et al. (2026), Steven Sloman and Almos Molnar argue that:

The target article provides an abundant collection of biases that influence scientists .... But all of this depends on viewing scientific activity according to what Chater and Loewenstein (2023) refer to as an 'i-frame,' that is, a focus on the behavior of individuals.

In contrast, they consider the collective adversarial dynamics of the scientific community and conclude that:

In this collective dynamic, there is even a ‘methodological justification for individual scientists to be biased’, because only when ideas are tenaciously defended by some and tenaciously criticized by others, can we discern their true merit (Popper, 1994: 93).

Sloman, S. A., & Molnar, A. C. (2026). Are scientists naïve humans? Computational Brain and Behavior. https://doi.org/10.1007/s42113-026-00296-6

Subscribe now

Share

They concluded that their findings are consistent with prior work that has found “low observed replication success rates” (Tyner et al., 2026, p. 143). But how do we know that an observed replication rate is “low”? As I’ve explained previously, “it is unclear how replication rates can be judged to be ‘low’ and in need of improvement in the absence of clear targets for ‘acceptable’ replication rates. Logically, this reasoning represents an incomplete comparison” (Rubin, 2023, p. 4). We need to consider relevant benchmarks against which to judge observed replication rates. Here, I consider two possibilities: the optimal replication rate and the expected replication rate.

The Optimal Replication Rate

One benchmark that might be used to establish whether an observed replication rate is “low” is the optimal replication rate. However, defining an optimal replication rate is problematic because it leads us to ask: “optimal for what purposes?” and different replication rates may be optimal for different purposes. To illustrate, I consider how different stages of research and different philosophies of science might warrant different optimal replication rates.

Stage of Research

The optimal replication rate needs to balance the desire to learn from our mistakes in the face of scientific ignorance with the desire to produce reliable and trustworthy scientific knowledge that can be used by society. This balance is often skewed more towards learning in the earlier, discovery and exploration stages of a research topic and more towards consolidation and verification in later, more applied and translational stages. As Nosek et al. (2022) explained:

It would be possible to achieve near-100% replicability by adopting an extremely conservative research agenda that studies phenomena that are already well understood or have extremely high prior odds. Such an approach would produce nearly zero research progress. Science exists to expand the boundaries of knowledge. In this pursuit, false starts and promising leads that turn out to be dead ends are inevitable. The occurrence of nonreplicability should decline with the maturation of a research topic, but a healthy, theoretically generative research enterprise will include nonreplicable findings. (p. 730)

Hence, optimal replication rates are likely to be lower in the earlier exploratory stages of a research topic than in the later confirmatory stages that are intended to translate findings into real-world applications.

Philosophy of Science

The tension between learning and confirming is also apparent in different philosophies of science. Hence, the optimal replication rate may also be lower in theory-centric falsificationist philosophies of science than in effect-centric confirmatory philosophies. In falsificationist philosophies, (reproducible) replication failures may entail logical refutations of theories that fuel scientific discovery and progress (Popper, 1966, p. 285; Rubin, 2025, p. 11). As Popper (1962) explained, “refutations have often been regarded as establishing the failure of a scientist, or at least of [their] theory. It should be stressed that this is an inductivist error. Every refutation should be regarded as a great success” (p. 243). Accordingly, high quality, reproducible replication failures should be seen as causes for celebration rather than grounds for a replication “crisis.” As Mayrhofer et al. (2024) explained:

From this [Popperian] perspective, the replication crisis is not a crisis at all but rather a process that increases our knowledge by demonstrating that certain theories are false or at least cannot be corroborated by repeated observations, increasing their probability of being false. (p. 4)

Similarly, from a Lakatosian perspective, replication failures may inspire theory development by suggesting the presence of unrecognized moderators and boundary conditions (e.g., Lakatos, 1978; Rubin, 2025). As Nosek (quoted in Jones, 2026) explained:

Replication, when it fails, is theoretically generative. It’s like: “Wait a second, did we change something that we didn’t realize was important? We had no reason to expect something different here, but something different happened. Now we have a mystery.” And that’s where discovery happens. (p. 38)

In contrast, effect-centric confirmatory philosophies focus more on establishing the existence of phenomena vis-à-vis their replicability before proceeding to testing theories that explain those phenomena. As Nak et al. (2026) explained:

The massive replication work that has ensued in the wake of the reproducibility crisis (Nosek et al., 2022) in many ways already exemplifies this turn towards phenomena, as it clearly was never aimed at theory testing but simply at assessing the replicability of psychological effects. For instance, the Open Science Collaboration (2015) set out to estimate the replicability of “effects” rather than to confirm or falsify theories. (p. 24)

From this effect-centric perspective, replication failures cast doubt on the existence of phenomena. Hence, the optimal replication rate should be higher under a confirmatory philosophy of “phenomena consolidation” (Nak et al., 2026) than under a falsificationist philosophy of theory development (Lakatos, 1978; Popper, 1962, 1966). (See also Devezer & Buzbas’, 2023, distinction between “result-centric” and “model-centric” science and Feest’s, 2024, distinction between “effect-seekers” and “complexity mongers.”)

In summary, the optimal replication rate may vary depending on both (a) stage of research and (b) philosophy of science. In particular, optimal replication rates should be lower in discovery and exploratory stages of research and research that follows a theory-centric falsificationist approach but higher in applied and translational stages and research that follows an effect-centric confirmatory approach.

The Expected Replication Rate

Given its variability and ambiguity, it is unclear how to formally specify the optimal replication rate. Indeed, Tyner et al. (2026) concluded that “the optimal replicability rate is not known” (p. 148; see also Nosek, quoted in Fox, 2026). In other words, it is unclear what the observed replication rate should be. Instead, we can consider what people expect the replication rate to be. We can then use this expected replication rate as a benchmark against which to judge whether the observed replication rate is too low, too high, or consistent with expectations. For example, commenting on Tyner et al.’s (2026) work, Nosek (quoted in Jones, 2026) assumed that scientists would be surprised by a lower than expected replication rate:

We don’t know what the optimal level of replicability is....The important part is not the number — it’s that scientists say “wow, I had presumed that published findings are more repeatable than they are.” (p. 38)

Indeed, several commentators have argued that the replication crisis occurred because observed replication rates are markedly lower than “expected or desired” (Nosek et al., 2022, p. 724; see also Munafò et al., 2017, p. 1; Open Science Collaboration, 2015, p. 7). Hence, as Tyner et al. (2026) explained, “the problem to solve is not unreplicability per se, it is overconfidence” (p. 148; see also Nosek, quoted in Fox, 2026).

Consistent with this view, there is evidence that researchers tend to overestimate replication rates. For example, Table 1 provides data from six studies that have used prediction markets and forecasting surveys to estimate expected replication rates and compare them with observed replication rates.

Table 1 shows that people tend to overestimate replication rates. Importantly, however, the extent of this overestimation does not appear to be particularly extreme. On average, people estimate the replication rate to be around 13 percentage points higher than the observed replication rate (i.e., ~63% rather than ~50%). It is debatable whether a 13-point overestimation represents sufficient grounds for a replication “crisis” (Rubin, 2023, p. 4).

It is also important to appreciate that the expected replication rate reflects people’s beliefs, and that their beliefs may not represent the optimal replication rate for scientific purposes. Hence, a discrepancy between the observed and expected replication rate does not necessarily imply that we need to change our research practices. Indeed, if community expectations are substantially different from both observed and optimal replication rates, then they will be not only unrealistic but also suboptimal! Hence, we end up in a situation in which (a) the optimal replication rate is unclear and (b) the expected replication rate may be misleading.

Comment from a Co-Author

is one of the authors of the Tyner et al. (2026) paper. He recently shared his thoughts about Tyner et al.’s (2026) work on BlueSky (Haber, 2026, April 10). His comments reflect some of the issues discussed here:

Question now becomes “does the replication rate differ from what people *expect* it would be.”

There’s definitely a problem if those don’t match, but unclear what the problem is.

There is no obviously correct benchmark for what that replication rate should be (and be skeptical of anyone who is sure they know what it is).

Heck, with all the time I’ve spent in this work, I am still truly not sure if/how much we should be bothered by the rates found.

Addressing the Replication Crisis

The typical response to the replication crisis has been to design and promote changes to research practices to try to increase observed replication rates. This response implies that the replication crisis is a methodological problem, and that methodological solutions are required to increase observed replication rates to a more optimal level. However, if the optimal replication rate is unknown, then the success or failure of this response will be unclear. Again, the problem here is one of a logically incomplete comparison (Rubin, 2023, p. 4).

An alternative perspective is to view the replication crisis as a problem of researchers’ collective “overconfidence” (Tyner et al., 2026, p. 148) caused by their unrealistically high expected replication rates. This more social psychological perspective avoids the problem of an incomplete comparison because we are able to measure both observed and expected replication rates. It also implies an alternative solution to the replication crisis. Specifically, the replication crisis can be addressed by not only increasing the observed replication rate but also decreasing the expected replication rate. From this perspective, it may be helpful to educate scientists, the public, and the media that (a) there is no clear optimal replication rate and (b) a ~50% replication rate is realistic.

But What About False Positives?

Some may balk at the proposal to reduce people’s expected replication rate because they view unexpectedly low replication rates as implying that original studies suffer from low quality methodology and questionable research practices that produced false positive results. However, it is worth referring to Tyner et al.’s (2026) sage advice when interpreting replication failures and successes:

Replication failures are not necessarily due to the original findings having low credibility. Low replication rates can also be due to false negatives, poorly designed replications, selecting only positive results for replication, and differences between original and replication studies that are initially perceived as unimportant. (p. 143)

A single failure to replicate does not justify concluding that the original research was wrong …. Even if the replication appeared to be testing the same research question, there could be differences in the methodology, sample or context that are unrecognized moderators of the outcome. In addition, even if the replication researchers were diligent in conducting the research, there could be unrecognized errors or flaws in implementing the replication protocol that interfered with observing the outcome. (p. 147)

A single successful replication does not justify concluding that the original research was correct …. The replicability of an effect is not the same as the validity of its interpretation. Original and replication studies may share confounds, faulty measures or other design weaknesses that produce replicable, but misinterpreted, outcomes. (p. 148)

Conclusion

In summary, it’s difficult to argue that replications rates are suboptimal because it is unclear what the optimal replication rate should be. It’s easier to argue that replication rates are lower than expected, but the evidence suggests that they are only around 13 percentage points below community expectations, and it’s unclear whether this degree of discrepancy is problematic, especially given that community expectations may not be the best guide for scientific practice. Consequently, the rationale for a replication “crisis” remains debatable. Finally, if the “crisis” is conceived as a problem of collective “overconfidence” (Tyner et al., 2026, p. 148) rather than inadequate methodology, then it may be more effective to adjust community expectations to better reflect realistic replication rates. In other words, the replication crisis may serve as a valuable lesson in scientific humility.

References

Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., ... & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. https://doi.org/10.1126/science.aaf091

Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., ... & Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637-644. https://doi.org/10.1038/s41562-018-0399-z

Chandrashekar, S. P., Viganola, D., Dreber, A., Johannesson, M., Pfeiffer, T., Siegel, A., & Feldman, G. (2026). Using prediction markets and forecasting surveys to predict 28 replication outcomes of classic articles in social psychology and judgement and decision making. Royal Society Open Science, 13(1), 250377. https://doi.org/10.1098/rsos.250377

Devezer, B., & Buzbas, E. O. (2023). Rigorous exploration in a model-centric science via epistemic iteration. Journal of Applied Research in Memory and Cognition, 12(2), 189–194. https://doi.org/10.1037/mac0000121

Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., ... & Johannesson, M. (2015). Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences, 112(50), 15343–15347. https://doi.org/10.1073/pnas.1516179112

Feest, U. (2024). What is the replication crisis a crisis of? Philosophy of Science, 91(5), 1361-1371. https://doi.org/10.1017/psa.2024.2

Fox, J. (2026, April 11). When being right less than half the time…is fine. Bloomberg. https://www.bloomberg.com/opinion/articles/2026-04-11/when-being-right-less-than-half-the-time-is-fine

Gordon, M., Viganola, D., Dreber, A., Johannesson, M., & Pfeiffer, T. (2021). Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects. PLOS One, 16(4), e0248780. https://doi.org/10.1371/journal.pone.0248780

Jones, N. (2026, April 2). Why science has a credibility problem — and how to address it. Nature, 652, 36. https://doi.org/10.1038/d41586-026-00972-4

Lakatos, I. (1978). The methodology of scientific research programmes (Philosophical Papers, Volume I). Cambridge University Press.

Mayrhofer, R., Büchner, I. C., & Hevesi, J. (2024). The quantitative paradigm and the nature of the human mind. The replication crisis as an epistemological crisis of quantitative psychology in view of the ontic nature of the psyche. Frontiers in Psychology, 15:1390233. https://doi.org/10.3389/fpsyg.2024.1390233

Mody, F., Wilkinson, D. P., Fraser, H., Fidler, F., Bishop, M. M., Burgman, M., … Pfeiffer, T. (2026). Large-scale human predictions of the replicability of published social and behavioural science papers – a multi- study analysis. MetaArXiv. https://doi.org/10.31222/osf.io/vgyed_v1

Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Percie du Sert, N., ... & Ioannidis, J. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 1-9. https://doi.org/10.1038/s41562-016-0021

Nak, J., van Dongen, N. N. N., Woodward, J. F., Scheel, A. M., Haig, B., van Bork, R., Blanken, T. F., de Ron, J., Finnemann, A., Haslbeck, J., Francken, J., Eronen, M., van Lissa, C. J., Dijstelbloem, H., & Borsboom, D. (2026). Establishing psychological phenomena: An alternative methodological perspective on scientific practice. PsyArXiv. https://doi.org/10.31234/osf.io/rh9cu_v1

Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., ... & Vazire, S. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73, 719–748. https://doi.org/10.1146/annurev-psych-020821-114157

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

Popper, K. R. (1962). Conjectures and refutations: The growth of scientific knowledge. Routledge.

Popper, K. R. (1966). Some comments on truth and the growth of knowledge. Studies in Logic and the Foundations of Mathematics, 44, 285-292. https://doi.org/10.1016/S0049-237X(09)70596-8

Rubin, M. (2023). Questionable metascience practices. Journal of Trial and Error, 4(1), 5–20. https://doi.org/10.36850/mr4

Rubin, M. (2025). The replication crisis is less of a “crisis” in Lakatos’ philosophy of science than it is in Popper’s. European Journal for Philosophy of Science, 15(5). https://doi.org/10.1007/s13194-024-00629-x

Tyner, A. H., Abatayo, A. L., Daley, M., et al. (2026). Investigating the replicability of the social and behavioural sciences. Nature, 652, 143–150. https://doi.org/10.1038/s41586-025-10078-y

Citation

Rubin, M. (2026, April 14). Is a 55% replication rate too low, too high, or just right? Critical Metascience. https://doi.org/10.59350/qfena-7xp96

Subscribe now

Share

• Workshop on the politics and finances of open science reform

• Symposium: “Who critiques the critique? Toward a reflexive metascience”

• Preprint calls for psychologists to establish phenomena before testing theories

• Systematic review of questionable research practices

Workshop: “The Politics & Finances of (Open) Science Reform”

Sheena Bartscherer, Sven Ulpts, and Joseph Bak-Coleman will be hosting a 2-day workshop in Berlin on 19th and 20th May about the socio-economic architecture of the open science movement. The workshop will address questions such as:

• Who funds open science?

• Who benefits from open science financially?

• How do private and political interests work against the proclaimed idea(l)s of open science?

Presenters include:

• Joseph Bak-Coleman

• Arwid Lund

• Philip Mirowski

• María de los Ángeles Crespo López

• Nicole Foti

• Osvaldo Gallardo

Online attendance is available. The deadline for registrations is Sunday 26th April.

More info: https://www.rmz.hu-berlin.de/de/termine/workshop-the-politics-finances-of-open-science-reform

Symposium: “Who Critiques the Critique? Toward a Reflexive Metascience”

Fabrice Gabarrot and I will be convening a symposium at the European Association of Social Psychology’s 2026 General Meeting on Wednesday 1st July in Strasbourg, France titled “Who Critiques the Critique? Toward a Reflexive Metascience.”

The symposium will explore critical metascience as a reflexive turn in social psychology that deepens the current reform movement by examining the scientific field through social-psychological and socio-historical perspectives.

Presenters include:

• Mark Rubin

• David Peterson and Aaron Panofsky

• Rizqy Zein and Mario Gollwitzer

• Fabrice Gabarrot

Attendance only available in person.

More info: https://easp2026strasbourg.com/

Establishing Phenomena Before Testing Theories

In an intriguing new preprint, Nak et al. (2026) called for psychologists to focus more on establishing phenomena before testing theories:

By and large, psychology should be more in a phase of documenting, characterizing and organizing phenomena (Eronen, & Bringmann, 2021; Haig, 2013; Rozin, 2001), and limit theory testing to situations where theories offer testable hypotheses (Mcphetres, et al., 2021; Scheel, et al., 2021). It is high time that we stop trying to fit all psychological research in the ill-fitting theory-testing mold, and instead adopt a framework that does justice to our empirical work.

From a critical metascience perspective, it’s interesting that the authors argued that their phenomena-oriented perspective provides a better fit to open science’s emphasis on “effects” rather than theory-testing:

Replication studies are also commonly phrased in terms of effects (Scheel, Schijen, & Lakens, 2021). Few of the replication studies that we know—single-study projects, many-labs, and many-studies projects—speak of reconfirming or falsifying theories based on their result (a notable exception is Coles et al., 2022). Instead, they speak of replicating or failing to replicate effects. In that sense, the turn towards open science and reproducibility already embodies a significant shift towards phenomena-oriented research, and many of the methodological innovations that have been implemented in the wake of the reproducibility crisis are best cast in that way. (p. 9; see also p. 24)

Nak, J., van Dongen, N. N. N., Woodward, J. F., Scheel, A. M., Haig, B., van Bork, R., Blanken, T. F., de Ron, J., Finnemann, A., Haslbeck, J., Francken, J., Eronen, M., van Lissa, C. J., Dijstelbloem, H., & Borsboom, D. (2026). Establishing psychological phenomena: An alternative methodological perspective on scientific practice. PsyArXiv. https://doi.org/10.31234/osf.io/rh9cu_v1

Systematic Review of Questionable Research Practices

Finally, Fanelli et al. (2026) recently reported the results of a major new systematic review of 23 surveys that asked researchers about their engagement in questionable research practices (QRPs). They found that:

The most commonly reported QRP types include selecting the analysis or the covariates within the analysis, selectively citing sources, withholding methodological details and HARKing (acronym for Hypothesizing After Results are Known). At the other extreme of the spectrum, we find FFP ([fabrication, falsification, & plagiarism] i.e. behaviours that qualify explicitly as scientific misconduct), denying authorship to someone who deserved it, and changing one’s methods or results in response to a funding source.

Hence…

Many of the highly reported QRPs tend to be the ones whose problematic nature is most ambiguous and context-dependent, including HARKing, p-hacking and selectively presenting analyses or results Rubin [2017]; Ulrich and Miller [2020]; Fanelli [2026].

Fanelli, D., Voodla, A., & Andres, S., & Uusberg, A. (2026). Questionable research practices: A principled classification and ranking based on survey data. Science and Engineering Ethics. https://doi.org/10.1007/s11948-026-00589-w

Subscribe now

Share

Welcome to Update #2, which summarizes work on:

• Barriers and enablers of reproducibility

• Cataloguing open research practices in the humanities and social sciences

• Reconsidering reproducibility and questionable research practices in animal-based biomedicine

• Naming individual studies in metascience work

Barriers and Enablers of Reproducibility

Serge Horbach, and colleagues published the results of a “future studies” analysis of reproducibility. Their 19 research participants included representatives of scholarly publishers, funding agencies, qualitative social scientists, and machine learning researchers.

The results indicated different views about making reproducibility a requirement:

Some participants forcefully argued that the only way to achieve higher levels of reproducibility was by mandating efforts to foster it, for example, through an explicit requirement in grant funding, journal publication or tenure processes. Others, however, were much more skeptical, arguing for the need to remain flexible. This skeptical view included maintaining the option to opt-out of reproducibility efforts and standards, in cases where these are not deemed relevant or feasible. This finding supports recent trends in the literature, which frame reproducibility (and replicability) as neither universally applicable nor feasible across diverse epistemic contexts (Cole et al., 2024; Leonelli, 2018; Penders et al., 2019; Ulpts & Schneider, 2023). Consequently, we recommend the development of guidelines for reproducibility (and/or transparency) practices tailored to specific domain, methodological and epistemic contexts.

Horbach, S. P., Cole, N. L., Kopeinik, S., Leitner, B., Ross-Hellauer, T., & Tijdink, J. (2026). How to get there from here? Barriers and enablers on the road towards reproducibility in research. Journal of Trial and Error, 1(1), 1-33. https://doi.org/10.36850/f4c4-4a38

Launch of MORPHSS Catalogue

Samuel Moore, and colleagues launched the Materialising Open Research Practices in the Humanities and Social Sciences (MORPHSS) catalogue: https://catalogue.morphss.work/

This is a living, iterative, and collaborative showcase of open research practices within the arts, humanities, and social sciences disciplines.

For further information, please see the accompanying report by Jenni Adams et al.:

We identify 30 open research practices in AHSS, among which we observe six key forms of openness: process openness, evidentiary openness, availability of outputs, the accessible communication of research, participatory openness and epistemic openness. Based on those identified, we find that open research practices in AHSS are diverse, extending beyond the suite of practices emphasised within dominant accounts of Open Science.

Adams, J., Barnes, M., Moore, S., Pinfield, S., & Soni, S. (2026). Openness in the arts, humanities and social sciences: Documenting open research practices beyond STEM (A MORPHSS Project Report) (p. 51). MORPHSS Project. https://doi.org/10.17613/h10rz-qk035

Reconsidering Reproducibility and Questionable Research Practices in Animal-Based Biomedicine

Simon Lohse engages with critical metascientific work as part of his discussion of reproducibility and questionable research practices in the area of animal-based biomedicine. He argues that mainstream metascientific treatments of reproducibility need refining in this area:

Critical analyses of meta-science emphasise the concept of epistemic diversity and question the extent to which general guidelines for improving replicability or validity can do justice to the heterogeneity of scientific practices and subject areas (esp. Leonelli, 2018, 2022). One of the main aims of this paper is thus to help shift the focus of the meta-scientific replication debate in animal-based research, so that it better reflects these aspects and concerns.

Lohse also calls for a more discipline- and context-specific consideration of questionable research practices:

In many cases, it will be highly context-dependent which methodological decisions are problematic – as their soundness depends on research aim, (evolving) local practices, specific challenges etc. (Fiedler & Schwarz, 2016; Sacco et al., 2019). As I have attempted to show, the assessment of a specific decision or procedure as problematic may, at times, also depend on ethical considerations. This conclusion casts further doubt on the usefulness of developing guidelines for good scientific practice with strongly universalistic tendencies, including those aimed at large scientific fields, such as medicine and psychology. Rather than relying pre-dominantly on such guidelines and a “top down approach” (Fletcher, 2021), we should pay more attention to local methodological norms engrained in specific epistemic practices and cultures; norms about which we do not know nearly enough, indicating the need for more qualitative studies informed by a philosophy of science/STS perspective.

Lohse, S. (2026). Reproducibility, questionable research practices and ethico-epistemic trade-offs in animal-based biomedicine. European Journal for Philosophy of Science, 16, 25. https://doi.org/10.1007/s13194-026-00727-y

Naming Individual Studies in Metascience Work

Kai Sassenberg and colleagues respond to Malte Elson et al.’s (2026) recent proposal that, “as a default, meta-scientific studies of published research artefacts need to include…a full, identifiable list of included studies.” In response, Sassenberg et al. argue that “publishing non-anonymized data might pose unjustified harm to original authors.” Part of this disagreement stems from different conceptualizations of the tasks and limits of metascience. As Sassenberg et al. explained:

The point on which we most strongly disagree with Elson et al. (2026) is their apparent understanding of metascience. Their description of the field implies that metascience is mainly about identifying flaws, errors, inconsistencies, and deviations, as epitomized in the very first sentences of their contribution: “Meta-science is the study of quality, bias, and efficiency in research. To this end, metascience often examines and analyses research artefacts from existing studies – such as published articles, preregistration documents, analysis code, and datasets” (p. 1). This suggests that metascience is basically a monitoring activity, by which researchers who know what is right and wrong (i.e., the correct way of “doing science”) evaluate original research and flag it. According to Elson et al. (2026), metascience is about appraising “..., for example, how complete preregistrations are, how often key reporting elements are missing, or how frequently analytic flexibility appears” (p. 2). This understanding of metascience conveys the view that it is objectively discernible what is right or wrong. We do not agree with this view. There is, of course, a blacklist of scientific practices that are clearly unethical (e.g., fabricating data, making research participants identifiable without explicit consent, plagiarizing ideas, etc.), but there is a much larger area of practices that are not scientifically questionable per se – at least not without knowing the specific underlying intentions (e.g., deciding to remove single cases in a dataset, deviating from a preregistration, etc.).

Elson, M., Hussey, I., Clarke, B., Norwood, S. F., Grinschgl, S., Arslan, R. C., … Cummins, J. (2026, February 13). Against anonymising meta-scientific data. PsyArXiv. https://doi.org/10.31234/osf.io/6eyjf_v1

Sassenberg, K., Hahn, L., Hellmann, J. H., & Gollwitzer, M. (2026, March 19). The perils of non-anonymized data in metascience. PsyArXiv. https://doi.org/10.31234/osf.io/zj8bk_v1

Subscribe now

Share

Papers Invited for Open Panel on Critical Metascience

Papers invited for an open panel on critical metascience at the 2026 meeting of the European Association for the Study of Science and Technology.

Many current users of metascience approaches are more interested in questions of speed, with innovation seen as an end in itself, than in questions of direction (see Fortunato et al (2018) for example. We think new approaches to, for example, scientometrics and topic modelling can be used to open up emerging science and innovation to scrutiny.

Convenors: , , , Tommaso Ciarli, Cian O’Donovan

When: 8-10 September 2026

Where: Krakow, Poland

More Info: https://nomadit.co.uk/conference/easst2026/p/18244

Papers Invited for Panel on Scientific Reform Movements

Papers invited for a Panel at the 2026 Annual Meeting of the Society for Social Studies of Science: “Science unbound: Scientific reform movements and their entanglement with (techno-)social configurations of power.”

Our panel…aims to examine the emergent forms of reform and governance agendas in science that they produce: tracing their underlying drivers, cultural situatedness, and implications for the wider science system in order to ask where these transformations might take us?

Convenors: David Joshua Schröder and

When: 7-10 October 2026

Where: Toronto, Canada

More Info: https://www.4sonline.org/call_for_submissions_toronto.php

Ongoing Debates About Social Psychology Effects

1. Mortality Salience: “MS effects on clinically relevant outcomes may exist, yet remain fragile and difficult to reproduce consistently.” Ramona Letzner, & Denisa Boroi (2026). Mortality salience: A critical synthesis of its effects on psychological symptoms. Death Studies, 1-12. https://doi.org/10.1080/07481187.2026.2630951

2. Cognitive Dissonance: “Yes, cognitive dissonance is still actually a thing. In the world of journalism, it would be a more exciting and clickable narrative if the answer were ‘no,’ but that’s the problem with narrative. Sometimes the truth—like turning blocks one quarter of a turn—is just boring." : https://skepticalinquirer.org/exclusive/yes-cognitive-dissonance-is-still-actually-a-thing/

3. Ego Depletion: “Our findings suggest that participant and experimenter expectations can influence performance in ego-depletion paradigms.” Zerhouni, O., Morgado, N., Urdapilleta, I. et al. (2026). Demand and experimenter bias and the ego-depletion effect. Current Psychology, 45, 609. https://doi.org/10.1007/s12144-025-08841-5

4. Stereotype Threat: “The report of stereotype threat’s demise has been greatly exaggerated.” https://marycmurphy.substack.com/p/the-report-of-stereotype-threats

Are Supplementary Materials the New File Drawer?

Paul Bertin has a very interesting paper questioning the way in which researchers use supplementary files:

“[Supplementary] files offer optimal settings for spinning research outcomes by sorting information between the main document and supplementary files to fit one’s narrative, making information technically available but effectively invisibilised.”

Bertin, P. (2026). Are supplementary materials the new file drawer? Theory & Society, 55, 17. https://doi.org/10.1007/s11186-025-09661-z

Disciplinary Differences in the Acceptability of Questionable Research Practices

New study by Amanda Lindkvist and colleagues finds:

• “Acceptability ratings of questionable practices were lowest among researchers and ethics reviewers in medical sciences.”

• “The largest difference was found for the item Conducting studies without publishing the results (in a journal or public repository), where researchers in the Natural Sciences were the most accepting and researchers in Medical and Health Sciences were the least accepting, on average.”

• Substantial differences were also found for:

1. Preregistering

2. Making data publicly available

3. Storing data in a way that makes it possible for others to understand or use

Lindkvist, A. M., Koppel, L., Andersson, D., Västfjäll, D., & Tinghög, G. (2026). Is research ethics discipline-specific? A survey of researchers’ and ethics reviewers’ views on research misconduct and questionable practices. Research Policy, 55(4), 105435. https://doi.org/10.1016/j.respol.2026.105435

Subscribe now

Share

Although p-hacking is often described as a “questionable research practice” (John et al., 2012), it is widely regarded as being statistically and ethically problematic (e.g., Miller et al., 2025; Pickett & Roche, 2018). It is also believed to be a major contributor to the replication crisis (e.g., Bishop, 2019). In particular, p-hacking is thought to inflate Type I error rates above their nominal conventional level, resulting in a larger proportion of false positive results in the literature than would otherwise be expected. This disproportionate number of false positives is then thought to cause unexpectedly low replication rates.

In this recent preprint, I aim to add some nuance to this view by distinguishing between two philosophies of significance testing — the error statistical approach (Mayo, 1996, 2018) and the formal inference approach (Rubin, 2021, 2024a, 2024b). I argue that, although p-hacking inflates Type I error rates in the error statistical approach, it does not inflate them in the formal inference approach.

To illustrate, imagine the following example of p-hacking: A researcher conducts a two-sided independent samples t test using a conventional alpha level of 0.05. They fail to find a significant result with regard to their first null hypothesis H_0,1: t(326) = 1.88, p = 0.061. For that reason, they remove an outlier from their sample and conduct a second test. The second test has a smaller degrees of freedom than the first, and it is formally specified by a different test procedure (i.e., outliers removed). Hence, it tests a different statistical null hypothesis, H_0,2. This time, the researcher finds a significant result, t(325) = 2.16, p = 0.032, which they then report without disclosing the nonsignificant result from their first test.

From an error statistical perspective, the researcher has conducted two significance tests (p₁;H_0,1 and p₂;H_0,2) and then selectively reported whichever test(s) yielded a significant result (Mayo, 1996, pp. 303–304, 348; Mayo, 2018, pp. 274–275). In this case, the “actual” sampling distribution is given under the “actual” test procedure’s “global” or “universal” intersection null hypothesis, H_0,1 ∩ H_0,2 (Mayo, 2018, p. 276). Consequently, the “actual” error rate is the familywise error rate 0.098 (i.e., 1 − [1 − 0.05]²), which is inflated above the nominal error rate of 0.05.

In contrast, from a formal inference perspective, the relevant sampling distribution is given under H_0,2, not H_0,1 ∩ H_0,2, because the researcher only reported an individual inference about H_0,2, not a union-intersection inference about H_0,1 ∩ H_0,2. Consequently, the relevant test procedure is the formally reported procedure for H_0,2, in which any outliers are removed, and the relevant Type I error rate is the nominal error rate (Rubin, 2021, p. 10991; Rubin, 2025, p. 10).

Importantly, from a formal inference perspective, it is not appropriate to compare the “actual” familywise error rate for a decision about H_0,1 ∩ H_0,2 with the nominal error rate for a decision about H_0,2 and argue that the former represents an “inflated” version of the latter. These two error rates are incommensurate with one another because they refer to two separate decisions about two different statistical null hypotheses based on two different statistical models. Arguing that a test’s Type I error rate has been inflated because another test has a larger error rate is like arguing that a person’s height has been inflated because their friend is taller than them!

It is also logically inconsistent to use a familywise error rate to license an individual inference. Specifically, we would be making a fallacy of division or ecological fallacy in this case because we would be misapplying an aggregate-level union probability to an individual member of the aggregate (Selvin, 1958; Waller, 2018). In particular, the union probability of obtaining at least one significant result given H_0,1 ∩ H_0,2 (i.e., the familywise error rate) does not represent the individual probability of obtaining a significant result given H_0,2 alone (García-Pérez, 2023; Rubin, 2021, pp. 10978–10983; Rubin, 2024a, p. 3; Rubin, 2024b, p. 51).

The familywise error rate would only be relevant if H_0,2 was treated as a logically exchangeable constituent of H_0,1 ∩ H_0,2, rather than as a distinct individual hypothesis. Here, the researcher would make a union-intersection inference about H_0,1 ∩ H_0,2 as a whole, rather than an individual inference about H_0,2 alone, and this inference would be based on at least one significant result among p₁;H_0,1 and p₂;H_0,2 (García-Pérez, 2023, p. 2; Rubin, 2021, p. 10981; Rubin, 2024a, p. 2). In this case, they would conclude that “either H_0,1 or H_0,2 or both are false” rather than “H_0,2 is false.”

In summary, from an error statistical perspective, (a) the “actual” test procedure includes two significance tests: p₁;H_0,1 and p₂;H_0,2; (b) the “actual” sampling distribution is given under the intersection null hypothesis H_0,1 ∩ H_0,2; and so (c) the “actual” Type I error rate is the familywise error rate of 0.098 (i.e., 1 − [1 − 0.05]²), which is “inflated” relative to the nominal error rate of 0.05. In contrast, from a formal inference perspective, (a) the formally reported inference is an individual inference about the individual null hypothesis H_0,2; (b) the relevant sampling distribution for this inference is given under H_0,2; and so (c) the relevant Type I error rate is p₂;H_0,2’s nominal error rate of 0.05, which is incommensurate with the familywise error rate for a decision about H_0,1 ∩ H_0,2. Figure 1 illustrates these differences.

The contrast between the error statistical and formal inference approaches has implications for the way in which we conceptualize p-hacking. In particular, p-hacking may be viewed as being more or less problematic for Type I error rates depending on one’s philosophy of significance testing. Accordingly, it would be helpful for discussions about the potential dangers of p-hacking to be more clearly situated within relevant philosophies of significance testing in order to articulate otherwise implicit assumptions.

Further Information

Article

Rubin, M. (2026). p-hacking inflates Type I error rates in the error statistical approach but not in the formal inference approach. ArXiv. https://doi.org/10.48550/arXiv.2602.21792

References

Bishop, D. V. (2019). Rein in the four horsemen of irreproducibility. Nature, 568(7753), 435–436. https://www.nature.com/articles/d41586-019-01307-2

García-Pérez, M. A. (2023). Use and misuse of corrections for multiple testing. Methods in Psychology, 8, 100120. https://doi.org/10.1016/j.metip.2023.100120

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

Mayo, D. G. (1996). Error and the growth of experimental knowledge. University of Chicago Press.

Mayo, D. G. (2018). Statistical inference as severe testing: How to get beyond the statistics wars. Cambridge University Press.

Miller, J. D., Phillips, N. L., & Lynam, D. R. (2025). Questionable research practices violate the American Psychological Association’s Code of Ethics. Journal of Psychopathology and Clinical Science, 134(2), 113–114. https://doi.org/10.1037/abn0000974

Pickett, J. T., & Roche, S. P. (2018). Questionable, objectionable or criminal? Public opinion on data fraud and selective reporting in science. Science and Engineering Ethics, 24, 151–171. https://doi.org/10.1007/s11948-017-9886-2

Rubin, M. (2021). When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing. Synthese, 199, 10969–11000. https://doi.org/10.1007/s11229-021-03276-4

Rubin, M. (2024a). Inconsistent multiple testing corrections: The fallacy of using family-based error rates to make inferences about individual hypotheses. Methods in Psychology, 10, 100140. https://doi.org/10.1016/j.metip.2024.100140

Rubin, M. (2024b). Type I error rates are not usually inflated. Journal of Trial and Error, 4(2), 46–71. https://doi.org/10.36850/4d35-44bd

Rubin, M. (2025). Preregistration does not improve the transparent evaluation of severity in Popper’s philosophy of science or when deviations are allowed. Synthese, 206, 111. https://doi.org/10.1007/s11229-025-05191-4

Selvin, H. C. (1958). Durkheim’s suicide and problems of empirical research. American Journal of Sociology, 63(6), 607–619. https://doi.org/10.1086/222356

Waller, J. (2018). Division. In R. Arp, S. Barbone & M. Bruce (Eds.), Bad arguments (pp. 259–260). Wiley. https://doi.org/10.1002/9781119165811.ch56

Share

Subscribe now

There have been repeated declarations of crisis in psychology’s history. But this history is often forgotten, resulting in repeating the same mistakes and raising similar questions over again. In our recent paper (Teymoori & Trappes, 2026), we recover this history to study psychology’s recurrent crisis discussions. We examine what generated crisis discussions, how they tended to unfold, and how they were resolved. And we derive some lessons from history for the current replication crisis.

Recurring Problems and Concerns

Historians of psychology, philosophers and psychologists alike identify three major disciplinary crisis discussions in the discipline. First is the “golden age” of crisis discussions in the early 20^th century (Sturm & Mülberger, 2012). In reams of pamphlets and books, psychologists and philosophers diagnosed a discipline in crisis. Many questioned whether psychology could or should use natural scientific methods. Others decried disciplinary fragmentation, problematic generalizations, a lack of theorizing, and the elimination of subjectivity. Sound familiar? Many of these issues come up again and again, including in the recent replication crisis.

Ultimately, the debates of the 1920s and 30s fizzled out, and psychology galvanized around experimental methods and the behaviorist paradigm. By the 1960s, experiments had become the gold standard, enforced by a regime of textbooks, journals, courses, editorial gatekeeping, and rhetoric. But this settled state quickly erupted into psychology’s second major crisis discussion. This time, psychologists diagnosed the discipline with a crisis resulting from problems with the experimental paradigm.

Again, many concerns ring familiar to today’s ears. For example, Rosenthal identified the issue of experimenter demand effects and social expectancies within experiments (Rosenthal, 1965, 1966), as well as the lack of replication and the now well-known “file drawer effect” (Rosenthal, 1979). Additional concerns included theoretical weaknesses, the social relevance and implications of research findings, poor research and publication practices, and a lack of ethical standards (Faye, 2012; Lakens, 2025; Teymoori & Trappes, 2026).

As in the 1930s, the crisis of the 1960s too came to a close. But this status quo was again disturbed, this time by psychology’s third major disciplinary crisis: the replication crisis of the 2010s. The replication crisis was triggered by several high-profile events, including cases of scientific fraud (such as the Diederik Stapel case), controversy surrounding Bem’s study on feeling the future, and a series of studies revealing a widespread lack of replicability in psychological research findings. Many viewed the problem as stemming from a lack of direct replication and insufficient checks and balances, giving a large degree of freedom for researchers, especially under the institutional pressure to “publish or perish”. Others diagnosed the discipline with a lack of strong theories and called for conceptual replication to strengthen theorizing.

Like the past two crises, the response to the present crisis has centered on administrative and methodological reforms. For example, psychologists have adopted pre-registration, accepted replication, and promoted open science practices such as open data and open review.

Underlying Issues Go Unresolved

It’s not just the concerns that seem to repeat throughout history. Across all three periods, we see a pattern of methodological and administrative solutions dominating in the resolution of crisis discussions. The first crisis ended with the widespread adoption of experimental and quantitative methods. In the subsequent crises, solutions focused on strengthening the internal validity of experiments and adding meta-analysis as a way to make sense of inconsistent findings across studies. In addition, psychologists introduced more checks and balances on researchers and experimental practice. In the 1960s this was training to reduce experimenter expectancy effects and to promote ethical guidelines for research. At present, it is largely open science practices such as pre-registration, open data, and open review.

Other proposed solutions have fared less well. During all three crises, researchers called for greater investment in theory development and tackling generalizability problems by diversifying research methods and approaches. However, these broader proposals have received less emphasis and adoption in mainstream psychology. Consistently over history, contrary perspectives were hived off into other disciplines or heterodox traditions that were mostly ignored or derided in the mainstream of the discipline.

Based on this history, we highlight two underlying issues that cut across the three major disciplinary crises in psychology. These issues aren’t properly addressed by methodological and administrative reforms. Yet addressing them is crucial for the discipline to avoid their eternal recurrence.

Questioning Epistemic Processes

First, the crisis discussions in psychology have all involved, often implicitly, a debate about how scientific knowledge is gained—what philosophers call epistemic processes. These range from questions about the very nature of psychological phenomena (their ontological status) to the specific challenges faced in developing scientific knowledge about them.

The diagram below shows the epistemic processes we engage in when producing knowledge about a psychic phenomenon. Psychic phenomena become hypothesized entities that are theorized in terms of psychological constructs. This process leads to heated discussions about how constructs should be conceptualized and defined, and how constructs are related to their referent.

Once a construct is established, psychologists create operational definitions to guide how it should be measured and studied empirically. This means empirical findings from measurement or experiment are at a distance from psychic phenomena themselves, with operationalizations and constructs standing in-between. Outcomes of empirical studies therefore cannot be equated with the hypothesized entity, nor can they be directly generalized to psychic phenomena. The problematic generalization of numerical outcomes in empirical studies to verbal hypothesis and by extension to hypothesized entities has been considered as part of the crisis psychology is facing (Teymoori & Trappes, 2026; Yarkoni, 2022).

You might argue that these are concepts and processes covered in undergraduate textbooks, and you may be correct. However, one issue is that measurement practices and this process from hypothesized entity to measurement and the generalization of findings have become ritualized in psychology—much like the use of statistics (aka statistical rituals).

The other issue is the difference between what we know in principle and what we do in practice. Lakens (2025), one of the major figures in the open science movement, recently stated that:

The fact that these issues [e.g., problematic research practices, lack of replication] seem obvious in hindsight, and yet persisted for decades, is a good reason to display intellectual humility. After all, who knows which practice that we currently all engage in will be at the root of the next crisis in psychology? (p. 7).

We believe epistemic processes have been one of the underlying issues across the three disciplinary crises and need to come to the fore in reform efforts.

Rethinking Scientific Ideals

The second issue is the restrictive ideal of science dominant in psychology. Take a look at one of the Scientific Utopia papers by Nosek and Bar-Anan (2012). They describe their reform effort as centering around a “practical template for improving the current model [of psychological science]” (p. 218, emphasis is ours). Just as in the 1960s, mainstream perspectives in the 2010s have largely avoided questioning whether the hypothetico-deductive model is adequate for understanding mental life and psychological phenomena.

The current, restrictive ideal of science tends to limit scientific legitimacy to findings produced only through the hypothetico-deductive model. As a result, it ignores other models of science and ways of producing psychological knowledge.

Alternative scientific ideals that emphasize interpretation, explanation, or emancipation—approaches that may extend beyond the scope of experimental psychology—have received little serious consideration or credibility. Instead, the reform movement has tended to focus on reinforcing the hypothetico-deductive model of scientific practice.

In The Recurrence of Fundamental Questions, we argue that psychology’s greatest challenge may be its willingness to confront its own limits and reimagine its foundations and ideals. Methodological and administrative solutions are valuable, but they are not enough. Rather, we need to engage with the epistemic processes and ideals at the heart of psychological knowledge production and engage very closely with critical perspectives that analyze and question those epistemic processes and scientific ideals.

References

Faye, C. (2012). American social psychology: Examining the contours of the 1970s crisis. Studies in History and Philosophy of Science, Special Section I: Assimilating Knowledge – Food and Nutrition in Early Modern Physiologies. II. Psychology, A Science in Crisis? A Century of Reflections and Debates, 43(2), 514–521. https://doi.org/10.1016/j.shpsc.2011.11.010

Lakens, D. (2025). Concerns about theorizing, relevance, generalizability, and methodology across two crises in social psychology. International Review of Social Psychology, 38(1), 1–17. https://doi.org/10.5334/irsp.1038

Nosek, B. A., & Bar-Anan, Y. (2012). Scientific Utopia: I. Opening scientific communication. Psychological Inquiry, 23(3), 217–243. https://doi.org/10.1080/1047840X.2012.692215

Rosenthal, R. (1965). The volunteer subject. Human Relations, 18(4), 389–406. https://doi.org/10.1177/001872676501800407

Rosenthal, R. (1966). Experimenter effects in behavioral research (pp. xiii, 500). Appleton-Century-Crofts.

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

Sturm, T., & Mülberger, A. (2012). Crisis discussions in psychology—New historical and philosophical perspectives. Studies in History and Philosophy of Science, Special Section I: Assimilating Knowledge – Food and Nutrition in Early Modern Physiologies. II. Psychology, A Science in Crisis? A Century of Reflections and Debates, 43(2), 425–433. https://doi.org/10.1016/j.shpsc.2011.11.001

Teymoori, A., & Trappes, R. (2026). The recurrence of fundamental questions: A historical and philosophical analysis of major disciplinary crises in psychology. Review of General Psychology. https://doi.org/10.1177/10892680261421875

Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, e1. https://doi.org/10.1017/S0140525X20001685

Share

Subscribe now