Questionable research practices may have little effect on replicability

2020/09/15 by Rolf Ulrich, Jeff Miller · 1 voice
Decision Sciences · Mathematics · Computer Science · #Meta-analysis and systematic reviews #Statistical Methods in Clinical Trials #Explainable Artificial Intelligence (XAI)

paper · doi:10.7554/elife.58237

openalex publication_date 2020/09/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/11

Abstract

This article examines why many studies fail to replicate statistically significant published results. We address this issue within a general statistical framework that also allows us to include various questionable research practices (QRPs) that are thought to reduce replicability. The analyses indicate that the base rate of true effects is the major factor that determines the replication rate of scientific results. Specifically, for purely statistical reasons, replicability is low in research domains where true effects are rare (e.g., search for effective drugs in pharmacology). This point is under-appreciated in current scientific and media discussions of replicability, which often attribute poor replicability mainly to QRPs.

Citations

• Estimating the reproducibility of psychological science
• Correlational effect size benchmarks.
• The file drawer problem and tolerance for null results.
• Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
• Power failure: why small sample size undermines the reliability of neuroscience
• Recommendations for Increasing Replicability in Psychology
• Why Most Published Research Findings Are False
• False-Positive Psychology
• Making replication mainstream
• Effect Size Estimation Fromt-Statistics in the Presence of Publication Bias
• P-curve: A key to the file-drawer.

Discussions

• The next paper: Questionable research practices may have little effect on replicability by Ulrich & Miller doi.org/10.7554/eLif... [bsky, 0 points, 1 comments]