Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results*

2018/11/19 by Alwyn Young · 616 citations
Mathematics · Psychology · #Biology #Clinical trial #Econometrics #Insignificance #Mathematics #Psychology #Randomization #Social psychology #Statistical Distribution Estimation and Applications #Statistical Methods and Bayesian Inference #Statistical Methods and Inference #Statistical analysis #Statistical hypothesis testing #Statistics

paper · open access · doi:10.1093/qje/qjy029

published in The Quarterly Journal of Economics 134(2), 557-598 (Oxford University Press)

openalex publication_date 2018/11/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

I follow R. A. Fisher's,The Design of Experiments (1935), using randomization statistical inference to test the null hypothesis of no treatment effects in a comprehensive sample of 53 experimental papers drawn from the journals of the American Economic Association. In the average paper, randomization tests of the significance of individual treatment effects find 13% to 22% fewer significant results than are found using authors’ methods. In joint tests of multiple treatment effects appearing together in tables, randomization tests yield 33% to 49% fewer statistically significant results than conventional tests. Bootstrap and jackknife methods support and confirm the randomization results.

Citations

• A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity
• Recent Developments in the Econometrics of Program Evaluation

Cited by

• Standard Errors for Difference‐in‐Difference Regression
• Trust in the Judiciary and Partisan Reactions to Judicial Checks: Evidence from Argentina
• How Credible Is the Credibility Revolution?
• How terrorism affects support for democracy
• The value of leadership: Evidence from a large-scale field experiment
• A Non-Bipartite Matching Framework for Difference-in-Differences with General Treatment Types
• Conditionality vs. autonomy: What works best in social assistance?
• Are Men Less Generous to a Smarter Woman? Evidence from a Dictator Game Experiment
• The Exact Variance of the Average Treatment Effect Estimator in Cluster Randomized Controlled Trials
• Sponsorship Effects in Online Surveys
• Good Politicians: Experimental Evidence on Motivations for Political Candidacy and Government Performance
• Some Finite Sample Properties of the Sign Test
• Understanding and misunderstanding randomized controlled trials
• A Simple Robust Procedure in Instrumental Variables Regression
• The Relative Effectiveness of Teachers and Learning Software: Evidence from a Field Experiment in El Salvador
• Government Transparency and Innovation: Evidence from Wireless Products
• Randomization inference for difference-in-differences with few treated clusters
• The Sound of Silence: Championing Democracy in an Authoritarian Society
• Data selection: at the interface of PDE-based inverse problem and randomized linear algebra
• Contact and Commitment to Development: Evidence from quasi‐random missionary assignments
• Bootstrap Diagnostic Tests
• Who increases emergency department use? New insights from the Oregon health insurance experiment
• The economic impacts of ecosystem disruptions: Costs from substituting biological pest control
• Can a light-touch graduation model enhance livelihood outcomes? Evidence from Ethiopia
• Beyond Bonferroni: Hierarchical Multiple Testing in Empirical Research
• Randomization Inference with Sample Attrition
• Second generation effects of an experimental conditional cash transfer program on early childhood human capital in Nicaragua
• Experimental long-term effects of early-childhood and school-age exposure to a conditional cash transfer program
• Cluster-robust inference: A guide to empirical practice
• The Perry Preschoolers at Late Midlife: A Study in Design-Specific Inference
• Matching Estimators with Few Treated and Many Control Observations
• Healthy, nudged, and wise: Experimental evidence on the role of information salience in reducing tobacco intake. [europepmc]
• Voluntary 'donations' versus reward-oriented 'contributions': two experiments on framing in funding mechanisms. [europepmc]
• The impact of mobile money on long-term poverty: evidence from Bangladesh. [europepmc]
• Impact of small farmers' access to improved seeds and deforestation in DR Congo. [europepmc]
• Public support for more stringent vaccine policies increases with vaccine effectiveness. [europepmc]
• Virtual reality is only mildly effective in improving forest conservation behaviors. [europepmc]
• MORE interpretable multi-omic regulatory networks to characterise phenotypes. [europepmc]
• Spurious precision in meta-analysis of observational research. [europepmc]

Related

• Fisher's Randomization Test for Causality with General Types of Treatments
• A randomization-based perspective of analysis of variance: a test statistic robust to treatment effect heterogeneity
• A paradox from randomization-based causal inference
• Leveraging the Fisher randomization test using confidence distributions: inference, combination and fusion learning
• Beyond the Sharp Null: Randomization Inference, Bounded Null Hypotheses,\n and Confidence Intervals for Maximum Effects