Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results*

2018/11/19 by Alwyn Young · 22 citations
Mathematics · #Statistical Methods and Inference #Statistical Methods and Bayesian Inference #Statistical Distribution Estimation and Applications

paper · doi:10.1093/qje/qjy029

Abstract

I follow R. A. Fisher's The Design of Experiments (1935), using randomization statistical inference to test the null hypothesis of no treatment effects in a comprehensive sample of 53 experimental papers drawn from the journals of the American Economic Association. In the average paper, randomization tests of the significance of individual treatment effects find 13% to 22% fewer significant results than are found using authors’ methods. In joint tests of multiple treatment effects appearing together in tables, randomization tests yield 33% to 49% fewer statistically significant results than conventional tests. Bootstrap and jackknife methods support and confirm the randomization results.

Citations

• A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity
• Recent Developments in the Econometrics of Program Evaluation

Cited by

• Standard Errors for Difference‐in‐Difference Regression
• Trust in the Judiciary and Partisan Reactions to Judicial Checks: Evidence from Argentina
• How Credible Is the Credibility Revolution?
• How terrorism affects support for democracy
• The value of leadership: Evidence from a large-scale field experiment
• A Non-Bipartite Matching Framework for Difference-in-Differences with General Treatment Types
• Conditionality vs. autonomy: What works best in social assistance?
• Are Men Less Generous to a Smarter Woman? Evidence from a Dictator Game Experiment
• The Exact Variance of the Average Treatment Effect Estimator in Cluster Randomized Controlled Trials
• Sponsorship Effects in Online Surveys
• Good Politicians: Experimental Evidence on Motivations for Political Candidacy and Government Performance
• Some Finite Sample Properties of the Sign Test
• Understanding and misunderstanding randomized controlled trials
• A Simple Robust Procedure in Instrumental Variables Regression
• The Relative Effectiveness of Teachers and Learning Software: Evidence from a Field Experiment in El Salvador
• Government Transparency and Innovation: Evidence from Wireless Products
• Randomization inference for difference-in-differences with few treated clusters
• The Sound of Silence: Championing Democracy in an Authoritarian Society
• Data selection: at the interface of PDE-based inverse problem and randomized linear algebra
• Contact and Commitment to Development: Evidence from quasi‐random missionary assignments
• Bootstrap Diagnostic Tests
• The economic impacts of ecosystem disruptions: Costs from substituting biological pest control
• Can a light-touch graduation model enhance livelihood outcomes? Evidence from Ethiopia

Related

• Fisher's Randomization Test for Causality with General Types of Treatments
• A randomization-based perspective of analysis of variance: a test statistic robust to treatment effect heterogeneity
• A paradox from randomization-based causal inference
• Leveraging the Fisher randomization test using confidence distributions: inference, combination and fusion learning
• Beyond the Sharp Null: Randomization Inference, Bounded Null Hypotheses, and Confidence Intervals for Maximum Effects