2024/08/15 by Kevin Lang · 1 voice · 4 citations
Biochemistry, Genetics and Molecular Biology · Mathematics · #Advanced Causal Inference Techniques #Biomedical Text Mining and Ontologies #Credibility #Economics #Labour economics #Law #Political science #Statistical Methods in Clinical Trials
paper · doi:10.1086/732772
published in Journal of Labor Economics 43(2), 635-663 (University of Chicago Press)
openalex publication_date 2024/08/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/19
Economists analyzing a well-conducted randomized controlled trial or natural experiment and finding a statistically significant effect conclude that the null of no effect is unlikely to be true. But how frequently is this conclusion warranted? The answer depends on the proportion of tested nulls that are true and the test's power. I model the distribution of t-statistics in leading economics journals. Using my preferred model, 65% of narrowly rejected null hypotheses and 41% of all rejected null hypotheses with |t|<10 are likely to be false rejections. For the null to have only a .05 probability of being true requires a t of 5.48.