Why "Redefining Statistical Significance" Will Not Improve\n Reproducibility and Could Make the Replication Crisis Worse

2017/11/21 by Harry Crane, Crane, Harry
Psychology · #Applications (stat.AP) #FOS: Computer and information sciences #Mental Health Research Topics #Methodology (stat.ME) #Other Statistics (stat.OT)

paper · pdf · doi:10.48550/arxiv.1711.07801

openalex publication_date 2017/11/21 · openalex created_date 2022/09/25 · openalex updated_date 2026/07/28

Abstract

A recent proposal to "redefine statistical significance" (Benjamin, et al.\nNature Human Behaviour, 2017) claims that false positive rates "would\nimmediately improve" by factors greater than two and replication rates would\ndouble simply by changing the conventional cutoff for 'statistical\nsignificance' from P<0.05 to P<0.005. I analyze the veracity of these claims,\nfocusing especially on how Benjamin, et al neglect the effects of P-hacking in\nassessing the impact of their proposal. My analysis shows that once P-hacking\nis accounted for the perceived benefits of the lower threshold all but\ndisappear, prompting two main conclusions: (i) The claimed improvements to\nfalse positive rate and replication rate in Benjamin, et al (2017) are\nexaggerated and misleading. (ii) There are plausible scenarios under which the\nlower cutoff will make the replication crisis worse.\n

Related

• The statistical significance filter leads to overoptimistic expectations of replicability
• Understanding the Replication Crisis as a Base Rate Fallacy
• The statistical significance filter leads to overconfident expectations of replicability
• The Certainty Bound: Structural Limits on Scientific Reliability
• Why multiple hypothesis test corrections provide poor control of false positives in the real world