2017/02/02 by Shravan Vasishth, Andrew Gelman, Vasishth, Shravan +1
Computer Science · Materials Science · Mathematics · Physics and Astronomy · Social Sciences · #Applications (stat.AP) #Bayesian Methods and Mixture Models #Computational and Text Analysis Methods #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning in Materials Science #Methodology (stat.ME) #Natural Language Processing Techniques #Opinion Dynamics and Social Influence #Statistical Methods and Inference #Statistics Theory (math.ST)
paper · pdf · doi:10.48550/arxiv.1702.00556
openalex publication_date 2017/02/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We show that publishing results using the statistical significance filter---publishing only when the p-value is less than 0.05---leads to a vicious cycle of overoptimistic expectation of the replicability of results. First, we show analytically that when true statistical power is relatively low, computing power based on statistically significant results will lead to overestimates of power. Then, we present a case study using 10 experimental comparisons drawn from a recently published meta-analysis in psycholinguistics (Jäger et al., 2017). We show that the statistically significant results yield an illusion of replicability. This illusion holds even if the researcher doesn't conduct any formal power analysis but just uses statistical significance to informally assess robustness (i.e., replicability) of results.