2020/10/16 by Fintan Costello, Costello, Fintan, Paul Watts +1
Agricultural and Biological Sciences · Mathematics · Psychology · #Alternative hypothesis #Computer science #Data mining #Econometrics #Epistemology #FOS: Computer and information sciences #Mathematics #Mental Health Research Topics #Methodology (stat.ME) #Null (SQL) #Null hypothesis #Point (geometry) #Replication (statistics) #Sensory Analysis and Statistical Methods #Simple (philosophy) #Statistical Methods in Clinical Trials #Statistical hypothesis testing #Statistics #stat.ME
paper · pdf · doi:10.48550/arxiv.2010.09209
published in arXiv (Cornell University) (Cornell University) · unpublished, 32 pages. arXiv admin note: text overlap with arXiv:2010.07813
arxiv created 2020/10/16 · openalex publication_date 2020/10/16 · arxiv updated 2020/10/20 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Null Hypothesis Significance Testing (NHST) has long been of central importance to psychology as a science, guiding theory development and underlying the application of evidence-based intervention and decision-making. Recent years, however, have seen growing awareness of serious problems with NHST as it is typically used; this awareness has led to proposals to limit the use of NHST techniques, to abandon these techniques and move to alternative statistical approaches, or even to ban the use of NHST entirely. These proposals are premature, because the observed problems with NHST arise as a consequence of an historically contingent, essentially unmotivated, and fundamentally incorrect, choice: that of NHST testing against point-form null hypotheses. Using simple counting experiments we give a detailed presentation of an alternative, more general approach: that of testing against distributional nulls. We show that this distributional approach is well-motivated mathematically, practically and experimentally, and that the use of distributional nulls addresses various problems with the standard point-form NHST approach, avoiding issues to do with sample size and allowing a coherent estimation of the probability of replication of a given experimental result. Rather than abandoning NHST, we should use the NHST approach in its most general form, with distributional rather than point-form null hypotheses.