2019/01/21 by Daniel J. Schad, Shravan Vasishth, Schad, Daniel J. +1
Mathematics · #Applications (stat.AP) #FOS: Computer and information sciences #Statistical Methods and Bayesian Inference #Statistical Methods and Inference #Statistical Methods in Clinical Trials
paper · pdf · doi:10.48550/arxiv.1901.06889
openalex publication_date 2019/01/21 · openalex created_date 2022/07/30 · openalex updated_date 2026/07/28
When researchers carry out a null hypothesis significance test, it is\ntempting to assume that a statistically significant result lowers Prob(H0), the\nprobability of the null hypothesis being true. Technically, such a statement is\nmeaningless for various reasons: e.g., the null hypothesis does not have a\nprobability associated with it. However, it is possible to relax certain\nassumptions to compute the posterior probability Prob(H0) under repeated\nsampling. We show in a step-by-step guide that the intuitively appealing\nbelief, that Prob(H0) is low when significant results have been obtained under\nrepeated sampling, is in general incorrect and depends greatly on: (a) the\nprior probability of the null being true; (b) type-I error rate, (c) type-II\nerror rate, and (d) replication of a result. Through step-by-step simulations\nusing open-source code in the R System of Statistical Computing, we show that\nuncertainty about the null hypothesis being true often remains high despite a\nsignificant result. To help the reader develop intuitions about this common\nmisconception, we provide a Shiny app\n(https://danielschad.shinyapps.io/probnull/). We expect that this tutorial will\nhelp researchers better understand and judge results from null hypothesis\nsignificance tests.\n