Abandon Statistical Significance
2017/09/30 by Blakeley B. McShane, David Gal, Andrew Gelman +2 · 1 voice · 819 citations
Computer Science · Decision Sciences · Mathematics · #Bayes' theorem #Data Analysis with R #Frequentist probability #Meta-analysis and systematic reviews #Novelty #Null hypothesis #Process (computing) #Replication (statistics) #Statistical Methods in Clinical Trials #Statistical hypothesis testing #Statistical model #msc:97K70 #stat.ME
paper · pdf · open access · doi:10.1080/00031305.2018.1527253
published in The American Statistician 73(sup1), 235-245 (Taylor & Francis)
openalex created_date 2017/10/06 · arxiv created 2018/09/08 · openalex publication_date 2019/03/20 · arxiv updated 2021/07/21 · openalex updated_date 2026/08/05
Abstract
We discuss problems the null hypothesis significance testing (NHST) paradigm poses for replication and more broadly in the biomedical and social sciences as well as how these problems remain unresolved by proposals involving modified p-value thresholds, confidence intervals, and Bayes factors. We then discuss our own proposal, which is to abandon statistical significance. We recommend dropping the NHST paradigm--and the p-value thresholds intrinsic to it--as the default statistical paradigm for research, publication, and discovery in the biomedical and social sciences. Specifically, we propose that the p-value be demoted from its threshold screening role and instead, treated continuously, be considered along with currently subordinate factors (e.g., related prior evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain) as just one among many pieces of evidence. We have no desire to "ban" p-values or other purely statistical measures. Rather, we believe that such measures should not be thresholded and that, thresholded or not, they should not take priority over the currently subordinate factors. We also argue that it seldom makes sense to calibrate evidence as a function of p-values or other purely statistical measures. We offer recommendations for how our proposal can be implemented in the scientific publication process as well as in statistical decision making more broadly.
Cited by
- Understanding the Replication Crisis as a Base Rate Fallacy
- Moving to a World Beyond “ p < 0.05”
- Prolonged Opioid Use and Pain Outcome and Associated Factors after Surgery under General Anesthesia: A Prospective Cohort Association Multicenter Study
- Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication
- Using and Interpreting Fixed Effects Models
- Recommendations to improve use and reporting of statistics in animal experiments
- What Good Is a Regression? Inference to the Best Explanation and the Practice of Political Science Research
- From significant to meaningful: ATOMizing the study of sex differences and similarities
- Why and how we should join the shift from significance testing to estimation
- With Low Power Comes Low Credibility? Toward a Principled Critique of Results From Underpowered Tests
- School’s sex-composition, teachers’ accuracy and gender stereotypes: explanations for teachers’ different teachability perceptions of boys and girls
- Power, precision, and sample size estimation in sport and exercise science research
- Achieving Statistical Significance with Control Variables and Without Transparency
- Anchoring effect in legal decision-making: A meta-analysis.
- A New Look at P Values for Randomized Clinical Trials
- Alternative to the statistical mass confusion of testing for “no effect”
- Death by p-value: the overreliance on p-values in critical care research
- The case for formal methodology in scientific reform
- Threats of a replication crisis in empirical computer science
- Divergence versus decision P ‐values: A distinction worth making in theory and keeping in practice: Or, how divergence P ‐values measure evidence even when decision P ‐values do not
- Aggressive policing and undermined legitimacy: assessing the impact of police stops at gunpoint on perceptions of police in São Paulo, Brazil
- Bayes factors for two-group comparisons in Cox regression with an application for reverse-engineering raw data from summary statistics
- The generalizability crisis
- How can we make sound replication decisions?
- Predicting replication outcomes and generalizability of results
- Reevaluating the Clinical Relevance of Bacterial Vaginosis–Associated Immune Responses in HIV Infection
- Cross-platform edge deployment of machine learning models: a model-driven approach
- Four principles for improved statistical ecology
- Environmental stress differentially affects phenotypic modularity and fluctuating asymmetry in generalist and specialist cactophilic Drosophila
- Testing maximum entropy models with e-values
- Measuring selection on reaction norms: Lack’s principle and plasticity in clutch size
- The Impact of Coronavirus (COVID-19) Related Public-Health Measures on Training Behaviours of Individuals Previously Participating in Resistance Training: A Cross-Sectional Survey Study
- A meta‐analysis on the effects of just‐below versus round prices
- Sample size evolution in neuroimaging research: An evaluation of highly-cited studies (1990–2012) and of latest practices (2017–2018) in high-impact journals
- Bayesian Hierarchical Models for Quantitative Estimates for Performance metrics applied to Saddle Search Algorithms
- Primate social organization evolved from a flexible pair-living ancestor
- Mindsets, contexts, and college enrollment: Taking the long view on growth mindset beliefs at the transition to high school
- A vállalatok egyes sajátosságai és a kutatás-fejlesztési tevékenység közötti összefüggések
- Neyman–Pearson lemma for Bayes factors
- Connecting simple and precise P‐values to complex and ambiguous realities (includes rejoinder to comments on “Divergence vs. decision P‐values”)
- Estimation for Better Inference in Neuroscience. [europepmc]
- A Bilingual Advantage? An Appeal for a Change in Perspective and Recommendations for Future Research. [europepmc]
- Comparing functional connectivity matrices: A geometry-aware approach applied to participant identification. [europepmc]
- Interactions between emotion and action in the brain. [europepmc]
- Do the anatomical and physiological properties of a muscle determine its adaptive response to different loading protocols? [europepmc]
- Dataset decay and the problem of sequential analyses on open datasets. [europepmc]
- Association of Histological and Clinical Chorioamnionitis With Neonatal Sepsis Among Preterm Infants: A Systematic Review, Meta-Analysis, and Meta-Regression. [europepmc]
- Replications in Comparative Cognition: What Should We Expect and How Can We Improve? [europepmc]
- Improving practices and inferences in developmental cognitive neuroscience. [europepmc]
- Harsh environments promote alloparental care across human societies. [europepmc]
- Effects of age on electrophysiological measures of cochlear synaptopathy in humans. [europepmc]
- Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise. [europepmc]
- Controllability over stressor decreases responses in key threat-related brain areas. [europepmc]
- The Impact of Coronavirus (COVID-19) Related Public-Health Measures on Training Behaviours of Individuals Previously Participating in Resistance Training: A Cross-Sectional Survey Study. [europepmc]
- The multiplicity of analysis strategies jeopardizes replicability: lessons learned across disciplines. [europepmc]
- Aversive Conditioning of Spatial Position Sharpens Neural Population-Level Tuning in Visual Cortex and Selectively Alters Alpha-Band Activity. [europepmc]
- The case for formal methodology in scientific reform. [europepmc]
- Bayesian inference of population prevalence. [europepmc]
- Slow and Steady, or Hard and Fast? A Systematic Review and Meta-Analysis of Studies Comparing Body Composition Changes between Interval Training and Moderate Intensity Continuous Training. [europepmc]
- Country-specific estimates of unintended pregnancy and abortion incidence: a global comparative analysis of levels in 2015-2019. [europepmc]
- Why and how we should join the shift from significance testing to estimation. [europepmc]
- Strengthening causal inference from randomised controlled trials of complex interventions. [europepmc]
- Ecological Momentary Assessment: A Systematic Review of Validity Research. [europepmc]
- Highlight results, don't hide them: Enhance interpretation, reduce biases and improve reproducibility. [europepmc]
- The Eccentric:Concentric Strength Ratio of Human Skeletal Muscle In Vivo: Meta-analysis of the Influences of Sex, Age, Joint Action, and Velocity. [europepmc]
Discussions
Related