P-Value Precision and Reproducibility

2011/11/01 by Dennis D. Boos, Leonard A. Stefanski · 170 citations
Computer Science · Decision Sciences · Mathematics · #Artificial intelligence #Computer science #Context (archaeology) #Contrast (vision) #Data Analysis with R #Econometrics #Geography #Mathematics #Meta-analysis and systematic reviews #Multiple comparisons problem #Null hypothesis #Replicate #Reproducibility #Rounding #Sample (material) #Sample size determination #Statistical Methods in Clinical Trials #Statistical hypothesis testing #Statistical power #Statistical significance #Statistics #Value (mathematics) #p-value

paper · pdf · doi:10.1198/tas.2011.10129

published in The American Statistician 65(4), 213-221 (Taylor & Francis)

openalex publication_date 2011/11/01 · openalex created_date 2016/06/24 · openalex updated_date 2026/08/05

Abstract

P-values are useful statistical measures of evidence against a null hypothesis. In contrast to other statistical estimates, however, their sample-to-sample variability is usually not considered or estimated, and therefore not fully appreciated. Via a systematic study of log-scale p-value standard errors, bootstrap prediction bounds, and reproducibility probabilities for future replicate p-values, we show that p-values exhibit surprisingly large variability in typical data situations. In addition to providing context to discussions about the failure of statistical results to replicate, our findings shed light on the relative value of exact p-values vis-a-vis approximate p-values, and indicate that the use of *, **, and *** to denote levels .05, .01, and .001 of statistical significance in subject-matter journals is about the right level of precision for reporting p-values when judged by widely accepted rules for rounding statistical estimates.

Citations

• The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant

Cited by

• Pitfalls of hypothesis tests and model selection on bootstrap samples: Causes and consequences in biometrical applications
• There are natural scores: Full comment on Shafer, "Testing by betting: A strategy for statistical and scientific communication"
• Evaluation of a meta-analysis of the association between red and processed meat and selected human health effects
• Seek for Success: A Visualization Approach for Understanding the Dynamics of Academic Careers
• The vulnerable male brain: Men's spatial abilities are condition-dependent, sexually selected traits
• Models to identify treatments for the acute and persistent effects of seizure‐inducing chemical threat agents
• From one environment to many: The problem of replicability of statistical inferences
• Common variants of the BRCA1 wild-type allele modify the risk of breast cancer in BRCA1 mutation carriers. [europepmc]
• Predictors of delay discounting among smokers: education level and a Utility Measure of Cigarette Reinforcement Efficacy are better predictors than demographics, smoking characteristics, executive functioning, impulsivity, or time perception. [europepmc]
• Models to identify treatments for the acute and persistent effects of seizure-inducing chemical threat agents. [europepmc]
• Three common misuses of P values. [europepmc]
• P value interpretations and considerations. [europepmc]
• Birth Month and Cardiovascular Disease Risk Association: Is meaningfulness in the eye of the beholder? [europepmc]
• KNN-MDR: a learning approach for improving interactions mapping performances in genome wide association studies. [europepmc]
• The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research. [europepmc]
• The more you test, the more you find: The smallest P-values become increasingly enriched with real findings as more tests are conducted. [europepmc]
• Aggression supersedes individual oxygen demand to drive group air-breathing in a social catfish. [europepmc]
• Bayesian prediction intervals for assessing P-value variability in prospective replication studies. [europepmc]
• Effects of cell-cell crosstalk on gene expression patterns in a cell model of renal cell carcinoma lung metastasis. [europepmc]
• Digestive and locomotor capacity show opposing responses to changing food availability in an ambush predatory fish. [europepmc]
• Compensatory Expression of Nur77 and Nurr1 Regulates NF- κ B-Dependent Inflammatory Signaling in Astrocytes. [europepmc]
• Enzymatic Extraction, Purification, and Characterization of Polysaccharides from Penthorum chinense Pursh : Natural Antioxidant and Anti-Inflammatory. [europepmc]
• The reign of the p-value is over: what alternative analyses could we employ to fill the power vacuum? [europepmc]
• Bayesian multilevel model of micro RNA levels in ovarian-cancer and healthy subjects. [europepmc]
• Hippocampus shape deformation: a potential diagnostic biomarker for chronic back pain in women. [europepmc]
• On Causal Inferences for Personalized Medicine: How Hidden Causal Assumptions Led to Erroneous Causal Claims About the D-Value. [europepmc]
• Resveratrol protects against myocardial ischemic injury via the inhibition of NF‑κB‑dependent inflammation and the enhancement of antioxidant defenses. [europepmc]
• Integrated Genomic and Transcriptomic Analysis reveals key genes for predicting dual-phenotype Hepatocellular Carcinoma Prognosis. [europepmc]
• Statistical inference through estimation: Recommendations from the International Society of Physiotherapy Journal Editors. [europepmc]
• Statistical inference through estimation: recommendations from the International Society of Physiotherapy Journal Editors. [europepmc]
• Reconceptualizing the p -value from a likelihood ratio test: a probabilistic pairwise comparison of models based on Kullback-Leibler discrepancy measures. [europepmc]
• A two-step pre-processing tool to remove Gaussian and ectopic noise for heart rate variability analysis. [europepmc]

Related

• Assessment of P-value variability in the current replicability crisis
• Valid P -Values Behave Exactly as They Should: Some Misleading Criticisms of P -Values and Their Resolution With S -Values
• What Have We (Not) Learnt from Millions of Scientific Papers with P Values?
• Thou Shalt Not Reject the P-value
• P value functions: An underused method to present research results and to promote quantitative reasoning