P-curve: A key to the file-drawer.

2013/07/15 by Uri Simonsohn, Leif D. Nelson, Joseph P. Simmons · 1 voice · 1,461 citations
Computer Science · Decision Sciences · Mathematics · Psychology · #Computer science #Computer security #Explainable Artificial Intelligence (XAI) #Hacker #Key (lock) #Mathematics #Meta-analysis and systematic reviews #Psychology #Set (abstract data type) #Statistics #Value (mathematics) #scientometrics and bibliometrics research

paper · doi:10.1037/a0033242

published in Journal of Experimental Psychology General 143(2), 534-547 (American Psychological Association)

openalex publication_date 2013/07/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that "work," readers must ask, "Are these effects true, or do they merely reflect selective reporting?" We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves-containing more low (.01s) than high (.04s) significant p values--only right-skewed p--curves are diagnostic of evidential value. By telling us whether we can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.

Citations

• The file drawer problem and tolerance for null results.
• Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis
• A power primer.
• Statistical power analysis for the behavioral sciences
• Consequences of prejudice against the null hypothesis.
• The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
• The case for motivated reasoning.
• False-Positive Psychology

Cited by

• Eating while distracted: a systematic review and meta-analysis of the effect of distraction on concurrent and later energy intake in adults
• The Certainty Bound: Structural Limits on Scientific Reliability
• Are supplementary materials the new file drawer?
• The Issue with Special Issues: when Guest Editors Publish in Support of Self
• Living Synthetic Benchmarks: A Neutral and Cumulative Framework for Simulation Studies
• Bestiary of Questionable Research Practices in Psychology
• From significant to meaningful: ATOMizing the study of sex differences and similarities
• One Decade Into the Replication Crisis, How Have Psychological Results Changed?
• International initiatives to enhance awareness and uptake of open research in psychology: a systematic mapping review
• Distinguishing the affective and cognitive bases of implicit attitudes to improve prediction of food choices
• Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology
• Are many sex/gender differences really power differences?
• In search of the lost interaction: A theoretical and methodological framework for researching interactions
• Threats of a replication crisis in empirical computer science
• Modelling in the midst of a replication crisis for embodied action semantics: a commentary on Carriere et al. (2026)
• Trash-talking: Competitive incivility motivates rivalry, performance, and unethical behavior
• A systematic review and meta-analysis of the impact of cash transfers on subjective well-being and mental health in low- and middle-income countries
• Responsible Research Assessment I: Implementing DORA and CoARA for hiring and promotion in psychology
• No meta-analytical effect of economic inequality on well-being or mental health
• Beyond the p Value Dichotomy: Alternatives for Statistical Inference—A Critical Review
• The replication crisis has led to positive structural, procedural, and community changes
• Assessing Publication Bias: a 7-Step User’s Guide with Best-Practice Recommendations
• A Validity-Based Framework for Understanding Replication in Psychology
• Transparency in Practice in Qualitative Research
• Someone Else’s Risk: Meta-analytical Evidence of a Self-Other Discrepancy in Climate Change-related Risk Perceptions
• Children’s Theory of Mind and Academic Achievement: a Meta-analysis on Reading Comprehension and Math Skills
• Bayesian stopping
• Meaningful Associations Redux: Quantifying and interpreting effect size in the context of the Adolescent Brain and Cognitive Development study
• Focused deterrence can reduce crime: A systematic review of randomized controlled trials and quasi‐experiments
• Lay belief about AI and its decision-making
• Trends in the distribution of P values in epidemiology journals: a statistical, P -curve, and simulation study
• A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics
• 281. MEVATRON77DX67 における電子線用可変型絞りについて(治療-6 電子線)
• Oral language interventions can improve language outcomes in children with neurodevelopmental disorders: A systematic review and meta‐analysis
• Anything goes for participant, patient and public involvement in youth mental health research
• Effects of reducing sedentary behaviour by increasing physical activity, on cognitive function, brain function and structure across the lifespan: a systematic review and meta-analysis
• Effect Size Estimation Fromt-Statistics in the Presence of Publication Bias
• Uncovering null effects in null fields: the case of homeopathy
• Academic integrity across educational levels: Exploring students’ engagement with grey-zone and non-compliant practices in four European countries
• What can we learn from 1,000 meta-analyses across 10 different disciplines?
• A Meta‐Analysis of the Correlation Between Socioeconomic Status and Direct Measures of Young Children's Word Comprehension
• Robust Bayesian meta‐analysis: Model‐averaging across complementary publication bias adjustment methods
• Replication in Social Science
• Looking our limitations in the eye: A call for more thorough and honest reporting of study limitations
• On the predicted replicability of two decades of experimental research on system justification: A Z‐curve analysis
• Increasing Transparency Through a Multiverse Analysis
• Open science, the replication crisis, and environmental public health
• Open minds, tied hands: Awareness, behavior, and reasoning on open science and irresponsible research behavior
• Big little lies: a compendium and simulation of p -hacking strategies
• Methods for testing publication bias in ecological and evolutionary meta‐analyses
• Cherry Picking with Synthetic Controls
• From review to synthesis: A step-by-step methodological guide to systematic reviews and multilevel meta-analyses
• The Importance of Clinical Relevance and the Limitations of P-Values in Plastic Surgery Research
• Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
• Small Telescopes
• A multiverse lack of replication for working memory capacity as moderator of the perceptual disfluency effect
• Predicting Replication Rates with Z-Curve: A Brief Exploratory Validation Study Using the Replication Database
• Apathy and Physical Activity: A Systematic Review and Meta-Analysis
• Publication bias, statistical power and reporting practices in the Journal of Sports Sciences: potential barriers to replicability
• Is time of the essence? A temporal meta-synthesis of seven media effects theories
• Relationship Between Fear of Movement and Physical Activity in Patients With Cardiac, Rheumatologic, Neurologic, Pulmonary, or Pain Conditions: A Systematic Review and Meta-Analysis
• Sex differences in trust and trustworthiness: A meta-analysis of the trust game and the gift-exchange game
• Ten simple rules for designing and conducting undergraduate replication projects
• Evaluation of a meta-analysis of the association between red and processed meat and selected human health effects
• Childhood maltreatment and intimate partner violence victimization: A meta-analysis
• Global p-Values in Multi-Design Studies
• Do subtle reminders of money change people’s political views?
• When is p-hacking detectable?
• Reliability of meta-analysis of an association between ambient air quality and development of asthma later in life
• Questionable research practices may have little effect on replicability
• A meta‐analysis on the effects of just‐below versus round prices
• Detecting p‐Hacking
• Why are replication rates so low?
• Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
• Pre-registration: Weighing costs and benefits for researchers
• Misstatements, misperceptions, and mistakes in controlling for covariates in observational research
• Presenting the Probabilities of Different Effect Sizes: Towards a Better Understanding and Communication of Statistical Uncertainty
• Dissecting the tension of open science standards implementation in management and organization journals
• The Bitter Truth About Sugar and Willpower
• Using large language models to generate silicon samples in consumer and marketing research: Challenges, opportunities, and guidelines
• Does lack of control lead to conspiracy beliefs? A meta‐analysis
• Large Cross-National Differences in Gene × Socioeconomic Status Interaction on Intelligence
• Replicability and Reproducibility in Comparative Psychology
• Kids These Days! Increasing delay of gratification ability over the past 50 years in children
• Brain volume and intelligence: The moderating role of intelligence measurement quality
• The prevalence of statistical reporting errors in psychology (1985–2013)
• The Extent and Consequences of P-Hacking in Science
• p -Curve and Effect Size
• Why does perfectionism confer risk for depressive symptoms? A meta-analytic test of the mediating role of stress and social disconnection
• Self-Enhancement and Psychological Adjustment: A Meta-Analytic Review
• Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
• Pair-bonding, fatherhood, and the role of testosterone: A meta-analytic review
• Publication bias [wikipedia]

Discussions

• Wie konnte das sein? Bad practice 1: p-Hacking. Daten so lange anders analysieren (andere Stichproben, Ausreißer raus, andere Tests), bis Signifikantes rauskommt (oft: p < .05). Heißt m.E.: Daten ausq [bsky, 4 points, 1 comments]

Related

• p -Curve and Effect Size
• When is p-hacking detectable?
• What Have We (Not) Learnt from Millions of Scientific Papers with P Values?
• The Extent and Consequences of P-Hacking in Science
• Valid P -Values Behave Exactly as They Should: Some Misleading Criticisms of P -Values and Their Resolution With S -Values