Estimating the reproducibility of psychological science

2015/08/27 · 2 voices · 295 citations
Decision Sciences · Psychology · #Meta-analysis and systematic reviews #scientometrics and bibliometrics research #Mental Health Research Topics

paper · doi:10.1126/science.aac4716

Abstract

Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

Citations

• Studies in the Logic of Explanation
• The file drawer problem and tolerance for null results.
• Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
• Consequences of prejudice against the null hypothesis.

Cited by

• Is economics self‐correcting? Replications in the American Economic Review
• Reproducibility, replicability, and robustness in corpus linguistics
• Toward open science in marketing research
• Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse
• Do researchers claim their findings are applicable to the real world? An empirical study of social and personality psychology articlesDo researchers claim their findings are applicable to the real world? An empirical study of social and personality psychology articles
• Empirically derived guidelines for effect size interpretation in social psychology
• Power, precision, and sample size estimation in sport and exercise science research
• “Excellence R Us”: university research and the fetishisation of excellence
• Navigating the Science System: Research Integrity and Academic Survival Strategies
• Replication – The ugly duckling of science?
• Enhancing precision in human neuroscience
• Nothing's plenty: The significance of null results in physics education research
• The same, only different: Smartphone‐based dietary Ecological Momentary Assessment tools vary in complexity, usability and active information processing
• ‘Replication games’ test the robustness of social-science studies
• The time has come for psychology to stop treating qualitative data as an embarrassing secret
• Statistical testing in a Linear Probability Space
• Perspective Getting and Cross-Party Support for Electoral Fairness
• Why and how we should join the shift from significance testing to estimation
• Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the “Many Analysts, One Data Set” Project
• Automated alignment is harder than you think
• When is enough enough? Empirical guidelines to determine participant sample size for scene viewing studies
• Critically Engaged Pragmatism: Scientific Norm and Social, Pragmatist Epistemology for AI Science Evaluation Tools
• The Prevalence of Inappropriate Image Duplication in Biomedical Research Publications
• Why Hypothesis Testers Should Spend Less Time Testing Hypotheses
• AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science
• Peer Community In and Peer Community Journal: a two-step diamond OA process giving research communities back control of publishing
• Investigating the replicability of the social and behavioural sciences
• Systematizing the Interpretation of Quantum Theory via Reconstruction
• Generalizability of choice architecture interventions
• Information transfer within and between autistic and non-autistic people
• Over-representation of phonological features in basic vocabulary doesn't replicate when controlling for spatial and phylogenetic effects
• The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network
• On the reliability of published findings using the regression discontinuity design in political science
• Good Fit Is Weak Evidence of Replication: Increasing Rigor Through Prior Predictive Similarity Checking
• Analyzing Brain Circuits in Population Neuroscience: A Case to Be a Bayesian
• An Evaluation of Four Solutions to the Forking Paths Problem: Adjusted Alpha, Preregistration, Sensitivity Analyses, and Abandoning the Neyman-Pearson Approach
• Bayesian probabilistic exploration of Bitcoin informational quanta and interactions under the GITT-VT paradigm
• Replication of “null results” – Absence of evidence or evidence of absence?
• Theory‐Refuting Findings in Psychology: How Much Should They Matter?
• Individual differences do not mask effects of unconscious processing
• From significance testing to estimation and Open Science: How esci can help
• Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses
• Peer Review at the Crossroads
• Meaningful Associations Redux: Quantifying and interpreting effect size in the context of the Adolescent Brain and Cognitive Development study
• Three-Sided Testing to Establish Practical Significance: A Tutorial
• Reproducibility in the Social Sciences
• Minimizing Mistakes in Psychological Science
• Authority Knows No Gender – Gender Effects in Exerting Obedience in Milgram’s Experiment
• Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies
• Phi, Fei, Fo, Fum: Effect Sizes for Categorical Data That Use the Chi-Squared Statistic
• Ten recommendations for reducing the carbon footprint of research computing in human neuroimaging
• Partnering With Authors to Enhance Reproducibility at JASA
• Is Open Access Criminology Influential?
• Between People and Paper: Inhabiting Experiment in a Journal Club
• Attention to Authenticity: An Essential Analogue to Focus on Rigor and Replicability
• Answers to 18 Questions About Open Science Practices
• Replication studies: a win-win for early-career training and behavioral ecology
• From Hypotheses to Systematicity and Back
• The Rise, Impact, and Imbalances of Big-Team Psychology
• Improving Social Science: Lessons from the Open Science Movement
• Breaking barriers towards credible research: Outcomes of a workshop at the BNA Festival of Neuroscience 2025
• Primer for Experimental Methods in Organization Theory
• With Low Power Comes Low Credibility? Toward a Principled Critique of Results From Underpowered Tests
• Laboratory mice engrafted with natural gut microbiota possess a wildling-like phenotype
• False-positive neuroimaging: Undisclosed flexibility in testing spatial hypotheses allows presenting anything as a replicated finding
• Better inference from size-selectivity and catch-comparison studies using simultaneous bootstrap confidence bands
• Interoceptive Mechanisms and Emotional Processing
• Choosing the correlation structure of mixed effect models for experiments with stimuli
• Inter-Rater Reliability in Assessing the Methodological Quality of Research Papers in Psychology
• Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies
• Extending the Cooperative Phenotype: Assessing the Stability of Cooperation across Countries
• The pivot penalty in research
• Reproducibility and robustness of economics and political science research
• The Replication Database: Documenting the Replicability of Psychological Science
• Reproducibility in chemistry research
• Psychosis and bipolar disorder risk in child and adolescent mental health services in the UK: population cohort study
• Novelty, Consistency, Transparency: The Trilemma of Psychological Sciences and its Consequences on Open Science Practices
• Linguistic alignment with an artificial agent: A commentary and re-analysis
• Statistics in service of metascience: Measuring replication distance with reproducibility rate
• Addressing two blind spots of commonly used experimental designs: The Highly‐Repeated Within‐Person approach
• The social psychology of everyday life
• Increasing Precision without Altering Treatment Effects: Repeated Measures Designs in Survey Experiments
• The case for formal methodology in scientific reform
• In search of the lost interaction: A theoretical and methodological framework for researching interactions
• Experts’ reasoning about the replication crisis: Apt epistemic performance and actor-oriented transfer
• ARTEM ‐ IS for ERP : Agreed Reporting Template for EEG Methodology—International Standard for Event‐Related Potential Experiments
• No Replication, No Trust? How Low Replicability Influences Trust in Psychology
• Towards More Robust Nanobioscience: Launching the NanoBubbles Replication Initiative
• Megastudy testing 25 treatments to reduce antidemocratic attitudes and partisan animosity
• The link between Empathy and Forgiveness: Replication and extensions Registered Report of McCullough et al. (1997)'s Study 1
• Editorial commitment to trust and integrity in science: implications for pain and anaesthesiology research*
• Editorial commitment to trust and integrity in science: Implications for pain and anesthesiology research
• Towards a critical‐conceptual analysis of ‘research culture’
• Ten simple rules for good research practice
• Default Bayesian Model Selection of Constrained Multivariate Normal Linear Models
• Peer Review as an Evolving Response to Organizational Constraint: Evidence from Sociology Journals, 1952–2018
• Integrating Media Selection and Media Effects Using Decision Theory
• The Artificial Social Agent Questionnaire (ASAQ) — Development and evaluation of a validated instrument for capturing human interaction experiences with artificial social agents
• Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology
• Registered report: Survey on attitudes and experiences regarding preregistration in psychological research
• Towards reproducible systematic reviews in Open, Distance, and Digital Education—An umbrella mapping review
• The ageing of the social mind: replicating the preservation of socio-affective and the decline of socio-cognitive processes in old age
• Fitting z-curves to estimate the size of the UESD file drawer and the replicability of published findings
• Registered Reports in School Psychology Research: Initial Experiences, Analyses, and Future
• The reproducibility crisis in phylogenetic analyses
• The Age of AI Agents Demands A New Scientific Paradigm To Sustain Trustworthy Science
• Reproduce to validate: A comprehensive study on the reproducibility of economics research
• <b>Rethinking where and how we publish in health sciences</b>
• Emphasizing Similarities Between Politically Opposed Groups and Their Influence in Perceptions of the Political Opposition: Evidence From Five Experiments
• The link between math anxiety and performance does not depend on working memory: A network analysis study
• Opening open science to all: demystifying reproducibility and transparency practices in linguistic research
• Registered Reports in Animal Biology: A new format
• Playing the long game: Carrying out principled tests of psychological phenomena before developing formal theories
• Brain Modelling as a Service: The Virtual Brain on EBRAINS
• The Challenges and Solutions of In‐Person Big‐Team Science
• Individual and collective epistemic virtue in science
• The myth of normative development
• Entia Non Sunt Multiplicanda … Shall I look for clusters in my cognitive data?
• Psychology's Questionable Research Fundamentals (QRFs): Key problems in quantitative psychology and psychological measurement beyond Questionable Research Practices (QRPs)
• Beyond the p Value Dichotomy: Alternatives for Statistical Inference—A Critical Review
• Contextual sensitivity in scientific reproducibility
• fNIRS reproducibility varies with data quality, analysis pipelines, and researcher experience
• From mess to method
• The competitive advantage of sanctioning institutions revisited: A multilab replication
• Automated Infant Eye Tracking: A Systematic Historical Review
• The replication crisis has led to positive structural, procedural, and community changes
• How measurement affects causal inference: Attenuation bias is (usually) more important than outcome scoring weights
• Searching for the backfire effect: Measurement and design considerations.
• Reproducibility and Research Integrity
• Common methodological mistakes
• A Validity-Based Framework for Understanding Replication in Psychology
• The statistical significance filter leads to overoptimistic expectations of replicability
• Collusive and adversarial replication
• Questionable and Improved Research Practices in Single-Case Experimental Design: Initial Investigation and Findings
• Sacrificing objects instead of persons: Order effects without emotional engagement
• Predicting the replicability of social and behavioural science claims in COVID-19 preprints
• The process of replication target selection in psychology: what to consider?
• Noise Versus Signal: What Can One Conclude When a Classic Finding Fails to Replicate?
• The Yet Underestimated Importance of Communicating Findings from Educational Trials to Teachers, Schools, School Authorities, or Policy Makers (Comment on Brady et al. (2023))
• Deliberate Lab: A Platform for Real-Time Human-AI Social Experiments
• Nostalgia encourages exploration and fosters uncertainty in response to AI technology
• Reproducibility: The New Frontier in AI Governance
• Directional replicability: when can the factor of two be omitted
• Almost, but not quite there: Research into the emergence of higher-order motivated behavior should fully embrace the dynamic systems approach
• Statistics in Service of Metascience: Measuring Replication Distance with Reproducibility Rate
• Beyond Statistics: Accepting the Null Hypothesis in Mature Sciences
• From ego depletion to self-control fatigue: A review of criticisms along with new perspectives for the investigation and replication of a multicomponent phenomenon.
• Open and Reproducible Research on Open Science Framework
• Facilitating bioinformatics reproducibility with QIIME 2 Provenance Replay
• A multi-study examination of intra-individual feedback loops between competence and value beliefs, procrastination, and goal achievement
• Replications in agricultural economics
• Metascience as a Scientific Social Movement
• Can an army of babies and dogs rescue psychology from its reproducibility crisis?
• “Well, Parts of Linguistics Is Open…”: Insights into Linguists’ Diverse Understandings of Open Science
• On Randomness in Agentic Evals
• Neural Bases of Affect-Based Impulsivity: A Decision Neuroscience Account
• The “Schizophrenia is 80% Heritable” Fallacy: Deconstructing a Twin Study Meta-Analysis
• The use and interpretation of unstandardized and standardized effect sizes in psychology: current practices and challenges
• What helps and hinders reproducible research? Researchers’ perspectives from a cross-disciplinary interview study
• Construct validity evidence reporting practices for the Reading the Mind in the Eyes Test: A systematic scoping review
• The role of theory in researching and understanding human communication
• Computationally reproducing results from meta-analyses in ecology and evolutionary biology using shared code and data
• Navigating evidence, legitimacy, and delivery: a three-dimensional framework for behavioral policy design
• The (in)efficacy of AI personas in deception detection experiments
• Generalizability is not optional: insights from a cross-cultural study of social discounting
• A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics
• Reproducible sex differences in personalised functional network topography in youth
• Initial evidence of research quality of registered reports compared with the standard publishing model
• Data sharing practices and data availability upon request differ across scientific disciplines
• Issues in Grounded Cognition and How to Solve Them – the Minimalist Account
• Improving transparency in observational social science research: A pre-analysis plan approach
• The institutional impact of research challenges and constraints on psychology and other social and behavioral sciences
• Building a collaborative psychological science: Lessons learned from ManyBabies 1.
• Sign-Congruence, External Validity, and Replication
• Both Questionable and Open Research Practices Are Prevalent in Education Research
• Prevalence of transparency and reproducibility-related research practices in music psychology (2017–2022)
• Are Social Media Ruining Our Lives? A Review of Meta-Analytic Evidence
• Replication value as a function of citation impact and sample size
• Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes
• 281. MEVATRON77DX67 における電子線用可変型絞りについて(治療-6 電子線)
• Navigating Open Science as Early Career Feminist Researchers
• Open Sharing of Data on Close Relationships and Other Sensitive Social Psychological Topics: Challenges, Tools, and Future Directions
• Open Science 2.0: Towards a truly collaborative research ecosystem
• Re‐examining the relationship between patience, risk‐taking, and human capital investment across countries*
• Problems of knowledge, problems of order: the open science field site
• Eleven years of student replication projects provide evidence on the correlates of replicability in psychology
• Metascience Is Not Enough – A Plea for Psychological Humanities in the Wake of the Replication Crisis
• When Can AI Models Explain Learning? Validity Criteria for AI as Cognitive Models in Education
• How can we make sound replication decisions?
• Replication, a Hallmark of Good Science: Unraveling the Factors That Predict Replication Success
• Is the Political Slant of Psychology Research Related to Scientific Replicability?
• Examining the replicability of online experiments selected by a decision market
• Many nonnormalities, one simulation: Do different data generation algorithms affect study results?
• Replication of null results: Absence of evidence or evidence of absence?
• method to streamline p-hacking
• Towards a cumulative science of vocal markers of autism: a cross-linguistic meta-analysis-based investigation of acoustic markers in American and Danish autistic children
• Title Pending 10808
• Effect Size Estimation Fromt-Statistics in the Presence of Publication Bias
• Does reading an anti-free will text affect beliefs related to free will over time? — A registered report
• Don’t Ditch the Laptop Just Yet: A Direct Replication of Mueller and Oppenheimer’s (2014) Study 1 Plus Mini Meta-Analyses Across Similar Studies
• Does synchronised singing enhance social bonding more than speaking does? A global experimental Stage 1 Registered Report
• A protocol for structured robustness reproductions and replicability assessments
• Predicting replication outcomes and generalizability of results
• Estimating the effect size of moral contagion in online networks: A pre-registered replication and meta-analysis
• How to become a Bayesian in eight easy steps: An annotated reading list
• Replication in Social Science
• A standardized framework to test event-based experiments
• Structural-functional brain network coupling predicts human cognitive ability
• How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice
• Changing Human Behavior to Conserve Biodiversity
• On the process and value of direct close replications: A rejoinder to Shafir and Cheek’s (2024) commentary on Chandrashekar et al. (2021)
• Emotions in conceptual spaces
• Influences on Data Quality in Developmental Children Studies
• Is archaeology a science? Insights and imperatives from 10,000 articles and a year of reproducibility reviews
• The Design of Current Replication Studies: A Systematic Literature Review on the Variation of Study Characteristics
• On the predicted replicability of two decades of experimental research on system justification: A Z‐curve analysis
• Should social science be more solution-oriented?
• Promoting Reproducibility and Replicability in Political Science
• Open science, the replication crisis, and environmental public health
• This human study did not involve human subjects: Validating LLM simulations as behavioral evidence
• Small Sample Size Solutions
• Tempered Expectations: A Tutorial for Calculating and Interpreting Prediction Intervals in the Context of Replications
• Four principles for improved statistical ecology
• Increasing Transparency Through a Multiverse Analysis
• The limitations to our understanding of peer review
• Open minds, tied hands: Awareness, behavior, and reasoning on open science and irresponsible research behavior
• Reproducibility in the Classroom
• Implications of the TASI taxonomy for understanding inconsistent effects pertaining to free will beliefs
• Philosophy of Open Science
• Representative bureaucracy and its symbolic effect on citizens: a conceptual replication
• On effects that do occur versus effects that can be made to occur
• Robust evidence for the simple view of second language reading: Secondary meta-analysis of Jeon and Yamashita (2022)
• When open data closes the door: A critical examination of the past, present and the potential future for open data guidelines in journals
• The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices: Five years later
• Accentuation and compatibility: Replication and extensions of Shafir (1993) to rethink choosing versus rejecting paradigms
• Optimizing the methodology of human sleep and memory research
• Emerging roles and responsibilities of libraries in support of reproducible research
• Economic Preferences and Personality Traits Among Finance Professionals and the General Population
• Developmental psychologists should adopt citizen science to improve generalization and reproducibility
• Large language models can predict the results of social science experiments
• Neglect of the individual as a neglected problem: The relevance of combined idiographic-nomothetic approaches for social psychology today
• Ivermectin isn’t a cancer miracle drug, but influencers claim otherwise – here’s how to avoid sprinting past scientific evidence
• Assessing replication success: a taxonomy and systematic comparison of replication success measures for prospective replications
• The Uptake of Preregistration, and Its Impact on the Proportion of Supported Hypotheses, in Sports Science
• Identificatory underpinnings of ethical research behavior for graduate students: Evidence from a baseline sample of a university research ethics training program
• Predicting Virtual Learning Environment adoption - A case study
• Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future
• Time to adjust: Improving replicability in experimental psychology by adjustment for evident selective inference
• Variability in the analysis of a single neuroimaging dataset by many teams
• The association of dispositional anxiety with the NoGo N2 under relaxation instruction vs. speed/accuracy instruction
• Power Calculations for Replication Studies
• Don’t fear ‘fear conditioning’: Methodological considerations for the design and analysis of studies on human fear acquisition, extinction, and return of fear
• Replicability: Terminology, Measuring Success, and Strategy
• The Effects of Exposure to Pleasant and Unpleasant Stimuli on Body Posture: A Systematic Review and Meta‐Analysis
• A solution to minimum sample size for regressions
• RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable
• Exploring Large Language Model Agents for Piloting Social Experiments
• Sight vs. Sound Judgments of Music Performance Depend on Relative Performer Quality: Cross-cultural Evidence From Classical Piano and Tsugaru Shamisen Competitions
• Contextual Changes, Credible Conclusions? A Direct and Conceptual Replication of Shen et al.'s (2019) Study on Online Image Credibility
• Exploring the Role of Randomization on Belief Rigidity in Online Social Networks
• A Tale of Open Science: Emergence of a New Normal
• Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication
• REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?
• Violating the normality assumption may be the lesser of two evils
• Measuring the predictability of life outcomes with a scientific mass collaboration
• Publication bias, statistical power and reporting practices in the Journal of Sports Sciences: potential barriers to replicability
• What Can Educational Psychology Learn From, and Contribute to, Theory Development Scholarship?
• Introducing 'inrep': an R package that facilitates fully reproducible research workfows for survey-based assessments
• A Structured Examination of Reproducibility: A Case Study for HTS Using the PRIMAD Model and BioCompute Object
• Advancing replicable and reproducible GIScience: an approach with KNIME
• Prediction and explanation in social systems
• Reproducibility and replicability of rodent phenotyping in preclinical studies
• More than just noise: Inter-individual differences in fear acquisition, extinction and return of fear in humans - Biological, experiential, temperamental factors, and methodological pitfalls
• Meaningful Work and Affective Commitment: A Moderated Mediation Model of Positive Work Reflection and Work Centrality
• The statistical significance filter leads to overconfident expectations of replicability
• Concerns About Replicability Across Two Crises in Social Psychology
• Addressing artifactual bias in large, automated MRI analyses of brain development
• “I Want to Know More!”: Children Are Sensitive to Explanation Quality When Exploring New Information
• Consumer segmentation based on situational appropriateness ratings: Partial replication and extension
• Public Health and Online Misinformation: Challenges and Recommendations
• Improving reproducibility in animal research by splitting the study population into several ‘mini-experiments’
• Sketch-and-test: picture-centered research with p5.js assisted crowdsourcing
• Enhancing the quality and reproducibility of research: Preferred Evaluation of Cognitive and Neuropsychological Studies - The PECANS statement for human studies
• Power and reproducibility in the external validation of brain-phenotype predictions
• The role of LLMs in theory building
• The priority of prediction in ecological understanding
• Randomized controlled trial for the Attempted Suicide Short Intervention Program (ASSIP): An independent non-replication study
• Bringing an Intersectional Lens to “Open” Science: An Analysis of Representation in the Reproducibility Project
• Variables associated with achievement in higher education: A systematic review of meta-analyses.
• LLM-Based Social Simulations Require a Boundary
• A Systematic Review of Reproducibility Research in Natural Language\n Processing
• Can you see how I learn? Human observers' inferences about Reinforcement Learning agents' learning processes
• Social physics
• Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences
• Questionable research practices may have little effect on replicability
• Psychology [wikipedia]
• Replication crisis [wikipedia]

Discussions

• Und der Klassiker zu dem Thema wäre vermutlich das große psychologische Replikationsprojekt: doi.org/10.1126/scie... ... So viel kann ich spoilern: Seitdem sind viele hilfreiche und positive Bewegunge [bsky, 12 points, 1 comments]
• Just to make sure I'm following you, would your detection system pick up this fake reference I just made up? Smith, A. A. (2015). Why many article citations are fake. Journal of Psychological Studies [bsky, 0 points, 1 comments]

Related

• A glass half full interpretation of the replicability of psychological science
• Replication, a Hallmark of Good Science: Unraveling the Factors That Predict Replication Success
• Eleven years of student replication projects provide evidence on the correlates of replicability in psychology
• Making replication mainstream
• Fostering transparency and reproducibility in psychological science.