A manifesto for reproducible science

2017/01/10 by Marcus R. Munafò, Brian A. Nosek, Dorothy Bishop +10 · 3,552 citations
Decision Sciences · Engineering · #Computer science #Computer security #Credibility #Data science #Engineering #Incentive #Management science #Manifesto #Meta-analysis and systematic reviews #Political science #Process (computing) #Reliability (semiconductor) #Scientific Computing and Data Management #Transparency (behavior) #scientometrics and bibliometrics research

paper · pdf · doi:10.1038/s41562-016-0021

published in Nature Human Behaviour 1(1), 0021 (Nature Portfolio)

openalex publication_date 2017/01/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.

Citations

• Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
• The natural selection of bad science

Cited by

• Advancing the psychology of social class with large-scale replications in four countries
• A tutorial for calculating field-specific effect size distributions
• The Recurrence of Fundamental Questions: A Historical and Philosophical Analysis of Major Disciplinary Crises in Psychology
• La "novedad" y carácter decisivo del cristianismo
• Examination of real-time fluctuations in suicidal ideation and its risk factors: Results from two ecological momentary assessment studies.
• Open Science in the Developing World: A Collection of Practical Guides for Researchers in Developing Countries
• What is critical metascience and why is it important?
• Women as the First Open Scientists: Five Stories of the Neglected Contributions of Women in (Social) Science Reform History
• Three Persistent Myths about Open Science
• Science becomes trustworthy by constantly questioning itself
• Spin bias in scientific publishing: upholding integrity in research reporting
• Bridging Neurodiversity and Open Scholarship: How Shared Values Can Guide Best Practices for Research Integrity, Social Justice, and Principled Education
• Grassroots networks can help implement and harmonize open research efforts
• Best practices in plant fluorescence imaging and reporting: A primer
• Report on reproducibility in condensed matter physics
• Causal relationships between gut microbiome and hundreds of age-related traits: evidence of a replicable effect on ApoM protein levels
• Prevalence of Transparent Research Practices in Psychology: A Cross-Sectional Study of Empirical Articles Published in 2022
• The replication crisis is less of a “crisis” in Lakatos’ philosophy of science than it is in Popper’s
• A guide for many authors: Writing manuscripts in large collaborations
• Metascience as a Scientific Social Movement
• Cycling on the Freeway: The perilous state of open-source neuroscience software
• Playing the long game: Carrying out principled tests of psychological phenomena before developing formal theories
• Estimating the reproducibility of social learning research published between 1955 and 2018
• A registered report survey of open research practices in psychology departments in the UK and Ireland
• Simulation studies for methodological research in psychology: A standardized template for planning, preregistration, and reporting.
• Reproducibility and Replicability in Geographical Analysis
• Shifting the Level of Selection in Science
• Reporting community involvement in autism research: Findings from the journal Autism
• Qualitative research at the crossroads of open science and big data: Ethical considerations
• International initiatives to enhance awareness and uptake of open research in psychology: a systematic mapping review
• Two-brain states characterize within- and between-brain connectivity during intergenerational collaborative drawing
• A Tutorial on Analyzing Ecological Momentary Assessment Data in Psychological Research With Bayesian (Generalized) Mixed-Effects Models
• Survey of open science practices and attitudes in the social sciences
• Scientific reform, citation politics and the bureaucracy of oblivion
• Wikimedia and universities: contributing to the global commons in the Age of Disinformation
• Reproducibility Failure in Biomedical Research: Problems and Solutions
• Effects of selective attention on the C1 ERP component: A systematic review and meta‐analysis
• Open Science
• Mitigating LLM-based p-Hacking by Preregistering for the Next LLM
• The Dead Salmons of AI Interpretability
• Open science saves lives: lessons from the COVID-19 pandemic
• Open science in agricultural economics
• Assessing computational reproducibility in Behavior Research Methods
• A Guide for Calculating Study-Level Statistical Power for Meta-Analyses
• Journals Operating Predatory Practices Are Systematically Eroding the Science Ethos: A Gate and Code Strategy to Minimise Their Operating Space and Restore Research Best Practice
• Re-evaluating Word Mover's Distance
• Toward open science in marketing research
• “Epistemic Disadvantage in Open Science Practices”
• Improving the reproducibility in geoscientific papers: lessons learned from a Hackathon in climate science
• Code sharing in ecology and evolution increases citation rates but remains uncommon
• Epistemic Norms for AI Safety and Alignment Research
• Investigating the replicability of the social and behavioural sciences
• Towards Nexus-Score: Metadata Gaps Limit Scholarly AI Attribution
• Support for preregistration and registered reports at Historical Biology
• Open Source Software and Data for Human Service Development: A Case Study on Predicting Housing Instability
• Over-representation of phonological features in basic vocabulary doesn't replicate when controlling for spatial and phylogenetic effects
• The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network
• Explosion of formulaic research articles, including inappropriate study designs and false discoveries, based on the NHANES US national health database
• Toward New Directions in Human Biology: A Roadmap for Anthropological Causal Inference With Observational Data
• Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
• A standard protocol for reporting species distribution models
• Addressing Reproducibility in Stem Cell and PRP Therapies
• Vision Token Masking Alone Cannot Prevent PHI Leakage in Medical Document OCR: A Systematic Evaluation
• Evaluating the efficacy and impact of a pilot programme for FAIR data stewardship at a UK university
• Standardising the NLP Workflow: A Framework for Reproducible Linguistic Analysis
• How Scientists Use Large Language Models to Program
• Robust rhythm reporting will advance ecological and evolutionary research
• Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses
• Reproducibility in the Social Sciences
• Biomedical researchers’ perspectives on the reproducibility of research
• Testing the Usability of the Psychological Research Preregistration-Quantitative (PRP-QUANT) Template
• Answers to 18 Questions About Open Science Practices
• Messing with Merton: The intersection between open science practices and Mertonian values
• Replication studies: a win-win for early-career training and behavioral ecology
• A manifesto for a globally diverse, equitable, and inclusive open science
• Primer for Experimental Methods in Organization Theory
• Addressing researcher degrees of freedom through minP adjustment
• Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
• False-positive neuroimaging: Undisclosed flexibility in testing spatial hypotheses allows presenting anything as a replicated finding
• Investigating the reproducibility of the social and behavioural sciences
• Reproducibility and robustness of economics and political science research
• From impact metrics and open science to communicating research: Journalists’ awareness of academic controversies
• Some lessons learned from genomic and epigenomic studies of the placenta
• Open Science for Evaluation Use: How Applying Open Science Principles Can Further Open up Evaluation
• A research agenda for GIScience in a time of disruptions
• The iterative development and refinement of health psychology theories through formal, dynamical systems modelling: a scoping review and initial expert-derived ‘best practice’ recommendations
• The case for formal methodology in scientific reform
• Invisible Hands and Fine Calipers: A Call to Use Formal Theory as a Toolkit for Theory Construction
• Engaging citizen scientists: designing an open research system for collaborative problem exploration
• Behavioral contagion on social media: Effects of social norms, design interventions, and critical media literacy on self-disclosure
• Ten simple rules for good research practice
• Open science needs a standardized data format: Suggestions for the field of psychoneuroendocrinology
• Registered report: Survey on attitudes and experiences regarding preregistration in psychological research
• Protecting against researcher bias in secondary data analysis: challenges and potential solutions
• Public engagement with research reproducibility
• Optimising the value of the critical appraisal skills programme (CASP) tool for quality appraisal in qualitative evidence synthesis
• The Age of AI Agents Demands A New Scientific Paradigm To Sustain Trustworthy Science
• Bugs as features (part 1): concepts and foundations for the compositional data analysis of the microbiome–gut–brain axis
• Sustainable data analysis with Snakemake
• Registered Reports in Animal Biology: A new format
• New developments in experience sampling methodology
• Towards cumulative forensic science: a commentary on ‘Methodological problems in every black-box study of forensic firearm comparisons’
• Unlocking the benefits of transparent and reusable science for climate risk management
• Individual and collective epistemic virtue in science
• The myth of normative development
• Data Visualization Using R for Researchers Who Do Not Use R
• fNIRS reproducibility varies with data quality, analysis pipelines, and researcher experience
• The replication crisis has led to positive structural, procedural, and community changes
• Reproducible brain-wide association studies require thousands of individuals
• Code Sharing in Healthcare Research: A Practical Guide and Recommendations for Good Practice
• Mendelian randomization
• Formalistic data and code availability policy in high-profile medical journals and pervasive policy-practice gaps in published articles: A meta-research study
• Questionable and Improved Research Practices in Single-Case Experimental Design: Initial Investigation and Findings
• Guidance framework to apply best practices in ecological data analysis: lessons learned from building Galaxy-Ecology
• Interactive Tools for Reproducible Science -- Understanding, Supporting, and Motivating Reproducible Science Practices
• Improving Science That Uses Code
• Open and Reproducible Research on Open Science Framework
• Believing is seeing – the deceptive influence of bias in quantitative microscopy
• Facilitating bioinformatics reproducibility with QIIME 2 Provenance Replay
• A template for preregistration of quantitative research in psychology: Report of the joint psychological societies preregistration task force.
• The use and interpretation of unstandardized and standardized effect sizes in psychology: current practices and challenges
• Rethinking (un)blinding in biomedical proposal peer review: A multi-stakeholder qualitative study
• Balancing scale and depth: Publishing models for large-scale and small-N research in systems neuroscience
• Towards better hypothesis tests in oxytocin research: Evaluating the validity of auxiliary assumptions
• Generalizability is not optional: insights from a cross-cultural study of social discounting
• The Sisyphean Cycle of Technology Panics
• SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
• Breakdowns in Scientific Practices: How and Why Some Accepted Scientific Claims May Have Little Actual Support
• Demystifying Open Science in health psychology and behavioral medicine: a practical guide to Registered Reports and Data Notes
• Prevalence of transparency and reproducibility-related research practices in music psychology (2017–2022)
• A call for modernisation of science, the case of anaerobic digestion: A scoping review
• Open Sharing of Data on Close Relationships and Other Sensitive Social Psychological Topics: Challenges, Tools, and Future Directions
• Digital open science—Teaching digital tools for reproducible and transparent research
• Populating the Data Ark: An attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles
• Why researchers should share their analytic code
• method to streamline p-hacking
• “The paper I wish I had written with Giacinto Scoles,” the cherry-picking of data, and other reflections on reproducibility issues
• Preregistering qualitative research
• Open Science in Suicide Research Is Open for Business
• A digital twin approach for the identification and update of ecological infrastructure
• Open Science at Clinical Psychological Science : Reflections on Progress, Lessons Learned, and Suggestions for Continued Improvement
• Bayesian statistics and modelling
• We need less animal research, better animal research, and animal research done for the right reasons
• Opening Pandora’s Box: Peeking inside Psychology’s data sharing practices, and seven recommendations for change
• A standardized framework to test event-based experiments
• Changing Human Behavior to Conserve Biodiversity
• Fostering urban nature: How identity dynamics and future envisioning drive biodiversity conservation in cities
• Online political adverts: The effect of disclosures and opportunities for clandestine campaigning
• An Expert Guide to Planning Experimental Tasks For Evidence-Accumulation Modeling
• Coping with Emotional Distress via Self-Disclosure to Robots: An Intervention with Caregivers
• Tempered Expectations: A Tutorial for Calculating and Interpreting Prediction Intervals in the Context of Replications
• An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)
• Dataset for a qualitative interview study exploring the barriers and facilitators to using and recommending aspirin for cancer prevention
• The limitations to our understanding of peer review
• Open science, the replication crisis, and environmental public health
• Open minds, tied hands: Awareness, behavior, and reasoning on open science and irresponsible research behavior
• Emerging roles and responsibilities of libraries in support of reproducible research
• Frontloading selectivity: A third way in scientific publishing?
• The Uptake of Preregistration, and Its Impact on the Proportion of Supported Hypotheses, in Sports Science
• Towards wide-scale adoption of open science practices: The role of open science communities
• A methodological synthesis and meta-analysis of judgment tasks in second language research
• The role of the ventrolateral anterior temporal lobes in social cognition
• The social determinants of mental health and disorder: evidence, prevention and recommendations
• The Importance of Clinical Relevance and the Limitations of P-Values in Plastic Surgery Research
• Replicability: Terminology, Measuring Success, and Strategy
• Teaching machines to understand data science code by semantic enrichment of dataflow graphs
• Open peer review urgently requires evidence: A call to action
• STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports
• Replication of the natural selection of bad science
• Evolution of Reporting P-values Across the Biomedical Literature, 1990-2025: an Updated Meta-Research Study
• Doubts on the efficacy of outliers correction methods
• Enhancing the trustworthiness of pain research: A call to action.
• Neural mechanisms underlying reward processing and social cognition: A replication study with a Japanese sample
• Making a case for enterprise modelling as a research method
• Investing in open and FAIR practices for more usable and equitable climate-risk research
• What Can Educational Psychology Learn From, and Contribute to, Theory Development Scholarship?
• Introducing 'inrep': an R package that facilitates fully reproducible research workfows for survey-based assessments
• Estimating the replicability of Brazilian biomedical science
• Robust research needs many lines of evidence
• A companion to the preclinical common data elements for rigor, reproducibility and transparency in basic and translational epilepsy research. A report of the TASK3‐WG1A: Pharmacology Working Group of the ILAE/AES Joint Translational Task Force
• The Next Generation of Quality Indicators for Group Design Research in Special Education
• Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models
• Benchmarking and Evaluation of AI Models in Biology: Outcomes and Recommendations from the CZI Virtual Cells Workshop
• Evaluating dental AI research papers: Key considerations for editors and reviewers
• Reproducibility in the Control of Autonomous Mobility-on-Demand Systems
• Social physics
• Estimating the replicability of Brazilian biomedical science
• Biological research on mental pain, social pain and other pains not primarily felt in the body: methodological systematic review
• Replicability of proton MR spectroscopic imaging findings in mild traumatic brain injury: Implications for clinical applications
• Sample size evolution in neuroimaging research: An evaluation of highly-cited studies (1990–2012) and of latest practices (2017–2018) in high-impact journals
• Replicating extensive brain structural heterogeneity in individuals with schizophrenia and bipolar disorder
• Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations
• The Tempo Giusto : A Call for Recalibrating Time in Human Movement Research
• Reproducibility in machine‐learning‐based research: Overview, barriers, and drivers
• TrialChain: A Blockchain-Based Platform to Validate Data Integrity in Large, Biomedical Research Studies
• A Systematic Review of Unsupervised Learning Techniques for Software Defect Prediction
• Ten common statistical mistakes to watch out for when writing or reviewing a manuscript
• Steps toward preregistration of research on research integrity
• Inclusive meritocracy: ability and descriptive representation among Danish politicians
• Actionable recommendations for narrowing the science-practice gap in open science
• Understanding the social and political dimensions of research(er) assessment: evaluative flexibility and hidden criteria in promotion processes at research institutes
• Reproducibility in systems biology modelling
• GitHub as an open electronic laboratory notebook for real-time sharing of knowledge and collaboration
• Changing Lanes Toward Open Science: Openness and Transparency in Automotive User Research
• Supporting replication research in management journals: Qualitative analysis of editorials published between 1970 and 2015
• Improving the trustworthiness, usefulness, and ethics of biomedical research through an innovative and comprehensive institutional initiative
• When iconicity stands in the way of abbreviation: No Zipfian effect for figurative signals
• Fast Lane to Slow Science
• Addressing the theory crisis in psychology
• Reproducible research practices and transparency across linguistics
• The empirical replicability of task-based fMRI as a function of sample size
• Pre-registration: Weighing costs and benefits for researchers
• Preregistration of psychology meta-analyses: A cross-sectional study of prevalence and practice
• Data Availability Statements and Data Sharing in Urology: A False Promise?
• GNN-Suite: a Graph Neural Network Benchmarking Framework for Biomedical Informatics
• A Systematic Scoping Review of the Choice Architecture Movement: Toward Understanding When and Why Nudges Work
• Real-world unexpected outcomes predict city-level mood states and risk-taking behavior
• The impact of using biased performance metrics on software defect prediction research
• Negative self-referent cognition predicts future depression symptom change: an intensive sampling approach
• The effect of preregistration on trust in empirical research findings: results of a registered report
• Poor work design begets poor work design: Capacity and willingness antecedents of individual work design behavior.
• AI Coding Agents Can Reproduce Social Science Findings
• Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation
• Indicators for the use of robotic labs in basic biomedical research: a literature analysis
• How effective are pharmaceuticals for cognitive enhancement in healthy adults? A series of meta-analyses of cognitive performance during acute administration of modafinil, methylphenidate and D-amphetamine
• Open science challenges, benefits and tips in early career and beyond
• A tale of two 'opens': intersections between Free and Open Source Software and Open Scholarship
• Foundations for Open Scholarship Strategy Development
• Billions at Stake: How Self-Citation Adjusted Metrics Can Transform Equitable Research Funding
• Small sample sizes reduce the replicability of task-based fMRI studies
• Cultural additivity: behavioural insights from the interaction of Confucianism, Buddhism and Taoism in folktales
• Ten Hot Topics around Scholarly Publishing
• The preregistration revolution
• A multi-disciplinary perspective on emergent and future innovations in peer review
• Democratizing online controlled experiments at Booking.com
• Why ex post peer review encourages high-risk research while ex ante review discourages it
• A weak scientific basis for gaming disorder: Let us err on the side of caution
• On the relevance of data science for flight delay research: a systematic review
• Notebook articles: Towards a transformative publishing experience in nonlinear science
• The Software Behind the Stats: A Student Exploration of Software Trends Across Disciplines
• Open access [wikipedia]
• Open science [wikipedia]
• P-value [wikipedia]
• Unexplored therapeutic opportunities in the human genome. [europepmc]
• The preregistration revolution. [europepmc]
• NeuroMatic: An Integrated Open-Source Software Toolkit for Acquisition, Analysis and Simulation of Electrophysiological Data. [europepmc]
• Bayesian alternatives for common null-hypothesis significance tests in psychiatry: a non-technical guide using JASP. [europepmc]
• Bioconda: sustainable and comprehensive software distribution for the life sciences. [europepmc]
• Representation, Pattern Information, and Brain Signatures: From Neurons to Neuroimaging. [europepmc]
• Small sample sizes reduce the replicability of task-based fMRI studies. [europepmc]
• Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017. [europepmc]
• Why rankings of biomedical image analysis competitions should be interpreted with care. [europepmc]
• General functional connectivity: Shared features of resting-state and task fMRI drive reliable and heritable individual differences in functional brain networks. [europepmc]
• Standards for distribution models in biodiversity assessments. [europepmc]
• Open science challenges, benefits and tips in early career and beyond. [europepmc]
• Brain structure and cognitive ability in healthy aging: a review on longitudinal correlated change. [europepmc]
• A decade of test-retest reliability of functional connectivity: A systematic review and meta-analysis. [europepmc]
• How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables. [europepmc]
• Ten common statistical mistakes to watch out for when writing or reviewing a manuscript. [europepmc]
• Establishment of Best Practices for Evidence for Prediction: A Review. [europepmc]
• Realistic in silico generation and augmentation of single-cell RNA-seq data using generative adversarial networks. [europepmc]
• Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0. [europepmc]
• The Hong Kong Principles for assessing researchers: Fostering research integrity. [europepmc]
• Advances in the field of intranasal oxytocin research: lessons learned and future directions for clinical research. [europepmc]
• Sustainable data analysis with Snakemake. [europepmc]
• Reproducible brain-wide association studies require thousands of individuals. [europepmc]
• Mendelian randomization. [europepmc]
• The social determinants of mental health and disorder: evidence, prevention and recommendations. [europepmc]

Related

• Rethinking Reproducibility as a Criterion for Research Quality
• Reproducibility starts with methods
• Reproducibility and Research Integrity
• Doing Things Twice (Or Differently): Strategies to Identify Studies for Targeted Validation
• Biomedical researchers’ perspectives on the reproducibility of research