A manifesto for reproducible science

2017/01/10 by Marcus R. Munafò, Brian A. Nosek, Dorothy Bishop +10 · 205 citations
Decision Sciences · #Meta-analysis and systematic reviews #Scientific Computing and Data Management #scientometrics and bibliometrics research

paper · pdf · doi:10.1038/s41562-016-0021

Abstract

Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.

Citations

• Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
• The natural selection of bad science

Cited by

• Advancing the psychology of social class with large-scale replications in four countries
• A tutorial for calculating field-specific effect size distributions
• The Recurrence of Fundamental Questions: A Historical and Philosophical Analysis of Major Disciplinary Crises in Psychology
• La "novedad" y carácter decisivo del cristianismo
• Examination of real-time fluctuations in suicidal ideation and its risk factors: Results from two ecological momentary assessment studies.
• Open Science in the Developing World: A Collection of Practical Guides for Researchers in Developing Countries
• What is critical metascience and why is it important?
• Women as the First Open Scientists: Five Stories of the Neglected Contributions of Women in (Social) Science Reform History
• Three Persistent Myths about Open Science
• Science becomes trustworthy by constantly questioning itself
• Spin bias in scientific publishing: upholding integrity in research reporting
• Bridging Neurodiversity and Open Scholarship: How Shared Values Can Guide Best Practices for Research Integrity, Social Justice, and Principled Education
• Grassroots networks can help implement and harmonize open research efforts
• Best practices in plant fluorescence imaging and reporting: A primer
• Report on reproducibility in condensed matter physics
• Causal relationships between gut microbiome and hundreds of age-related traits: evidence of a replicable effect on ApoM protein levels
• Prevalence of Transparent Research Practices in Psychology: A Cross-Sectional Study of Empirical Articles Published in 2022
• The replication crisis is less of a “crisis” in Lakatos’ philosophy of science than it is in Popper’s
• A guide for many authors: Writing manuscripts in large collaborations
• Metascience as a Scientific Social Movement
• Cycling on the Freeway: The perilous state of open-source neuroscience software
• Playing the long game: Carrying out principled tests of psychological phenomena before developing formal theories
• Estimating the reproducibility of social learning research published between 1955 and 2018
• A registered report survey of open research practices in psychology departments in the UK and Ireland
• Simulation studies for methodological research in psychology: A standardized template for planning, preregistration, and reporting.
• Reproducibility and Replicability in Geographical Analysis
• Shifting the Level of Selection in Science
• Reporting community involvement in autism research: Findings from the journal Autism
• Qualitative research at the crossroads of open science and big data: Ethical considerations
• International initiatives to enhance awareness and uptake of open research in psychology: a systematic mapping review
• Two-brain states characterize within- and between-brain connectivity during intergenerational collaborative drawing
• A Tutorial on Analyzing Ecological Momentary Assessment Data in Psychological Research With Bayesian (Generalized) Mixed-Effects Models
• Survey of open science practices and attitudes in the social sciences
• Scientific reform, citation politics and the bureaucracy of oblivion
• Wikimedia and universities: contributing to the global commons in the Age of Disinformation
• Reproducibility Failure in Biomedical Research: Problems and Solutions
• Effects of selective attention on the C1 ERP component: A systematic review and meta‐analysis
• Open Science
• Mitigating LLM-based p-Hacking by Preregistering for the Next LLM
• The Dead Salmons of AI Interpretability
• Open science saves lives: lessons from the COVID-19 pandemic
• Open science in agricultural economics
• Assessing computational reproducibility in Behavior Research Methods
• A Guide for Calculating Study-Level Statistical Power for Meta-Analyses
• Journals Operating Predatory Practices Are Systematically Eroding the Science Ethos: A Gate and Code Strategy to Minimise Their Operating Space and Restore Research Best Practice
• Re-evaluating Word Mover's Distance
• Toward open science in marketing research
• “Epistemic Disadvantage in Open Science Practices”
• Improving the reproducibility in geoscientific papers: lessons learned from a Hackathon in climate science
• Code sharing in ecology and evolution increases citation rates but remains uncommon
• Epistemic Norms for AI Safety and Alignment Research
• Investigating the replicability of the social and behavioural sciences
• Towards Nexus-Score: Metadata Gaps Limit Scholarly AI Attribution
• Support for preregistration and registered reports at Historical Biology
• Open Source Software and Data for Human Service Development: A Case Study on Predicting Housing Instability
• Over-representation of phonological features in basic vocabulary doesn't replicate when controlling for spatial and phylogenetic effects
• The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network
• Explosion of formulaic research articles, including inappropriate study designs and false discoveries, based on the NHANES US national health database
• Toward New Directions in Human Biology: A Roadmap for Anthropological Causal Inference With Observational Data
• Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
• A standard protocol for reporting species distribution models
• Addressing Reproducibility in Stem Cell and PRP Therapies
• Vision Token Masking Alone Cannot Prevent PHI Leakage in Medical Document OCR: A Systematic Evaluation
• Evaluating the efficacy and impact of a pilot programme for FAIR data stewardship at a UK university
• Standardising the NLP Workflow: A Framework for Reproducible Linguistic Analysis
• How Scientists Use Large Language Models to Program
• Robust rhythm reporting will advance ecological and evolutionary research
• Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses
• Reproducibility in the Social Sciences
• Biomedical researchers’ perspectives on the reproducibility of research
• Testing the Usability of the Psychological Research Preregistration-Quantitative (PRP-QUANT) Template
• Answers to 18 Questions About Open Science Practices
• Messing with Merton: The intersection between open science practices and Mertonian values
• Replication studies: a win-win for early-career training and behavioral ecology
• A manifesto for a globally diverse, equitable, and inclusive open science
• Primer for Experimental Methods in Organization Theory
• Addressing researcher degrees of freedom through minP adjustment
• Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
• False-positive neuroimaging: Undisclosed flexibility in testing spatial hypotheses allows presenting anything as a replicated finding
• Investigating the reproducibility of the social and behavioural sciences
• Reproducibility and robustness of economics and political science research
• From impact metrics and open science to communicating research: Journalists’ awareness of academic controversies
• Some lessons learned from genomic and epigenomic studies of the placenta
• Open Science for Evaluation Use: How Applying Open Science Principles Can Further Open up Evaluation
• A research agenda for GIScience in a time of disruptions
• The iterative development and refinement of health psychology theories through formal, dynamical systems modelling: a scoping review and initial expert-derived ‘best practice’ recommendations
• Sustainable data analysis with Snakemake
• The case for formal methodology in scientific reform
• Invisible Hands and Fine Calipers: A Call to Use Formal Theory as a Toolkit for Theory Construction
• Engaging citizen scientists: designing an open research system for collaborative problem exploration
• Behavioral contagion on social media: Effects of social norms, design interventions, and critical media literacy on self-disclosure
• Ten simple rules for good research practice
• Open science needs a standardized data format: Suggestions for the field of psychoneuroendocrinology
• Registered report: Survey on attitudes and experiences regarding preregistration in psychological research
• Protecting against researcher bias in secondary data analysis: challenges and potential solutions
• Public engagement with research reproducibility
• Optimising the value of the critical appraisal skills programme (CASP) tool for quality appraisal in qualitative evidence synthesis
• The Age of AI Agents Demands A New Scientific Paradigm To Sustain Trustworthy Science
• Bugs as features (part 1): concepts and foundations for the compositional data analysis of the microbiome–gut–brain axis
• Sustainable data analysis with Snakemake
• Registered Reports in Animal Biology: A new format
• New developments in experience sampling methodology
• Towards cumulative forensic science: a commentary on ‘Methodological problems in every black-box study of forensic firearm comparisons’
• Unlocking the benefits of transparent and reusable science for climate risk management
• Individual and collective epistemic virtue in science
• The myth of normative development
• Data Visualization Using R for Researchers Who Do Not Use R
• fNIRS reproducibility varies with data quality, analysis pipelines, and researcher experience
• The replication crisis has led to positive structural, procedural, and community changes
• Reproducible brain-wide association studies require thousands of individuals
• Code Sharing in Healthcare Research: A Practical Guide and Recommendations for Good Practice
• Mendelian randomization
• Formalistic data and code availability policy in high-profile medical journals and pervasive policy-practice gaps in published articles: A meta-research study
• Questionable and Improved Research Practices in Single-Case Experimental Design: Initial Investigation and Findings
• Guidance framework to apply best practices in ecological data analysis: lessons learned from building Galaxy-Ecology
• Interactive Tools for Reproducible Science -- Understanding, Supporting, and Motivating Reproducible Science Practices
• Improving Science That Uses Code
• Open and Reproducible Research on Open Science Framework
• Believing is seeing – the deceptive influence of bias in quantitative microscopy
• Facilitating bioinformatics reproducibility with QIIME 2 Provenance Replay
• A template for preregistration of quantitative research in psychology: Report of the joint psychological societies preregistration task force.
• The use and interpretation of unstandardized and standardized effect sizes in psychology: current practices and challenges
• Rethinking (un)blinding in biomedical proposal peer review: A multi-stakeholder qualitative study
• Balancing scale and depth: Publishing models for large-scale and small-N research in systems neuroscience
• Towards better hypothesis tests in oxytocin research: Evaluating the validity of auxiliary assumptions
• Generalizability is not optional: insights from a cross-cultural study of social discounting
• The Sisyphean Cycle of Technology Panics
• SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
• Breakdowns in Scientific Practices: How and Why Some Accepted Scientific Claims May Have Little Actual Support
• Demystifying Open Science in health psychology and behavioral medicine: a practical guide to Registered Reports and Data Notes
• Prevalence of transparency and reproducibility-related research practices in music psychology (2017–2022)
• A call for modernisation of science, the case of anaerobic digestion: A scoping review
• Open Sharing of Data on Close Relationships and Other Sensitive Social Psychological Topics: Challenges, Tools, and Future Directions
• Digital open science—Teaching digital tools for reproducible and transparent research
• Populating the Data Ark: An attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles
• Why researchers should share their analytic code
• method to streamline p-hacking
• “The paper I wish I had written with Giacinto Scoles,” the cherry-picking of data, and other reflections on reproducibility issues
• Preregistering qualitative research
• Open Science in Suicide Research Is Open for Business
• A digital twin approach for the identification and update of ecological infrastructure
• Open Science at Clinical Psychological Science : Reflections on Progress, Lessons Learned, and Suggestions for Continued Improvement
• Bayesian statistics and modelling
• We need less animal research, better animal research, and animal research done for the right reasons
• Opening Pandora’s Box: Peeking inside Psychology’s data sharing practices, and seven recommendations for change
• A standardized framework to test event-based experiments
• Changing Human Behavior to Conserve Biodiversity
• Fostering urban nature: How identity dynamics and future envisioning drive biodiversity conservation in cities
• Online political adverts: The effect of disclosures and opportunities for clandestine campaigning
• An Expert Guide to Planning Experimental Tasks For Evidence-Accumulation Modeling
• Coping with Emotional Distress via Self-Disclosure to Robots: An Intervention with Caregivers
• Tempered Expectations: A Tutorial for Calculating and Interpreting Prediction Intervals in the Context of Replications
• An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)
• Dataset for a qualitative interview study exploring the barriers and facilitators to using and recommending aspirin for cancer prevention
• The limitations to our understanding of peer review
• Open science, the replication crisis, and environmental public health
• Open minds, tied hands: Awareness, behavior, and reasoning on open science and irresponsible research behavior
• Emerging roles and responsibilities of libraries in support of reproducible research
• Frontloading selectivity: A third way in scientific publishing?
• The Uptake of Preregistration, and Its Impact on the Proportion of Supported Hypotheses, in Sports Science
• Towards wide-scale adoption of open science practices: The role of open science communities
• A methodological synthesis and meta-analysis of judgment tasks in second language research
• The role of the ventrolateral anterior temporal lobes in social cognition
• The social determinants of mental health and disorder: evidence, prevention and recommendations
• The Importance of Clinical Relevance and the Limitations of P-Values in Plastic Surgery Research
• Replicability: Terminology, Measuring Success, and Strategy
• Teaching machines to understand data science code by semantic enrichment\n of dataflow graphs
• Open peer review urgently requires evidence: A call to action
• STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports
• Replication of the natural selection of bad science
• Evolution of Reporting P-values Across the Biomedical Literature, 1990-2025: an Updated Meta-Research Study
• Doubts on the efficacy of outliers correction methods
• Enhancing the trustworthiness of pain research: A call to action.
• Neural mechanisms underlying reward processing and social cognition: A replication study with a Japanese sample
• Making a case for enterprise modelling as a research method
• Investing in open and FAIR practices for more usable and equitable climate-risk research
• What Can Educational Psychology Learn From, and Contribute to, Theory Development Scholarship?
• Introducing 'inrep': an R package that facilitates fully reproducible research workfows for survey-based assessments
• Estimating the replicability of Brazilian biomedical science
• Robust research needs many lines of evidence
• A companion to the preclinical common data elements for rigor, reproducibility and transparency in basic and translational epilepsy research. A report of the TASK3‐WG1A: Pharmacology Working Group of the ILAE/AES Joint Translational Task Force
• Unexplored therapeutic opportunities in the human genome. [europepmc]
• The preregistration revolution. [europepmc]
• NeuroMatic: An Integrated Open-Source Software Toolkit for Acquisition, Analysis and Simulation of Electrophysiological Data. [europepmc]
• Bayesian alternatives for common null-hypothesis significance tests in psychiatry: a non-technical guide using JASP. [europepmc]
• Bioconda: sustainable and comprehensive software distribution for the life sciences. [europepmc]
• Representation, Pattern Information, and Brain Signatures: From Neurons to Neuroimaging. [europepmc]
• Small sample sizes reduce the replicability of task-based fMRI studies. [europepmc]
• Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017. [europepmc]
• Why rankings of biomedical image analysis competitions should be interpreted with care. [europepmc]
• General functional connectivity: Shared features of resting-state and task fMRI drive reliable and heritable individual differences in functional brain networks. [europepmc]
• Standards for distribution models in biodiversity assessments. [europepmc]
• Open science challenges, benefits and tips in early career and beyond. [europepmc]
• Brain structure and cognitive ability in healthy aging: a review on longitudinal correlated change. [europepmc]
• A decade of test-retest reliability of functional connectivity: A systematic review and meta-analysis. [europepmc]
• How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables. [europepmc]
• Ten common statistical mistakes to watch out for when writing or reviewing a manuscript. [europepmc]
• Establishment of Best Practices for Evidence for Prediction: A Review. [europepmc]
• Realistic in silico generation and augmentation of single-cell RNA-seq data using generative adversarial networks. [europepmc]
• Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0. [europepmc]
• The Hong Kong Principles for assessing researchers: Fostering research integrity. [europepmc]
• Advances in the field of intranasal oxytocin research: lessons learned and future directions for clinical research. [europepmc]
• Sustainable data analysis with Snakemake. [europepmc]
• Reproducible brain-wide association studies require thousands of individuals. [europepmc]
• Mendelian randomization. [europepmc]
• The social determinants of mental health and disorder: evidence, prevention and recommendations. [europepmc]

Related

• Reproducibility starts with methods
• Reproducibility and Research Integrity
• Doing Things Twice (Or Differently): Strategies to Identify Studies for Targeted Validation
• Biomedical researchers’ perspectives on the reproducibility of research
• Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability