An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
2021/04/01 by Anne M. Scheel, Mitchell Schijen, Daniël Lakens · 2 voices · 42 citations
Decision Sciences · #Meta-analysis and systematic reviews #scientometrics and bibliometrics research
paper · pdf · doi:10.1177/25152459211007467
Abstract
Selectively publishing results that support the tested hypotheses (“positive” results) distorts the available evidence for scientific claims. For the past decade, psychological scientists have been increasingly concerned about the degree of such distortion in their literature. A new publication format has been developed to prevent selective reporting: In Registered Reports (RRs), peer review and the decision to publish take place before results are known. We compared the results in published RRs ( N = 71 as of November 2018) with a random sample of hypothesis-testing studies from the standard literature ( N = 152) in psychology. Analyzing the first hypothesis of each article, we found 96% positive results in standard reports but only 44% positive results in RRs. We discuss possible explanations for this large difference and suggest that a plausible factor is the reduction of publication bias and/or Type I error inflation in the RR literature.
Citations
Cited by
- Political ideology and trust in scientists in the USA
- Bridging theory and data: A computational workflow for cultural evolution
- An Inception-Cohort Study Quantifying How Many Registered Studies Are Publicly Shared
- Toward a More Powerful Experimental Communication Science: An Assessment of Two Decades’ Research (2001–2023)
- The Recurrence of Fundamental Questions: A Historical and Philosophical Analysis of Major Disciplinary Crises in Psychology
- Replicability and transparency in physical therapy research: Time to wake up
- To Be FAIR: Theory Specification Needs an Update
- “Do Your Own Research”
- Footprint of publication selection bias on meta‐analyses in medicine, environmental sciences, psychology, and economics
- Peer Community In and Peer Community Journal: a two-step diamond OA process giving research communities back control of publishing
- Elite Partisan Disagreement and Military Victory: Evidence from South Korean Battle Experiments
- Support for preregistration and registered reports at Historical Biology
- Peer Review at the Crossroads
- Novelty, Consistency, Transparency: The Trilemma of Psychological Sciences and its Consequences on Open Science Practices
- Linguistic alignment with an artificial agent: A commentary and re-analysis
- A framework for assessing the trustworthiness of scientific research findings
- The publish or perish, publish and perish, publish then perish, and now retract and perish cultures in academia
- Protecting against researcher bias in secondary data analysis: challenges and potential solutions
- Registered Reports in School Psychology Research: Initial Experiences, Analyses, and Future
- Registered Reports in Animal Biology: A new format
- Towards cumulative forensic science: a commentary on ‘Methodological problems in every black-box study of forensic firearm comparisons’
- The replication crisis has led to positive structural, procedural, and community changes
- Prosocial motives underlie scientific censorship by scientists: A perspective and research agenda
- Fixing science means an end to gaming the system
- Registration of research on research integrity is still not common: Findings from the Hong Kong, Cape Town, and Athens editions of the World Conference on Research Integrity
- Initial evidence of research quality of registered reports compared with the standard publishing model
- The future of academic publishing
- Demystifying Open Science in health psychology and behavioral medicine: a practical guide to Registered Reports and Data Notes
- Examining the replicability of online experiments selected by a decision market
- Ending publication bias: A values-based approach to surface null and negative results
- Robust Bayesian meta‐analysis: Model‐averaging across complementary publication bias adjustment methods
- Handling conflicts of interest concerns more than transparency
- Editorial Perspective: Misaligned incentives in mental health research – the case for Registered Reports
- Ten simple rules for writing a Registered Report
- Developmental psychologists should adopt citizen science to improve generalization and reproducibility
- Implementing open science and reproducible research practices in mental health research through registered reports
- The Uptake of Preregistration, and Its Impact on the Proportion of Supported Hypotheses, in Sports Science
- Z-Curve Plot: A Visual Diagnostic for Publication Bias in Meta-Analysis
- Reputation, Risk, and Visibility
- fNIRS reproducibility varies with data quality, analysis pipelines, and researcher experience
- Publication bias, statistical power and reporting practices in the Journal of Sports Sciences: potential barriers to replicability
- Practical routes to preregistration: a guide to enhanced transparency and rigour in neuropsychological research
Discussions
Related