The natural selection of bad science
2016/09/01 by Paul E. Smaldino, Richard McElreath · 6 voices · 88 citations
Environmental Science · Decision Sciences · Social Sciences · #Species Distribution and Climate Change #scientometrics and bibliometrics research #Experimental Behavioral Economics Studies
paper · doi:10.1098/rsos.160384
Abstract
Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, suggesting that they result from something more than just misunderstanding. The persistence of poor methods results partly from incentives that favour them, leading to the natural selection of bad science. This dynamic requires no conscious strategizing-no deliberate cheating nor loafing-by scientists, only that publication is a principal factor for career advancement. Some normative methods of analysis have almost certainly been selected to further publication instead of discovery. In order to improve the culture of science, a shift must be made away from correcting misunderstandings and towards rewarding understanding. We support this argument with empirical evidence and computational modelling. We first present a 60-year meta-analysis of statistical power in the behavioural sciences and show that power has not improved despite repeated demonstrations of the necessity of increasing power. To demonstrate the logical consequences of structural incentives, we then present a dynamic model of scientific communities in which competing laboratories investigate novel or previously published hypotheses using culturally transmitted research methods. As in the real world, successful labs produce more 'progeny,' such that their methods are more often copied and their students are more likely to start labs of their own. Selection for high output leads to poorer methods and increasingly high false discovery rates. We additionally show that replication slows but does not stop the process of methodological deterioration. Improving the quality of research requires change at the institutional level.
Citations
Cited by
- Critical Values Robust to P-Hacking
- That’s a Lot to Process! Pitfalls of Popular Path Models
- Is Peer Review a Good Idea?
- A manifesto for reproducible science
- Reducing bias, increasing transparency and calibrating confidence with preregistration
- The Recurrence of Fundamental Questions: A Historical and Philosophical Analysis of Major Disciplinary Crises in Psychology
- Grassroots networks can help implement and harmonize open research efforts
- Citation self-awareness for a fairer academic publishing landscape
- Are our 95% CIs only worth 45% confidence? A heterogeneity-aware method for extending inference
- A guide for social science journal editors on easing into open science
- Estimating the reproducibility of social learning research published between 1955 and 2018
- Shifting the Level of Selection in Science
- Eleven strategies for making reproducible research and open science training the norm at research institutions
- Inferring the causal effect of journals on citations
- One Decade Into the Replication Crisis, How Have Psychological Results Changed?
- Defining and classifying models of groups: The social ontology of higher-order networks
- Why and how we should join the shift from significance testing to estimation
- Do Musicians Have Better Short-Term Memory Than Nonmusicians? A Multilab Study
- Changing the Landscape of Archaeological Publishing
- Scientific Reliance on AI may Reflect Broader Failures in Science Communication
- Dead rats, dopamine, performance metrics, and peacock tails: Proxy failure is an inherent risk in goal-oriented systems
- Power Analysis for Conjoint Experiments
- Counterfactual LLM-based Framework for Measuring Rhetorical Style
- Reproducibility in the Social Sciences
- The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support
- Responsible Research Assessment I: Implementing DORA and CoARA for hiring and promotion in psychology
- How bad gatekeepers undermine good science
- Improving Social Science: Lessons from the Open Science Movement
- Regional and institutional trends in assessment for academic promotion
- An evaluation of reproducibility and errors in published sample size calculations performed using G*Power
- Statistics in service of metascience: Measuring replication distance with reproducibility rate
- ChatGPT in Education: An Effect in Search of a Cause
- A framework for assessing the trustworthiness of scientific research findings
- The publish or perish, publish and perish, publish then perish, and now retract and perish cultures in academia
- What Incentive Structures Do to Political Philosophy: Epistemic Pathologies and Institutional Design
- Making replication mainstream
- Individual and collective epistemic virtue in science
- Scientific misconduct, questionable research practices, and research climate: empirical approaches to integrity problems
- Self-plagiarism: A retrospective study of its prevalence and patterns across scientific disciplines
- Amazon Mechanical Turk: The Human Sciences’ Labor Problem
- The generalizability crisis
- Statistics in Service of Metascience: Measuring Replication Distance with Reproducibility Rate
- Behavioral and Experimental Agri-Environmental Research: Methodological Challenges, Literature Gaps, and Recommendations
- ‘Homo Metricus’: The New Academic Worker. How Quantitative Research Evaluation Practices Reshape the Intellectual Capital Needed to Succeed in Contemporary Universities?
- Verification of nucleotide sequence reagent identities in original publications in high impact factor cancer research journals
- Eleven years of student replication projects provide evidence on the correlates of replicability in psychology
- (Re)Integrating classical natural history approaches into contemporary field primatology: toward a more holistic understanding of primates
- The gaming of citation and authorship in academic journals: a warning from medicine
- Reclaiming mendelian randomization from the deluge of papers and misleading findings
- Open science, the replication crisis, and environmental public health
- Cross-national analyses require additional controls to account for the non-independence of nations
- Turning a blind eye to the reality of academic publishing
- The Varieties of Schelling Model Experience
- Lessons from the Evolutionary Computation Bestiary
- The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
- STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports
- A comprehensive public health approach is needed to study the impact of digital technology on health
- Replication of the natural selection of bad science
- Enacting Equality in Organizations: How Facilitating a Prosocial Bystander Intervention Curriculum Turned Engineering Faculty Members into Agents of Change
- Estimating the replicability of Brazilian biomedical science
- Scientists in silico?
- Structure in scientific networks: towards predictions of research\n dynamism
- Enhancing the quality and reproducibility of research: Preferred Evaluation of Cognitive and Neuropsychological Studies - The PECANS statement for human studies
- Questionable science and reproducibility in electrical brain stimulation research. [europepmc]
- The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research. [europepmc]
- The reproducibility of research and the misinterpretation of p -values. [europepmc]
- Modelling science trustworthiness under publish or perish pressure. [europepmc]
- Prestigious Science Journals Struggle to Reach Even Average Reliability. [europepmc]
- Promoting Replicability in Developmental Research Through Meta-analyses: Insights From Language Acquisition Research. [europepmc]
- Questionable research practices in ecology and evolution. [europepmc]
- Poor statistical reporting, inadequate data presentation and spin persist despite editorial advice. [europepmc]
- A systematic review and meta-analysis of effects of early life non-cognitive skills on academic, psychosocial, cognitive and health outcomes. [europepmc]
- On the prevalence of uninformative parameters in statistical models applying model selection in applied ecology. [europepmc]
- Power Analysis and Effect Size in Mixed Effects Models: A Tutorial. [europepmc]
- How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables. [europepmc]
- A Collaborative Approach to Infant Research: Promoting Reproducibility, Best Practices, and Theory-Building. [europepmc]
- An Overview of Scientific Reproducibility: Consideration of Relevant Issues for Behavior Science/Analysis. [europepmc]
- NIH peer review: Criterion scores completely account for racial disparities in overall impact scores. [europepmc]
- The generalizability crisis. [europepmc]
- Misinformation in and about science. [europepmc]
- A manifesto for reproducible science. [europepmc]
- Preferred reporting items for systematic reviews and meta-analyses in ecology and evolutionary biology: a PRISMA extension. [europepmc]
- Video game play is positively correlated with well-being. [europepmc]
- Challenges for assessing replicability in preclinical cancer biology. [europepmc]
- Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field. [europepmc]
- The Concept of Scaffold-Guided Bone Regeneration for the Treatment of Long Bone Defects: Current Clinical Application and Future Perspective. [europepmc]
- Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions. [europepmc]
- Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition. [europepmc]
Discussions
- Richard @rmcelreath.bsky.social and I wrote The Natural Selection of Bad Science over a decade ago, and it's been very frustrating to watch the process get worse rather than better in many cases. doi [bsky, 18 points, 1 comments]
- Wer viel publiziert, macht Karriere. Wer sorgfältig repliziert, aktuell noch nicht. Smaldino & McElreath (2016): Fragwürdige Forschungsmethoden können sich auch ohne per se betrügerische Intentionen d [bsky, 4 points, 1 comments]
- So another example of this perhaps doi.org/10.1098/rsos... [bsky, 2 points, 0 comments]
- 🔵Warsaw Thursday, 5 December, 7:00pm CET Article: Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society open science, 3(9), 160384. doi.org/10.1098/rsos... Mor [bsky, 1 points, 3 comments]
- doi.org/10.1098/rsos... [bsky, 0 points, 1 comments]
- I think about this paper a lot. "The natural selection of bad science" https://doi.org/10.1098/rsos.160384 [bsky, 0 points, 0 comments]
Related