Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling

2012/04/16 by Leslie K. John, George Loewenstein, Drazen Prelec +1 · 1 voice · 124 citations
Psychology · Social Sciences · #Academic integrity and plagiarism #Deception detection and forensic psychology #Misinformation and Its Impacts

paper · doi:10.1177/0956797611430953

openalex publication_date 2012/04/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/03

Abstract

Cases of clear scientific misconduct have received significant media attention recently, but less flagrantly questionable research practices may be more prevalent and, ultimately, more damaging to the academic enterprise. Using an anonymous elicitation format supplemented by incentives for honest reporting, we surveyed over 2,000 psychologists about their involvement in questionable research practices. The impact of truth-telling incentives on self-admissions of questionable research practices was positive, and this impact was greater for practices that respondents judged to be less defensible. Combining three different estimation methods, we found that the percentage of respondents who have engaged in questionable practices was surprisingly high. This finding suggests that some questionable practices may constitute the prevailing research norm.

Citations

• Why Most Published Research Findings Are False
• The case for motivated reasoning.
• False-Positive Psychology
• Exploring Scientific Misconduct: Isolated Individuals, Impure Institutions, or an Inevitable Idiom of Modern Science?

Cited by

• Error needs culture! Exploring the relationship between error culture and openness
• A framework for assessing the trustworthiness of scientific research findings
• Scientific misconduct, questionable research practices, and research climate: empirical approaches to integrity problems
• Individual and collective epistemic virtue in science
• Quality research needs good working conditions
• Metascience as a Scientific Social Movement
• Clarifications on the application and interpretation of the test for excess significance and its extensions
• A template for preregistration of quantitative research in psychology: Report of the joint psychological societies preregistration task force.
• Making replication mainstream
• A Validity-Based Framework for Understanding Replication in Psychology
• A solution to the single-question crowd wisdom problem
• The Baby Factory
• Initial evidence of research quality of registered reports compared with the standard publishing model
• Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
• Both Questionable and Open Research Practices Are Prevalent in Education Research
• 281. MEVATRON77DX67 における電子線用可変型絞りについて(治療-6 電子線)
• How can we make sound replication decisions?
• Proper Proxy Scoring Rules
• Examining the replicability of online experiments selected by a decision market
• Selective outcome reporting in trials of behavioural health interventions in health psychology and behavioural medicine journals: a review
• Faculty-student differences in authorship perceptions before and after authorship ethics interventions
• Effect Size Estimation Fromt-Statistics in the Presence of Publication Bias
• Considering Registered Reports at C&RL
• Game-Theoretic Statistics and Safe Anytime-Valid Inference
• Predicting replication outcomes and generalizability of results
• Replication, statistical consistency, and publication bias
• Bayesian data analysis for newcomers
• Robust Bayesian meta‐analysis: Model‐averaging across complementary publication bias adjustment methods
• Method Reporting with Initials for Transparency (MeRIT) promotes more granularity and accountability for author contributions
• Reliability Underreported in Verbal Memory Tests for Spanish-Speaking Populations: A Systematic Review
• The Chrysalis Effect
• Voir La Vie Sans Rose-Colored Glasses
• Aligning Scientific Values and Research Integrity: A Study of Researchers’ Perceptions and Practices in Four Countries
• Editorial Perspective: Misaligned incentives in mental health research – the case for Registered Reports
• On the predicted replicability of two decades of experimental research on system justification: A Z‐curve analysis
• Is something rotten in the state of Denmark? Cross-national evidence for widespread involvement but not systematic use of questionable research practices across all fields of research
• Open science, the replication crisis, and environmental public health
• An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)
• Increasing Transparency Through a Multiverse Analysis
• Open minds, tied hands: Awareness, behavior, and reasoning on open science and irresponsible research behavior
• Big little lies: a compendium and simulation of p -hacking strategies
• Developmental psychologists should adopt citizen science to improve generalization and reproducibility
• Neglect of the individual as a neglected problem: The relevance of combined idiographic-nomothetic approaches for social psychology today
• Identificatory underpinnings of ethical research behavior for graduate students: Evidence from a baseline sample of a university research ethics training program
• Improving Social Science: Lessons from the Open Science Movement
• Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future
• On the Limitations of Manipulation Checks: An Obstacle Toward Cumulative Science
• Time to adjust: Improving replicability in experimental psychology by adjustment for evident selective inference
• Inter-Rater Reliability in Assessing the Methodological Quality of Research Papers in Psychology
• Contextual sensitivity in scientific reproducibility
• Genome-wide association studies
• Z-Curve Plot: A Visual Diagnostic for Publication Bias in Meta-Analysis
• Recommendations for Increasing Replicability in Psychology
• Language Development Research Editorial: Why do we need another journal?
• Psychology's Questionable Research Fundamentals (QRFs): Key problems in quantitative psychology and psychological measurement beyond Questionable Research Practices (QRPs)
• Experimental Standards for Deep Learning in Natural Language Processing Research
• Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
• A Tale of Open Science: Emergence of a New Normal
• Humans can find rhythm in randomly timed sounds
• From Hypotheses to Systematicity and Back
• False-Positive Psychology
• The ‘shades of grey’ in research integrity—Researchers admit to questionable research practices that they do not perceive to be serious
• Is time of the essence? A temporal meta-synthesis of seven media effects theories
• Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses
• The benefits of preregistration and Registered Reports
• Incentives in surveys
• Does Truth Pay? Investigating the Effectiveness of the Bayesian Truth Serum With an Interim Payment: A Registered Report
• The validity of individual Rorschach variables: Systematic reviews and meta-analyses of the comprehensive system.
• Questionable and Improved Research Practices in Single-Case Experimental Design: Initial Investigation and Findings
• Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses
• Incentivizing Evaluation via Limited Access to Ground Truth: Peer-Prediction Makes Things Worse
• Global p-Values in Multi-Design Studies
• Can visualization alleviate dichotomous thinking? Effects of visual representations on the cliff effect
• Minimizing Mistakes in Psychological Science
• What Can We Learn from the Retraction of Medical Articles
• Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences
• Interactions between infant characteristics and parenting factors rarely replicate across cohorts and developmental domains
• Questionable research practices may have little effect on replicability
• Qualitative research at the crossroads of open science and big data: Ethical considerations
• Investigating the effects of single-dose intranasal testosterone on economic preferences in a large randomized trial of men
• Same model, same data, but different outcomes: Evaluating the impact of method choices in structural equation modeling
• The replication crisis has led to positive structural, procedural, and community changes
• Effort-aware Fairness: Incorporating a Philosophy-informed, Human-centered Notion of Effort into Algorithmic Fairness Metrics
• The Solution to Science's Replication Crisis
• The Case for Preregistering Quasi-Experimental Program and Policy Evaluations
• What meta‐research has taught us about research and changes to research practices
• KORELASI PANJANG TUNGKAI DAN DAYA LEDAK OTOTTUNGKAI TERHADAP JAUHNYA HASIL TENDANGANBOLA PADA SISWA EKSTRAKURIKULER SEPAK BOLASMA NEGERI 1 NGRAMBE KABUPATEN NGAWITAHUN 2008/2009
• Smart but Evil? Student-Teachers’ Perception of Educational Researchers’ Epistemic Trustworthiness
• Threats of a replication crisis in empirical computer science
• Actionable recommendations for narrowing the science-practice gap in open science
• Answers to 18 Questions About Open Science Practices
• Towards a taxonomy of research misconduct: The case of business school research
• The extent and causes of academic text recycling or ‘self-plagiarism’
• Supporting replication research in management journals: Qualitative analysis of editorials published between 1970 and 2015
• Reproducible research practices and transparency across linguistics
• Questionable Research Practices Revisited
• Investigating the replicability of preclinical cancer biology
• Challenges for assessing replicability in preclinical cancer biology
• The empirical replicability of task-based fMRI as a function of sample size
• Data transparency and citation in the journal Gesture
• Metaresearch for Evaluating Reproducibility in Ecology and Evolution
• Striving for transparent and credible research: practical guidelines for behavioral ecologists
• Pre-registration: Weighing costs and benefits for researchers
• Decisions, Decisions, Decisions: an ethnographic study of researcher discretion in practice
• Scientific Utopia
• Misconduct and Questionable Research Practices: The Ethics of Quantitative Data Handling and Reporting in Applied Linguistics
• Navigating the Science System: Research Integrity and Academic Survival Strategies
• Dissecting the tension of open science standards implementation in management and organization journals
• Cross-Cultural Psychology and the Rise of Academic Capitalism: Linguistic Changes in CCR and JCCP Articles, 1970-2014
• The effect of preregistration on trust in empirical research findings: results of a registered report
• Moving beyond self-reports to estimate the prevalence of commercial contract cheating: an Australian study
• A framework for open policy analysis
• Construct jangle or construct mangle? Thinking straight about (nonredundant) psychological constructs
• Testing for Questionable Research Practices in a Meta-Analysis: An Example from Experimental Parapsychology
• Facts Are More Important Than Novelty
• Underreporting in Psychology Experiments
• p-Hacking Inflates Type I Error Rates in the Error Statistical Approach but not in the Formal Inference Approach
• An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
• The academic, economic and societal impacts of Open Access: an evidence-based review
• The preregistration revolution
• The prevalence of statistical reporting errors in psychology (1985–2013)
• The Extent and Consequences of P-Hacking in Science
• p -Curve and Effect Size
• STAR: Statistical Tests with Auditable Results
• Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
• Why Selective Publication of Statistically Significant Results Can Be Effective
• Replication crisis [wikipedia]

Discussions

• Perhaps: Flake, J. K., & Fried, E. I. (2020). doi.org/10.1177/2515... Janke, S., et al. (2019). doi.org/10.1177/1948... John, L. K., Loewenstein, G., & Prelec, D. (2012). doi.org/10.1177/0956... S [bsky, 3 points, 0 comments]

Related

• Questionable Research Practices Revisited
• Bestiary of Questionable Research Practices in Psychology
• Bending Our Ethics Code
• Questionable research practices violate the American Psychological Association’s Code of Ethics.
• Participant carelessness and fraud: Consequences for clinical research and potential solutions.