Threats of a replication crisis in empirical computer science

2020/07/22 by Andy Cockburn, Pierre Dragicevic, Lonni Besançon +1 · 1 voice · 10 citations
Computer Science · Decision Sciences · #Explainable Artificial Intelligence (XAI) #Scientific Computing and Data Management #Topic Modeling

paper · doi:10.1145/3360311

openalex publication_date 2020/07/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/11

Abstract

Research replication only works if there is confidence built into the results.

Citations

• The file drawer problem and tolerance for null results.
• Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
• False-Positive Psychology
• Abandon Statistical Significance
• The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective
• P-curve: A key to the file-drawer.
• CRITICAL QUESTIONS FOR BIG DATA
• Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
• Why Most Published Research Findings Are False

Cited by

• Open science saves lives: lessons from the COVID-19 pandemic
• State of the Art of LLM-Enabled Interaction with Visualization
• Feature-Guided Analysis of Neural Networks: A Replication Study
• Hyperparameter Optimization and Reproducibility in Deep Learning Model Training
• Benchmarking Crisis in Social Media Analytics: A Solution for the Data-Sharing Problem
• Voice user interfaces for effortless navigation in medical virtual reality environments
• How can we make sound replication decisions?
• Re-evaluating GPT-4’s bar exam performance
• Initiating and Replicating the Observations of Interactional Properties by User Studies Optimizing Applicative Prototypes
• Can visualization alleviate dichotomous thinking? Effects of visual representations on the cliff effect
• Should we use the NASA-TLX in HCI? A review of theoretical and methodological issues around Mental Workload Measurement
• A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research

Discussions

• I'm delighted to see my article on the repro crisis (doi.org/10.1145/3360...) featured in this press article! modernsciences.org/research-rep... [bsky, 4 points, 0 comments]

Related

• Predicting replication outcomes and generalizability of results
• Understanding the Replication Crisis as a Base Rate Fallacy
• The reliability of replications: a study in computational reproductions
• ‘Replication games’ test the robustness of social-science studies
• Computational Reproducibility in Computational Social Science