Transparency, Reproducibility, and the Credibility of Economics Research

2018/09/01 by Garret Christensen, Edward Miguel · 37 citations
Decision Sciences · Mathematics · #scientometrics and bibliometrics research #Meta-analysis and systematic reviews #Advanced Causal Inference Techniques

paper · pdf · doi:10.1257/jel.20171350

Abstract

There is growing interest in enhancing research transparency and reproducibility in economics and other scientific fields. We survey existing work on these topics within economics and discuss the evidence suggesting that publication bias, inability to replicate, and specification searching remain widespread in the discipline. We next discuss recent progress in this area, including through improved research design, study registration and pre-analysis plans, disclosure standards, and open sharing of data and materials, drawing on experiences in both economics and other social sciences. We discuss areas where consensus is emerging on new practices, as well as approaches that remain controversial, and speculate about the most effective ways to make economics research more credible in the future. ( JEL A11, C18, I23)

Citations

• Adaptive linear step-up procedures that control the false discovery rate
• Imitation Costs and Patents: An Empirical Study
• Social Security, Induced Retirement, and Aggregate Capital Accumulation
• Economic Shocks and Civil Conflict: An Instrumental Variables Approach
• Prediction Policy Problems
• Revisiting the Minimum Wage—Employment Debate: Throwing Out the Baby with the Bathwater?

Cited by

• A guide for social science journal editors on easing into open science
• Replication code availability over time and across fields: Evidence from the German Socio‐Economic Panel
• What drives financial development? A Meta-regression analysis
• Selective reporting of placebo tests in top economics journals
• Survey of open science practices and attitudes in the social sciences
• pAI-Econ-claude: A Gated Human-in-the-Loop Multi-Agent Architecture for AI-Assisted Economic Theory Development
• Does Multi-Agent Debate Improve AI Feedback on Research Papers?
• Randomness in large language models: What researchers need to know (and report)
• There must be an error here! Experimental evidence on coding errors' biases
• Selective and (mis)leading economics journals: Meta‐research evidence
• Is economics self‐correcting? Replications in the <i>American Economic Review</i>
• Rural electrification, the credibility revolution, and the limits of evidence-based policy
• Open Research: Examples of good practice, and resources across disciplines (2025 edition)
• The Case for Preregistering Quasi-Experimental Program and Policy Evaluations
• ‘Replication games’ test the robustness of social-science studies
• Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the “Many Analysts, One Data Set” Project
• Introducing the Journal Editors Discussion Interface
• Long-term effects of the Targeting the Ultra Poor program—A reproducibility and replicability assessment of Banerjee et al. (2021)
• Prior-Free Information Design
• Relative Risk Aversion: A Meta‐Analysis
• Publication Bias and Editorial Statement on Negative Findings
• Reproducibility and robustness of economics and political science research
• Destructive Behaviour, Judgement and Economic Decision-making under Thermal Stress
• What meta‐research has taught us about research and changes to research practices
• Reproduce to validate: A comprehensive study on the reproducibility of economics research
• Predicting the replicability of social and behavioural science claims in COVID-19 preprints
• Reproducibility: The New Frontier in AI Governance
• Replications in agricultural economics
• Deepening transparency about value-laden assumptions in energy and environmental modelling: improving best practices for both modellers and non-modellers
• A protocol for structured robustness reproductions and replicability assessments
• Promoting Reproducibility and Replicability in Political Science
• Inaccurate forecasting of a randomized controlled trial
• Cherry Picking with Synthetic Controls
• Reputation, Risk, and Visibility
• Data Practices for Studying the Impacts of Environmental Amenities and Hazards with Nationwide Property Data
• STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports
• Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication

Related

• Reproduce to validate: A comprehensive study on the reproducibility of economics research
• An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)
• ‘Replication games’ test the robustness of social-science studies
• A Reputation Economy: Results from an Empirical Survey on Academic Data\n Sharing
• Synthetic Data Generation for Economists