How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice Based on 67 Replicated Studies

2024/05/03 by Apoorva Lal, Mackenzie Lockhart, Yiqing Xu +1 · 1 voice · 7 citations
Mathematics · Social Sciences · #Advanced Causal Inference Techniques #Electoral Systems and Political Participation #Qualitative Comparative Analysis Research

paper · pdf · doi:10.1017/pan.2024.2

openalex publication_date 2024/05/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/21

Abstract

Abstract Instrumental variable (IV) strategies are widely used in political science to establish causal relationships, but the identifying assumptions required by an IV design are demanding, and assessing their validity remains challenging. In this paper, we replicate 67 articles published in three top political science journals from 2010 to 2022 and identify several concerning patterns. First, researchers often overestimate the strength of their instruments due to non-i.i.d. error structures such as clustering. Second, IV estimates are often highly uncertain, and the commonly used t -test for two-stage-least-squares (2SLS) estimates frequently underestimate the uncertainties. Third, in most replicated studies, 2SLS estimates are significantly larger in magnitude than ordinary-least-squares estimates, and their absolute ratio is inversely related to the strength of the instrument in observational studies—a pattern not observed in experimental ones—suggesting potential violations of unconfoundedness or the exclusion restriction in the former. We provide a checklist and software to help researchers avoid these pitfalls and improve their practice.

Citations

• Large Sample Properties of Generalized Method of Moments Estimators
• A Practitioner’s Guide to Cluster-Robust Inference
• Estimation of the Parameters of a Single Equation in a Complete System of Stochastic Equations
• Instrumental Variables Regression with Weak Instruments
• Profiling Compliers and Noncompliers for Instrumental-Variable Analysis
• Bootstrap-Based Improvements for Inference with Clustered Errors
• On the reliability of published findings using the regression discontinuity design in political science
• Plausibly Exogenous

Cited by

• Rural electrification, the credibility revolution, and the limits of evidence-based policy
• For God, Tsar and fatherland? The church’s role in modern autocracy
• The business case for demographic diversity in strategic leadership teams: A systematic and critical review of the causal evidence
• Resale Royalty in Non-Fungible Token Marketplaces: Blessing or Burden for Creators and Platforms?
• Attitudinal and Behavioral Legacies of Wartime Violence: A Meta-Analysis
• The Development of an Issue Public: Evidence from The Eras Tour
• Causal Panel Analysis under Parallel Trends: Lessons from a Large Reanalysis Study
• Estimating Causal Effects With Observational Data: Guidelines for Agricultural and Applied Economists

Discussions

• 3/ Why? Because a key bottleneck to scaling transparency practices in empirical research is heterogeneous replication packages. I previously led a project reanalyzing 67 IV studies (APSR/AJPS/JOP, 20 [bsky, 5 points, 1 comments]

Related

• How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice Based on Over 60 Replicated Studies
• On the reliability of published findings using the regression discontinuity design in political science
• On the reliability of published findings using the regression discontinuity design in political science
• What to Observe When Assuming Selection on Observables
• Confidence Intervals for Causal Effects with Invalid Instruments using Two-Stage Hard Thresholding with Voting