The Dangers of Extreme Counterfactuals

2005/11/03 by Gary King, Langche Zeng · 1 voice · 24 citations
Economics, Econometrics and Finance · Mathematics · #Advanced Causal Inference Techniques #Health Systems, Economic Evaluations, Quality of Life #Statistical Methods and Inference

paper · pdf · doi:10.1093/pan/mpj004

openalex publication_date 2005/11/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

We address the problem that occurs when inferences about counterfactuals—predictions, “what-if” questions, and causal effects—are attempted far from the available data. The danger of these extreme counterfactuals is that substantive conclusions drawn from statistical models that fit the data well turn out to be based largely on speculation hidden in convenient modeling assumptions that few would be willing to defend. Yet existing statistical strategies provide few reliable means of identifying extreme counterfactuals. We offer a proof that inferences farther from the data allow more model dependence and then develop easy-to-apply methods to evaluate how model dependent our answers would be to specified counterfactuals. These methods require neither sensitivity testing over specified classes of models nor evaluating any specific modeling assumptions. If an analysis fails the simple tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. Free software that accompanies this article implements all the methods developed.

Citations

• Association, Causation, And Marginal Structural Models
• Some distance properties of latent root and vector methods used in multivariate analysis
• A General Coefficient of Similarity and Some of Its Properties
• Estimating causal effects of treatments in randomized and nonrandomized studies.
• The central role of the propensity score in observational studies for causal effects
• The Consquences of Adjustment for a Concomitant Variable That Has Been Affected by the Treatment
• Making the Most of Statistical Analyses: Improving Interpretation and Presentation
• Causal Diagrams for Epidemiologic Research

Cited by

• How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
• Evaluating the impact of anti-tobacco legislation in Spain with a gender perspective: a synthetic control approach
• Matching Methods for Confounder Adjustment: An Addition to the Epidemiologist’s Toolbox
• More than just transition: Uncovering heterogeneous socio-economic outcomes of climate policies
• How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice
• Vote buying, turnout and trust: democratic consequences of electoral clientelism in the Polish 2023 election
• Everything all at once: On choosing an estimand for multi-component environmental exposures
• Regularizing Extrapolation in Causal Inference
• Does Decapitation Work? Assessing the Effectiveness of Leadership Targeting in Counterinsurgency Campaigns
• What to Observe When Assuming Selection on Observables
• The varied impact of direct democracy on local housing policies: unravelling the role of socioeconomic context
• Fragility in Average Treatment Effect on the Treated under Limited Covariate Support
• Estimating Average Treatment Effects with Support Vector Machines
• Why Propensity Scores Should Not Be Used for Matching
• Causal Estimands for Policy Evaluation and Beyond
• Kernel Regularized Least Squares: Reducing Misspecification Bias with a Flexible and Interpretable Machine Learning Approach
• Causal Inference without Balance Checking: Coarsened Exact Matching
• Varying impacts of letters of recommendation on college admissions: Approximate balancing weights for subgroup effects in observational studies
• Do fiscal rules constrain political budget cycles?
• A Primary Cause of Partisanship? Nomination Systems and Legislator Ideology
• The Washington consensus works: Causal effects of reform, 1970-2015
• Matching Methods for Causal Inference: A Review and a Look Forward
• Estimating the Direct Economic Damages of the Earthquake in Haiti
• Information Equivalence in Survey Experiments

Discussions

• There are also some tools for saying, "nah, that counterfactual is too extreme" doi.org/10.1093/pan/... [bsky, 3 points, 1 comments]

Related

• When Counterfactual Reasoning Fails: Chaos and Real-World Complexity
• The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations
• Advancing Counterfactual Inference through Nonlinear Quantile Regression
• Evaluating counterfactual explanations using Pearl's counterfactual method
• Counterfactual Realizability