Challenges in Obtaining Valid Causal Effect Estimates With Machine Learning Algorithms

2021/07/08 by Ashley I Naimi, Ashley I. Naimi, Alan E Mishler +3 · 28 citations
Mathematics · #Advanced Causal Inference Techniques #Algorithm #Computer science #Confidence interval #Confounding #Estimator #Machine learning #Mathematics #Monte Carlo method #Nonparametric statistics #Parametric statistics #Statistical Methods and Bayesian Inference #Statistical Methods and Inference #Statistics

paper · open access · doi:10.1093/aje/kwab201

published in American Journal of Epidemiology 192(9), 1536-1544 (Oxford University Press)

openalex publication_date 2021/07/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Unlike parametric regression, machine learning (ML) methods do not generally require precise knowledge of the true data generating mechanisms. As such, numerous authors have advocated for ML methods to estimate causal effects. Unfortunately, ML algorithmscan perform worse than parametric regression. We demonstrate the performance of ML-based single- and double-robust estimators. We use 100 Monte Carlo samples with sample sizes of 200, 1200, and 5000 to investigate bias and confidence interval coverage under several scenarios. In a simple confounding scenario, confounders were related to the treatment and the outcome via parametric models. In a complex confounding scenario, the simple confounders were transformed to induce complicated nonlinear relationships. In the simple scenario, when ML algorithms were used, double-robust estimators were superior to single-robust estimators. In the complex scenario, single-robust estimators with ML algorithms were at least as biased as estimators using misspecified parametric models. Double-robust estimators were less biased, but coverage was well below nominal. The use of sample splitting, inclusion of confounder interactions, reliance on a richly specified ML algorithm, and use of doubly robust estimators was the only explored approach that yielded negligible bias and nominal coverage. Our results suggest that ML based singly robust methods should be avoided.

Citations

• Constructing Inverse Probability Weights for Marginal Structural Models
• Targeted Maximum Likelihood Estimation for Causal Inference in Observational Studies

Cited by

• SensIAT: An R Package for Conducting Sensitivity Analysis of Randomized Trials with Irregular Assessment Times
• Evaluation and comparison of covariate balance metrics in studies with time‐dependent confounding
• Debiased machine learning for combining probability and non-probability survey data
• Doubly Robust Estimation in Observational Studies with Partial Interference
• Incremental Propensity Score Effects for Time-fixed Exposures. [europepmc]
• AIPW: An R Package for Augmented Inverse Probability-Weighted Estimation of Average Causal Effects. [europepmc]
• Performance Evaluation of Parametric and Nonparametric Methods When Assessing Effect Measure Modification. [europepmc]
• Analyses of child cardiometabolic phenotype following assisted reproductive technologies using a pragmatic trial emulation approach. [europepmc]
• The Role of the Natural Course in Causal Analysis. [europepmc]
• Revisiting the g-null Paradox. [europepmc]
• Use of Machine Learning to Estimate the Per-Protocol Effect of Low-Dose Aspirin on Pregnancy Outcomes: A Secondary Analysis of a Randomized Clinical Trial. [europepmc]
• Use of a Doubly Robust Machine-Learning-Based Approach to Evaluate Body Mass Index as a Modifier of the Association Between Fruit and Vegetable Intake and Preeclampsia. [europepmc]
• Synthetic Negative Controls: Using Simulation to Screen Large-scale Propensity Score Analyses. [europepmc]
• Emulating a target trial of intensive nurse home visiting in the policy-relevant population using linked administrative data. [europepmc]
• Machine learning can improve the development of evidence-based dietary guidelines. [europepmc]
• Estimation of the Average Causal Effect in Longitudinal Data With Time-Varying Exposures: The Challenge of Nonpositivity and the Impact of Model Flexibility. [europepmc]
• A Simulation Study Comparing the Performance of Time-Varying Inverse Probability Weighting and G-Computation in Survival Analysis. [europepmc]
• Association of statin use with outcomes of patients admitted with COVID-19: an analysis of electronic health records using superlearner. [europepmc]
• Periconceptional Dietary Patterns and Adverse Pregnancy and Birth Outcomes. [europepmc]
• Handling missing data when estimating causal effects with targeted maximum likelihood estimation. [europepmc]
• Targeted learning with an undersmoothed LASSO propensity score model for large-scale covariate adjustment in health-care database studies. [europepmc]
• Pseudo-random Number Generator Influences on Average Treatment Effect Estimates Obtained with Machine Learning. [europepmc]
• Bridged treatment comparisons: an illustrative application in HIV treatment. [europepmc]
• Machine learning in causal inference for epidemiology. [europepmc]
• Towards robust causal inference in epidemiologic research: employing double cross-fit TMLE in right heart catheterization data. [europepmc]
• Step-by-step causal analysis of EHRs to ground decision-making. [europepmc]
• Performance of Cross-Validated Targeted Maximum Likelihood Estimation. [europepmc]
• Covariate selection strategies and estimands - a review of current practice of risk factor analysis from a causal perspective. [europepmc]
• Recanting Twins: Addressing Intermediate Confounding in Mediation Analysis. [europepmc]

Related

• Challenges in Obtaining Valid Causal Effect Estimates with Machine Learning Algorithms
• Machine learning for causal inference: on the use of cross-fit estimators
• Machine Learning for Causal Inference
• Robust Causal Learning for the Estimation of Average Treatment Effects
• The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation