Model-Free and Model-Based Policy Evaluation when Causality is Uncertain

2022/04/02 by David Bruns-Smith, Bruns-Smith, David · 1 citation
Computer Science · Economics, Econometrics and Finance · Mathematics · #Advanced Causal Inference Techniques #Bayesian Modeling and Causal Inference #Economic Policies and Impacts #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2204.00956

openalex publication_date 2022/04/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

When decision-makers can directly intervene, policy evaluation algorithms give valid causal estimates. In off-policy evaluation (OPE), there may exist unobserved variables that both impact the dynamics and are used by the unknown behavior policy. These "confounders" will introduce spurious correlations and naive estimates for a new policy will be biased. We develop worst-case bounds to assess sensitivity to these unobserved confounders in finite horizons when confounders are drawn iid each period. We demonstrate that a model-based approach with robust MDPs gives sharper lower bounds by exploiting domain knowledge about the dynamics. Finally, we show that when unobserved confounders are persistent over time, OPE is far more difficult and existing techniques produce extremely conservative bounds.

Cited by

• Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees

Related

• Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
• Generalizing Off-Policy Evaluation From a Causal Perspective For Sequential Decision-Making
• An Instrumental Variable Approach to Confounded Off-Policy Evaluation
• Uncertainty Quantification and Causal Considerations for Off-Policy Decision Making
• Off-Policy Evaluation with Out-of-Sample Guarantees