Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning

2021/02/19 by Luofeng Liao, Liao, Luofeng, Zuyue Fu +8 · 2 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2102.09907

openalex publication_date 2021/02/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In offline reinforcement learning (RL) an optimal policy is learned solely from a priori collected observational data. However, in observational data, actions are often confounded by unobserved variables. Instrumental variables (IVs), in the context of RL, are the variables whose influence on the state variables is all mediated by the action. When a valid instrument is present, we can recover the confounded transition dynamics through observational data. We study a confounded Markov decision process where the transition dynamics admit an additive nonlinear functional form. Using IVs, we derive a conditional moment restriction through which we can identify transition dynamics based on observational data. We propose a provably efficient IV-aided Value Iteration (IVVI) algorithm based on a primal-dual reformulation of the conditional moment restriction. To our knowledge, this is the first provably efficient algorithm for instrument-aided offline RL.

Citations

• A Stochastic Approximation Method
• Instrumental Variable Estimation of Nonparametric Models
• Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach
• Causal inference and data fusion in econometrics
• Minimax Estimation of Conditional Moment Models
• Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with\n Latent Confounders
• Nonparametric Instrumental Regression

Cited by

• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Causality-informed Anomaly Detection in Partially Observable Sensor Networks: Moving beyond Correlations

Related

• Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data
• Causal Reinforcement Learning using Observational and Interventional Data
• Causal Deep Reinforcement Learning Using Observational Data
• Near-optimal Offline Reinforcement Learning with Linear Representation:\n Leveraging Variance Information with Pessimism