Provably Efficient Causal Reinforcement Learning with Confounded Observational Data

2020/06/22 by Lingxiao Wang, Zhuoran Yang, Wang, Lingxiao +3 · 13 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Computer science #FOS: Computer and information sciences #Function (biology) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Mathematics #Observational learning #Observational study #Regret #Reinforcement Learning in Robotics #Reinforcement learning #Sample (material) #Statistics #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.12311

published in arXiv (Cornell University) (Cornell University) · 42 pages, 4 figures

arxiv created 2020/06/22 · openalex publication_date 2020/06/22 · arxiv updated 2020/06/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Empowered by expressive function approximators such as neural networks, deep reinforcement learning (DRL) achieves tremendous empirical successes. However, learning expressive function approximators requires collecting a large dataset (interventional data) by interacting with the environment. Such a lack of sample efficiency prohibits the application of DRL to critical scenarios, e.g., autonomous driving and personalized medicine, since trial and error in the online setting is often unsafe and even unethical. In this paper, we study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. To incorporate the possibly confounded observational data, we propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner. More specifically, DOVI explicitly adjusts for the confounding bias in the observational data, where the confounders are partially observed or unobserved. In both cases, such adjustments allow us to construct the bonus based on a notion of information gain, which takes into account the amount of information acquired from the offline setting. In particular, we prove that the regret of DOVI is smaller than the optimal regret achievable in the pure online setting by a multiplicative factor, which decreases towards zero when the confounded observational data are more informative upon the adjustments. Our algorithm and analysis serve as a step towards causal reinforcement learning.

Citations

• Contextual Markov Decision Processes
• Generalization and Exploration via Randomized Value Functions
• Deep Reinforcement Learning for Dialogue Generation
• Policy Evaluation and Optimization with Continuous Treatments
• Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Who Make Drivers Stop? Towards Driver-centric Risk Assessment: Risk Object Identification via Causal Inference
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Provably Efficient Reinforcement Learning with Linear Function Approximation

Cited by

• The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportability
• Causal Reinforcement Learning based Agent-Patient Interaction with Clinical Domain Knowledge
• Expediting Reinforcement Learning by Incorporating Knowledge About Temporal Causality in the Environment
• Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Critic

Related

• Causal Deep Reinforcement Learning Using Observational Data
• Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Data
• Causal Reinforcement Learning using Observational and Interventional Data
• Deconfounding Reinforcement Learning in Observational Settings
• Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domains