Model-Based Reinforcement Learning Under Confounding

2025/12/08 by Venkatesh, Nishanth, Malikopoulos, Andreas A.
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2512.07528

Abstract

We investigate model-based reinforcement learning in contextual Markov decision processes (C-MDPs) in which the context is unobserved and induces confounding in the offline dataset. In such settings, conventional model-learning methods are fundamentally inconsistent, as the transition and reward mechanisms generated under a behavioral policy do not correspond to the interventional quantities required for evaluating a state-based policy. To address this issue, we adapt a proximal off-policy evaluation approach that identifies the confounded reward expectation using only observable state-action-reward trajectories under mild invertibility conditions on proxy variables. When combined with a behavior-averaged transition model, this construction yields a surrogate MDP whose Bellman operator is well defined and consistent for state-based policies, and which integrates seamlessly with the maximum causal entropy (MaxCausalEnt) model-learning framework. The proposed formulation enables principled model learning and planning in confounded environments where contextual information is unobserved, unavailable, or impractical to collect.

Citations

• AI Recommendation Systems for Lane-Changing Using Adherence-Aware Reinforcement Learning
• Connected and Automated Vehicles in Mixed-Traffic: Learning Human Driver Behavior for Effective On-Ramp Merging
• Offline Policy Evaluation and Optimization under Confounding
• Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes
• Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
• Off-Policy Evaluation in Partially Observable Environments
• Learning Latent Dynamics for Planning from Pixels
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Where Do You Think You're Going?: Inferring Beliefs about Dynamics from\n Behavior
• Neural Network Dynamics for Model-Based Deep Reinforcement Learning with\n Model-Free Fine-Tuning
• Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder
• Inverse Reinforcement Learning with Simultaneous Estimation of Rewards\n and Dynamics

Related

• Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes
• Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with\n Latent Confounders
• Reinforcement Learning with Continuous Actions Under Unmeasured Confounding
• Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
• Generalization in Monitored Markov Decision Processes (Mon-MDPs)