Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with\n Latent Confounders

2020/07/27 by Andrew Bennett, Bennett, Andrew, Nathan Kallus +5 · 4 citations
Computer Science · Environmental Science · Economics, Econometrics and Finance · #Reinforcement Learning in Robotics #Environmental Impact and Sustainability #Health Systems, Economic Evaluations, Quality of Life

paper · pdf · doi:10.48550/arxiv.2007.13893

Abstract

Off-policy evaluation (OPE) in reinforcement learning is an important problem\nin settings where experimentation is limited, such as education and healthcare.\nBut, in these very same settings, observed actions are often confounded by\nunobserved variables making OPE even more difficult. We study an OPE problem in\nan infinite-horizon, ergodic Markov decision process with unobserved\nconfounders, where states and actions can act as proxies for the unobserved\nconfounders. We show how, given only a latent variable model for states and\nactions, policy value can be identified from off-policy data. Our method\ninvolves two stages. In the first, we show how to use proxies to estimate\nstationary distribution ratios, extending recent work on breaking the curse of\nhorizon to the confounded setting. In the second, we show optimal balancing can\nbe combined with such learned ratios to obtain policy value while avoiding\ndirect modeling of reward functions. We establish theoretical guarantees of\nconsistency, and benchmark our method empirically.\n

Citations

• Large Sample Properties of Generalized Method of Moments Estimators
• Planning and acting in partially observable stochastic domains
• Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes

Cited by

• Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
• Confounding Robust Deep Reinforcement Learning: A Causal Approach
• Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
• Causal Reinforcement Learning using Observational and Interventional Data

Related

• Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
• Projected State-action Balancing Weights for Offline Reinforcement Learning
• Reinforcement Learning with Continuous Actions Under Unmeasured Confounding
• Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning
• Model-Based Reinforcement Learning Under Confounding