Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction

2022/12/14 by Brahma S. Pavse, Josiah P. Hanna, Pavse, Brahma S. +1 · 1 citation
Computer Science · #Age of Information Optimization #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2212.07486

openalex publication_date 2022/12/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider the problem of off-policy evaluation (OPE) in reinforcement learning (RL), where the goal is to estimate the performance of an evaluation policy, π_e, using a fixed dataset, D, collected by one or more policies that may be different from π_e. Current OPE algorithms may produce poor OPE estimates under policy distribution shift i.e., when the probability of a particular state-action pair occurring under π_e is very different from the probability of that same pair occurring in D (Voloshin et al. 2021, Fu et al. 2021). In this work, we propose to improve the accuracy of OPE estimators by projecting the high-dimensional state-space into a low-dimensional state-space using concepts from the state abstraction literature. Specifically, we consider marginalized importance sampling (MIS) OPE algorithms which compute state-action distribution correction ratios to produce their OPE estimate. In the original ground state-space, these ratios may have high variance which may lead to high variance OPE. However, we prove that in the lower-dimensional abstract state-space the ratios can have lower variance resulting in lower variance OPE. We then highlight the challenges that arise when estimating the abstract ratios from data, identify sufficient conditions to overcome these issues, and present a minimax optimization problem whose solution yields these abstract ratios. Finally, our empirical evaluation on difficult, high-dimensional state-space OPE tasks shows that the abstract ratios can make MIS OPE estimators achieve lower mean-squared error and more robust to hyperparameter tuning than the ground ratios.

Cited by

• Where to Intervene: Action Selection in Deep Reinforcement Learning

Related

• Low Variance Off-policy Evaluation with State-based Importance Sampling
• A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• Marginalized Operators for Off-policy Reinforcement Learning
• Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning