Marginalized Operators for Off-policy Reinforcement Learning

2022/03/30 by Yunhao Tang, Mark Rowland, Tang, Yunhao +5
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2203.16177

openalex publication_date 2022/03/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this work, we propose marginalized operators, a new class of off-policy evaluation operators for reinforcement learning. Marginalized operators strictly generalize generic multi-step operators, such as Retrace, as special cases. Marginalized operators also suggest a form of sample-based estimates with potential variance reduction, compared to sample-based estimates of the original multi-step operators. We show that the estimates for marginalized operators can be computed in a scalable way, which also generalizes prior results on marginalized importance sampling as special cases. Finally, we empirically demonstrate that marginalized operators provide performance gains to off-policy evaluation and downstream policy optimization algorithms.

Related

• Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction
• Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning
• Minimax Weight and Q-Function Learning for Off-Policy Evaluation
• Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
• A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation