Average-Reward Off-Policy Policy Evaluation with Function Approximation

2021/01/08 by Shangtong Zhang, Zhang, Shangtong, Yi Wan +5 · 5 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Optimization and Search Problems #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2101.02808

openalex publication_date 2021/01/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

We consider off-policy policy evaluation with function approximation (FA) in average-reward MDPs, where the goal is to estimate both the reward rate and the differential value function. For this problem, bootstrapping is necessary and, along with off-policy learning and FA, results in the deadly triad (Sutton & Barto, 2018). To address the deadly triad, we propose two novel algorithms, reproducing the celebrated success of Gradient TD algorithms in the average-reward setting. In terms of estimating the differential value function, the algorithms are the first convergent off-policy linear function approximation algorithms. In terms of estimating the reward rate, the algorithms are the first convergent off-policy linear function approximation algorithms that do not require estimating the density ratio. We demonstrate empirically the advantage of the proposed algorithms, as well as their nonlinear variants, over a competitive density-ratio-based approach, in a simple domain as well as challenging robot simulation tasks.

Citations

• Stochastic Approximation: A Dynamical Systems Viewpoint
• Artificial intelligence: a modern approach
• Finite-Sample Analysis of Proximal Gradient TD Algorithms
• Consistent On-Line Off-Policy Evaluation
• Addressing Function Approximation Error in Actor-Critic Methods
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Planning with Expectation Models
• Challenges of Real-World Reinforcement Learning
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Minimax Weight and Q-Function Learning for Off-Policy Evaluation
• GenDICE: Generalized Offline Estimation of Stationary Values
• MOReL : Model-Based Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
• Learning and Planning in Average-Reward Markov Decision Processes

Cited by

• Average-Reward Reinforcement Learning with Trust Region Methods
• Hardware-Software Collaborative Computing of Photonic Spiking Reinforcement Learning for Robotic Continuous Control
• Towards Formalizing Reinforcement Learning Theory
• Implicit Updates for Average-Reward Temporal Difference Learning
• Breaking the Deadly Triad with a Target Network

Related

• Off-Policy Average Reward Actor-Critic with Deterministic Policy Search
• Model-free Policy Learning with Reward Gradients
• A Nonparametric Off-Policy Policy Gradient
• On-Policy Deep Reinforcement Learning for the Average-Reward Criterion
• Performance Bounds for Policy-Based Average Reward Reinforcement Learning Algorithms