Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning

2021/08/19 by Andrea Zanette, Martin J. Wainwright, Zanette, Andrea +3 · 6 citations
Computer Science · #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2108.08812

openalex publication_date 2021/08/19 · openalex created_date 2021/08/30 · openalex updated_date 2026/07/28

Abstract

Actor-critic methods are widely used in offline reinforcement learning practice, but are not so well-understood theoretically. We propose a new offline actor-critic algorithm that naturally incorporates the pessimism principle, leading to several key advantages compared to the state of the art. The algorithm can operate when the Bellman evaluation operator is closed with respect to the action value function of the actor's policies; this is a more general setting than the low-rank MDP model. Despite the added generality, the procedure is computationally tractable as it involves the solution of a sequence of second-order programs. We prove an upper bound on the suboptimality gap of the policy returned by the procedure that depends on the data coverage of any arbitrary, possibly data dependent comparator policy. The achievable guarantee is complemented with a minimax lower bound that is matching up to logarithmic factors.

Citations

• Learning Continuous Control Policies by Stochastic Value Gradients
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Behavior Regularized Offline Reinforcement Learning
• Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MOReL : Model-Based Offline Reinforcement Learning
• Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound
• Variational Policy Gradient Method for Reinforcement Learning with General Utilities
• Batch Policy Learning in Average Reward Markov Decision Processes
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• Batch Value-function Approximation with Only Realizability
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• Is Pessimism Provably Efficient for Offline RL?
• Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms
• Model-free Representation Learning and Exploration in Low-rank MDPs
• Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning

Cited by

• Convergence of actor-critic for entropy regularised MDPs in general action spaces
• On the optimization dynamics of RLVR: Gradient gap and step size thresholds
• Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Representation Learning for Online and Offline RL in Low-rank MDPs
• Multi-Agent Reinforcement Learning in Intelligent Transportation Systems: A Comprehensive Survey
• A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory

Related

• Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
• Off-Policy Actor-Critic
• Adversarially Trained Actor Critic for Offline Reinforcement Learning
• Actor-Critics Can Achieve Optimal Sample Efficiency
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality