Combining policy gradient and Q-learning

2016/11/05 by Brendan O'Donoghue, Brendan O’Donoghue, Remi Munos +7 · 11 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Reinforcement Learning in Robotics #cs.AI #cs.LG #math.OC #stat.ML

paper · pdf · doi:10.48550/arxiv.1611.01626

openalex publication_date 2016/11/05 · arxiv created 2017/04/07 · arxiv updated 2017/04/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experience from a replay buffer. This is motivated by making a connection between the fixed points of the regularized policy gradient algorithm and the Q-values. This connection allows us to estimate the Q-values from the action preferences of the policy, to which we apply Q-learning updates. We refer to the new technique as 'PGQL', for policy gradient and Q-learning. We also establish an equivalency between action-value fitting techniques and actor-critic algorithms, showing that regularized policy gradient techniques can be interpreted as advantage function learning algorithms. We conclude with some numerical examples that demonstrate improved data efficiency and stability of PGQL. In particular, we tested PGQL on the full suite of Atari games and achieved performance exceeding that of both asynchronous advantage actor-critic (A3C) and Q-learning.

Citations

• End-to-End Training of Deep Visuomotor Policies
• Continuous control with deep reinforcement learning
• Prioritized Experience Replay
• Policy Gradient Methods for Off-policy Control
• Reward Augmented Maximum Likelihood for Neural Structured Prediction

Cited by

• Learning Boltzmann Generators via Constrained Mass Transport
• Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games
• Reasoning with Exploration: An Entropy Perspective
• Direct and indirect reinforcement learning
• Generalized Off-Policy Actor-Critic
• An Introduction to Deep Reinforcement Learning
• Regret Minimization for Partially Observable Deep Reinforcement Learning
• Energy-based Surprise Minimization for Multi-Agent Value Factorization
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning

Related

• PG-Rainbow: Using Distributional Reinforcement Learning in Policy Gradient Methods
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning
• Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approach
• Equivalence Between Policy Gradients and Soft Q-Learning