GRAC: Self-Guided and Self-Regularized Actor-Critic

2020/09/18 by Lin Shao, Yifan You, Shao, Lin +7 · 5 citations
Computer Science · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Games #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Robotic Path Planning Algorithms #Robotics (cs.RO) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2009.08973

openalex publication_date 2020/09/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep reinforcement learning (DRL) algorithms have successfully been demonstrated on a range of challenging decision making and control tasks. One dominant component of recent deep reinforcement learning algorithms is the target network which mitigates the divergence when learning the Q function. However, target networks can slow down the learning process due to delayed function updates. Our main contribution in this work is a self-regularized TD-learning method to address divergence without requiring a target network. Additionally, we propose a self-guided policy improvement method by combining policy-gradient with zero-order optimization to search for actions associated with higher Q-values in a broad neighborhood. This makes learning more robust to local noise in the Q function approximation and guides the updates of our actor network. Taken together, these components define GRAC, a novel self-guided and self-regularized actor critic algorithm. We evaluate GRAC on the suite of OpenAI gym tasks, achieving or outperforming state of the art in every environment tested.

Citations

• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Deep Reinforcement Learning and the Deadly Triad
• Towards Characterizing Divergence in Deep Q-Learning
• Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

Cited by

• Multi-Agent Cross-Entropy Method with Monotonic Nonlinear Critic Decomposition
• Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning
• Actor-Free Continuous Control via Structurally Maximizable Q-Functions
• Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning
• Towards Automatic Actor-Critic Solutions to Continuous Control

Related

• Relative Entropy Regularized Policy Iteration
• XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
• The Role of Deep Learning Regularizations on Actors in Offline RL
• OPAC: Opportunistic Actor-Critic
• Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm