Regularly Updated Deterministic Policy Gradient Algorithm

2020/07/01 by Shuai Han, Shuai D. Han, Han, Shuai +6 · 1 citation
Computer Science · Mathematics · #Adaptive Dynamic Programming Control #Algorithm #Artificial intelligence #Computer science #FOS: Computer and information sciences #Function (biology) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Machine learning #Mathematical optimization #Mathematics #Reinforcement Learning in Robotics #Reinforcement learning #Stability (learning theory) #Value (mathematics) #Variance (accounting) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2007.00169

published in arXiv (Cornell University) (Cornell University)

arxiv created 2020/07/01 · openalex publication_date 2020/07/01 · arxiv updated 2020/07/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Deep Deterministic Policy Gradient (DDPG) algorithm is one of the most well-known reinforcement learning methods. However, this method is inefficient and unstable in practical applications. On the other hand, the bias and variance of the Q estimation in the target function are sometimes difficult to control. This paper proposes a Regularly Updated Deterministic (RUD) policy gradient algorithm for these problems. This paper theoretically proves that the learning procedure with RUD can make better use of new data in replay buffer than the traditional procedure. In addition, the low variance of the Q value in RUD is more suitable for the current Clipped Double Q-learning strategy. This paper has designed a comparison experiment against previous methods, an ablation experiment with the original DDPG, and other analytical experiments in Mujoco environments. The experimental results demonstrate the effectiveness and superiority of RUD.

Citations

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• On the Theory of the Brownian Motion
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Count-Based Exploration with Neural Density Models
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Evolution-Guided Policy Gradient in Reinforcement Learning
• Recruitment-imitation Mechanism for Evolutionary Reinforcement Learning

Related

• Deterministic Value-Policy Gradients
• DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning
• Combining policy gradient and Q-learning
• Accelerating Policy Gradient by Estimating Value Function from Prior Computation in Deep Reinforcement Learning
• Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning