Policy Search by Target Distribution Learning for Continuous Control

2019/05/27 by Chuheng Zhang, Zhang, Chuheng, Yuanqi Li +3
Computer Science · Engineering · Mathematics · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Fuel Cells and Related Materials #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1905.11041

AAAI-20 (oral)

openalex publication_date 2019/05/27 · arxiv created 2019/11/18 · arxiv updated 2019/11/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We observe that several existing policy gradient methods (such as vanilla policy gradient, PPO, A2C) may suffer from overly large gradients when the current policy is close to deterministic (even in some very simple environments), leading to an unstable training process. To address this issue, we propose a new method, called target distribution learning (TDL), for policy improvement in reinforcement learning. TDL alternates between proposing a target distribution and training the policy network to approach the target distribution. TDL is more effective in constraining the KL divergence between updated policies, and hence leads to more stable policy improvements over iterations. Our experiments show that TDL algorithms perform comparably to (or better than) state-of-the-art algorithms for most continuous control tasks in the MuJoCo environment while being more stable in training.

Citations

• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation
• Trust Region Policy Optimization
• On Information and Sufficiency
• Continuous control with deep reinforcement learning
• Prioritized Experience Replay
• Sample Efficient Actor-Critic with Experience Replay
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Maximum a Posteriori Policy Optimisation
• Simple random search provides a competitive approach to reinforcement learning
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Remember and Forget for Experience Replay
• Relative Entropy Regularized Policy Iteration
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift

Related

• The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
• Learning Optimal Deterministic Policies with Stochastic Policy Gradients
• DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning
• Distributional Policy Optimization: An Alternative Approach for Continuous Control
• Local Search for Policy Iteration in Continuous Control