DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning

2020/06/26 by Rasool Fakoor, Pratik Chaudhari, Fakoor, Rasool +3 · 3 citations
Computer Science · Engineering · Mathematics · Psychology · #Adaptive Dynamic Programming Control #Artificial intelligence #Computer science #Control (management) #Epistemology #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Philosophy #Psychology #Reinforcement #Reinforcement Learning in Robotics #Reinforcement learning #Simplicity #Social psychology #Traffic control and management #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.15199

published in ArXiv.org

arxiv created 2020/06/26 · openalex publication_date 2020/06/26 · arxiv updated 2020/06/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper prescribes a suite of techniques for off-policy Reinforcement Learning (RL) that simplify the training process and reduce the sample complexity. First, we show that simple Deterministic Policy Gradient works remarkably well as long as the overestimation bias is controlled. This is contrast to existing literature which creates sophisticated off-policy techniques. Second, we pinpoint training instabilities, typical of off-policy algorithms, to the greedy policy update step; existing solutions such as delayed policy updates do not mitigate this issue. Third, we show that ideas in the propensity estimation literature can be used to importance-sample transitions from the replay buffer and selectively update the policy to prevent deterioration of performance. We make these claims using extensive experimentation on a set of challenging MuJoCo tasks. A short video of our results can be seen at https://tinyurl.com/scs6p5m .

Citations

• Playing Atari with Deep Reinforcement Learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Off-Policy Deep Reinforcement Learning without Exploration

Cited by

• StorageXTuner: An LLM Agent-Driven Automatic Tuning Framework for Heterogeneous Storage Systems

Related

• Off-Policy Policy Gradient Algorithms by Constraining the State Distribution Shift
• On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling
• Proximal Deterministic Policy Gradient
• Adaptive Trade-Offs in Off-Policy Learning
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction