GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms

2018/02/14 by Cédric Colas, Olivier Sigaud, Colas, Cédric +4 · 4 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Optimization and Search Problems #Reinforcement Learning in Robotics #Robotic Path Planning Algorithms #cs.LG

paper · pdf · doi:10.48550/arxiv.1802.05054

accepted at ICML 2018, 14 pages

openalex publication_date 2018/02/14 · arxiv created 2018/09/20 · arxiv updated 2018/09/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In continuous action domains, standard deep reinforcement learning algorithms like DDPG suffer from inefficient exploration when facing sparse or deceptive reward problems. Conversely, evolutionary and developmental methods focusing on exploration like Novelty Search, Quality-Diversity or Goal Exploration Processes explore more robustly but are less efficient at fine-tuning policies using gradient descent. In this paper, we present the GEP-PG approach, taking the best of both worlds by sequentially combining a Goal Exploration Process and two variants of DDPG. We study the learning performance of these components and their combination on a low dimensional deceptive reward problem and on the larger Half-Cheetah benchmark. We show that DDPG fails on the former and that GEP-PG improves over the best DDPG variant in both environments. Supplementary videos and discussion can be found at http://frama.link/gep_pg, the code at http://github.com/flowersteam/geppg.

Citations

• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Continuous control with deep reinforcement learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Sample Efficient Actor-Critic with Experience Replay
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• Parameter Space Noise for Exploration
• Noisy Networks for Exploration
• Hindsight Experience Replay
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic

Cited by

• Multi-Agent Guided Policy Optimization
• Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning

Related

• ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Control
• Exploring More When It Needs in Deep Reinforcement Learning
• Is Exploration or Optimization the Problem for Deep Reinforcement Learning?
• GA+DDPG+HER: Genetic Algorithm-Based Function Optimizer in Deep Reinforcement Learning for Robotic Manipulation Tasks
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents