Temporally-Extended ε-Greedy Exploration

2020/06/02 by Will Dabney, Georg Ostrovski, Dabney, Will +4 · 4 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Mathematics · #Diffusion and Search Dynamics #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Metaheuristic Optimization Algorithms Research #Reinforcement Learning in Robotics #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.01782

arxiv created 2020/06/02 · openalex publication_date 2020/06/02 · arxiv updated 2020/06/03 · openalex created_date 2020/06/19 · openalex updated_date 2026/07/28

Abstract

Recent work on exploration in reinforcement learning (RL) has led to a series of increasingly complex solutions to the problem. This increase in complexity often comes at the expense of generality. Recent empirical studies suggest that, when applied to a broader set of domains, some sophisticated exploration methods are outperformed by simpler counterparts, such as ε-greedy. In this paper we propose an exploration algorithm that retains the simplicity of ε-greedy while reducing dithering. We build on a simple hypothesis: the main limitation of ε-greedy exploration is its lack of temporal persistence, which limits its ability to escape local optima. We propose a temporally extended form of ε-greedy that simply repeats the sampled action for a random duration. It turns out that, for many duration distributions, this suffices to improve exploration on a large set of domains. Interestingly, a class of distributions inspired by ecological models of animal foraging behaviour yields particularly strong performance.

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• (More) Efficient Reinforcement Learning via Posterior Sampling
• Dueling Network Architectures for Deep Reinforcement Learning
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Noisy Networks for Exploration
• DeepMind Control Suite
• Exploration by Random Network Distillation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Never Give Up: Learning Directed Exploration Strategies

Cited by

• RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Design
• Leveraging Temporally Extended Behavior Sharing for Multi-task Reinforcement Learning
• Delphos: A reinforcement learning framework for assisting discrete choice model specification
• TempoRL: Learning When to Act

Related

• Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization
• Exploration by Random Reward Perturbation
• RBED: Reward Based Epsilon Decay
• ε-BMC: A Bayesian Ensemble Approach to Epsilon-Greedy Exploration in Model-Free Reinforcement Learning
• Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximation