Self-Imitation Advantage Learning

2020/12/22 by Johan Ferret, Olivier Pietquin, Ferret, Johan +3 · 1 citation
Computer Science · #Adaptive Dynamic Programming Control #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #cs.LG

paper · pdf · doi:10.48550/arxiv.2012.11989

AAMAS 2021

arxiv created 2020/12/22 · openalex publication_date 2020/12/22 · arxiv updated 2020/12/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Self-imitation learning is a Reinforcement Learning (RL) method that encourages actions whose returns were higher than expected, which helps in hard exploration and sparse reward problems. It was shown to improve the performance of on-policy actor-critic methods in several discrete control tasks. Nevertheless, applying self-imitation to the mostly action-value based off-policy RL methods is not straightforward. We propose SAIL, a novel generalization of self-imitation learning for off-policy RL, based on a modification of the Bellman optimality operator that we connect to Advantage Learning. Crucially, our method mitigates the problem of stale returns by choosing the most optimistic return estimate between the observed return and the current action-value for self-imitation. We demonstrate the empirical effectiveness of SAIL on the Arcade Learning Environment, with a focus on hard exploration games.

Citations

• Playing Atari with Deep Reinforcement Learning
• Dueling Network Architectures for Deep Reinforcement Learning
• Prioritized Experience Replay
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Hindsight Experience Replay
• Generative Adversarial Self-Imitation Learning
• Go-Explore: a New Approach for Hard-Exploration Problems
• Challenges of Real-World Reinforcement Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• Never Give Up: Learning Directed Exploration Strategies
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Cited by

• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning

Related

• Self-Imitation Learning
• Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
• Reinforced Imitation in Heterogeneous Action Space
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• Generative Adversarial Self-Imitation Learning