Self-Imitation Learning via Generalized Lower Bound Q-learning

2020/06/12 by Yunhao Tang, Tang, Yunhao · 2 citations
Computer Science · Mathematics · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.07442

Accepted at NeurIPS (Neural Information Processing Systems) 2020, Vancouver, Canada. Code is available at https://github.com/robintyh1/nstep-sil

openalex publication_date 2020/06/12 · arxiv created 2021/02/14 · arxiv updated 2021/02/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Self-imitation learning motivated by lower-bound Q-learning is a novel and effective approach for off-policy learning. In this work, we propose a n-step lower bound which generalizes the original return-based lower-bound Q-learning, and introduce a new family of self-imitation learning algorithms. To provide a formal motivation for the potential performance gains provided by self-imitation learning, we show that n-step lower bound Q-learning achieves a trade-off between fixed point bias and contraction rate, drawing close connections to the popular uncorrected n-step Q-learning. We finally show that n-step lower bound Q-learning is a more robust alternative to return-based self-imitation learning and uncorrected n-step, over a wide range of continuous control benchmark tasks.

Citations

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• A Markovian Decision Process
• Continuous control with deep reinforcement learning
• Prioritized Experience Replay
• Model-Free Episodic Control
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• DeepMind Control Suite
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Addressing Function Approximation Error in Actor-Critic Methods
• Distributed Distributional Deterministic Policy Gradients
• Learning Self-Imitating Diverse Policies
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning

Cited by

• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
• GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning

Related

• Augmented Q Imitation Learning (AQIL)
• Quantile Filtered Imitation Learning
• IQ-Learn: Inverse soft-Q Learning for Imitation
• Inverse Q-Learning Done Right: Offline Imitation Learning in Q^π-Realizable MDPs
• Scalable Causal Imitation Learning