REPAINT: Knowledge Transfer in Deep Reinforcement Learning

2020/11/24 by Yunzhe Tao, Tao, Yunzhe, Şahika Genç +8 · 3 citations
Computer Science · Decision Sciences · Psychology · #Adaptive Dynamic Programming Control #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Artificial intelligence #Benchmark (surveying) #Computer science #FOS: Computer and information sciences #Knowledge management #Knowledge transfer #Machine Learning (cs.LG) #Machine learning #Policy learning #Psychology #Reinforcement #Reinforcement Learning in Robotics #Reinforcement learning #Representation (politics) #Robotics (cs.RO) #Selection (genetic algorithm) #Similarity (geometry) #Task (project management) #Transfer of learning #cs.AI #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2011.11827

published in arXiv (Cornell University) (Cornell University) · Published at ICML 2021

openalex publication_date 2020/11/24 · arxiv created 2021/05/26 · arxiv updated 2021/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Accelerating learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target tasks is low. This work proposes REPresentation And INstance Transfer (REPAINT) algorithm for knowledge transfer in deep reinforcement learning. REPAINT not only transfers the representation of a pre-trained teacher policy in the on-policy learning, but also uses an advantage-based experience selection approach to transfer useful samples collected following the teacher policy in the off-policy learning. Our experimental results on several benchmark tasks show that REPAINT significantly reduces the total training time in generic cases of task similarity. In particular, when the source tasks are dissimilar to, or sub-tasks of, the target tasks, REPAINT outperforms other baselines in both training-time reduction and asymptotic performance of return scores.

Citations

• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation
• Trust Region Policy Optimization
• Reinforcement learning in robotics: A survey
• A Survey on Transfer Learning
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Prioritized Experience Replay
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• StarCraft II: A New Challenge for Reinforcement Learning
• Kickstarting Deep Reinforcement Learning
• Universal Successor Representations for Transfer Reinforcement Learning
• Decoupling Dynamics and Reward for Transfer Learning
• Universal Successor Features Approximators
• Generalized Off-Policy Actor-Critic
• Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
• Finite Sample Analysis of Two-Timescale Stochastic Approximation with Applications to Reinforcement Learning
• Transfer in Deep Reinforcement Learning Using Successor Features and\n Generalised Policy Improvement
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning
• Learning dexterous in-hand manipulation
• Dota 2 with Large Scale Deep Reinforcement Learning
• Critic Regularized Regression

Related

• Transfer Learning in Deep Reinforcement Learning: A Survey
• Transfer in Deep Reinforcement Learning Using Successor Features and\n Generalised Policy Improvement
• Provable Benefits of Representational Transfer in Reinforcement Learning
• Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning
• An advantage based policy transfer algorithm for reinforcement learning with measures of transferability