Periodic Intra-Ensemble Knowledge Distillation for Reinforcement Learning

2020/02/01 by Zhang-Wei Hong, Hong, Zhang-Wei, Prabhat Nagarajan +3 · 1 citation
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Autonomous Vehicle Technology and Safety #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mobile Crowdsensing and Crowdsourcing #Reinforcement Learning in Robotics #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2002.00149

8 pages

arxiv created 2020/02/01 · openalex publication_date 2020/02/01 · arxiv updated 2020/02/04 · openalex created_date 2020/02/07 · openalex updated_date 2026/07/28

Abstract

Off-policy ensemble reinforcement learning (RL) methods have demonstrated impressive results across a range of RL benchmark tasks. Recent works suggest that directly imitating experts' policies in a supervised manner before or during the course of training enables faster policy improvement for an RL agent. Motivated by these recent insights, we propose Periodic Intra-Ensemble Knowledge Distillation (PIEKD). PIEKD is a learning framework that uses an ensemble of policies to act in the environment while periodically sharing knowledge amongst policies in the ensemble through knowledge distillation. Our experiments demonstrate that PIEKD improves upon a state-of-the-art RL method in sample efficiency on several challenging MuJoCo benchmark tasks. Additionally, we perform ablation studies to better understand PIEKD.

Citations

• Distilling the Knowledge in a Neural Network
• Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Deep Exploration via Randomized Value Functions
• Divide-and-Conquer Reinforcement Learning
• Evolution-Guided Policy Gradient in Reinforcement Learning
• Soft Actor-Critic Algorithms and Applications
• Deep Q-learning from Demonstrations
• Population-Guided Parallel Policy Search for Reinforcement Learning

Cited by

• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions

Related

• Distilling Policy Distillation
• Distilling Reinforcement Learning into Single-Batch Datasets
• How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning
• In-context Reinforcement Learning with Algorithm Distillation
• Online Policy Distillation with Decision-Attention