Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation

2021/07/05 by Yao Yao, Xiao Li, Li Xiao +8 · 3 citations
Computer Science · Mathematics · #Action (physics) #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Artificial intelligence #Benchmark (surveying) #Computer science #Ensemble forecasting #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Machine learning #Mathematics #Novelty #Novelty detection #Reinforcement Learning in Robotics #Reinforcement learning #Sample (material) #Set (abstract data type) #Tuple #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2107.01825

published in arXiv (Cornell University) (Cornell University) · 7 pages, 5 figures, accepted by IEEE International Conference on Robotics and Automation 2021 (IEEE ICRA 2021)

arxiv created 2021/07/05 · openalex publication_date 2021/07/05 · arxiv updated 2021/07/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Model-based deep reinforcement learning has achieved success in various domains that require high sample efficiencies, such as Go and robotics. However, there are some remaining issues, such as planning efficient explorations to learn more accurate dynamic models, evaluating the uncertainty of the learned models, and more rational utilization of models. To mitigate these issues, we present MEEE, a model-ensemble method that consists of optimistic exploration and weighted exploitation. During exploration, unlike prior methods directly selecting the optimal action that maximizes the expected accumulative return, our agent first generates a set of action candidates and then seeks out the optimal action that takes both expected return and future observation novelty into account. During exploitation, different discounted weights are assigned to imagined transition tuples according to their model uncertainty respectively, which will prevent model predictive error propagation in agent training. Experiments on several challenging continuous control benchmark tasks demonstrated that our approach outperforms other model-free and model-based state-of-the-art methods, especially in sample complexity.

Citations

• Variational Dropout and the Local Reparameterization Trick
• Learning to Perform Physics Experiments via Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Model-Ensemble Trust-Region Policy Optimization
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• Combating the Compounding-Error Problem with a Multi-step Model
• RLCard: A Toolkit for Reinforcement Learning in Card Games
• Learning dexterous in-hand manipulation
• Planning to Explore via Self-Supervised World Models
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning

Cited by

• Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Method
• Avoiding Death through Fear Intrinsic Conditioning
• Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control

Related

• Learning Off-policy with Model-based Intrinsic Motivation For Active Online Exploration
• Off-policy Reinforcement Learning with Model-based Exploration Augmentation
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• High-Accuracy Model-Based Reinforcement Learning, a Survey
• Model-Based Active Exploration