Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning

2021/04/09 by Wenzhen Huang, Qiyue Yin, Huang, Wenzhen +5
Biochemistry, Genetics and Molecular Biology · Computer Science · #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Viral Infectious Diseases and Gene Expression in Insects #cs.LG

paper · pdf · doi:10.48550/arxiv.2104.04174

arxiv created 2021/04/09 · openalex publication_date 2021/04/09 · arxiv updated 2021/04/12 · openalex created_date 2021/04/26 · openalex updated_date 2026/07/28

Abstract

Model-based reinforcement learning (RL) is more sample efficient than model-free RL by using imaginary trajectories generated by the learned dynamics model. When the model is inaccurate or biased, imaginary trajectories may be deleterious for training the action-value and policy functions. To alleviate such problem, this paper proposes to adaptively reweight the imaginary transitions, so as to reduce the negative effects of poorly generated trajectories. More specifically, we evaluate the effect of an imaginary transition by calculating the change of the loss computed on the real samples when we use the transition to train the action-value and policy functions. Based on this evaluation criterion, we construct the idea of reweighting each imaginary transition by a well-designed meta-gradient algorithm. Extensive experimental results demonstrate that our method outperforms state-of-the-art model-based and model-free RL algorithms on multiple tasks. Visualization of our changing weights further validates the necessity of utilizing reweight scheme.

Citations

• Learning Continuous Control Policies by Stochastic Value Gradients
• Model-Ensemble Trust-Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Meta-Gradient Reinforcement Learning
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Exploring Model-based Planning with Policy Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Benchmarking Model-Based Reinforcement Learning
• Discovery of Useful Questions as Auxiliary Tasks

Related

• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Model Imitation for Model-Based Reinforcement Learning
• On-Policy Model Errors in Reinforcement Learning
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning