Model-Augmented Actor-Critic: Backpropagating through Paths

2020/05/16 by Ignasi Clavera, Clavera, Ignasi, Violet Fu +3 · 5 citations
Computer Science · Decision Sciences · Mathematics · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model-Driven Software Engineering Techniques #Reinforcement Learning in Robotics #Simulation Techniques and Applications #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2005.08068

Accepted paper at ICLR 2020

arxiv created 2020/05/16 · openalex publication_date 2020/05/16 · arxiv updated 2020/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function. We show that our approach (i) is consistently more sample efficient than existing state-of-the-art model-based algorithms, (ii) matches the asymptotic performance of model-free algorithms, and (iii) scales to long horizons, a regime where typically past model-based approaches have struggled.

Citations

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Model-Ensemble Trust-Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• MPC-Inspired Neural Network Policies for Sequential Decision Making
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Benchmarking Model-Based Reinforcement Learning

Cited by

• First Order Model-Based RL through Decoupled Backpropagation
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• TD-GRPC: Temporal Difference Learning with Group Relative Policy Constraint for Humanoid Locomotion
• Contrastive Active Inference
• Iterative Amortized Policy Optimization

Related

• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
• The Actor-Advisor: Policy Gradient With Off-Policy Advice
• On the Sample Complexity of Actor-Critic Method for Reinforcement Learning with Function Approximation
• Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation
• Modified Actor-Critics