Iterative Amortized Policy Optimization

2020/10/20 by Joseph Marino, Alexandre Piché, Marino, Joseph +5
Computer Science · Mathematics · Physics and Astronomy · #Adversarial Robustness in Machine Learning #Amortized analysis #Artificial intelligence #Benchmark (surveying) #Computer science #Data structure #FOS: Computer and information sciences #Inference #Machine Learning (cs.LG) #Mathematical optimization #Mathematics #Model Reduction and Neural Networks #Reinforcement Learning in Robotics #cs.LG

paper · pdf · doi:10.48550/arxiv.2010.10670

Advances in Neural Processing Systems (NeurIPS) 2021

openalex publication_date 2020/10/20 · arxiv created 2021/10/22 · arxiv updated 2021/10/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Policy networks are a central feature of deep reinforcement learning (RL) algorithms for continuous control, enabling the estimation and sampling of high-value actions. From the variational inference perspective on RL, policy networks, when used with entropy or KL regularization, are a form of amortized optimization, optimizing network parameters rather than the policy distributions directly. However, direct amortized mappings can yield suboptimal policy estimates and restricted distributions, limiting performance and exploration. Given this perspective, we consider the more flexible class of iterative amortized optimizers. We demonstrate that the resulting technique, iterative amortized policy optimization, yields performance improvements over direct amortization on benchmark continuous control tasks.

Citations

• Training Very Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Matplotlib: A 2D Graphics Environment
• Long Short-Term Memory
• A New Approach to Linear Filtering and Prediction Problems
• Self-improving reactive agents based on reinforcement learning, planning and teaching
• Mastering the game of Go with deep neural networks and tree search
• Adaptive Computation Time for Recurrent Neural Networks
• Reinforcement Learning with Deep Energy-Based Policies
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Safe end-to-end imitation learning for model predictive control
• Model-Based Planning with Discrete and Continuous Actions
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• A General Method for Amortizing Variational Filtering
• Boosting Trust Region Policy Optimization by Normalizing Flows Policy
• Learning Latent Dynamics for Planning from Pixels
• Soft Actor-Critic Algorithms and Applications
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Model-Augmented Actor-Critic: Backpropagating through Paths
• Array programming with NumPy

Related

• Regularization Matters in Policy Optimization
• Robust Policy Optimization in Deep Reinforcement Learning
• Can We Optimize Deep RL Policy Weights as Trajectory Modeling?
• Modified Actor-Critics
• Relative Entropy Regularized Policy Iteration