Maximum a Posteriori Policy Optimisation

2018/06/14 by Abdolmaleki, Abbas, Springenberg, Jost Tobias, Tassa, Yuval +3 · 28 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Robotics (cs.RO)

paper · doi:10.48550/arxiv.1806.06920

Abstract

We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are competitive with the state-of-the-art in deep reinforcement learning. In particular, for continuous control, our method outperforms existing methods with respect to sample efficiency, premature convergence and robustness to hyperparameter settings while achieving similar or better final performance.

Cited by

• Generative Actor Critic
• Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
• π^*_0.6: a VLA That Learns From Experience
• NFQ2.0: The CartPole Benchmark Revisited
• Model Predictive Control via Probabilistic Inference: A Tutorial
• On Flow Matching KL Divergence
• Off-policy Reinforcement Learning with Model-based Exploration Augmentation
• D2 Actor Critic: Diffusion Actor Meets Distributional Critic
• Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning
• Learning Boltzmann Generators via Constrained Mass Transport
• ADPO: Anchored Direct Preference Optimization
• Offline Reinforcement Learning with Generative Trajectory Policies
• MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inference
• Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions
• Incoherence in Goal-Conditioned Autoregressive Models
• A KL-regularization Framework for Learning to Plan with Adaptive Priors
• TROLL: Trust Regions improve Reinforcement Learning for Large Language Models
• Diffusion Alignment as Variational Expectation-Maximization
• Memory-Driven Self-Improvement for Decision Making with Large Language Models
• Discrete Variational Autoencoding via Policy Search
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
• Failure Modes of Maximum Entropy RLHF
• SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
• Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedback
• The Geometry of Nonlinear Reinforcement Learning
• Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learning
• LyLA-Therm: Lyapunov-based Langevin Adaptive Thermodynamic Neural Network Controller
• Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inference

Related

• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Relative Entropy Regularized Policy Iteration
• Robust Policy Optimization in Deep Reinforcement Learning
• Absolute Policy Optimization
• Near Optimal Policy Optimization via REPS