Reinforcement Learning as Iterative and Amortised Inference

2020/06/13 by Beren Millidge, Millidge, Beren, Alexander Tschantz +5
Computer Science · #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2006.10524

openalex publication_date 2020/06/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

There are several ways to categorise reinforcement learning (RL) algorithms, such as either model-based or model-free, policy-based or planning-based, on-policy or off-policy, and online or offline. Broad classification schemes such as these help provide a unified perspective on disparate techniques and can contextualise and guide the development of new algorithms. In this paper, we utilise the control as inference framework to outline a novel classification scheme based on amortised and iterative inference. We demonstrate that a wide range of algorithms can be classified in this manner providing a fresh perspective and highlighting a range of existing similarities. Moreover, we show that taking this perspective allows us to identify parts of the algorithmic design space which have been relatively unexplored, suggesting new routes to innovative RL algorithms.

Citations

• Stochastic Backpropagation and Approximate Inference in Deep Generative\n Models
• Black Box Variational Inference
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Universal Planning Networks
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Soft Actor-Critic Algorithms and Applications
• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Related

• Making Sense of Reinforcement Learning and Probabilistic Inference
• Control as Hybrid Inference
• A Crash Course on Reinforcement Learning
• Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization
• Statistical and Algorithmic Foundations of Reinforcement Learning