Bayesian Policy Gradients via Alpha Divergence Dropout Inference

2017/12/06 by Peter Henderson, Thang Doan, Henderson, Peter +5 · 1 citation
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #cs.LG

paper · pdf · doi:10.48550/arxiv.1712.02037

Accepted to Bayesian Deep Learning Workshop at NIPS 2017

arxiv created 2017/12/06 · openalex publication_date 2017/12/06 · arxiv updated 2017/12/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Policy gradient methods have had great success in solving continuous control tasks, yet the stochastic nature of such problems makes deterministic value estimation difficult. We propose an approach which instead estimates a distribution by fitting the value function with a Bayesian Neural Network. We optimize an α-divergence objective with Bayesian dropout approximation to learn and estimate this distribution. We show that using the Monte Carlo posterior mean of the Bayesian value function distribution, rather than a deterministic network, improves stability and performance of policy gradient methods in continuous control MuJoCo simulations.

Citations

• Continuous control with deep reinforcement learning
• Deep Exploration via Bootstrapped DQN
• Parameter Space Noise for Exploration
• A Distributional Perspective on Reinforcement Learning
• Weight Uncertainty in Neural Networks

Cited by

• Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning

Related

• Learning Continuous Control Policies by Stochastic Value Gradients
• Deterministic Value-Policy Gradients
• Trajectory-Based Off-Policy Deep Reinforcement Learning
• Variational Inference for Policy Gradient
• Value-Gradient based Formulation of Optimal Control Problem and Machine Learning Algorithm