Stein Variational Policy Gradient

2017/04/07 by Yang Liu, Prajit Ramachandran, Liu, Yang +5 · 14 citations
Computer Science · Decision Sciences · Social Sciences · #Reinforcement Learning in Robotics #Advanced Bandit Algorithms Research #Insurance, Mortality, Demography, Risk Management

paper · pdf · doi:10.48550/arxiv.1704.02399

Abstract

Policy gradient methods have been successfully applied to many complex reinforcement learning problems. However, policy gradient methods suffer from high variance, slow convergence, and inefficient exploration. In this work, we introduce a maximum entropy policy optimization framework which explicitly encourages parameter exploration, and show that this framework can be reduced to a Bayesian inference problem. We then propose a novel Stein variational policy gradient method (SVPG) which combines existing policy gradient methods and a repulsive functional to generate a set of diverse but well-behaved policies. SVPG is robust to initialization and can easily be implemented in a parallel manner. On continuous control problems, we find that implementing SVPG on top of REINFORCE and advantage actor-critic algorithms improves both average return and data efficiency.

Citations

• Deep Exploration via Bootstrapped DQN
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic

Cited by

• Soft Quality-Diversity Optimization
• Improved Contrastive Divergence Training of Energy Based Models
• Learning Structural Weight Uncertainty for Sequential Decision-Making
• Active Domain Randomization
• Bayesian Model-Agnostic Meta-Learning
• Stein Variational Gradient Descent Without Gradient
• Adaptive Kernel Selection for Stein Variational Gradient Descent
• Distributionally Adversarial Attack
• Curriculum in Gradient-Based Meta-Reinforcement Learning
• A Non-Asymptotic Analysis for Stein Variational Gradient Descent
• Kernel Variational Inference Flow for Nonlinear Filtering Problem
• A Unified Particle-Optimization Framework for Scalable Bayesian Sampling
• Learning Self-Imitating Diverse Policies
• Online reinforcement learning with sparse rewards through an active inference capsule

Related

• Action-depedent Control Variates for Policy Optimization via Stein's Identity
• Variational Inference for Policy Gradient
• Variational Policy Gradient Method for Reinforcement Learning with General Utilities
• Policy Optimization Reinforcement Learning with Entropy Regularization
• S²AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic