Efficient Wasserstein Natural Gradients for Reinforcement Learning

2020/10/12 by Ted Moskovitz, Michael Arbel, Moskovitz, Ted +5 · 1 citation
Computer Science · Engineering · #FOS: Computer and information sciences #Lattice Boltzmann Simulation Studies #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2010.05380

openalex publication_date 2020/10/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A novel optimization approach is proposed for application to policy gradient methods and evolution strategies for reinforcement learning (RL). The procedure uses a computationally efficient Wasserstein natural gradient (WNG) descent that takes advantage of the geometry induced by a Wasserstein penalty to speed optimization. This method follows the recent theme in RL of including a divergence penalty in the objective to establish a trust region. Experiments on challenging tasks demonstrate improvements in both computational cost and performance over advanced baselines.

Citations

• On Information and Sufficiency
• Stochastic Optimization for Large-scale Optimal Transport
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Implicit Policy for Reinforcement Learning
• Ricci curvature for parametric statistics via optimal transport
• First-Order Preconditioning via Hypergradient Descent
• Optimal transport natural gradient for statistical manifolds with continuous sample space

Cited by

• Constrained Sliced Wasserstein Embedding

Related

• Policy Optimization as Wasserstein Gradient Flows
• Wasserstein Policy Optimization
• Wasserstein Robust Reinforcement Learning
• Learning to Score Behaviors for Guided Policy Optimization
• Natural Policy Gradients In Reinforcement Learning Explained