Stochastic Recursive Momentum for Policy Gradient Methods

2020/03/09 by Huizhuo Yuan, Yuan, Huizhuo, Xiangru Lian +5 · 3 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Search Problems #Reinforcement Learning in Robotics #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2003.04302

openalex publication_date 2020/03/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we propose a novel algorithm named STOchastic Recursive Momentum for Policy Gradient (STORM-PG), which operates a SARAH-type stochastic recursive variance-reduced policy gradient in an exponential moving average fashion. STORM-PG enjoys a provably sharp O(1/ε³) sample complexity bound for STORM-PG, matching the best-known convergence rate for policy gradient algorithm. In the mean time, STORM-PG avoids the alternations between large batches and small batches which persists in comparable variance-reduced policy gradient methods, allowing considerably simpler parameter tuning. Numerical experiments depicts the superiority of our algorithm over comparative policy gradient algorithms.

Citations

• Playing Atari with Deep Reinforcement Learning
• Continuous control with deep reinforcement learning
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Deep Reinforcement Learning that Matters
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction

Cited by

• Reusing Trajectories in Policy Gradients Enables Fast Convergence
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction
• Stochastic Recursive Momentum Method for Non-Convex Compositional Optimization

Related

• Momentum-Based Policy Gradient with Second-Order Information
• Stability and Generalization for Stochastic Recursive Momentum-based Algorithms for (Strongly-)Convex One to K-Level Stochastic Optimizations
• On the Global Optimum Convergence of Momentum-based Policy Gradient
• Momentum-Based Policy Gradient Methods
• Fast Stochastic Policy Gradient: Negative Momentum for Reinforcement Learning