Stochastic Variance Reduction for Policy Gradient Estimation

2017/10/17 by Tian-Bing Xu, Qiang Liu, Xu, Tianbing +3 · 1 citation
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1710.06034

openalex publication_date 2017/10/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advances in policy gradient methods and deep learning have demonstrated their applicability for complex reinforcement learning problems. However, the variance of the performance gradient estimates obtained from the simulation is often excessive, leading to poor sample efficiency. In this paper, we apply the stochastic variance reduced gradient descent (SVRG) to model-free policy gradient to significantly improve the sample-efficiency. The SVRG estimation is incorporated into a trust-region Newton conjugate gradient framework for the policy optimization. On several Mujoco tasks, our method achieves significantly better performance compared to the state-of-the-art model-free policy gradient methods in robotic continuous control such as trust region policy optimization (TRPO)

Citations

• Benchmarking Deep Reinforcement Learning for Continuous Control
• Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations
• Numerical optimization

Cited by

• Stabilizing Policy Gradient Methods via Reward Profiling

Related

• SVRG for Policy Evaluation with Fewer Gradient Evaluations
• An Empirical Analysis of Measure-Valued Derivatives for Policy Gradients
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient
• An Analysis of Measure-Valued Derivatives for Policy Gradients