Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator

2019/10/02 by James A. Preiss, Sébastien M. R. Arnold, Preiss, James A. +5 · 1 citation
Computer Science · Decision Sciences · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Reinforcement Learning in Robotics #Simulation Techniques and Applications

paper · pdf · doi:10.48550/arxiv.1910.01249

openalex publication_date 2019/10/02 · openalex created_date 2019/10/10 · openalex updated_date 2026/07/28

Abstract

We study the variance of the REINFORCE policy gradient estimator in environments with continuous state and action spaces, linear dynamics, quadratic cost, and Gaussian noise. These simple environments allow us to derive bounds on the estimator variance in terms of the environment and noise parameters. We compare the predictions of our bounds to the empirical variance in simulation experiments.

Citations

• Trust Region Policy Optimization
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• A Tour of Reinforcement Learning: The View from Continuous Control

Cited by

• Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach

Related

• How are policy gradient methods affected by the limits of control?
• Learning robust control for LQR systems with multiplicative noise via policy gradient
• Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data
• An Empirical Analysis of Measure-Valued Derivatives for Policy Gradients
• An Analysis of Measure-Valued Derivatives for Policy Gradients