Sample-Efficient Model-Free Policy Gradient Methods for Stochastic LQR via Robust Linear Regression

2025/12/03 by Song, Bowen, Gros, Sebastien, Iannelli, Andrea
#FOS: Electrical engineering #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2512.03764

Abstract

Policy gradient algorithms are widely used in reinforcement learning and belong to the class of approximate dynamic programming methods. This paper studies two key policy gradient algorithms - the Natural Policy Gradient and the Gauss-Newton Method - for solving the Linear Quadratic Regulator (LQR) problem in unknown stochastic linear systems. The main challenge lies in obtaining an unbiased gradient estimate from noisy data due to errors-in-variables in linear regression. This issue is addressed by employing a primal-dual estimation procedure. Using this novel gradient estimation scheme, the paper establishes convergence guarantees with a sample complexity of order O(1/epsilon). Theoretical results are further supported by numerical experiments, which demonstrate the effectiveness of the proposed algorithms.

Citations

• Policy Gradient Adaptive Control for the LQR: Indirect and Direct Approaches
• Robustness of Online Identification-based Policy Iteration to Noisy Data
• Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data
• Sample Complexity of the Linear Quadratic Regulator: A Reinforcement Learning Lens
• Stochastic Online Optimization for Cyber-Physical and Robotic Systems
• The Role of Identification in Data-driven Policy Iteration: A System Theoretic Study
• Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Control
• Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guarantees
• Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon
• Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator

Related

• Convergence Guarantees of Model-free Policy Gradient Methods for LQR with Stochastic Data
• Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon
• Oracle Complexity Reduction for Model-free LQR: A Stochastic Variance-Reduced Policy Gradient Approach
• Model-Free Design of Stochastic LQR Controller from Reinforcement Learning and Primal-Dual Optimization Perspective
• A model-free first-order method for linear quadratic regulator with O(1/ε) sampling complexity