Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs

2021/04/11 by Feiran Zhao, Zhao, Feiran, Keyou You +3 · 3 citations
Computer Science · Engineering · #Adaptive Dynamic Programming Control #Advanced Control Systems Optimization #FOS: Electrical engineering #FOS: Mathematics #Optimization and Control (math.OC) #Reinforcement Learning in Robotics #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2104.04901

openalex publication_date 2021/04/11 · openalex created_date 2021/04/26 · openalex updated_date 2026/07/28

Abstract

While the techniques in optimal control theory are often model-based, the policy optimization (PO) approach directly optimizes the performance metric of interest. Even though it has been an essential approach for reinforcement learning problems, there is little theoretical understanding on its performance. In this paper, we focus on the risk-constrained linear quadratic regulator (RC-LQR) problem via the PO approach, which requires addressing a challenging non-convex constrained optimization problem. To solve it, we first build on our earlier result that an optimal policy has a time-invariant affine structure to show that the associated Lagrangian function is coercive, locally gradient dominated and has local Lipschitz continuous gradient, based on which we establish strong duality. Then, we design policy gradient primal-dual methods with global convergence guarantees in both model-based and sample-based settings. Finally, we use samples of system trajectories in simulations to validate our methods.

Citations

• Constrained Markov Decision Processes
• Constrained Policy Optimization
• Reward Constrained Policy Optimization
• Risk Averse Robust Adversarial Reinforcement Learning
• LQR through the Lens of First Order Methods: Discrete-time Case
• Convergent Policy Optimization for Safe Reinforcement Learning
• Primal-dual Learning for the Model-free Risk-constrained Linear Quadratic Regulator
• Infinite-horizon Risk-constrained Linear Quadratic Regulator with Average Cost

Cited by

• Actor-Critic Learning for Risk-Constrained Linear Quadratic Regulation
• Policy Optimization in the Linear Quadratic Gaussian Problem: A Frequency Domain Perspective
• On the Robustness of Derivative-free Methods for Linear Quadratic Regulator

Related

• Policy Gradient Methods for the Cost-Constrained LQR: Strong Duality and Global Convergence
• Primal-dual Learning for the Model-free Risk-constrained Linear Quadratic Regulator
• Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Learning robust control for LQR systems with multiplicative noise via policy gradient