Learning-based primal-dual optimal control of discrete-time stochastic systems with multiplicative noise

2025/06/03 by Xiushan Jiang, Jiang, Xiushan, Weihai Zhang +1
Engineering · #Advanced Control Systems Optimization #FOS: Mathematics #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.2506.02613

openalex publication_date 2025/06/03 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) is an effective approach for solving optimal control problems without knowing the exact information of the system model. However, the classical Q-learning method, a model-free RL algorithm, has its limitations, such as lack of strict theoretical analysis and the need for artificial disturbances during implementation. This paper explores the partially model-free stochastic linear quadratic regulator (SLQR) problem for a system with multiplicative noise from the primal-dual perspective to address these challenges. This approach lays a strong theoretical foundation for understanding the intrinsic mechanisms of classical RL algorithms. We reformulate the SLQR into a non-convex primal-dual optimization problem and derive a strong duality result, which enables us to provide model-based and model-free algorithms for SLQR optimal policy design based on the Karush-Kuhn-Tucker (KKT) conditions. An illustrative example demonstrates the proposed model-free algorithm's validity, showcasing the central nervous system's learning mechanism in human arm movement.

Citations

• Data-Driven Output Prediction and Control of Stochastic Systems: An Innovation-Based Approach
• Inverse linear-quadratic discrete-time finite-horizon optimal control for indistinguishable homogeneous agents: a convex optimization approach
• Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems
• Bounded Synthesis and Reinforcement Learning of Supervisors for Stochastic Discrete Event Systems with LTL Specifications
• Model-free optimal control of discrete-time systems with additive and multiplicative noises
• Finite-time Analysis of Approximate Policy Iteration for the Linear Quadratic Regulator
• The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint
• Model-Free Linear Quadratic Control via Reduction to Expert Prediction
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator

Related

• Model-Free Design of Stochastic LQR Controller from Reinforcement Learning and Primal-Dual Optimization Perspective
• Model-free stochastic linear quadratic control for discrete-time systems with multiplicative and additive noises via semidefinite programming
• Primal-dual policy learning for mean-field stochastic LQR problem
• Stochastic Primal-Dual Q-Learning
• Model-free optimal control of discrete-time systems with additive and multiplicative noises