Mirror descent for constrained stochastic control problems

2025/06/03 by Deven Sethi, David Šiška, Sethi, Deven +1
Computer Science · Engineering · #Advanced Control Systems Optimization #Aerospace Engineering and Control Systems #FOS: Mathematics #Optimization and Control (math.OC) #Optimization and Search Problems

paper · pdf · doi:10.48550/arxiv.2506.02564

openalex publication_date 2025/06/03 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Mirror descent is a well established tool for solving convex optimization problems with convex constraints. This article introduces continuous-time mirror descent dynamics for approximating optimal Markov controls for stochastic control problems with the action space being bounded and convex. We show that if the Hamiltonian is uniformly convex in its action variable then mirror descent converges linearly while if it is uniformly strongly convex relative to an appropriate Bregman divergence, then the mirror flow converges exponentially. The two fundamental difficulties that must be overcome to prove such results are: first, the inherent lack of convexity of the map from Markov controls to the corresponding value function. Second, maintaining sufficient regularity of the value function and the Markov controls along the mirror descent updates. The first issue is handled using the performance difference lemma, while the second using careful Sobolev space estimates for the solutions of the associated linear PDEs.

Citations

• Entropy annealing for policy mirror descent in continuous time and space
• Mirror Descent for Stochastic Control Problems with Measure-valued Controls
• A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces
• Policy Optimization over General State and Action Spaces
• Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems
• Convergence of Policy Iteration for Entropy-Regularized Stochastic Control Problems
• q-Learning in Continuous Time
• Linear convergence of a policy gradient method for some finite horizon continuous time control problems
• Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regime
• Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms
• Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation
• Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
• Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classes
• On the Global Convergence Rates of Softmax Policy Gradient Methods
• Understanding the impact of entropy on policy optimization
• Deep Learning Approximation for Stochastic Control Problems

Related

• Mirror Descent for Stochastic Control Problems with Measure-valued Controls
• Adaptive Stochastic Mirror Descent for Constrained Optimization
• Variational Principles for Mirror Descent and Mirror Langevin Dynamics
• Stochastic Mirror Descent for Convex Optimization with Consensus Constraints
• Mirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise Variance