Continuous Policy and Value Iteration for Stochastic Control Problems and Its Convergence

2025/06/09 by Qi Feng, Feng, Qi, Wang Gu +1 · 1 citation
Computer Science · Physics and Astronomy · #Bellman equation #Control (management) #Convergence (economics) #Function (biology) #Iterative learning control #Model Reduction and Neural Networks #Monotonic function #Optimal control #Reinforcement Learning in Robotics #Stochastic Gradient Optimization Techniques #Stochastic control #Stochastic differential equation

paper · pdf · doi:10.48550/arxiv.2506.08121

published in arXiv (Cornell University) (Cornell University)

Abstract

We introduce a continuous policy-value iteration algorithm where the approximations of the value function of a stochastic control problem and the optimal control are simultaneously updated through Langevin-type dynamics. This framework applies to both the entropy-regularized relaxed control problems and the classical control problems, with infinite horizon. We establish policy improvement and demonstrate convergence to the optimal control under the monotonicity condition of the Hamiltonian. By utilizing Langevin-type stochastic differential equations for continuous updates along the policy iteration direction, our approach enables the use of distribution sampling and non-convex learning techniques in machine learning to optimize the value function and identify the optimal control simultaneously.

Citations

• Learning Mean-Field Games through Mean-Field Actor-Critic Flow
• Continuous-time optimal investment with portfolio constraints: a reinforcement learning approach
• Regret of exploratory policy improvement and q-learning
• A policy iteration algorithm for non-Markovian control problems
• Exploratory Optimal Stopping: A Singular Control Formulation
• Policy Iteration for Exploratory Hamilton--Jacobi--Bellman Equations
• Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching
• Solving Time-Continuous Stochastic Optimal Control Problems: Algorithm Design and Convergence Analysis of Actor-Critic Flow
• Fisher information dissipation for time inhomogeneous stochastic differential equations
• Policy Optimization for Continuous Reinforcement Learning
• A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee
• Uniform-in-time propagation of chaos for mean field Langevin dynamics
• Mean-Field Langevin Dynamics: Exponential Convergence and Annealing
• Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms
• Exploratory HJB equations and their convergence
• Exploratory HJB Equations and Their Convergence
• Simulated annealing from continuum to discretization: a convergence analysis via the Eyring--Kramers law
• A modified MSA for stochastic control problems
• Non-convex Learning via Replica Exchange Stochastic Gradient MCMC
• Gradient Flows for Regularized Stochastic Control Problems
• Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
• Exponential Convergence and stability of Howards's Policy Improvement Algorithm for Controlled Diffusions
• On sampling from a log-concave density using kinetic Langevin diffusions
• Underdamped Langevin MCMC: A non-asymptotic analysis
• Couplings and quantitative contraction rates for Langevin dynamics
• A Hitting Time Analysis of Stochastic Gradient Langevin Dynamics
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• On the policy improvement algorithm in continuous time
• A Complete Recipe for Stochastic Gradient MCMC
• Consistency and fluctuations for stochastic gradient Langevin dynamics
• Recursive Stochastic Algorithms for Global Optimization in ℝ^d
• Diffusion for Global Optimization in ℝⁿ
• A new proof of Moser's parabolic harnack inequality using the old ideas of Nash
• On the Convergence of Policy Iteration in Stationary Dynamic Programming

Cited by

• Neural Actor-Critic Methods for Hamilton-Jacobi-Bellman PDEs: Asymptotic Analysis and Numerical Studies

Related

• Convergence of Policy Iteration for Entropy-Regularized Stochastic Control Problems
• A policy iteration algorithm for non-Markovian control problems
• Robust policy iteration for continuous-time stochastic H_∞ control problem with unknown dynamics
• Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approach
• Learning Continuous Control Policies by Stochastic Value Gradients