Solving Time-Continuous Stochastic Optimal Control Problems: Algorithm Design and Convergence Analysis of Actor-Critic Flow

2024/02/27 by Mo Zhou, Zhou, Mo, Jianfeng Lu +1 · 3 citations
Decision Sciences · Engineering · #49L12 49M25 (secondary) #93E20 (Primary) #FOS: Mathematics #G.1.6 #G.1.8 #Optimization and Control (math.OC) #Scheduling and Optimization Algorithms #Simulation Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2402.17208

openalex publication_date 2024/02/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose an actor-critic framework to solve the time-continuous stochastic optimal control problem. A least square temporal difference method is applied to compute the value function for the critic. The policy gradient method is implemented as policy improvement for the actor. Our key contribution lies in establishing a linear rate of convergence for our proposed actor-critic flow. Theoretical findings are further validated through numerical examples, showing the efficacy of our approach in practical applications.

Cited by

• Continuous Policy and Value Iteration for Stochastic Control Problems and Its Convergence
• Simulating Fokker-Planck equations via mean field control of score-based normalizing flows
• Model-Based Closed-Loop Control Algorithm for Stochastic Partial Differential Equation Control

Related

• On the policy improvement algorithm in continuous time
• An Actor Critic Method for Free Terminal Time Optimal Control
• A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee
• A policy iteration algorithm for non-Markovian control problems
• Continuous Policy and Value Iteration for Stochastic Control Problems and Its Convergence