A Reinforcement Learning Framework for Some Singular Stochastic Control Problems

2025/06/27 by Zongxia Liang, Xiaodong Luo, Liang, Zongxia +3 · 2 citations
Computer Science · #49K45 #93B47 #93E20 #Adaptive Dynamic Programming Control #FOS: Mathematics #Optimization and Control (math.OC) #Optimization and Variational Analysis #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2506.22203

openalex publication_date 2025/06/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

We develop a continuous-time reinforcement learning framework for a class of singular stochastic control problems without entropy regularization. The optimal singular control is characterized as the optimal singular control law, which is a pair of regions of time and the augmented states. The goal of learning is to identify such an optimal region via the trial-and-error procedure. In this context, we generalize the existing policy evaluation theories with regular controls to learn our optimal singular control law and develop a policy improvement theorem via the region iteration. To facilitate the model-free policy iteration procedure, we further introduce the zero-order and first-order q-functions arising from singular control problems and establish the martingale characterization for the pair of q-functions together with the value function. Based on our theoretical findings, some q-learning algorithms are devised accordingly and a numerical example based on simulation experiment is presented.

Citations

• Continuous-time reinforcement learning for optimal switching over multiple regimes
• Exploratory Optimal Stopping: A Singular Control Formulation
• Learning to Optimally Stop Diffusion Processes, with Financial Applications
• Unified continuous-time q-learning for mean-field game and mean-field control problems
• Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approach
• Policy Iteration for Exploratory Hamilton--Jacobi--Bellman Equations
• Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty
• On optimal tracking portfolio in incomplete markets: The reinforcement learning approach
• Equilibria for Time-inconsistent Singular Control Problems
• Continuous-time q-learning for mean-field control problems
• Convergence of Policy Iteration for Entropy-Regularized Stochastic Control Problems
• Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithm
• q-Learning in Continuous Time
• Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms
• Policy Evaluation and Temporal-Difference Learning in Continuous Time\n and Space: A Martingale Approach
• An Optimal Extraction Problem with Price Impact
• Mean Field Forward-Backward Stochastic Differential Equations
• Optimization of the flow of dividends

Cited by

• Continuous-time reinforcement learning for optimal switching over multiple regimes
• Optimal dividend and capital injection under self-exciting claims

Related

• Reinforcement learning for irreversible reinsurance problems: the randomized singular control approach
• Exploratory Optimal Stopping: A Singular Control Formulation
• Reinforcement Learning Framework For Stochastic Optimal Control Problem Under Model Uncertainty
• Continuous-time q-learning for mean-field control problems
• Stochastic Primal-Dual Q-Learning