Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems

2026/06/10 by Xin Guo, Yijie Huang, Xiang Yu · 1 voice
Economics, Econometrics and Finance · Computer Science · Mathematics · #q-fin.CP #cs.LG #math.OC

paper · pdf

Abstract

In this paper, we develop a continuous-time model-free reinforcement learning algorithm to learn deterministic equilibrium policies in general time-inconsistent control problems. Utilizing the extended Hamilton-Jacobi-Bellman system, we recast the original time-inconsistent problem into an equivalent two-stage problem. In the first stage, for given auxiliary functions, we employ the deterministic policy gradient approach to learn an optimal policy in an auxiliary time-consistent control problem. In the second stage, given the updated policy, we exploit the inner fixed point iterations and some martingale characterizations to learn the auxiliary functions. As a theoretical contribution, we provide some mild model assumptions and establish the convergence of inner fixed point iterations. By repeating this actor-critic style of iterations across two stages, our algorithm aims to learn the equilibrium under different sources of time-inconsistency in a unified manner. The superior effectiveness of the proposed algorithm are illustrated in two classical financial applications with time-inconsistency: mean-variance portfolio management and optimal tracking portfolio under non-exponential discounting.

Citations

• Continuous-time reinforcement learning for optimal switching over multiple regimes
• Bridging Discrete and Continuous RL: Stable Deterministic Policy Gradient with Martingale Characterization
• A Two-fold Randomization Framework for Impulse Control Problems
• Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning
• Exploratory Optimal Stopping: A Singular Control Formulation
• Learning to Optimally Stop Diffusion Processes, with Financial Applications
• Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropy
• Entropy annealing for policy mirror descent in continuous time and space
• On optimal tracking portfolio in incomplete markets: The reinforcement learning approach
• Continuous-time q-learning for mean-field control problems
• Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithm
• q-Learning in Continuous Time
• Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms
• Exploratory HJB equations and their convergence
• Exploratory HJB Equations and Their Convergence
• Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach
• Entropy Regularization for Mean Field Games with Learning
• Being serious about non-commitment: subgame perfect equilibrium in continuous time
• Myopia and Inconsistency in Dynamic Utility Maximization

Discussions

• Xin Guo, Yijie Huang, Xiang Yu: Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems https://arxiv.org/abs/2606.11798 https://arxiv.org/pdf/2606.11798 https://a [bsky, 0 points, 0 comments]

Related

• A Subgame Perfect Equilibrium Reinforcement Learning Approach to Time-inconsistent Problems
• Bridging Discrete and Continuous RL: Stable Deterministic Policy Gradient with Martingale Characterization
• A Temporal Difference Method for Stochastic Continuous Dynamics
• Exploratory Mean-Variance with Jumps: An Equilibrium Approach
• Deterministic Policy Gradients With General State Transitions