On Bellman equations for continuous-time policy evaluation I: discretization and approximation

2024/07/08 by Mou, Wenlong, Zhu, Yuhua
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Numerical Analysis (math.NA) #Optimization and Control (math.OC) #Probability (math.PR)

paper · doi:10.48550/arxiv.2407.05966

Abstract

We study the problem of computing the value function from a discretely-observed trajectory of a continuous-time diffusion process. We develop a new class of algorithms based on easily implementable numerical schemes that are compatible with discrete-time reinforcement learning (RL) with function approximation. We establish high-order numerical accuracy as well as the approximation error guarantees for the proposed approach. In contrast to discrete-time RL problems where the approximation factor depends on the effective horizon, we obtain a bounded approximation factor using the underlying elliptic structures, even if the effective horizon diverges to infinity.

Related

• PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluation
• Temporal Difference Learning with Continuous Time and State in the Stochastic Setting
• Operator Models for Continuous-Time Offline Reinforcement Learning
• A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models
• Optimal-PhiBE: A PDE-based Model-free framework for Continuous-time Reinforcement Learning