Solving nonconvex Hamilton--Jacobi--Isaacs equations with PINN-based policy iteration

2025/07/21 by Hee Jun Yang, Yang, Hee Jun, Minjung Gim +3
Computer Science · Physics and Astronomy · #35Q93 #49L25 #49N70 #68T07 #Adaptive Dynamic Programming Control #Analysis of PDEs (math.AP) #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Model Reduction and Neural Networks #Numerical Analysis (math.NA) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2507.15455

openalex publication_date 2025/07/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a mesh-free policy iteration framework that combines classical dynamic programming with physics-informed neural networks (PINNs) to solve high-dimensional, nonconvex Hamilton--Jacobi--Isaacs (HJI) equations arising in stochastic differential games and robust control. The method alternates between solving linear second-order PDEs under fixed feedback policies and updating the controls via pointwise minimax optimization using automatic differentiation. Under standard Lipschitz and uniform ellipticity assumptions, we prove that the value function iterates converge locally uniformly to the unique viscosity solution of the HJI equation. The analysis establishes equi-Lipschitz regularity of the iterates, enabling provable stability and convergence without requiring convexity of the Hamiltonian. Numerical experiments demonstrate the accuracy and scalability of the method. In a two-dimensional stochastic path-planning game with a moving obstacle, our method matches finite-difference benchmarks with relative L²-errors below %10^-2%. In five- and ten-dimensional publisher-subscriber differential games with anisotropic noise, the proposed approach consistently outperforms direct PINN solvers, yielding smoother value functions and lower residuals. Our results suggest that integrating PINNs with policy iteration is a practical and theoretically grounded method for solving high-dimensional, nonconvex HJI equations, with potential applications in robotics, finance, and multi-agent reinforcement learning.

Citations

• Policy iteration for nonconvex viscous Hamilton--Jacobi equations
• Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learning
• Policy Iteration for Exploratory Hamilton--Jacobi--Bellman Equations
• Policy iteration for the deterministic control problems -- a viscosity approach
• DeepReach: A Deep Learning Approach to High-Dimensional Reachability
• Implicit Neural Representations with Periodic Activation Functions
• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Exponential Convergence and Stability of Howard's Policy Improvement Algorithm for Controlled Diffusions
• An Algorithmic Introduction to Numerical Simulation of Stochastic Differential Equations
• Artificial neural networks for solving ordinary and partial differential equations

Related

• Value Approximation for Two-Player General-Sum Differential Games with State Constraints
• Physics-informed approach for exploratory Hamilton--Jacobi--Bellman equations via policy iterations
• A neural network based policy iteration algorithm with global H²-superlinear convergence for stochastic games on domains
• Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learning
• Neural Policy Iteration for Stochastic Optimal Control: A Physics-Informed Approach