2023/12/29 by Haoyu Han, Heng Yang, Han, Haoyu +1
Engineering · Physics and Astronomy · #FOS: Mathematics #Model Reduction and Neural Networks #Oil and Gas Production Techniques #Optimization and Control (math.OC)
paper · pdf · doi:10.48550/arxiv.2312.17467
openalex publication_date 2023/12/29 · openalex created_date 2024/01/02 · openalex updated_date 2026/07/28
We revisit the inverted pendulum problem with the goal of understanding and computing the true optimal value function. We start with an observation that the true optimal value function must be nonsmooth (i.e., not globally C1) due to the symmetry of the problem. We then give a result that can certify the optimality of a candidate piece-wise C1 value function. Further, for a candidate value function obtained via numerical approximation, we provide a bound of suboptimality based on its Hamilton-Jacobi-Bellman (HJB) equation residuals. Inspired by Holzhuter (2004), we then design an algorithm that solves backward the Pontryagin's minimum principle (PMP) ODE from terminal conditions provided by the locally optimal LQR value function. This numerical procedure leads to a piece-wise C1 value function whose nonsmooth region contains periodic spiral lines and smooth regions attain HJB residuals about 10-4, hence certified to be the optimal value function up to minor numerical inaccuracies. This optimal value function checks the power of optimality: (i) it sits above a polynomial lower bound; (ii) its induced controller globally swings up and stabilizes the pendulum, and (iii) attains lower trajectory cost than baseline methods such as energy shaping, model predictive control (MPC), and proximal policy optimization (with MPC attaining almost the same cost). We conclude by distilling the optimal value function into a simple neural network. Our code is avilable in https://github.com/ComputationalRobotics/InvertedPendulumOptimalValue.