Off Policy Lyapunov Stability in Reinforcement Learning

2025/09/11 by Gill, Sarvan, Daniela Constantinescu, Constantinescu, Daniela
Computer Science · #Adaptive Dynamic Programming Control

paper · pdf · doi:10.48550/arxiv.2509.09863

Abstract

Traditional reinforcement learning lacks the ability to provide stability guarantees. More recent algorithms learn Lyapunov functions alongside the control policies to ensure stable learning. However, the current self-learned Lyapunov functions are sample inefficient due to their on-policy nature. This paper introduces a method for learning Lyapunov functions off-policy and incorporates the proposed off-policy Lyapunov function into the Soft Actor Critic and Proximal Policy Optimization algorithms to provide them with a data efficient stability certificate. Simulations of an inverted pendulum and a quadrotor illustrate the improved performance of the two algorithms when endowed with the proposed off-policy Lyapunov function.

Citations

• Critic as Lyapunov function (CALF): a model-free, stability-ensuring agent
• Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
• Physics-Informed Neural Network Lyapunov Functions: PDE Characterization, Learning, and Verification
• Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions
• Stable and Safe Reinforcement Learning via a Barrier-Lyapunov Actor-Critic Approach
• Refining Control Barrier Functions through Hamilton-Jacobi Reachability
• Model-free Neural Lyapunov Control for Safe Robot Navigation
• Safe Control with Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction methods
• Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning
• Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics
• Learning Certified Control using Contraction Metric
• Neural Lyapunov Control
• Actor-Critic Reinforcement Learning for Control with Stability Guarantee
• End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks
• Almost Lyapunov Functions for Nonlinear Systems
• A Lyapunov-based Approach to Safe Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Proximal Policy Optimization Algorithms
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Reinforcement Learning: An Introduction

Related

• Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions
• A Policy Optimization Method Towards Optimal-time Stability
• Stability Enhancement in Reinforcement Learning via Adaptive Control Lyapunov Function
• Critic as Lyapunov function (CALF): a model-free, stability-ensuring agent
• Neural Lyapunov and Optimal Control