Quasi-Newton Iteration in Deterministic Policy Gradient

2022/03/25 by Kordabad, Arash Bahari, Esfahani, Hossein Nejatbakhsh, Cai, Wenqi +1 · 2 citations
#FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2203.13854

Abstract

This paper presents a model-free approximation for the Hessian of the performance of deterministic policies to use in the context of Reinforcement Learning based on Quasi-Newton steps in the policy parameters. We show that the approximate Hessian converges to the exact Hessian at the optimal policy, and allows for a superlinear convergence in the learning, provided that the policy parametrization is rich. The natural policy gradient method can be interpreted as a particular case of the proposed method. We analytically verify the formulation in a simple linear case and compare the convergence of the proposed method with the natural policy gradient in a nonlinear example.

Cited by

• Second-Order Policy Gradient Methods for the Linear Quadratic Regulator
• Computationally efficient Gauss-Newton reinforcement learning for model predictive control

Related

• Quasi-Newton Compatible Actor-Critic for Deterministic Policies
• Quasi-Newton Trust Region Policy Optimization
• Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction
• Elementary Analysis of Policy Gradient Methods
• Policy Gradient Algorithms Implicitly Optimize by Continuation