Accelerating Primal-dual Methods for Regularized Markov Decision Processes

2022/02/21 by Li, Haoya, Yu, Hsiang-fu, Ying, Lexing +1 · 1 citation
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2202.10506

Abstract

Entropy regularized Markov decision processes have been widely used in reinforcement learning. This paper is concerned with the primal-dual formulation of the entropy regularized problems. Standard first-order methods suffer from slow convergence due to the lack of strict convexity and concavity. To address this issue, we first introduce a new quadratically convexified primal-dual formulation. The natural gradient ascent descent of the new formulation enjoys global convergence guarantee and exponential convergence rate. We also propose a new interpolating metric that further accelerates the convergence significantly. Numerical results are provided to demonstrate the performance of the proposed methods under multiple settings.

Cited by

• A Two-Timescale Primal-Dual Framework for Reinforcement Learning via Online Dual Variable Guidance

Related

• A unified view of entropy-regularized Markov decision processes
• A Dual Approach to Constrained Markov Decision Processes with Entropy Regularization
• A Theory of Regularized Markov Decision Processes
• A Two-Timescale Primal-Dual Framework for Reinforcement Learning via Online Dual Variable Guidance
• A Two-Timescale Primal-Dual Framework for Reinforcement Learning via Online Dual Variable Guidance