Primal-Dual Algorithm for Distributed Reinforcement Learning: Distributed GTD

2018/03/21 by Lee, Donghwan, Yoon, Hyungjin, Hovakimyan, Naira
#FOS: Mathematics #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.1803.08031

Abstract

The goal of this paper is to study a distributed version of the gradient temporal-difference (GTD) learning algorithm for multi-agent Markov decision processes (MDPs). The temporal difference (TD) learning is a reinforcement learning (RL) algorithm which learns an infinite horizon discounted cost function (or value function) for a given fixed policy without the model knowledge. In the distributed RL case each agent receives local reward through a local processing. Information exchange over sparse communication network allows the agents to learn the global value function corresponding to a global reward, which is a sum of local rewards. In this paper, the problem is converted into a constrained convex optimization problem with a consensus constraint. Then, we propose a primal-dual distributed GTD algorithm and prove that it almost surely converges to a set of stationary points of the optimization problem.

Related

• Primal-Dual Distributed Temporal Difference Learning
• A primal-dual perspective for distributed TD-learning
• Distributed Value Function Approximation for Collaborative Multi-Agent Reinforcement Learning
• Finite-Time Analysis of Distributed TD(0) with Linear Function\n Approximation for Multi-Agent Reinforcement Learning
• Fast Multi-Agent Temporal-Difference Learning via Homotopy Stochastic Primal-Dual Optimization