A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

2019/03/15 by Suttle, Wesley, Yang, Zhuoran, Zhang, Kaiqing +3 · 2 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1903.06372

Abstract

This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy. To this end, the paper develops a multi-agent version of emphatic temporal difference learning for off-policy policy evaluation, and proves convergence under linear function approximation. The paper then leverages this result, in conjunction with a novel multi-agent off-policy policy gradient theorem and recent work in both multi-agent on-policy and single-agent off-policy actor-critic methods, to develop and give convergence guarantees for a new multi-agent off-policy actor-critic algorithm.

Cited by

• A Communication-Efficient Decentralized Actor-Critic Algorithm
• Policy Optimization in Multi-Agent Settings under Partially Observable Environments

Related

• Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus
• Multi-agent Off-policy Actor-Critic Reinforcement Learning for Partially Observable Environments
• Multi-agent Natural Actor-critic Reinforcement Learning Algorithms
• Decentralized Multi-Agent Reinforcement Learning: An Off-Policy Method
• Distributed Policy Evaluation Under Multiple Behavior Strategies