Agent Modeling as Auxiliary Task for Deep Reinforcement Learning

2019/07/22 by Pablo Hernández-Leal, Hernandez-Leal, Pablo, Bilal Kartal +3 · 1 citation
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multiagent Systems (cs.MA) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1907.09597

openalex publication_date 2019/07/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper we explore how actor-critic methods in deep reinforcement learning, in particular Asynchronous Advantage Actor-Critic (A3C), can be extended with agent modeling. Inspired by recent works on representation learning and multiagent deep reinforcement learning, we propose two architectures to perform agent modeling: the first one based on parameter sharing, and the second one based on agent policy features. Both architectures aim to learn other agents' policies as auxiliary tasks, besides the standard actor (policy) and critic (values). We performed experiments in both cooperative and competitive domains. The former is a problem of coordinated multiagent object transportation and the latter is a two-player mini version of the Pommerman game. Our results show that the proposed architectures stabilize learning and outperform the standard A3C architecture when learning a best response in terms of expected rewards.

Cited by

• Adapting Auxiliary Losses Using Gradient Similarity
• Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?

Related

• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Contrastive learning-based agent modeling for deep reinforcement learning
• Deep Multiagent Reinforcement Learning: Challenges and Directions
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Learning Policy Representations in Multiagent Systems