On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning

2024/11/22 by Till Freihaut, Freihaut, Till, Giorgia Ramponi +1
Engineering · #Advanced Research in Systems and Signal Processing #Elevator Systems and Control

paper · pdf · doi:10.48550/arxiv.2411.15046

Abstract

Multi-agent Inverse Reinforcement Learning (MAIRL) aims to recover agent reward functions from expert demonstrations. We characterize the feasible reward set in Markov games, identifying all reward functions that rationalize a given equilibrium. However, equilibrium-based observations are often ambiguous: a single Nash equilibrium can correspond to many reward structures, potentially changing the game's nature in multi-agent systems. We address this by introducing entropy-regularized Markov games, which yield a unique equilibrium while preserving strategic incentives. For this setting, we provide a sample complexity analysis detailing how errors affect learned policy performance. Our work establishes theoretical foundations and practical insights for MAIRL.

Related

• Multi-Agent Adversarial Inverse Reinforcement Learning
• Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts
• Multi-Agent Inverse Q-Learning from Demonstrations
• Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal Demonstrations
• Computational Rationalization: The Inverse Equilibrium Problem