Inverse Reinforcement Learning with the Average Reward Criterion

2023/05/24 by Wu, Feiyang, Ke, Jingyang, Wu, Anqi · 2 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2305.14608

Abstract

We study the problem of Inverse Reinforcement Learning (IRL) with an average-reward criterion. The goal is to recover an unknown policy and a reward function when the agent only has samples of states and actions from an experienced agent. Previous IRL methods assume that the expert is trained in a discounted environment, and the discount factor is known. This work alleviates this assumption by proposing an average-reward framework with efficient learning algorithms. We develop novel stochastic first-order methods to solve the IRL problem under the average-reward setting, which requires solving an Average-reward Markov Decision Process (AMDP) as a subproblem. To solve the subproblem, we develop a Stochastic Policy Mirror Descent (SPMD) method under general state and action spaces that needs O(1/ε) steps of gradient computation. Equipped with SPMD, we propose the Inverse Policy Mirror Descent (IPMD) method for solving the IRL problem with a O(1/ε²) complexity. To the best of our knowledge, the aforementioned complexity results are new in IRL. Finally, we corroborate our analysis with numerical experiments using the MuJoCo benchmark and additional control tasks.

Cited by

• Distributional Inverse Reinforcement Learning
• Approximate Solution Methods for the Average Reward Criterion in Optimal Tracking Control of Linear Systems

Related

• Inverse Reinforcement Learning with Simultaneous Estimation of Rewards\n and Dynamics
• Stochastic Inverse Reinforcement Learning
• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
• Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids
• On the Performance of Maximum Likelihood Inverse Reinforcement Learning