Versatile Inverse Reinforcement Learning via Cumulative Rewards

2021/01/01 by Niklas Freymuth, Freymuth, Niklas, Philipp Becker +3
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #I.2.6 #Machine Learning (cs.LG) #Machine Learning and Data Classification #Reinforcement Learning in Robotics

paper · doi:10.48550/arxiv.2111.07667

openalex publication_date 2021/01/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Inverse Reinforcement Learning infers a reward function from expert demonstrations, aiming to encode the behavior and intentions of the expert. Current approaches usually do this with generative and uni-modal models, meaning that they encode a single behavior. In the common setting, where there are various solutions to a problem and the experts show versatile behavior this severely limits the generalization capabilities of these methods. We propose a novel method for Inverse Reinforcement Learning that overcomes these problems by formulating the recovered reward as a sum of iteratively trained discriminators. We show on simulated tasks that our approach is able to recover general, high-quality reward functions and produces policies of the same quality as behavioral cloning approaches designed for versatile behavior.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Remarks on Some Nonparametric Estimates of a Density Function
• On Estimation of a Probability Density Function and Mode
• Reinforcement Learning with Deep Energy-Based Policies
• Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations
• Variational Inference: A Review for Statisticians

Related

• Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
• Multi Task Inverse Reinforcement Learning for Common Sense Reward
• Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
• Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids
• Toward Computationally Efficient Inverse Reinforcement Learning via Reward Shaping