Reward function shape exploration in adversarial imitation learning: an empirical study

2021/04/14 by Yawei Wang, Xiu Li, Wang, Yawei +1
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2104.06687

openalex publication_date 2021/04/14 · openalex created_date 2021/04/26 · openalex updated_date 2026/07/28

Abstract

For adversarial imitation learning algorithms (AILs), no true rewards are obtained from the environment for learning the strategy. However, the pseudo rewards based on the output of the discriminator are still required. Given the implicit reward bias problem in AILs, we design several representative reward function shapes and compare their performances by large-scale experiments. To ensure our results' reliability, we conduct the experiments on a series of Mujoco and Box2D continuous control tasks based on four different AILs. Besides, we also compare the performance of various reward function shapes using varying numbers of expert trajectories. The empirical results reveal that the positive logarithmic reward function works well in typical continuous control tasks. In contrast, the so-called unbiased reward function is limited to specific kinds of tasks. Furthermore, several designed reward functions perform excellently in these environments as well.

Citations

• Trust Region Policy Optimization
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning
• Addressing reward bias in Adversarial Imitation Learning with neutral reward functions

Related

• Addressing reward bias in Adversarial Imitation Learning with neutral reward functions
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Reward
• Support-weighted Adversarial Imitation Learning
• On Discovering Algorithms for Adversarial Imitation Learning