Adversarial Option-Aware Hierarchical Imitation Learning

2021/06/10 by Mingxuan Jing, Jing, Mingxuan, Wenbing Huang +11 · 2 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Reinforcement Learning in Robotics #Robot Manipulation and Learning #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2106.05530

accepted by ICML 2021

arxiv created 2021/06/11 · arxiv updated 2021/06/14

Abstract

It has been a challenge to learning skills for an agent from long-horizon unannotated demonstrations. Existing approaches like Hierarchical Imitation Learning(HIL) are prone to compounding errors or suboptimal solutions. In this paper, we propose Option-GAIL, a novel method to learn skills at long horizon. The key idea of Option-GAIL is modeling the task hierarchy by options and train the policy via generative adversarial optimization. In particular, we propose an Expectation-Maximization(EM)-style algorithm: an E-step that samples the options of expert conditioned on the current learned policy, and an M-step that updates the low- and high-level policies of agent simultaneously to minimize the newly proposed option-occupancy measurement between the expert and the agent. We theoretically prove the convergence of the proposed algorithm. Experiments show that Option-GAIL outperforms other counterparts consistently across a variety of tasks.

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning

Cited by

• Intention-aware Hierarchical Diffusion Model for Long-term Trajectory Anomaly Detection
• ILCL: Inverse Logic-Constraint Learning from Temporally Constrained Demonstrations

Related

• Provable Hierarchical Imitation Learning via EM
• Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control
• Hindsight Generative Adversarial Imitation Learning
• Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information
• Hierarchical Imitation and Reinforcement Learning