Truly Batch Apprenticeship Learning with Deep Successor Features

2019/03/24 by Lee, Donghun, Srinivasan, Srivatsan, Doshi-Velez, Finale · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1903.10077

Abstract

We introduce a novel apprenticeship learning algorithm to learn an expert's underlying reward structure in off-policy model-free batch settings. Unlike existing methods that require a dynamics model or additional data acquisition for on-policy evaluation, our algorithm requires only the batch data of observed expert behavior. Such settings are common in real-world tasks---health care, finance or industrial processes ---where accurate simulators do not exist or data acquisition is costly. To address challenges in batch settings, we introduce Deep Successor Feature Networks(DSFN) that estimate feature expectations in an off-policy setting and a transition-regularized imitation network that produces a near-expert initial policy and an efficient feature representation. Our algorithm achieves superior results in batch settings on both control benchmarks and a vital clinical task of sepsis management in the Intensive Care Unit.

Cited by

• Language Model Distillation: A Temporal Difference Imitation Learning Perspective

Related

• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement\n Learning
• Safe Policy Improvement with Baseline Bootstrapping
• Knowledge Transfer from Teachers to Learners in Growing-Batch Reinforcement Learning
• A Strong Baseline for Batch Imitation Learning