Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic

2019/01/08 by Mikael Henaff, Henaff, Mikael, Alfredo Canziani +3 · 8 citations
Computer Science · #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Adversarial Robustness in Machine Learning

paper · pdf · doi:10.48550/arxiv.1901.02705

Abstract

Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on. We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks. We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.

Citations

• Auto-Encoding Variational Bayes
• End to End Learning for Self-Driving Cars
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Imagination-Augmented Agents for Deep Reinforcement Learning
• A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning

Cited by

• Planning as Descent: Goal-Conditioned Latent Trajectory Synthesis in Learned Energy Landscapes
• Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference
• Reset-Free Lifelong Learning with Skill-Space Planning
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors
• A review of uncertainty quantification in deep learning: Techniques, applications and challenges
• Batch Policy Learning under Constraints
• Scalable Offline Metrics for Autonomous Driving
• Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets

Related

• A Bayesian approach to quantifying uncertainties and improving generalizability in traffic prediction models
• Simultaneous Policy Learning and Latent State Inference for Imitating Driver Behavior
• Identify, Estimate and Bound the Uncertainty of Reinforcement Learning for Autonomous Driving
• Graph-based Prediction and Planning Policy Network (GP3Net) for scalable self-driving in dynamic environments using Deep Reinforcement Learning
• UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning