Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic

2019/01/08 by Mikael Henaff, Henaff, Mikael, Alfredo Canziani +3 · 15 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1901.02705

arxiv created 2019/01/08 · arxiv updated 2019/01/10

Abstract

Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on. We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks. We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.

Citations

• Auto-Encoding Variational Bayes
• End to End Learning for Self-Driving Cars
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Imagination-Augmented Agents for Deep Reinforcement Learning
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Cited by

• Planning as Descent: Goal-Conditioned Latent Trajectory Synthesis in Learned Energy Landscapes
• Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference
• Reset-Free Lifelong Learning with Skill-Space Planning
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Batch Policy Learning under Constraints
• Scalable Offline Metrics for Autonomous Driving
• Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets
• SAFARI: Safe and Active Robot Imitation Learning with Imagination
• Visual-based Autonomous Driving Deployment from a Stochastic and Uncertainty-aware Perspective
• Simple and Effective VAE Training with Calibrated Decoders
• Dynamics-aware Embeddings
• Multi-Turn Beam Search for Neural Dialogue Modeling
• Deliberative acting, planning and learning with hierarchical operational models
• Learning the smoothness of noisy curves with application to online curve estimation

Related

• A Bayesian approach to quantifying uncertainties and improving generalizability in traffic prediction models
• Simultaneous Policy Learning and Latent State Inference for Imitating Driver Behavior
• Identify, Estimate and Bound the Uncertainty of Reinforcement Learning for Autonomous Driving
• Graph-based Prediction and Planning Policy Network (GP3Net) for scalable self-driving in dynamic environments using Deep Reinforcement Learning
• UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning