Causal Confusion in Imitation Learning

2019/05/28 by Pim de Haan, Dinesh Jayaraman, de Haan, Pim +3 · 50 citations
Computer Science · Engineering · Mathematics · #AI-based Problem Solving and Planning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Robot Manipulation and Learning #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1905.11979

Published at NeurIPS 2019 9 pages, plus references and appendices

openalex publication_date 2019/05/28 · arxiv created 2019/11/04 · arxiv updated 2019/11/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Behavioral cloning reduces policy learning to supervised learning by training a discriminative model to predict expert actions given observations. Such discriminative models are non-causal: the training procedure is unaware of the causal structure of the interaction between the expert and the environment. We point out that ignoring causality is particularly damaging because of the distributional shift in imitation learning. In particular, it leads to a counter-intuitive "causal misidentification" phenomenon: access to more information can yield worse performance. We investigate how this problem arises, and propose a solution to combat it through targeted interventions---either environment interaction or expert queries---to determine the correct causal model. We show that causal misidentification occurs in several benchmark control domains as well as realistic driving settings, and validate our solution against DAgger and other baselines and ablations.

Citations

• Playing Atari with Deep Reinforcement Learning
• Variational Dropout and the Local Reparameterization Trick
• A survey of robot learning from demonstration
• End to End Learning for Self-Driving Cars
• Categorical Reparameterization with Gumbel-Softmax
• Causal Effect Inference with Deep Latent-Variable Models
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
• An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
• ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst

Cited by

• Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
• Human-like working memory signatures emerge from intrinsically plastic artificial neurons for robust dynamic vision
• Rethinking Camera Choice: An Empirical Study on Fisheye Camera Properties in Robotic Manipulation
• End-to-end Autonomous Vehicle Following System using Monocular Fisheye Camera
• AutoFocus-IL: VLM-based Saliency Maps for Data-Efficient Visual Imitation Learning without Extra Human Annotations
• Improving Robustness to Out-of-Distribution States in Imitation Learning via Deep Koopman-Boosted Diffusion Policy
• Constructing the Umwelt: Cognitive Planning through Belief-Intent Co-Evolution
• Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents
• RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation
• Learning to play: A Multimodal Agent for 3D Game-Play
• ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context
• CroSTAta: Cross-State Transition Attention Transformer for Robotic Manipulation
• HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy
• VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators
• Fidelity-Aware Data Composition for Robust Robot Generalization
• It's Not Just More Demos: Counterfactual Action Sensitivity Coverage for Data-Efficient Robust Robot Imitation
• Ratatouille: Imitation Learning Ingredients for Real-world Social Robot Navigation
• SEG-Parking: Towards Safe, Efficient, and Generalizable Autonomous Parking via End-to-End Offline Reinforcement Learning
• TreeIRL: Safe Urban Driving with Tree Search and Inverse Reinforcement Learning
• Synthetic vs. Real Training Data for Visual Navigation
• Grasp Like Humans: Learning Generalizable Multi-Fingered Grasping from Human Proprioceptive Sensorimotor Integration
• Gentle Object Retraction in Dense Clutter Using Multimodal Force Sensing and Imitation Learning
• Pixels to Play: A Foundation Model for 3D Gameplay
• Human Centric General Physical Intelligence for Agile Manufacturing Automation
• Rational Inverse Reasoning: Few-Shot Imitation by Inferring Intent through Planning
• Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation
• Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving
• Improving Generalization Ability of Robotic Imitation Learning by Resolving Causal Confusion in Observations
• GABRIL: Gaze-Based Regularization for Mitigating Causal Confusion in Imitation Learning
• Behavioral Exploration: Learning to Explore via In-Context Adaptation
• CUPID: Curating Data your Robot Loves with Influence Functions
• Efficient and Generalizable Environmental Understanding for Visual Navigation
• Breaking Habits: On the Role of the Advantage Function in Learning Causal State Representations
• Preference Learning for AI Alignment: a Causal Perspective
• ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Driving
• What Can RL Bring to VLA Generalization? An Empirical Study
• What Do You Need for Compositional Generalization in Diffusion Planning?
• SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving
• Guided Policy Optimization under Partial Observability
• Object-Focus Actor for Data-efficient Robot Generalization Dexterous Manipulation
• Learning Long-Context Diffusion Policies via Past-Token Prediction
• Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges
• Geometric Action Model for Robot Policy Learning
• CADET: Physics-Grounded Causal Auditing and Training-Free Deconfounding of End-to-End Driving Planners
• Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAs
• Scaling Self-Play for End-to-End Driving
• How Should Vision-Language-Action Models Use Proprioceptive State?
• CIVIL: Causal and Intuitive Visual Imitation Learning
• Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models
• FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions

Related

• Causal Imitation Learning with Unobserved Confounders
• Initial State Interventions for Deconfounded Imitation Learning
• Scalable Causal Imitation Learning
• Fighting Copycat Agents in Behavioral Cloning from Observation Histories
• Causal Imitation Learning under Expert-Observable and Expert-Unobservable Confounding