DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
2023/12/14 by Wenhai Wang, Cui, Erfei, Wang, Wenhai +15 · 79 citations
Computer Science · #Topic Modeling #Multimodal Machine Learning Applications #Advanced Neural Network Applications
paper · pdf · doi:10.48550/arxiv.2312.09245
Abstract
Large language models (LLMs) have opened up new possibilities for intelligent agents, endowing them with human-like thinking and cognitive abilities. In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD). We introduce DriveMLM, an LLM-based AD framework that can perform close-loop autonomous driving in realistic simulators. To this end, (1) we bridge the gap between the language decisions and the vehicle control commands by standardizing the decision states according to the off-the-shelf motion planning module. (2) We employ a multimodal LLM (MLLM) to model the behavior planning module of a module AD system, which uses driving rules, user commands, and inputs from various sensors (e.g., camera, lidar) as input and makes driving decisions and provide explanations; This model can plug-and-play in existing AD systems such as Autopilot and Apollo for close-loop driving. (3) We design an effective data engine to collect a dataset that includes decision state and corresponding explanation annotation for model training and evaluation. We conduct extensive experiments and show that replacing the decision-making modules of the Autopilot and Apollo with DriveMLM resulted in significant improvements of 3.2 and 4.7 points on the CARLA Town05 Long respectively, demonstrating the effectiveness of our model. We hope this work can serve as a baseline for autonomous driving with LLMs.
Cited by
- GamiBench: Evaluating Spatial Reasoning and 2D-to-3D Planning Capabilities of MLLMs with Origami Folding Tasks
- RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic
- RESPOND: Risk-Enhanced Structured Pattern for LLM-driven Online Node-level Decision-making
- NL2CA: Auto-formalizing Cognitive Decision-Making from Natural Language Using an Unsupervised CriticNL2LTL Framework
- Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation
- Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
- MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion
- DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
- UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving
- OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic
- RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
- Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving
- CorrectAD: A Self-Correcting Agentic System to Improve End-to-end Planning in Autonomous Driving
- PlanT 2.0: Exposing Biases and Structural Flaws in Closed-Loop Driving
- AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving
- VLDrive: Vision-Augmented Lightweight MLLMs for Efficient Language-grounded Autonomous Driving
- Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
- Large Language Model-assisted Autonomous Vehicle Recovery from Immobilization
- RoCA: Robust Cross-Domain End-to-End Autonomous Driving
- Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
- Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study
- SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Driving
- DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment
- SanDRA: Safe Large-Language-Model-Based Decision Making for Automated Vehicles Using Reachability Analysis
- FuncPoison: Poisoning Function Library to Hijack Multi-agent Autonomous Driving Systems
- Semantic Edge-Cloud Communication for Real-Time Urban Traffic Surveillance with ViT and LLMs over Mobile Networks
- MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Cases
- Large Language Models for Pedestrian Safety: An Application to Predicting Driver Yielding Behavior at Unsignalized Intersections
- iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning
- Are VLMs Ready for Lane Topology Awareness in Autonomous Driving?
- Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
- OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision
- AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Driving
- 2nd Place Solution for CVPR2024 E2E Challenge: End-to-End Autonomous Driving Using Vision Language Model
- CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Driving
- LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
- ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving
- VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization
- Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving
- Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models
- VLMPlanner: Integrating Visual Language Models with Motion Planning
- BEV-LLM: Leveraging Multimodal BEV Maps for Scene Captioning in Autonomous Driving
- ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Driving
- VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Process
- Box-QAymo: Box-Referring VQA Dataset for Autonomous Driving
- A Survey on Vision-Language-Action Models for Autonomous Driving
- ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Models
- Case-based Reasoning Augmented Large Language Model Framework for Decision Making in Realistic Safety-Critical Driving Scenarios
- A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospects
- NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Driving
- A Hierarchical Test Platform for Vision Language Model (VLM)-Integrated Real-World Autonomous Driving
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
- Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving
- Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving
- Hierarchical Question-Answering for Driving Scene Understanding Using Vision-Language Models
- Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models
- Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects
- ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Driving
- Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
- DiffE2E: Rethinking End-to-End Driving with a Hybrid Action Diffusion and Supervised Policy
- Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs
- LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarios
- DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving
- Unlocking Smarter Device Control: Foresighted Planning with a World Model-Driven Code Execution Approach
- VERDI: VLM-Embedded Reasoning for Autonomous Driving
- TinyDrive: Multiscale Visual Question Answering with Selective Token Routing for Autonomous Driving
- AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving
- TS-VLM: Text-Guided SoftSort Pooling for Vision-Language Models in Multi-View Driving Reasoning
- STAR-VLM: Spatiotemporal Grounding Vision-Language Models for Motion and Velocity Estimation via Automotive Radar Supervision
- Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving
- PADriver: Towards Personalized Autonomous Driving
- LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving
- Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAs
- Toward Fully Autonomous Driving: AI, Challenges, Opportunities, and Needs
- UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space
- OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning
- Planning Safety Trajectories with Dual-Phase, Physics-Informed, and Transportation Knowledge-Driven Large Language Models
- Manipulating Multimodal Agents via Cross-Modal Prompt Injection
Related