OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving

2024/12/19 by Chengyuan Qian, Xing, Shuo, Yu‐Ping Wang +10 · 23 citations
Engineering · #Transportation and Mobility Innovations

paper · pdf · doi:10.48550/arxiv.2412.15208

Abstract

Since the advent of Multimodal Large Language Models (MLLMs), they have made a significant impact across a wide range of real-world applications, particularly in Autonomous Driving (AD). Their ability to process complex visual data and reason about intricate driving scenarios has paved the way for a new paradigm in end-to-end AD systems. However, the progress of developing end-to-end models for AD has been slow, as existing fine-tuning methods demand substantial resources, including extensive computational power, large-scale datasets, and significant funding. Drawing inspiration from recent advancements in inference computing, we propose OpenEMMA, an open-source end-to-end framework based on MLLMs. By incorporating the Chain-of-Thought reasoning process, OpenEMMA achieves significant improvements compared to the baseline when leveraging a diverse range of MLLMs. Furthermore, OpenEMMA demonstrates effectiveness, generalizability, and robustness across a variety of challenging driving scenarios, offering a more efficient and effective approach to autonomous driving. We release all the codes in https://github.com/taco-group/OpenEMMA.

Cited by

• LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
• WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Driving
• NexusFlow: Unifying Disparate Tasks under Partial Supervision via Invertible Flow Networks
• WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Driving
• E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving
• MindDrive: An All-in-One Framework Bridging World Models and Vision-Language Model for End-to-End Autonomous Driving
• MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
• RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving
• From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models
• WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios
• Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Driving
• Advancing Off-Road Autonomous Driving: The Large-Scale ORAD-3D Dataset and Comprehensive Benchmarks
• Addressing Corner Cases in Autonomous Driving: A World Model-based Approach with Mixture of Experts and LLMs
• SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Driving
• RDD: Retrieval-Based Demonstration Decomposer for Planner Alignment in Long-Horizon Tasks
• Q-Router: Agentic Video Quality Assessment with Expert Model Routing and Artifact Localization
• AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Driving
• Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
• Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation
• KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Models
• DriveQA: Passing the Driving Knowledge Test
• ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving

Related

• LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving
• X-Driver: Explainable Autonomous Driving with Vision-Language Models
• DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
• Application of Multimodal Large Language Models in Autonomous Driving
• EMMA: End-to-End Multimodal Model for Autonomous Driving