OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model

2025/03/30 by Xingcheng Zhou, Zhou, Xingcheng, Han, Xuyuan +6 · 56 citations
Computer Science · Psychology · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Social Robot Interaction and HRI

paper · pdf · doi:10.48550/arxiv.2503.23463

openalex publication_date 2025/03/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present OpenDriveVLA, a Vision Language Action model designed for end-to-end autonomous driving, built upon open-source large language models. OpenDriveVLA generates spatially grounded driving actions by leveraging multimodal inputs, including 2D and 3D instance-aware visual representations, ego vehicle states, and language commands. To bridge the modality gap between driving visual representations and language embeddings, we introduce a hierarchical vision language alignment process, projecting both 2D and 3D structured visual tokens into a unified semantic space. Furthermore, we incorporate structured agent environment ego interaction modeling into the autoregressive decoding process, enabling the model to capture fine-grained spatial dependencies and behavior-aware dynamics critical for reliable trajectory planning. Extensive experiments on the nuScenes dataset demonstrate that OpenDriveVLA achieves state-of-the-art results across open-loop trajectory planning and driving-related question answering tasks. Qualitative analyses further illustrate its capability to follow high-level driving commands and generate trajectories under challenging scenarios, highlighting its potential for next-generation end-to-end autonomous driving.

Cited by

• ObsDriveBench: Benchmarking Multimodal Understanding under Adverse Weather with Observability Awareness
• Development of Vision-Language Model-based GNSS Spoofing Detection for Autonomous Vehicle Navigation
• GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understanding
• LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
• Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation
• DVGT: Driving Visual Geometry Transformer
• SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
• DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
• From Human Intention to Action Prediction: A Comprehensive Benchmark for Intention-driven End-to-End Autonomous Driving
• SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
• Latent Chain-of-Thought World Modeling for End-to-End Driving
• WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Driving
• E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
• CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
• OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic
• SUPER-AD: Semantic Uncertainty-aware Planning for End-to-End Robust Autonomous Driving
• WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
• Enhancing End-to-End Autonomous Driving with Risk Semantic Distillaion from VLM
• DAP: A Discrete-token Autoregressive Planner for Autonomous Driving
• Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
• EBT-Policy: Energy Unlocks Emergent Physical Reasoning Capabilities
• Constructing the Umwelt: Cognitive Planning through Belief-Intent Co-Evolution
• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
• Endowing GPT-4 with a Humanoid Body: Building the Bridge Between Off-the-Shelf VLMs and the Physical World
• A Survey on Efficient Vision-Language-Action Models
• VDRive: Leveraging Reinforced VLA and Diffusion Policy for End-to-end Autonomous Driving
• DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving
• Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
• Bring the Apple, Not the Sofa: Impact of Irrelevant Context in Embodied AI Commands on VLA Models
• BEV-VLM: Trajectory Planning via Unified BEV Abstraction
• MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Cases
• FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Models
• Pure Vision Language Action (VLA) Models: A Comprehensive Survey
• CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refine
• Maps for Autonomous Driving: Full-process Survey and Frontiers
• Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
• OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision
• AutoDrive-R²: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Driving
• ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving
• IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model
• Static and Plugged: Make Embodied Evaluation Simple
• FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning
• VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving
• LaViPlan : Language-Guided Visual Path Planning with RLVR
• Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving
• Agentic Vehicles for Human-Centered Mobility: Definition, Prospects, and Synergistic Co-Development with Vehicle Autonomy
• A Survey on Vision-Language-Action Models for Autonomous Driving
• V2X-REALM: Vision-Language Model-Based Robust End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modeling
• NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Driving
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
• Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving
• DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models
• STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving

Related

• A Survey on Vision-Language-Action Models for Autonomous Driving
• CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
• Doe-1: Closed-Loop Autonomous Driving with Large World Model
• VLA-R: Vision-Language Action Retrieval toward Open-World End-to-End Autonomous Driving