dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning

2025/12/04 by Ma, Yingzi, Cao, Yulong, Ding, Wenhao +6 · 3 citations
Computer Science · Engineering · #Autonomous Vehicle Technology and Safety #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics

paper · doi:10.48550/arxiv.2512.04459

openalex publication_date 2025/12/04 · openalex created_date 2025/12/06 · openalex updated_date 2026/07/28

Abstract

The autonomous driving community is increasingly focused on addressing the challenges posed by out-of-distribution (OOD) driving scenarios. A dominant research trend seeks to enhance end-to-end (E2E) driving systems by integrating vision-language models (VLMs), leveraging their rich world knowledge and reasoning abilities to improve generalization across diverse environments. However, most existing VLMs or vision-language agents (VLAs) are built upon autoregressive (AR) models. In this paper, we observe that existing AR-based VLMs -- limited by causal attention and sequential token generation -- often fail to maintain consistency and controllability between high-level reasoning and low-level planning. In contrast, recent discrete diffusion VLMs equipped with bidirectional attention exhibit superior controllability and reliability through iterative denoising. Building on these observations, we introduce dVLM-AD, a diffusion-based vision-language model that unifies perception, structured reasoning, and low-level planning for end-to-end driving. Evaluated on nuScenes and WOD-E2E, dVLM-AD yields more consistent reasoning-action pairs and achieves planning performance comparable to existing driving VLM/VLA systems despite a modest backbone, outperforming AR-based baselines with a 9 percent improvement in behavior-trajectory consistency and a 6 percent increase in RFS on long-tail WOD-E2E scenarios. These results suggest a controllable and reliable pathway for scalable end-to-end driving.

Citations

• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
• WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios
• Path Drift in Large Reasoning Models:How First-Person Commitments Override Safety
• dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thought
• Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving
• AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Driving
• LLaDA-VLA: Vision Language Diffusion Action Models
• Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models
• AutoDrive-R²: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Driving
• CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification
• Diffusion Language Models Know the Answer Before Decoding
• Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policies
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• Dream 7B: Diffusion Large Language Models
• IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model
• Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models
• Unveiling the Potential of Diffusion Large Language Model in Controllable Generation
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
• ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
• DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models
• LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Tasks
• Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models
• ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledge
• DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving
• LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
• FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
• Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding
• MMaDA: Multimodal Large Diffusion Language Models
• LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving
• NORA: A Small Open-Sourced Generalist Vision Language Action Model for Embodied Tasks
• π_0.5: a Vision-Language-Action Model with Open-World Generalization
• OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
• DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• Qwen2.5-VL Technical Report
• H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking
• Large Language Diffusion Models
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision
• EMMA: End-to-End Multimodal Model for Autonomous Driving
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving
• LLaVA-OneVision: Easy Visual Task Transfer
• Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving
• Simple and Effective Masked Diffusion Language Models
• Simplified and Generalized Masked Diffusion for Discrete Data
• DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving
• DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
• DriveLM: Driving with Graph Visual Question Answering
• Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
• Improved Baselines with Visual Instruction Tuning
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• Likelihood-Based Diffusion Language Models
• Visual Instruction Tuning
• VAD: Vectorized Scene Representation for Efficient Autonomous Driving
• Planning-oriented Autonomous Driving
• Diffusion-LM Improves Controllable Text Generation
• Flamingo: a Visual Language Model for Few-Shot Learning
• Structured Denoising Diffusion Models in Discrete State-Spaces
• nuScenes: A multimodal dataset for autonomous driving

Cited by

• Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
• UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving

Related

• ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving
• VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision
• ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
• DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving
• DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking