Vision Language Models in Autonomous Driving: A Survey and Outlook

2023/10/22 by Xingcheng Zhou, Zhou, Xingcheng, Mingyu Liu +9 · 33 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Text and Document Classification Technologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.14414

openalex publication_date 2023/10/22 · openalex created_date 2023/10/25 · openalex updated_date 2026/07/28

Abstract

The applications of Vision-Language Models (VLMs) in the field of Autonomous Driving (AD) have attracted widespread attention due to their outstanding performance and the ability to leverage Large Language Models (LLMs). By incorporating language data, driving systems can gain a better understanding of real-world environments, thereby enhancing driving safety and efficiency. In this work, we present a comprehensive and systematic survey of the advances in vision language models in this domain, encompassing perception and understanding, navigation and planning, decision-making and control, end-to-end autonomous driving, and data generation. We introduce the mainstream VLM tasks in AD and the commonly utilized metrics. Additionally, we review current studies and applications in various areas and summarize the existing language-enhanced autonomous driving datasets thoroughly. Lastly, we discuss the benefits and challenges of VLMs in AD and provide researchers with the current research gaps and future trends.

Cited by

• Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
• Defect-aware Hybrid Prompt Optimization via Progressive Tuning for Zero-Shot Multi-type Anomaly Detection and Segmentation
• RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies
• New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehicles
• RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving
• Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
• TrafficLens: Multi-Camera Traffic Video Analysis Using LLMs
• RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System
• MM-Telco: Benchmarks and Multimodal Large Language Models for Telecom Applications
• Prompt-Driven Domain Adaptation for End-to-End Autonomous Driving via In-Context RL
• Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
• A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models
• Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Driving
• Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
• Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
• SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Driving
• SAVANT: Semantic Analysis with Vision-Augmented Anomaly deTection
• DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment
• Towards General Urban Monitoring with Vision-Language Models: A Review, Evaluation, and a Research Agenda
• Task-Aware Resolution Optimization for Visual Large Language Models
• NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Driving
• How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective
• InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning
• Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
• A Knowledge-Driven Diffusion Policy for End-to-End Autonomous Driving Based on Expert Routing
• CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Driving
• GENNAV: Polygon Mask Generation for Generalized Referring Navigable Regions
• GM-Skip: Metric-Guided Transformer Block Skipping for Efficient Vision-Language Models
• Traffic-R1: Reinforced LLMs Bring Human-Like Reasoning to Traffic Signal Control Systems
• Context-based Motion Retrieval using Open Vocabulary Methods for Autonomous Driving
• LRR-Bench: Left, Right or Rotate? Vision-Language models Still Struggle With Spatial Understanding Tasks
• Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues

Related

• A Survey on Vision-Language-Action Models for Autonomous Driving
• CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving
• LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
• A Survey on Multimodal Large Language Models for Autonomous Driving
• Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving