Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects
2025/05/26 by Cui, Yixin, Lin, Haotian, Yang, Shuo +3 · 6 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO)
paper · doi:10.48550/arxiv.2505.20223
Abstract
The rapid evolution of large language models in natural language processing has substantially elevated their semantic understanding and logical reasoning capabilities. Such proficiencies have been leveraged in autonomous driving systems, contributing to significant improvements in system performance. Models such as OpenAI o1 and DeepSeek-R1, leverage Chain-of-Thought (CoT) reasoning, an advanced cognitive method that simulates human thinking processes, demonstrating remarkable reasoning capabilities in complex tasks. By structuring complex driving scenarios within a systematic reasoning framework, this approach has emerged as a prominent research focus in autonomous driving, substantially improving the system's ability to handle challenging cases. This paper investigates how CoT methods improve the reasoning abilities of autonomous driving models. Based on a comprehensive literature review, we present a systematic analysis of the motivations, methodologies, challenges, and future research directions of CoT in autonomous driving. Furthermore, we propose the insight of combining CoT with self-learning to facilitate self-evolution in driving systems. To ensure the relevance and timeliness of this study, we have compiled a dynamic repository of literature and open-source projects, diligently updated to incorporate forefront developments. The repository is publicly available at https://github.com/cuiyx1720/Awesome-CoT4AD.
Citations
- AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving
- X-Driver: Explainable Autonomous Driving with Vision-Language Models
- DriveAgent: Multi-Agent Structured Reasoning with LLM and Multimodal Sensor Fusion for Autonomous Driving
- LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving
- PRIMEDrive-CoT: A Precognitive Chain-of-Thought Framework for Uncertainty-Aware Object Interaction in Driving Scene Scenario
- OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
- A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond
- ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation
- DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding
- Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
- CoT-Drive: Efficient Motion Forecasting for Autonomous Driving with LLMs and Chain-of-Thought Prompting
- AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning
- CoT-VLM4Tar: Chain-of-Thought Guided Vision-Language Models for Traffic Anomaly Resolution
- Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactions
- Chain of Draft: Thinking Faster by Writing Less
- From System 1 to System 2: A Survey of Reasoning Large Language Models
- Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learning
- The Role of World Models in Shaping Autonomous Driving: A Comprehensive Survey
- Latent Thought Models with Variational Bayes Inference-Time Computation
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
- LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinking
- Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios
- Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought
- Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
- SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Driving
- Visual Large Language Models for Generalized and Specialized Applications
- OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving
- AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving
- SafeDrive: Knowledge- and Data-Driven Risk-Sensitive Decision-Making for Autonomous Vehicles with Large Language Models
- WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model
- CALMM-Drive: Confidence-Aware Autonomous Driving with Large Multimodal Model
- PKRD-CoT: A Unified Chain-of-thought Prompting for Multi-Modal Large Language Models in Autonomous Driving
- On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
- EMMA: End-to-End Multimodal Model for Autonomous Driving
- Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving
- GPT-4o System Card
- Supervised Chain of Thought
- CoT-TL: Low-Resource Temporal Knowledge Representation of Planning Instructions Using Chain-of-Thought Reasoning
- Enhancing Advanced Visual Reasoning Ability of Large Language Models
- Towards Interactive and Learnable Cooperative Driving Automation: a Large Language Model-Driven Decision-Making Framework
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
- Making Large Language Models Better Planners with Reasoning-Decision Alignment
- Large Language Models for Human-like Autonomous Driving: A Survey
- KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Models
- OpenVLA: An Open-Source Vision-Language-Action Model
- PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning
- A Survey on Large Language Models for Code Generation
- M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought
- Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving
- From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step
- LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Models
- DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
- DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
- A Survey for Foundation Models in Autonomous Driving
- MM-LLMs: Recent Advances in MultiModal Large Language Models
- A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook
- DriveLM: Driving with Graph Visual Question Answering
- LingoQA: Visual Question Answering for Autonomous Driving
- DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
- LMDrive: Closed-Loop End-to-End Driving with Large Language Models
- NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations
- Prospective Role of Foundation Models in Advancing Autonomous Vehicles
- Towards Knowledge-driven Autonomous Driving
- Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving
- Dolphins: Multimodal Language Model for Driving
- A Survey on Multimodal Large Language Models for Autonomous Driving
- A Language Agent for Autonomous Driving
- LLM4Drive: A Survey of Large Language Models for Autonomous Driving
- Vision Language Models in Autonomous Driving: A Survey and Outlook
- Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles
- Towards Better Chain-of-Thought Prompting Strategies: A Survey
- LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving
- Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
- Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving
- DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
- GPT-Driver: Learning to Drive with GPT
- DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models
- Qwen Technical Report
- Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning
- Language Prompt for Autonomous Driving
- Graph of Thoughts: Solving Elaborate Problems with Large Language Models
- Explaining Autonomous Driving Actions with Visual Question Answering
- Drive Like a Human: Rethinking Autonomous Driving with Large Language Models
- Preference Ranking Optimization for Human Alignment
- End-to-end Autonomous Driving: Challenges and Frontiers
- Let's Verify Step by Step
- NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- Visual Instruction Tuning
- When Brain-inspired AI Meets AGI
- Referring Multi-Object Tracking
- LLaMA: Open and Efficient Foundation Language Models
- Multimodal Chain-of-Thought Reasoning in Language Models
- Planning-oriented Autonomous Driving
- Towards Reasoning in Large Language Models: A Survey
- Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
- On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
- ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
- DRAMA: Joint Risk Localization and Captioning in Driving
- Language models show human-like content effects on reasoning tasks
- Large Language Models are Zero-Shot Reasoners
- Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Survey of Hallucination in Natural Language Generation
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- AUTO-DISCERN: Autonomous Driving Using Common Sense Reasoning
- End-to-End Urban Driving by Imitating a Reinforcement Learning Coach
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
- One Million Scenes for Autonomous Driving: ONCE Dataset
- Rule-based Optimal Control for Autonomous Driving
- CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptions
- A Reinforcement Learning-Based Adaptive Path Tracking Approach for Autonomous Driving
- Language Models are Few-Shot Learners
- Explainable Object-induced Action Decision for Autonomous Vehicles
- Deep Reinforcement Learning for Autonomous Driving: A Survey
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset
- nuScenes: A multimodal dataset for autonomous driving
- CityFlow: A City-Scale Benchmark for Multi-Target Multi-Camera Vehicle Tracking and Re-Identification
- The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems
- Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation
- BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
- BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
- A critical evaluation of the Next Generation Simulation (NGSIM) vehicle trajectory dataset
- CARLA: An Open Urban Driving Simulator
- Attention Is All You Need
- Curiosity-driven Exploration by Self-supervised Prediction
- CIDEr: Consensus-based Image Description Evaluation
- Vision meets robotics: The KITTI dataset
- Congested Traffic States in Empirical Observations and Microscopic Simulations
- LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
- WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving
- OpenAI o1 System Card
Cited by
Related