Drive Like a Human: Rethinking Autonomous Driving with Large Language Models

2023/07/14 by Daocheng Fu, Xin Li, Fu, Daocheng +11 · 1 voice · 40 citations
Computer Science · Psychology · #Artificial intelligence #Code (set theory) #Cognitive psychology #Comprehension #Computer science #Computer security #Face (sociological concept) #Human–computer interaction #Interpretation (philosophy) #Key (lock) #Memorization #Modular design #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Programming language #Psychology #Topic Modeling #cs.CL #cs.RO

paper · pdf · doi:10.48550/arxiv.2307.07162

openalex publication_date 2023/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we explore the potential of using a large language model (LLM) to understand the driving environment in a human-like manner and analyze its ability to reason, interpret, and memorize when facing complex scenarios. We argue that traditional optimization-based and modular autonomous driving (AD) systems face inherent performance limitations when dealing with long-tail corner cases. To address this problem, we propose that an ideal AD system should drive like a human, accumulating experience through continuous driving and using common sense to solve problems. To achieve this goal, we identify three key abilities necessary for an AD system: reasoning, interpretation, and memorization. We demonstrate the feasibility of employing an LLM in driving scenarios by building a closed-loop system to showcase its comprehension and environment-interaction abilities. Our extensive experiments show that the LLM exhibits the impressive ability to reason and solve long-tailed cases, providing valuable insights for the development of human-like autonomous driving. The related code are available at https://github.com/PJLab-ADG/DriveLikeAHuman .

Cited by

• VLN-AVP: Zero-Shot Vision-Language Navigation with Hybrid Long-Short-Term Memory for Autonomous Valet Parking
• Leaner Transformers: More Heads, Less Depth
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
• OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving
• A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach
• SocialDriveGen: Generating Diverse Traffic Scenarios with Controllable Social Interactions
• MTR-VP: Towards End-to-End Trajectory Planning through Context-Driven Image Encoding and Multiple Trajectory Prediction
• Geometrically-Constrained Agent for Spatial Reasoning
• TrafficLens: Multi-Camera Traffic Video Analysis Using LLMs
• WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
• Visionary Co-Driver: Enhancing Driver Perception of Potential Risks with LLM and HUD
• Are LLMs The Way Forward? A Case Study on LLM-Guided Reinforcement Learning for Decentralized Autonomous Driving
• AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving
• VLDrive: Vision-Augmented Lightweight MLLMs for Efficient Language-grounded Autonomous Driving
• Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
• DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment
• Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Driving
• DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation
• Agentic AI Reasoning for Mobile Edge General Intelligence: Fundamentals, Approaches, and Directions
• iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning
• KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Models
• DriveQA: Passing the Driving Knowledge Test
• Driving Style Recognition Like an Expert Using Semantic Privileged Information from Large Language Models
• Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
• A Survey on Deep Multi-Task Learning in Connected Autonomous Vehicles
• PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behavior
• AD²-Bench: A Hierarchical CoT Benchmark for MLLM in Autonomous Driving under Adverse Conditions
• A Survey on Vision-Language-Action Models for Autonomous Driving
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
• DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models
• Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving
• CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
• ROAD: Responsibility-Oriented Reward Design for Reinforcement Learning in Autonomous Driving
• Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects
• RealEngine: Simulating Autonomous Driving in Realistic Context
• SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving
• AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving
• Generative AI for Autonomous Driving: A Review
• PADriver: Towards Personalized Autonomous Driving
• LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecture

Discussions

• Rethinking Autonomous Driving with Large Language Models [hn, 39 points, 42 comments]

Related

• Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles
• Large Language Models for Human-like Autonomous Driving: A Survey
• LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving
• A Language Agent for Autonomous Driving
• DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving