VAD: Vectorized Scene Representation for Efficient Autonomous Driving
2023/03/21 by Bo Jiang, Jiang, Bo, Shaoyu Chen +17 · 1 voice · 228 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Autonomous Vehicle Technology and Safety #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotic Path Planning Algorithms #Robotics (cs.RO) #cs.CV #cs.RO
paper · pdf · doi:10.48550/arxiv.2303.12077
openalex publication_date 2023/03/21 · arxiv published 2023/03/21 · arxiv updated 2023/08/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
Autonomous driving requires a comprehensive understanding of the surrounding environment for reliable trajectory planning. Previous works rely on dense rasterized scene representation (e.g., agent occupancy and semantic map) to perform planning, which is computationally intensive and misses the instance-level structure information. In this paper, we propose VAD, an end-to-end vectorized paradigm for autonomous driving, which models the driving scene as a fully vectorized representation. The proposed vectorized paradigm has two significant advantages. On one hand, VAD exploits the vectorized agent motion and map elements as explicit instance-level planning constraints which effectively improves planning safety. On the other hand, VAD runs much faster than previous end-to-end planning methods by getting rid of computation-intensive rasterized representation and hand-designed post-processing steps. VAD achieves state-of-the-art end-to-end planning performance on the nuScenes dataset, outperforming the previous best method by a large margin. Our base model, VAD-Base, greatly reduces the average collision rate by 29.0% and runs 2.5x faster. Besides, a lightweight variant, VAD-Tiny, greatly improves the inference speed (up to 9.3x) while achieving comparable planning performance. We believe the excellent performance and the high efficiency of VAD are critical for the real-world deployment of an autonomous driving system. Code and models are available at https://github.com/hustvl/VAD for facilitating future research.
Cited by
- Test-Time Coverage: Test-Conditioned Data Curation for Deployment-Aware Learning
- KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System
- WorldRFT: Latent World Model Planning with Reinforcement Fine-Tuning for Autonomous Driving
- LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
- Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments
- TakeAD: Preference-based Post-optimization for End-to-end Autonomous Driving with Expert Takeover Data
- DVGT: Driving Visual Geometry Transformer
- Driving in Corner Case: A Real-World Adversarial Closed-Loop Evaluation Platform for End-to-End Autonomous Driving
- Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
- LADY: Linear Attention for Autonomous Driving Efficiency without Transformers
- DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
- From Human Intention to Action Prediction: Intention-Driven End-to-End Autonomous Driving
- FutureX: Enhance End-to-End Autonomous Driving via Latent Chain-of-Thought World Model
- Empowering Dynamic Urban Navigation with Stereo and Mid-Level Vision
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
- Closing the Navigation Compliance Gap in End-to-end Autonomous Driving
- UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving
- TrajMoE: Scene-Adaptive Trajectory Planning with Mixture of Experts and Reinforcement Learning
- Mimir: Hierarchical Goal-Driven Diffusion with Uncertainty Propagation for End-to-End Autonomous Driving
- DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving
- Spatial Retrieval Augmented Autonomous Driving
- WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Driving
- NexusFlow: Unifying Disparate Tasks under Partial Supervision via Invertible Flow Networks
- BeLLA: End-to-End Birds Eye View Large Language Assistant for Autonomous Driving
- FreeGen: Feed-Forward Reconstruction-Generation Co-Training for Free-Viewpoint Driving Scene Synthesis
- dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
- CSMapping: Scalable Crowdsourced Semantic Mapping and Topology Inference for Autonomous Driving
- ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understanding
- CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
- nuScenes Revisited: Progress and Challenges in Autonomous Driving
- OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic
- RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving
- TrajDiff: End-to-end Autonomous Driving without Perception Annotation
- SimScale: Learning to Drive via Real-World Simulation at Scale
- SUPER-AD: Semantic Uncertainty-aware Planning for End-to-End Robust Autonomous Driving
- Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving
- LaGen: Towards Autoregressive LiDAR Scene Generation
- AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models
- Map-World: Masked Action planning and Path-Integral World Model for Autonomous Driving
- DeeAD: Dynamic Early Exit of Vision-Language Action for Efficient Autonomous Driving
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
- Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving
- WPT: World-to-Policy Transfer via Online World Model Distillation
- Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
- GuideFlow: Constraint-Guided Flow Matching for Planning in End-to-End Autonomous Driving
- DiffRefiner: Coarse to Fine Trajectory Planning via Diffusion Refinement with Semantic Interaction for End to End Autonomous Driving
- GUIDE: Gaussian Unified Instance Detection for Enhanced Obstacle Perception in Autonomous Driving
- Enhancing End-to-End Autonomous Driving with Risk Semantic Distillaion from VLM
- Enhancing All-to-X Backdoor Attacks with Optimized Target Class Mapping
- DAP: A Discrete-token Autoregressive Planner for Autonomous Driving
- CorrectAD: A Self-Correcting Agentic System to Improve End-to-end Planning in Autonomous Driving
- ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Experts
- Argus: Resilience-Oriented Safety Assurance Framework for End-to-End ADSs
- Embodied Cognition Augmented End2End Autonomous Driving
- SafeDrive: Fine-Grained Safety Reasoning for End-to-End Driving in a Sparse World
- Dataset Safety in Autonomous Driving: Requirements, Risks, and Assurance
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving
- HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Driving
- LaneDiffusion: Improving Centerline Graph Learning via Prior Injected BEV Feature Generation
- SAFe-Copilot: Unified Shared Autonomy Framework
- UniLION: Towards Unified Autonomous Driving Model with Linear Group RNNs
- Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
- Constructing the Umwelt: Cognitive Planning through Belief-Intent Co-Evolution
- Beyond Imitation: Constraint-Aware Trajectory Generation with Flow Matching For End-to-End Autonomous Driving
- Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
- Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Driving
- RoCA: Robust Cross-Domain End-to-End Autonomous Driving
- ZTRS: Zero-Imitation End-to-end Autonomous Driving with Trajectory Scoring
- SynAD: Enhancing Real-World End-to-End Autonomous Driving Models through Synthetic Data Integration
- VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting
- From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction
- Perfect Prediction or Plenty of Proposals? What Matters Most in Planning for Autonomous Driving
- Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks
- VDRive: Leveraging Reinforced VLA and Diffusion Policy for End-to-end Autonomous Driving
- SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queries
- SimpleVSF: VLM-Scoring Fusion for Trajectory Prediction of End-to-End Autonomous Driving
- DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment
- Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models
- MACE: Mixture-of-Experts Accelerated Coordinate Encoding for Large-Scale Scene Localization and Rendering
- CoIRL-AD: Collaborative-Competitive Imitation-Reinforcement Learning in Latent World Models for Autonomous Driving
- DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving
- Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modeling
- Stability Under Scrutiny: Benchmarking Representation Paradigms for Online HD Mapping
- Scalable Offline Metrics for Autonomous Driving
- ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving
- HyPlan: Hybrid Learning-Assisted Planning Under Uncertainty for Safe Autonomous Driving
- Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models
- More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
- RAP: 3D Rasterization Augmented End-to-End Planning
- DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation
- BridgeDrive: Diffusion Bridge Policy for Closed-Loop Trajectory Planning in Autonomous Driving
- BEV-VLM: Trajectory Planning via Unified BEV Abstraction
- Autoregressive End-to-End Planning with Time-Invariant Spatial Alignment and Multi-Objective Policy Refinement
- AnchDrive: Bootstrapping Diffusion Policies with Hybrid Trajectory Anchors for End-to-End Driving
- OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Driving
- Driving on Registers
- PIE: Perception and Interaction Enhanced End-to-End Motion Planning for Autonomous Driving
- TinyBEV: Cross Modal Knowledge Distillation for Efficient Multi Task Bird's Eye View Perception and Planning
- DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
- RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generation
- CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refine
- AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Driving
- MAP: End-to-End Autonomous Driving with Map-Assisted Planning
- FlowDrive: Energy Flow Field for End-to-End Autonomous Driving
- Bridging Perception and Planning: Towards End-to-End Planning for Signal Temporal Logic Tasks
- Embodied Navigation Foundation Model
- M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments
- LLM-based Human-like Traffic Simulation for Self-driving Tests
- Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
- Asymmetry Vulnerability and Physical Attacks on Online Map Construction for Autonomous Driving
- OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision
- A Knowledge-Driven Diffusion Policy for End-to-End Autonomous Driving Based on Expert Routing
- OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction
- KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Models
- Can the Waymo Open Motion Dataset Support Realistic Behavioral Modeling? A Validation Study with Naturalistic Trajectories
- AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Driving
- CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Driving
- To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software
- Self-Supervised Sparse Sensor Fusion for Long Range Perception
- LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
- EvoPSF: Online Evolution of Autonomous Driving Models via Planning-State Feedback
- ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving
- SpaRC-AD: A Baseline for Radar-Camera Fusion in End-to-End Autonomous Driving
- ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Scene Reconstruction
- Risk Map As Middleware: Towards Interpretable Cooperative End-to-end Autonomous Driving for Risk-Aware Planning
- ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting
- GMF-Drive: Gated Mamba Fusion with Spatial-Aware BEV Representation for End-to-End Autonomous Driving
- ReSim: Reliable World Simulation for Autonomous Driving
- DistillDrive: End-to-End Multi-Mode Autonomous Driving Distillation by Isomorphic Hetero-Source Planning Model
- IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model
- TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction
- RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Case
- EvaDrive: Evolutionary Adversarial Policy Optimization for End-to-End Autonomous Driving
- MetAdv: A Unified and Interactive Adversarial Testing Platform for Autonomous Driving
- Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving
- DiffSemanticFusion: Semantic Raster BEV Fusion for Autonomous Driving via Online HD Map Diffusion
- FASTopoWM: Fast-Slow Lane Segment Topology Reasoning with Latent World Models
- FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning
- Research Challenges and Progress in the End-to-End V2X Cooperative Autonomous Driving Competition
- VLMPlanner: Integrating Visual Language Models with Motion Planning
- VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving
- A Concept for Efficient Scalability of Automated Driving Allowing for Technical, Legal, Cultural, and Ethical Differences
- Advancing Complex Wide-Area Scene Understanding with Hierarchical Coresets Selection
- GEMINUS: Dual-aware Global and Scene-Adaptive Mixture-of-Experts for End-to-End Autonomous Driving
- PRIX: Learning to Plan from Raw Pixels for End-to-End Autonomous Driving
- ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Driving
- From Binary to Semantic: Utilizing Large-Scale Binary Occupancy Data for 3D Semantic Occupancy Prediction
- Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Driving
- Self-supervised Pretraining for Integrated Prediction and Planning of Automated Vehicles
- Online Navigation Refinement: Achieving Lane-Level Guidance by Associating Standard-Definition and Online Perception Maps
- Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planning
- NavigScene: Bridging Local Perception and Global Navigation for Beyond-Visual-Range Autonomous Driving
- Breaking Imitation Bottlenecks: Reinforced Diffusion Powers Diverse Trajectory Generation
- VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Process
- World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model
- Epona: Autoregressive Diffusion World Model for Autonomous Driving
- A Survey on Vision-Language-Action Models for Autonomous Driving
- DriveMRP: Enhancing Vision-Language Models with Synthetic Motion Data for Motion Risk Prediction
- ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Models
- BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
- ParkFormer: A Transformer-Based Parking Policy with Goal Embedding and Pedestrian-Aware Control
- ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
- NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Driving
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
- FocalAD: Local Motion Planning for End-to-End Autonomous Driving
- Efficient Multi-Camera Tokenization with Triplanes for End-to-End Driving
- Generalized Trajectory Scoring for End-to-end Multimodal Planning
- DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning
- DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving
- HMVLM: Multistage Reasoning-Enhanced Vision-Language Model for Long-Tailed Driving Scenarios
- RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulation
- RealDrive: Retrieval-Augmented Driving with Diffusion Models
- SAH-Drive: A Scenario-Aware Hybrid Planner for Closed-Loop Vehicle Trajectory Generation
- S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation
- Autoregressive Meta-Actions for Unified Controllable Trajectory Generation
- Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models
- HMAD: Advancing E2E Driving with Anchored Offset Proposals and Simulation-Supervised Multi-target Scoring
- From Failures to Fixes: LLM-Driven Scenario Repair for Self-Evolving Autonomous Driving
- CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Driving
- GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Driving
- DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving
- ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Driving
- DriveCamSim: Generalizable Camera Simulation via Explicit Camera Modeling for Autonomous Driving
- DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving
- DiffE2E: Rethinking End-to-End Driving with a Hybrid Action Diffusion and Supervised Policy
- DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving
- Echo Planning for Autonomous Driving: From Current Observations to Future Trajectories and Back
- Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs
- DecoupleGS: Interactive 3D Gaussian Splatting for End-to-End Autonomous Driving Testing
- FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
- RealEngine: Simulating Autonomous Driving in Realistic Context
- SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving
- Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)
- DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving
- CodeMerge: Codebook-Guided Model Merging for Robust Test-Time Adaptation in Autonomous Driving
- Challenger: Affordable Adversarial Driving Video Generation
- VERDI: VLM-Embedded Reasoning for Autonomous Driving
- iPad: Iterative Proposal-centric End-to-End Autonomous Driving
- Generative AI for Autonomous Driving: A Review
- InstanceBEV: Unifying Instance and BEV Representation for 3D Panoptic Segmentation
- Safety2Drive: Safety-Critical Scenario Benchmark for the Evaluation of Autonomous Driving
- PRISM: Privileged Probabilistic Latent Supervision for End-to-End Autonomous Driving Motion Planning
- Scene-Adaptive Motion Planning with Explicit Mixture of Experts and Interaction-Oriented Optimization
- SEPT: Standard-Definition Map Enhanced Scene Perception and Topology Reasoning for Autonomous Driving
- SafePath: Conformal Prediction for Safe LLM-Based Autonomous Navigation
- Latent-Centroid Steering: Single-Pass Classifier-Free Guidance for Command-Aligned Autonomous Driving
- TransDiffuser: Diverse Trajectory Generation with Decorrelated Multi-modal Representation for End-to-end Autonomous Driving
- M3CAD: Towards Generic Cooperative Autonomous Driving Benchmark
- DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Driving
- DSDrive: Distilling Large Language Model for Lightweight End-to-End Autonomous Driving with Unified Reasoning and Planning
- PADriver: Towards Personalized Autonomous Driving
- UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
- CADET: Physics-Grounded Causal Auditing and Training-Free Deconfounding of End-to-End Driving Planners
- Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAs
- Scaling Self-Play for End-to-End Driving
- Unleashing the Potential of Diffusion Models for End-to-End Autonomous Driving
- ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Driving
- Toward Fully Autonomous Driving: AI, Challenges, Opportunities, and Needs
- SUV: Future Scene Understanding as Video Generation for End-to-End Driving
- When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit
- Data Scaling Laws for End-to-End Autonomous Driving
- OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning
- Exposing the Copycat Problem of Imitation-based Planner: A Novel Closed-Loop Simulator, Causal Benchmark and Joint IL-RL Baseline
- Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Prediction
- UncAD: Towards Safe End-to-end Autonomous Driving via Online Map Uncertainty
- Fully Unified Motion Planning for End-to-End Autonomous Driving
- MapTCL: Temporal Consistency Learning via Bidirectional Alignment for Vectorized HD Map Construction
- Drive in Corridors: Enhancing the Safety of End-to-end Autonomous Driving via Corridor Learning and Planning
Discussions
Related