VAD: Vectorized Scene Representation for Efficient Autonomous Driving
2023/03/21 by Bo Jiang, Jiang, Bo, Shaoyu Chen +17 · 139 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Autonomous Vehicle Technology and Safety #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotic Path Planning Algorithms #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.2303.12077
openalex publication_date 2023/03/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
Autonomous driving requires a comprehensive understanding of the surrounding environment for reliable trajectory planning. Previous works rely on dense rasterized scene representation (e.g., agent occupancy and semantic map) to perform planning, which is computationally intensive and misses the instance-level structure information. In this paper, we propose VAD, an end-to-end vectorized paradigm for autonomous driving, which models the driving scene as a fully vectorized representation. The proposed vectorized paradigm has two significant advantages. On one hand, VAD exploits the vectorized agent motion and map elements as explicit instance-level planning constraints which effectively improves planning safety. On the other hand, VAD runs much faster than previous end-to-end planning methods by getting rid of computation-intensive rasterized representation and hand-designed post-processing steps. VAD achieves state-of-the-art end-to-end planning performance on the nuScenes dataset, outperforming the previous best method by a large margin. Our base model, VAD-Base, greatly reduces the average collision rate by 29.0% and runs 2.5x faster. Besides, a lightweight variant, VAD-Tiny, greatly improves the inference speed (up to 9.3x) while achieving comparable planning performance. We believe the excellent performance and the high efficiency of VAD are critical for the real-world deployment of an autonomous driving system. Code and models are available at https://github.com/hustvl/VAD for facilitating future research.
Cited by
- Test-Time Coverage: Test-Conditioned Data Curation for Deployment-Aware Learning
- KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System
- WorldRFT: Latent World Model Planning with Reinforcement Fine-Tuning for Autonomous Driving
- LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
- Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments
- TakeAD: Preference-based Post-optimization for End-to-end Autonomous Driving with Expert Takeover Data
- DVGT: Driving Visual Geometry Transformer
- Driving in Corner Case: A Real-World Adversarial Closed-Loop Evaluation Platform for End-to-End Autonomous Driving
- Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
- LADY: Linear Attention for Autonomous Driving Efficiency without Transformers
- DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
- From Human Intention to Action Prediction: A Comprehensive Benchmark for Intention-driven End-to-End Autonomous Driving
- FutureX: Enhance End-to-End Autonomous Driving via Latent Chain-of-Thought World Model
- Empowering Dynamic Urban Navigation with Stereo and Mid-Level Vision
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
- Closing the Navigation Compliance Gap in End-to-end Autonomous Driving
- UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving
- TrajMoE: Scene-Adaptive Trajectory Planning with Mixture of Experts and Reinforcement Learning
- Mimir: Hierarchical Goal-Driven Diffusion with Uncertainty Propagation for End-to-End Autonomous Driving
- DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving
- Spatial Retrieval Augmented Autonomous Driving
- WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Driving
- NexusFlow: Unifying Disparate Tasks under Partial Supervision via Invertible Flow Networks
- BeLLA: End-to-End Birds Eye View Large Language Assistant for Autonomous Driving
- FreeGen: Feed-Forward Reconstruction-Generation Co-Training for Free-Viewpoint Driving Scene Synthesis
- dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
- CSMapping: Scalable Crowdsourced Semantic Mapping and Topology Inference for Autonomous Driving
- ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understanding
- CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
- nuScenes Revisited: Progress and Challenges in Autonomous Driving
- OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic
- RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving
- TrajDiff: End-to-end Autonomous Driving without Perception Annotation
- SimScale: Learning to Drive via Real-World Simulation at Scale
- SUPER-AD: Semantic Uncertainty-aware Planning for End-to-End Robust Autonomous Driving
- Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving
- LaGen: Towards Autoregressive LiDAR Scene Generation
- AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models
- Map-World: Masked Action planning and Path-Integral World Model for Autonomous Driving
- DeeAD: Dynamic Early Exit of Vision-Language Action for Efficient Autonomous Driving
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
- Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving
- WPT: World-to-Policy Transfer via Online World Model Distillation
- Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
- GuideFlow: Constraint-Guided Flow Matching for Planning in End-to-End Autonomous Driving
- DiffRefiner: Coarse to Fine Trajectory Planning via Diffusion Refinement with Semantic Interaction for End to End Autonomous Driving
- GUIDE: Gaussian Unified Instance Detection for Enhanced Obstacle Perception in Autonomous Driving
- Enhancing End-to-End Autonomous Driving with Risk Semantic Distillaion from VLM
- Enhancing All-to-X Backdoor Attacks with Optimized Target Class Mapping
- DAP: A Discrete-token Autoregressive Planner for Autonomous Driving
- CorrectAD: A Self-Correcting Agentic System to Improve End-to-end Planning in Autonomous Driving
- ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Experts
- Argus: Resilience-Oriented Safety Assurance Framework for End-to-End ADSs
- Embodied Cognition Augmented End2End Autonomous Driving
- SafeDrive: Fine-Grained Safety Reasoning for End-to-End Driving in a Sparse World
- Dataset Safety in Autonomous Driving: Requirements, Risks, and Assurance
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving
- HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Driving
- LaneDiffusion: Improving Centerline Graph Learning via Prior Injected BEV Feature Generation
- SAFe-Copilot: Unified Shared Autonomy Framework
- UniLION: Towards Unified Autonomous Driving Model with Linear Group RNNs
- Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
- Constructing the Umwelt: Cognitive Planning through Belief-Intent Co-Evolution
- Beyond Imitation: Constraint-Aware Trajectory Generation with Flow Matching For End-to-End Autonomous Driving
- Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
- Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Driving
- ZTRS: Zero-Imitation End-to-end Autonomous Driving with Trajectory Scoring
- SynAD: Enhancing Real-World End-to-End Autonomous Driving Models through Synthetic Data Integration
- VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting
- From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction
- Perfect Prediction or Plenty of Proposals? What Matters Most in Planning for Autonomous Driving
- Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks
- VDRive: Leveraging Reinforced VLA and Diffusion Policy for End-to-end Autonomous Driving
- SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queries
- SimpleVSF: VLM-Scoring Fusion for Trajectory Prediction of End-to-End Autonomous Driving
- DiffVLA++: Bridging Cognitive Reasoning and End-to-End Driving through Metric-Guided Alignment
- Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models
- MACE: Mixture-of-Experts Accelerated Coordinate Encoding for Large-Scale Scene Localization and Rendering
- CoIRL-AD: Collaborative-Competitive Imitation-Reinforcement Learning in Latent World Models for Autonomous Driving
- DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving
- Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modeling
- Stability Under Scrutiny: Benchmarking Representation Paradigms for Online HD Mapping
- Scalable Offline Metrics for Autonomous Driving
- ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving
- HyPlan: Hybrid Learning-Assisted Planning Under Uncertainty for Safe Autonomous Driving
- Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models
- More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
- RAP: 3D Rasterization Augmented End-to-End Planning
- DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation
- BridgeDrive: Diffusion Bridge Policy for Closed-Loop Trajectory Planning in Autonomous Driving
- BEV-VLM: Trajectory Planning via Unified BEV Abstraction
- Autoregressive End-to-End Planning with Time-Invariant Spatial Alignment and Multi-Objective Policy Refinement
- AnchDrive: Bootstrapping Diffusion Policies with Hybrid Trajectory Anchors for End-to-End Driving
- OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Driving
- Driving on Registers
- PIE: Perception and Interaction Enhanced End-to-End Motion Planning for Autonomous Driving
- TinyBEV: Cross Modal Knowledge Distillation for Efficient Multi Task Bird's Eye View Perception and Planning
- DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
- RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generation
- CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refine
- AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Driving
- MAP: End-to-End Autonomous Driving with Map-Assisted Planning
- FlowDrive: Energy Flow Field for End-to-End Autonomous Driving
- Bridging Perception and Planning: Towards End-to-End Planning for Signal Temporal Logic Tasks
- Embodied Navigation Foundation Model
- M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments
- LLM-based Human-like Traffic Simulation for Self-driving Tests
- Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
- Asymmetry Vulnerability and Physical Attacks on Online Map Construction for Autonomous Driving
- OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision
- A Knowledge-Driven Diffusion Policy for End-to-End Autonomous Driving Based on Expert Routing
- OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction
- KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Models
- Can the Waymo Open Motion Dataset Support Realistic Behavioral Modeling? A Validation Study with Naturalistic Trajectories
- AutoDrive-R2: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Driving
- CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Driving
- To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software
- Self-Supervised Sparse Sensor Fusion for Long Range Perception
- LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
- EvoPSF: Online Evolution of Autonomous Driving Models via Planning-State Feedback
- ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving
- SpaRC-AD: A Baseline for Radar-Camera Fusion in End-to-End Autonomous Driving
- ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Scene Reconstruction
- Risk Map As Middleware: Towards Interpretable Cooperative End-to-end Autonomous Driving for Risk-Aware Planning
- ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting
- GMF-Drive: Gated Mamba Fusion with Spatial-Aware BEV Representation for End-to-End Autonomous Driving
- DistillDrive: End-to-End Multi-Mode Autonomous Driving Distillation by Isomorphic Hetero-Source Planning Model
- IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model
- TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction
- RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Case
- EvaDrive: Evolutionary Adversarial Policy Optimization for End-to-End Autonomous Driving
- MetAdv: A Unified and Interactive Adversarial Testing Platform for Autonomous Driving
- Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving
- DiffSemanticFusion: Semantic Raster BEV Fusion for Autonomous Driving via Online HD Map Diffusion
- FASTopoWM: Fast-Slow Lane Segment Topology Reasoning with Latent World Models
- FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning
- Research Challenges and Progress in the End-to-End V2X Cooperative Autonomous Driving Competition
- VLMPlanner: Integrating Visual Language Models with Motion Planning
- A Concept for Efficient Scalability of Automated Driving Allowing for Technical, Legal, Cultural, and Ethical Differences
Related