Scalability in Perception for Autonomous Driving: Waymo Open Dataset
2020/06/01 by Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla +20 · 177 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Autonomous Vehicle Technology and Safety #Robotics and Sensor-Based Localization
paper · doi:10.1109/cvpr42600.2020.00252
openalex publication_date 2020/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29
Abstract
The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the over-all viability of the technology. In an effort to help align the research community’s contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.
Cited by
- CARRADA Dataset: Camera and Automotive Radar with Range-Angle-Doppler Annotations
- Mitigating Bias in Calibration Error Estimation
- Robust 4D Driving Scene Reconstruction from Imperfect Visual Priors
- Multiple Object Tracking with Correlation Learning
- VEGA: Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision
- AerialMetric: Benchmarking and Adapting UAV Monocular Metric Depth Estimation in the Real World
- Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detection
- Scene-Centric Unsupervised Video Panoptic Segmentation
- SynFlow: Scaling Up LiDAR Scene Flow Estimation with Synthetic Data
- Towards Semantic Segmentation of Urban-Scale 3D Point Clouds: A Dataset, Benchmarks and Challenges
- Progressive Coordinate Transforms for Monocular 3D Object Detection
- Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping
- Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts
- D2GS: Dense Depth Regularization for LiDAR-free Urban Scene Reconstruction
- MIC-BEV: Multi-Infrastructure Camera Bird's-Eye-View Transformer with Relation-Aware Fusion for 3D Object Detection
- UniPlanner: A Unified Motion Planning Framework for Autonomous Vehicle Decision-Making Systems via Multi-Dataset Integration
- ZTRS: Zero-Imitation End-to-end Autonomous Driving with Trajectory Scoring
- Neural USD: An object-centric framework for iterative editing and control
- Which LiDAR scanning pattern is better for roadside perception: Repetitive or Non-repetitive?
- Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations
- UrbanIng-V2X: A Large-Scale Multi-Vehicle, Multi-Infrastructure Dataset Across Multiple Intersections for Cooperative Perception
- 3D Spatial Recognition without Spatially Labeled 3D
- Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study
- Monocular Visual 8D Pose Estimation for Articulated Bicycles and Cyclists
- Advances in 4D Representation: Geometry, Motion, and Interaction
- X-Ego: Acquiring Team-Level Tactical Situational Awareness via Cross-Egocentric Contrastive Video Representation Learning
- Rethinking Trajectory Forecasting Evaluation
- Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
- OmniNWM: Omniscient Driving Navigation World Models
- A Comprehensive Survey on World Models for Embodied AI
- Open-set 3D Object Detection
- CALM-Net: Curvature-Aware LiDAR Point Cloud-based Multi-Branch Neural Network for Vehicle Re-Identification
- David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generation
- ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection
- Leveraging 2D Priors and SDF Guidance for Dynamic Urban Scene Rendering
- DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models
- An Analytical Framework to Enhance Autonomous Vehicle Perception for Smart Cities
- BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
- SimULi: Real-Time LiDAR and Camera Simulation with Unscented Transforms
- SIMSplat: Predictive Driving Scene Editing with Language-aligned 4D Gaussian Splatting
- DrivingScene: A Multi-Task Online Feed-Forward 3D Gaussian Splatting Method for Dynamic Driving Scenes
- IPS300+: a Challenging Multimodal Dataset for Intersection Perception System
- Scalable Primitives for Generalized Sensor Fusion in Autonomous Vehicles
- SNAP: Towards Segmenting Anything in Any Point Cloud
- Full segmentation annotations of 3D time-lapse microscopy images of MDA231 cells
- EC3R-SLAM: Efficient and Consistent Monocular Dense SLAM with Feed-Forward 3D Reconstruction
- Explainable Human-in-the-Loop Segmentation via Critic Feedback Signals
- Calibrating the Full Predictive Class Distribution of 3D Object Detectors for Autonomous Driving
- Nav-EE: Navigation-Guided Early Exiting for Efficient Vision-Language Models in Autonomous Driving
- Visibility-Aware Densification for 3D Gaussian Splatting in Dynamic Urban Scenes
- Exploring Single Domain Generalization of LiDAR-based Semantic Segmentation under Imperfect Labels
- CVD-STORM: Cross-View Video Diffusion with Spatial-Temporal Reconstruction Model for Autonomous Driving
- XYZCylinder: Towards Compatible Feed-Forward 3D Gaussian Splatting for Driving Scenes via Unified Cylinder Lifting Method
- Yaw-Guided Imitation Learning for Autonomous Driving in Urban\n Environments
- ARTDECO: Towards Efficient and High-Fidelity On-the-Fly 3D Reconstruction with Structured Scene Representation
- Geometry-Aware Cross Modal Alignment for Light Field-LiDAR Semantic Segmentation
- LogSTOP: Temporal Scores over Prediction Sequences for Matching and Retrieval
- The Safety Challenge of World Models for Embodied AI Agents: A Review
- ALISE: Annotation-Free LiDAR Instance Segmentation for Autonomous Driving
- Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI
- Progressive Gaussian Transformer with Anisotropy-aware Sampling for Open Vocabulary Occupancy Prediction
- AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networks
- Learning Adaptive Pseudo-Label Selection for Semi-Supervised 3D Object Detection
- From Static to Dynamic: a Survey of Topology-Aware Perception in Autonomous Driving
- GeLoc3r: Enhancing Relative Camera Pose Regression with Geometric Consistency Regularization
- Self-driving cars: Are we there yet?
- Residual Vector Quantization For Communication-Efficient Multi-Agent Perception
- 4D Driving Scene Generation With Stereo Forcing
- MAD: Motion Appearance Decoupling for efficient Driving World Models
- TESO: Online Tracking of Essential Matrix by Stochastic Optimization
- Driving on Registers
- Multi-view Tracking, Re-ID, and Social Network Analysis of a Flock of Visually Similar Birds in an Outdoor Aviary
- DAGE: Dual-Stream Architecture for Efficient and Fine-Grained Geometry Estimation
- Deep Learning for 3D Point Clouds: A Survey
- Pure Vision Language Action (VLA) Models: A Comprehensive Survey
- Combining Projected Uncertainty for Self-Supervised Visual Odometry: From Two-Frame to Multi-Frame
- MAESTRO: Task-Relevant Optimization via Adaptive Feature Enhancement and Suppression for Multi-task 3D Perception
- PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
- Neural Atlas Graphs for Dynamic Scene Decomposition and Editing
- See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model
- Robust Object Detection for Autonomous Driving via Curriculum-Guided Group Relative Policy Optimization
- Self-Supervised Cross-Modal Learning for Image-to-Point Cloud Registration
- RangeSAM: On the Potential of Visual Foundation Models for Range-View represented LiDAR segmentation
- How many classes do we need to see for novel class discovery?
- STEP: Structured Training and Evaluation Platform for benchmarking trajectory prediction models
- SimCoachCorpus: A naturalistic dataset with language and trajectories for embodied teaching
- Deep Learning for 3D Point Cloud Understanding: A Survey
- MCGS-SLAM: A Multi-Camera SLAM Framework Using Gaussian Splatting for High-Fidelity Mapping
- BEVUDA++: Geometric-aware Unsupervised Domain Adaptation for Multi-View 3D Object Detection
- Weakly and Self-Supervised Class-Agnostic Motion Prediction for Autonomous Driving
- From Machine Learning to Robotics: Challenges and Opportunities for\n Embodied Intelligence
- TeraSim-World: Worldwide Safety-Critical Data Synthesis for End-to-End Autonomous Driving
- 4DRadar-GS: Self-Supervised Dynamic Driving Scene Reconstruction with 4D Radar
- OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling
- DeltaFlow: An Efficient Multi-frame Scene Flow Estimation Method
- 3DAeroRelief: The first 3D Benchmark UAV Dataset for Post-Disaster Assessment
- Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
- Diffusion-Based Action Recognition Generalizes to Untrained Domains
- Towards Open-Vocabulary Multimodal 3D Object Detection with Attributes
- Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
- Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes
- Ekya: Continuous Learning of Video Analytics Models on Edge Compute\n Servers
- 3DPillars: Pillar-based two-stage 3D object detection
- Visibility-Aware Language Aggregation for Open-Vocabulary Segmentation in 3D Gaussian Splatting
- Guideline-Consistent Segmentation via Multi-Agent Refinement
- OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction
- Are We Ready for Unmanned Surface Vehicles in Inland Waterways? The USVInland Multisensor Dataset and Benchmark
- Domain Adaptation for Different Sensor Configurations in 3D Object Detection
- Differentiable Entropy Regularization: A Complexity-Aware Approach for Neural Optimization
- UrbanTwin: Building High-Fidelity Digital Twins for Sim2Real LiDAR Perception and Evaluation
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- High-Fidelity Digital Twins for Bridging the Sim2Real Gap in LiDAR-Based ITS Perception
- Deep Learning for 3D Point Clouds: A Survey
- Machine Learning in Micromobility: A Systematic Review of Datasets, Techniques, and Applications
- A Unified Voxel Diffusion Module for Point Cloud 3D Object Detection
- PointSlice: Accurate and Efficient Slice-Based Representation for 3D Object Detection from Point Clouds
- CoVeRaP: Cooperative Vehicular Perception through mmWave FMCW Radars
- FLUID: A Fine-Grained Lightweight Urban Signalized-Intersection Dataset of Dense Conflict Trajectories
- Domain Adaptation-Based Crossmodal Knowledge Distillation for 3D Semantic Segmentation
- DriveQA: Passing the Driving Knowledge Test
- VoCap: Video Object Captioning and Segmentation from Any Prompt
- To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software
- Enhancing Pseudo-Boxes via Data-Level LiDAR-Camera Fusion for Unsupervised 3D Object Detection
- Realistic and Controllable 3D Gaussian-Guided Object Editing for Driving Video Generation
- A Survey on Automated Driving System Testing: Landscapes and Trends
- SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving
- Data-Efficient Point Cloud Semantic Segmentation Pipeline for Unimproved Roads
- LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Grounding
- DriveSplat: Decoupled Driving Scene Reconstruction with Geometry-enhanced Partitioned Neural Gaussians
- BasketLiDAR: The First LiDAR-Camera Multimodal Dataset for Professional Basketball MOT
- LookOut: Real-World Humanoid Egocentric Navigation
- FastTracker: Real-Time and Accurate Visual Tracking
- MoVieDrive: Multi-Modal Multi-View Urban Scene Video Generation
- Self-Supervised Sparse Sensor Fusion for Long Range Perception
- Multimodal Data Storage and Retrieval for Embodied AI: A Survey
- ROVR-Open-Dataset: A Large-Scale Depth Dataset for Autonomous Driving
- CMF-IoU: Multi-Stage Cross-Modal Fusion 3D Object Detection with IoU Joint Prediction
- Video Super-Resolution with Long-Term Self-Exemplars
- Neural Rendering for Sensor Adaptation in 3D Object Detection
- CenterAtt: Fast 2-stage Center Attention Network
- Transferable Class Statistics and Multi-scale Feature Approximation for 3D Object Detection
- G-CUT3R: Guided 3D Reconstruction with Camera and Depth Prior Integration
- Delving into Dynamic Scene Cue-Consistency for Robust 3D Multi-Object Tracking
- STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer
- Improving Robustness of Learning-based Autonomous Steering Using Adversarial Images
- From Micro to Macro Flow Modeling: Characterizing Heterogeneity of Mixed-Autonomy Traffic
- ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Scene Reconstruction
- GRASPTrack: Geometry-Reasoned Association via Segmentation and Projection for Multi-Object Tracking
- ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting
- ETA: Energy-based Test-time Adaptation for Depth Completion
- B4DL: A Benchmark for 4D LiDAR LLM in Spatio-Temporal Understanding
- CSRAP: Enhanced Canvas Attention Scheduling for Real-Time Mission Critical Perception
- Perceive-Sample-Compress: Towards Real-Time 3D Gaussian Splatting
- Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens
- BEVCon: Advancing Bird's Eye View Perception with Contrastive Learning
- RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Case
- Prediction-Oriented Subsampling from Data Streams
- La La LiDAR: Large-Scale Layout Generation from LiDAR Data
- Veila: Panoramic LiDAR Generation from a Monocular RGB Image
- Robust Single-Stage Fully Sparse 3D Object Detection via Detachable Latent Diffusion
- Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents
- MIDAR: Mimicking LiDAR Detection for Traffic Applications with a Lightweight Plug-and-Play Model
- Dream-to-Recon: Monocular 3D Reconstruction with Diffusion-Depth Distillation from Single Images
- On-the-Fly Object-aware Representative Point Selection in Point Cloud
- VDEGaussian: Video Diffusion Enhanced 4D Gaussian Splatting for Dynamic Urban Scenes Modeling
- Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving
- AID4AD: Aerial Image Data for Automated Driving Perception
- ModelNet40-E: An Uncertainty-Aware Benchmark for Point Cloud Classification
- RoadMamba: A Dual Branch Visual State Space Model for Road Surface Classification
- Context-based Motion Retrieval using Open Vocabulary Methods for Autonomous Driving
- Cooperative Perception: A Resource-Efficient Framework for Multi-Drone 3D Scene Reconstruction Using Federated Diffusion and NeRF
- I2V-GS: Infrastructure-to-Vehicle View Transformation with Gaussian Splatting for Autonomous Driving Data Generation
- OmniTraj: Pre-Training on Heterogeneous Data for Adaptive and Zero-Shot Human Trajectory Prediction
- UAVScenes: A Multi-Modal Dataset for UAVs
- Object Recognition Datasets and Challenges: A Review
- Goal-Based Vision-Language Driving
- Vision-Language Cross-Attention for Real-Time Autonomous Driving
- Research Challenges and Progress in the End-to-End V2X Cooperative Autonomous Driving Competition
- The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
Related