NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario

2023/05/24 by Tianwen Qian, Jingjing Chen, Qian, Tianwen +7 · 52 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2305.14836

openalex publication_date 2023/05/24 · openalex created_date 2023/05/27 · openalex updated_date 2026/07/28

Abstract

We introduce a novel visual question answering (VQA) task in the context of autonomous driving, aiming to answer natural language questions based on street-view clues. Compared to traditional VQA tasks, VQA in autonomous driving scenario presents more challenges. Firstly, the raw visual data are multi-modal, including images and point clouds captured by camera and LiDAR, respectively. Secondly, the data are multi-frame due to the continuous, real-time acquisition. Thirdly, the outdoor scenes exhibit both moving foreground and static background. Existing VQA benchmarks fail to adequately address these complexities. To bridge this gap, we propose NuScenes-QA, the first benchmark for VQA in the autonomous driving scenario, encompassing 34K visual scenes and 460K question-answer pairs. Specifically, we leverage existing 3D detection annotations to generate scene graphs and design question templates manually. Subsequently, the question-answer pairs are generated programmatically based on these templates. Comprehensive statistics prove that our NuScenes-QA is a balanced large-scale benchmark with diverse question formats. Built upon it, we develop a series of baselines that employ advanced 3D detection and VQA techniques. Our extensive experiments highlight the challenges posed by this new task. Codes and dataset are available at https://github.com/qiantianwen/NuScenes-QA.

Cited by

• ObsDriveBench: Benchmarking Multimodal Understanding under Adverse Weather with Observability Awareness
• Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation
• SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning
• OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving
• MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion
• DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
• From Segments to Scenes: Temporal Understanding in Autonomous Driving via Vision-Language Model
• WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Driving
• BeLLA: End-to-End Birds Eye View Large Language Assistant for Autonomous Driving
• StreamEQA: Towards Streaming Video Understanding for Embodied Scenarios
• CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
• nuScenes Revisited: Progress and Challenges in Autonomous Driving
• RoadSceneBench: A Lightweight Benchmark for Mid-Level Road Scene Understanding
• WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
• Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
• Thinking Ahead: Foresight Intelligence in MLLMs and World Models
• RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System
• What Your Features Reveal: Data-Efficient Black-Box Feature Inversion Attack for Split DNNs
• Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
• A Low-Rank Method for Vision Language Model Hallucination Mitigation in Autonomous Driving
• MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs
• 3EED: Ground Everything Everywhere in 3D
• Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
• All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles
• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
• MMEdge: Accelerating On-device Multimodal Inference via Pipelined Sensing and Encoding
• Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study
• Robust Driving QA through Metadata-Grounded Context and Task-Specific Prompts
• Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysis
• Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
• EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmark
• More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
• Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval
• AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyond
• From Static to Dynamic: a Survey of Topology-Aware Perception in Autonomous Driving
• Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
• OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Driving
• AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering
• V2V-GoT: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models and Graph-of-Thoughts
• M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer
• AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Driving
• MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillance
• OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision
• Multimodal AI Teacher: Integrating Edge Computing and Reasoning Models for Enhanced Student Error Analysis
• DriveQA: Passing the Driving Knowledge Test
• LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
• ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving
• STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes
• JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics
• A Study of the Framework and Real-World Applications of Language Embedding for 3D Scene Understanding
• Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving
• SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generation

Related

• NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Driving
• NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations
• STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes
• Hierarchical Question-Answering for Driving Scene Understanding Using Vision-Language Models
• LingoQA: Visual Question Answering for Autonomous Driving