Robust Driving QA through Metadata-Grounded Context and Task-Specific Prompts

2025/10/21 by Shimeng Yu, Junsung Park, Yu, Seungjun +5
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Robotics (cs.RO) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.19001

openalex publication_date 2025/10/21 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28

Abstract

We present a two-phase vision-language QA system for autonomous driving that answers high-level perception, prediction, and planning questions. In Phase-1, a large multimodal LLM (Qwen2.5-VL-32B) is conditioned on six-camera inputs, a short temporal window of history, and a chain-of-thought prompt with few-shot exemplars. A self-consistency ensemble (multiple sampled reasoning chains) further improves answer reliability. In Phase-2, we augment the prompt with nuScenes scene metadata (object annotations, ego-vehicle state, etc.) and category-specific question instructions (separate prompts for perception, prediction, planning tasks). In experiments on a driving QA benchmark, our approach significantly outperforms the baseline Qwen2.5 models. For example, using 5 history frames and 10-shot prompting in Phase-1 yields 65.1% overall accuracy (vs.62.61% with zero-shot); applying self-consistency raises this to 66.85%. Phase-2 achieves 67.37% overall. Notably, the system maintains 96% accuracy under severe visual corruption. These results demonstrate that carefully engineered prompts and contextual grounding can greatly enhance high-level driving QA with pretrained vision-language models.

Citations

• Box-QAymo: Box-Referring VQA Dataset for Autonomous Driving
• 3D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation
• LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving
• ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness
• Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
• R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?
• Multi-Object Hallucination in Vision-Language Models
• Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models
• OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning
• DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
• MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
• Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes
• DriveLM: Driving with Graph Visual Question Answering
• DUSt3R: Geometric 3D Vision Made Easy
• LingoQA: Visual Question Answering for Autonomous Driving
• LMDrive: Closed-Loop End-to-End Driving with Large Language Models
• HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
• DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
• Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning
• Lost in the Middle: How Language Models Use Long Contexts
• NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• nuScenes: A multimodal dataset for autonomous driving
• Graph-Structured Representations for Visual Question Answering

Related

• Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving
• ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Models
• Enhancing Vision-Language Models for Autonomous Driving through Task-Specific Prompting and Spatial Reasoning
• Hierarchical Question-Answering for Driving Scene Understanding Using Vision-Language Models
• Box-QAymo: Box-Referring VQA Dataset for Autonomous Driving