Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases

2024/04/16 by Kai Chen, Yanze Li, Chen, Kai +23 · 12 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2404.10595

openalex publication_date 2024/04/16 · openalex created_date 2024/04/18 · openalex updated_date 2026/07/28

Abstract

Large Vision-Language Models (LVLMs) have received widespread attention for advancing the interpretable self-driving. Existing evaluations of LVLMs primarily focus on multi-faceted capabilities in natural circumstances, lacking automated and quantifiable assessment for self-driving, let alone the severe road corner cases. In this work, we propose CODA-LM, the very first benchmark for the automatic evaluation of LVLMs for self-driving corner cases. We adopt a hierarchical data structure and prompt powerful LVLMs to analyze complex driving scenes and generate high-quality pre-annotations for the human annotators, while for LVLM evaluation, we show that using the text-only large language models (LLMs) as judges reveals even better alignment with human preferences than the LVLM judges. Moreover, with our CODA-LM, we build CODA-VLM, a new driving LVLM surpassing all open-sourced counterparts on CODA-LM. Our CODA-VLM performs comparably with GPT-4V, even surpassing GPT-4V by +21.42% on the regional perception task. We hope CODA-LM can become the catalyst to promote interpretable self-driving empowered by LVLMs.

Cited by

• Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation
• A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach
• From Segments to Scenes: Temporal Understanding in Autonomous Driving via Vision-Language Model
• Seeing before Observable: Potential Risk Reasoning in Autonomous Driving via Vision Language Models
• Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
• RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System
• All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles
• SAVANT: Semantic Analysis with Vision-Augmented Anomaly deTection
• Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception
• iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning
• RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Case
• SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generation

Related

• Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding
• Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving
• STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving
• Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Driving
• Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving