City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning

2025/07/17 by Penglei Sun, Sun, Penglei, Yaoxian Song +17
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2507.12795

openalex publication_date 2025/07/17 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Scene understanding enables intelligent agents to interpret and comprehend their environment. While existing large vision-language models (LVLMs) for scene understanding have primarily focused on indoor household tasks, they face two significant limitations when applied to outdoor large-scale scene understanding. First, outdoor scenarios typically encompass larger-scale environments observed through various sensors from multiple viewpoints (e.g., bird view and terrestrial view), while existing indoor LVLMs mainly analyze single visual modalities within building-scale contexts from humanoid viewpoints. Second, existing LVLMs suffer from missing multidomain perception outdoor data and struggle to effectively integrate 2D and 3D visual information. To address the aforementioned limitations, we build the first multidomain perception outdoor scene understanding dataset, named \textbf\underlineSVM-City, deriving from multi\textbf\underlineScale scenarios with multi\textbf\underlineView and multi\textbf\underlineModal instruction tuning data. It contains 420k images and 4, 811M point clouds with 567k question-answering pairs from vehicles, low-altitude drones, high-altitude aerial planes, and satellite. To effectively fuse the multimodal data in the absence of one modality, we introduce incomplete multimodal learning to model outdoor scene understanding and design the LVLM named \textbf\underlineCity-VLM. Multimodal fusion is realized by constructing a joint probabilistic distribution space rather than implementing directly explicit fusion operations (e.g., concatenation). Experimental results on three typical outdoor scene understanding tasks show City-VLM achieves 18.14 % performance surpassing existing LVLMs in question-answering tasks averagely. Our method demonstrates pragmatic and generalization performance across multiple outdoor scenes.

Citations

• Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks
• The Llama 3 Herd of Models
• 3D Question Answering for City Scene Understanding
• Robust Multimodal Learning via Representation Decoupling
• Multi-Task Domain Adaptation for Language Grounding with 3D Objects
• Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning
• MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
• 3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
• LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding
• EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answering
• GeoChat: Grounded Large Vision-Language Model for Remote Sensing
• Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI
• Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
• Uni3D: Exploring Unified 3D Representation at Scale
• 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Towards Label-free Scene Understanding by Vision Foundation Models
• NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
• Human-in-the-loop Robotic Grasping using BERT Scene Representation
• Text2Pos: Text-to-Point-Cloud Cross-Modal Localization
• SensatUrban: Learning Semantics from Urban-Scale Photogrammetric Point Clouds
• SensatUrban: Learning Semantics from Urban-Scale Photogrammetric Point Clouds
• BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View
• LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation
• LoRA: Low-Rank Adaptation of Large Language Models
• Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation
• Center-based 3D Object Detection and Tracking
• Data Uncertainty Learning in Face Recognition
• ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language
• Deep Modular Co-Attention Networks for Visual Question Answering
• nuScenes: A multimodal dataset for autonomous driving
• Bilinear Attention Networks
• Compositional Attention Networks for Machine Reasoning
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Stacked Attention Networks for Image Question Answering
• Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers

Related

• UrbanFeel: A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspective
• How Well Do Vision--Language Models Understand Cities? A Comparative Study on Spatial Reasoning from Street-View Images
• UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios
• SpatialMosaic: A Multiview VLM Dataset for Partial Visibility
• 3D Question Answering for City Scene Understanding