SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models

2025/09/22 by Pingyi Chen, Chen, Pingyi, Yujing Lou +15 · 3 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Constraint Satisfaction and Optimization #FOS: Computer and information sciences #Satellite Image Processing and Photogrammetry

paper · pdf · doi:10.48550/arxiv.2509.17664

openalex publication_date 2025/09/22 · openalex created_date 2025/10/16 · openalex updated_date 2026/07/28

Abstract

While vision language models (VLMs) excel in 2D semantic visual understanding, their ability to quantitatively reason about 3D spatial relationships remains under-explored, due to the deficiency of 2D images' spatial representation ability. In this paper, we analyze the problem hindering VLMs' spatial understanding abilities and propose SD-VLM, a novel framework that significantly enhances fundamental spatial perception abilities of VLMs through two key contributions: (1) propose Massive Spatial Measuring and Understanding (MSMU) dataset with precise spatial annotations, and (2) introduce a simple depth positional encoding method strengthening VLMs' spatial awareness. MSMU dataset covers massive quantitative spatial tasks with 700K QA pairs, 2.5M physical numerical annotations, and 10K chain-of-thought augmented samples. We have trained SD-VLM, a strong generalist VLM which shows superior quantitative spatial measuring and understanding capability. SD-VLM not only achieves state-of-the-art performance on our proposed MSMU-Bench, but also shows spatial generalization abilities on other spatial understanding benchmarks including Q-Spatial and SpatialRGPT-Bench. Extensive experiments demonstrate that SD-VLM outperforms GPT-4o and Intern-VL3-78B by 26.91% and 25.56% respectively on MSMU-Bench. Code and models are released at https://github.com/cpystan/SD-VLM.

Citations

• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
• MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
• Qwen2.5-VL Technical Report
• iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs
• GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models
• Qwen2.5 Technical Report
• RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics
• An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models
• Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes
• GPT-4o System Card
• Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities
• Does Spatial Cognition Emerge in Frontier Models?
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models
• LLaVA-OneVision: Easy Visual Task Transfer
• SpatialBot: Precise Spatial Understanding with Vision Language Models
• MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
• VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications
• Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers
• BLINK: Multimodal Large Language Models Can See but Not Perceive
• UniDepth: Universal Monocular Metric Depth Estimation
• PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
• SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
• Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
• DriveLM: Driving with Graph Visual Question Answering
• LMDrive: Closed-Loop End-to-End Driving with Large Language Models
• LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• An Embodied Generalist Agent in 3D World
• What's "up" with vision-language models? Investigating their struggle with spatial reasoning
• Improved Baselines with Visual Instruction Tuning
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
• Multi3DRefer: Grounding Text Description to Multiple 3D Objects
• Physically Grounded Vision-Language Models for Robotic Manipulation
• ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes
• MMBench: Is Your Multi-modal Model an All-around Player?
• Visual Instruction Tuning
• GPT-4 Technical Report
• SQA3D: Situated Question Answering in 3D Scenes
• Visual Spatial Reasoning
• ScanQA: 3D Question Answering for Spatial Scene Understanding
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language
• SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition
• Towards VQA Models That Can Read
• VizWiz Grand Challenge: Answering Visual Questions from Blind People
• Attention Is All You Need
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
• A flexible new technique for camera calibration
• 3D-LLM: Injecting the 3D World into Large Language Models
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

Cited by

• 4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation
• Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling
• Traj-VLN: Learning Pixel-Space Interaction via Autoregressive Trajectory Generation

Related

• SpatialBot: Precise Spatial Understanding with Vision Language Models
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
• GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation
• SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning
• MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs