2025/09/29 by Haijier Chen, Chen, Haijier, Bo Xu +8 · 1 citation
Computer Science · #3D modeling #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Consistency (knowledge bases) #FOS: Computer and information sciences #Geometric modeling #Metric (unit) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Prior probability #Scalability #Solid modeling #Video Analysis and Summarization #Visualization
paper · pdf · doi:10.48550/arxiv.2509.24385
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/09/29 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28
Recent developments in Multimodal Large Language Models (MLLMs) have significantly improved Vision-Language (VL) reasoning in 2D domains. However, extending these capabilities to 3D scene understanding remains a major challenge. Existing 3D Multimodal Large Language Models (3D-MLLMs) often depend on 3D data inputs, which limits scalability and generalization. To address this limitation, we propose Vid-LLM, a video-based 3D-MLLM that directly processes video inputs without requiring external 3D data, making it practical for real-world deployment. In our method, the geometric prior are directly used to improve the performance of the sceen perception. To integrate the geometric cues into the MLLM compactly, we design a Cross-Task Adapter (CTA) module to align the 3D geometric priors with the vision-language representations. To ensure geometric consistency and integrity, we introduce a Metric Depth Model that recovers real-scale geometry from the reconstruction outputs. Finally, the model is fine-tuned with a two-stage distillation optimization strategy, realizing fast convergence and stabilizes training. Extensive experiments across diverse benchmarks verified the effectiveness of our method on 3D Question Answering, 3D Dense Captioning and 3D Visual Grounding tasks, demonstrating the superior multi-task capabilities.