Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning

2024/03/18 by Rao Fu, Fu, Rao, Jingyu Liu +7 · 73 citations
Computer Science · Engineering · #3D Modeling in Geospatial Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Motion and Animation #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2403.11401

openalex publication_date 2024/03/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper introduces Scene-LLM, a 3D-visual-language model that enhances embodied agents' abilities in interactive 3D indoor environments by integrating the reasoning strengths of Large Language Models (LLMs). Scene-LLM adopts a hybrid 3D visual feature representation, that incorporates dense spatial information and supports scene state updates. The model employs a projection layer to efficiently project these features in the pre-trained textual embedding space, enabling effective interpretation of 3D visual information. Unique to our approach is the integration of both scene-level and ego-centric 3D information. This combination is pivotal for interactive planning, where scene-level data supports global planning and ego-centric data is important for localization. Notably, we use ego-centric 3D frame features for feature alignment, an efficient technique that enhances the model's ability to align features of small objects within the scene. Our experiments with Scene-LLM demonstrate its strong capabilities in dense captioning, question answering, and interactive planning. We believe Scene-LLM advances the field of 3D visual understanding and reasoning, offering new possibilities for sophisticated agent interactions in indoor settings.

Cited by

• SpatialMosaic: A Multiview VLM Dataset for Partial Visibility
• Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos
• SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
• CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoning
• SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagery
• Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understanding
• Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling
• Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding
• S²-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance
• Words into World: A Task-Adaptive Agent for Language-Guided Spatial Retrieval in AR
• SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models
• HMR3D: Hierarchical Multimodal Representation for 3D Scene Understanding with Large Vision-Language Model
• MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding
• SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition
• POMA-3D: The Point Map Way to 3D Scene Understanding
• Part-X-MLLM: Part-aware 3D Multimodal Large Language Model
• Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
• AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models
• Sparse3DPR: Training-Free 3D Hierarchical Scene Parsing and Task-Adaptive Subgraph Reasoning from Sparse RGB Views
• Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks
• DIV-Nav: Open-Vocabulary Spatial Relationships for Multi-Object Navigation
• Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
• SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenes
• VO-DP: Semantic-Geometric Adaptive Diffusion Policy for Vision-Only Robotic Manipulation
• Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model
• Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graph
• Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergy
• How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective
• LCMF: Lightweight Cross-Modality Mambaformer for Embodied Robotics VQA
• Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understanding
• TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route
• 3D Aware Region Prompted Vision Language Model
• M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments
• DeepThink3D: Enhancing Large Language Models with Programmatic Reasoning in Complex 3D Situated Reasoning Tasks
• UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
• How Does a Virtual Agent Decide Where to Look? -- Symbolic Cognitive Reasoning for Embodied Head Rotation
• MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Mesh
• Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting
• Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions
• Advancing Complex Wide-Area Scene Understanding with Hierarchical Coresets Selection
• DreamScene: 3D Gaussian-based End-to-end Text-to-3D Scene Generation
• City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning
• Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding
• StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling
• 4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration
• Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations
• General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Acting
• SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence
• FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understanding
• AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making
• LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
• From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes
• Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions
• Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs
• 3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding
• ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary
• Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
• Zero-Shot 3D Visual Grounding from Vision-Language Models
• 3D Question Answering via only 2D Vision-Language Models
• Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts
• Increasing Computation Resolves Conflicts in Vision Language Models
• AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoning
• SpatialLLM: From Multi-modality Data to Urban Spatial Intelligence
• Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation
• Vision language models have difficulty recognizing virtual objects
• SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Models
• SpaceMind++: Toward Allocentric Cognitive Maps for Spatially Grounded Video MLLMs
• ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation
• Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding
• What Language Does and What the Evidence Supports: A Functional Role Taxonomy and Evidence Audit of Language Grounding in Embodied Agents
• Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
• A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Science
• How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM

Related

• Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understanding
• Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation
• When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
• SceneGPT: A Language Model for 3D Scene Understanding
• IL3D: A Large-Scale Indoor Layout Dataset for LLM-Driven 3D Scene Generation