LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning

2023/11/30 by Sijin Chen, Xin Chen, Chen, Sijin +14 · 38 citations
Computer Science · #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Human Pose and Action Recognition

paper · pdf · doi:10.48550/arxiv.2311.18651

Abstract

Recent advances in Large Multimodal Models (LMM) have made it possible for various applications in human-machine interactions. However, developing LMMs that can comprehend, reason, and plan in complex and diverse 3D environments remains a challenging topic, especially considering the demand for understanding permutation-invariant point cloud 3D representations of the 3D scene. Existing works seek help from multi-view images, and project 2D features to 3D space as 3D scene representations. This, however, leads to huge computational overhead and performance degradation. In this paper, we present LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and respond to both textual-instructions and visual-prompts. This help LMMs better comprehend human interactions and further help to remove the ambiguities in cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results, and surpasses various 3D vision-language models on both 3D Dense Captioning and 3D Question Answering.

Cited by

• Chorus: Multi-Teacher Pretraining for Holistic 3D Gaussian Scene Encoding
• N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models
• Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos
• Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding
• CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligence
• CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoning
• Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling
• NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Prediction
• S²-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance
• MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding
• SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition
• Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding
• Vision-Language Memory for Spatial Reasoning
• LAST: LeArning to Think in Space and Time for Generalist Vision-Language Models
• Disc3D: Automatic Curation of High-Quality 3D Dialog Data via Discriminative Object Referring
• POMA-3D: The Point Map Way to 3D Scene Understanding
• Part-X-MLLM: Part-aware 3D Multimodal Large Language Model
• AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models
• Sparse3DPR: Training-Free 3D Hierarchical Scene Parsing and Task-Adaptive Subgraph Reasoning from Sparse RGB Views
• URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model
• Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks
• SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenes
• Reasoning in Space via Grounding in the World
• Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergy
• Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generation
• JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation
• ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA
• SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
• Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understanding
• 3D Aware Region Prompted Vision Language Model
• M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments
• Reg3D: Reconstructive Geometry Instruction Tuning for 3D Scene Understanding
• UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
• Advancing 3D Scene Understanding with MV-ScanQA Multi-View Reasoning Evaluation and TripAlign Pre-training Dataset
• Spatial-ORMLLM: Improve Spatial Relation Understanding in the Operating Room with Multimodal Large Language Model
• Understanding Dynamic Scenes in Ego Centric 4D Point Clouds
• A Study of the Framework and Real-World Applications of Language Embedding for 3D Scene Understanding
• 3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding

Related

• 3D-LLM: Injecting the 3D World into Large Language Models
• 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer
• 3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
• Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following
• When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models