Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model

2024/05/27 by Kuan‐Chih Huang, Xiangtai Li, Huang, Kuan-Chih +7 · 7 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2405.17427

openalex publication_date 2024/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements in multimodal large language models (LLMs) have demonstrated significant potential across various domains, particularly in concept reasoning. However, their applications in understanding 3D environments remain limited, primarily offering textual or numerical outputs without generating dense, informative segmentation masks. This paper introduces Reason3D, a novel LLM designed for comprehensive 3D understanding. Reason3D processes point cloud data and text prompts to produce textual responses and segmentation masks, enabling advanced tasks such as 3D reasoning segmentation, hierarchical searching, express referring, and question answering with detailed mask outputs. We propose a hierarchical mask decoder that employs a coarse-to-fine approach to segment objects within expansive scenes. It begins with a coarse location estimation, followed by object mask estimation, using two unique tokens predicted by LLMs based on the textual query. Experimental results on large-scale ScanNet and Matterport3D datasets validate the effectiveness of our Reason3D across various tasks.

Cited by

• DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotation
• Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding
• AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models
• REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting
• Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model
• SeqVLM: Proposal-Guided Multi-View Sequences Reasoning via VLM for Zero-Shot 3D Visual Grounding
• Multimodal Referring Segmentation: A Survey

Related

• OpenMaskDINO3D : Reasoning 3D Segmentation via Large Language Model
• MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation
• Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models
• PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model
• Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentation