OneFormer3D: One Transformer for Unified Point Cloud Segmentation

2023/11/24 by Maxim Kolodiazhnyi, Kolodiazhnyi, Maxim, Анна Воронцова +5 · 17 citations
Engineering · Environmental Science · Computer Science · #3D Shape Modeling and Analysis #Remote Sensing and LiDAR Applications #Advanced Neural Network Applications

paper · pdf · doi:10.48550/arxiv.2311.14405

Abstract

Semantic, instance, and panoptic segmentation of 3D point clouds have been addressed using task-specific models of distinct design. Thereby, the similarity of all segmentation tasks and the implicit relationship between them have not been utilized effectively. This paper presents a unified, simple, and effective model addressing all these tasks jointly. The model, named OneFormer3D, performs instance and semantic segmentation consistently, using a group of learnable kernels, where each kernel is responsible for generating a mask for either an instance or a semantic category. These kernels are trained with a transformer-based decoder with unified instance and semantic queries passed as an input. Such a design enables training a model end-to-end in a single run, so that it achieves top performance on all three segmentation tasks simultaneously. Specifically, our OneFormer3D ranks 1st and sets a new state-of-the-art (+2.1 mAP50) in the ScanNet test leaderboard. We also demonstrate the state-of-the-art results in semantic, instance, and panoptic segmentation of ScanNet (+21 PQ), ScanNet200 (+3.8 mAP50), and S3DIS (+0.8 mIoU) datasets.

Cited by

• PointCHR: Point Cloud Analysis via Curvature-Aware Hyperbolic Rectification
• Online Segment Any 3D Thing as Instance Tracking
• LLM-Guided Material Inference for 3D Point Clouds
• OpenTrack3D: Towards Accurate and Generalizable Open-Vocabulary 3D Instance Segmentation
• Panda: Self-distillation of Reusable Sensor-level Representations for High Energy Physics
• DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotation
• MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding
• Benchmarking individual tree segmentation using multispectral airborne laser scanning data: the FGI-EMIT dataset
• Registration is a Powerful Rotation-Invariance Learner for 3D Anomaly Detection
• BEEP3D: Box-Supervised End-to-End Pseudo-Mask Generation for 3D Instance Segmentation
• OVSeg3R: Learn Open-vocabulary Instance Segmentation from 2D via 3D Reconstruction
• Deep learning-based 3D morphological segmentation and quantitative growth analysis of field-grown cabbage across the full cycle
• Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model
• StripDet: Strip Attention-Based Lightweight 3D Object Detection from Point Cloud
• Robotic 3D Flower Pose Estimation for Small-Scale Urban Farms
• Open-world Point Cloud Semantic Segmentation: A Human-in-the-loop Framework
• Details Matter for Indoor Open-vocabulary 3D Instance Segmentation

Related

• PSGformer: Enhancing 3D Point Cloud Instance Segmentation via Precise Semantic Guidance
• Few-Shot 3D Point Cloud Semantic Segmentation via Stratified Class-Specific Attention Based Transformer Network
• Mask3D: Mask Transformer for 3D Semantic Instance Segmentation
• DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution
• PUPS: Point Cloud Unified Panoptic Segmentation