2024/09/10 by Ali Tourani, Saad Ejaz, Tourani, Ali +7
Computer Science · Earth and Planetary Sciences · Engineering · #3D Surveying and Cultural Heritage #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #I.2.10 #I.2.9 #I.4.9 #Image Processing and 3D Reconstruction #Manufacturing Process and Optimization #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.2409.06625
openalex publication_date 2024/09/10 · openalex created_date 2024/10/22 · openalex updated_date 2026/07/28
RGB-D cameras supply rich and dense visual and spatial information for various robotics tasks such as scene understanding, map reconstruction, and localization. Integrating depth and visual information can aid robots in localization and element mapping, advancing applications like 3D scene graph generation and Visual Simultaneous Localization and Mapping (VSLAM). While point cloud data containing such information is primarily used for enhanced scene understanding, exploiting their potential to capture and represent rich semantic information has yet to be adequately targeted. This paper presents a real-time pipeline for localizing building components, including wall and ground surfaces, by integrating geometric calculations for pure 3D plane detection followed by validating their semantic category using point cloud data from RGB-D cameras. It has a parallel multi-thread architecture to precisely estimate poses and equations of all the planes detected in the environment, filters the ones forming the map structure using a panoptic segmentation validation, and keeps only the validated building components. Incorporating the proposed method into a VSLAM framework confirmed that constraining the map with the detected environment-driven semantic elements can improve scene understanding and map reconstruction accuracy. It can also ensure (re-)association of these detected components into a unified 3D scene graph, bridging the gap between geometric accuracy and semantic understanding. Additionally, the pipeline allows for the detection of potential higher-level structural entities, such as rooms, by identifying the relationships between building components based on their layout.