Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image

2020/02/27 by Yinyu Nie, Nie, Yinyu, Xiaoguang Han +10 · 21 citations
Computer Science · Earth and Planetary Sciences · Engineering · #3D Shape Modeling and Analysis #3D Surveying and Cultural Heritage #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #cs.CV

paper · pdf · doi:10.48550/arxiv.2002.12212

Accepted by CVPR 2020

arxiv created 2020/02/27 · openalex publication_date 2020/02/27 · arxiv updated 2020/02/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Semantic reconstruction of indoor scenes refers to both scene understanding and object reconstruction. Existing works either address one part of this problem or focus on independent objects. In this paper, we bridge the gap between understanding and reconstruction, and propose an end-to-end solution to jointly reconstruct room layout, object bounding boxes and meshes from a single image. Instead of separately resolving scene understanding and object reconstruction, our method builds upon a holistic scene context and proposes a coarse-to-fine hierarchy with three components: 1. room layout with camera pose; 2. 3D object bounding boxes; 3. object meshes. We argue that understanding the context of each component can assist the task of parsing the others, which enables joint understanding and reconstruction. The experiments on the SUN RGB-D and Pix3D datasets demonstrate that our method consistently outperforms existing methods in indoor layout estimation, 3D object detection and mesh reconstruction.

Citations

• Deep Level Sets: Implicit Surface Representations for 3D Shape Inference
• Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense

Cited by

• ShapeR: Robust Conditional 3D Shape Generation from Casual Captures
• 3D-RE-GEN: 3D Reconstruction of Indoor Scenes with a Generative Framework
• SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Model
• SRAM: Shape-Realism Alignment Metric for No Reference 3D Shape Evaluation
• CC-FMO: Camera-Conditioned Zero-Shot Single Image to 3D Scene Generation with Foundation Model Orchestration
• Visual Spatial Tuning
• PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image
• Joint Learning of Pose Regression and Denoising Diffusion with Score Scaling Sampling for Category-level 6D Pose Estimation
• MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning
• TUN3D: Towards Real-World Scene Understanding from Unposed Images
• 3D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Image
• HLG: Comprehensive 3D Room Construction via Hierarchical Layout Generation
• PixCuboid: Room Layout Estimation from Multi-view Featuremetric Alignment
• DepR: Depth Guided Single-view Scene Reconstruction with Instance-level Diffusion
• 3D Software Synthesis Guided by Constraint-Expressive Intermediate Representation
• LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans
• FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts
• MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objects
• Pose-Aware Diffusion for 3D Generation
• Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation
• PolyLayout: Multi-room Manhattan Layout Estimation
• Digital Twin Generation from Visual Data: A Survey
• Detect Anything 3D in the Wild

Related

• Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes
• Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
• Learning to Reconstruct and Understand Indoor Scenes from Sparse Views
• A Modular Framework for Single-View 3D Reconstruction of Indoor Environments
• Room Envelopes: A Synthetic Dataset for Indoor Layout Reconstruction from Images