2021/03/31 by Yawar Siddiqui, Justus Thies, Siddiqui, Yawar +9 · 2 citations
Computer Science · Engineering · Mathematics · #3D Shape Modeling and Analysis #3D reconstruction #Advanced Vision and Imaging #Artificial intelligence #Artificial neural network #Computer Graphics and Visualization Techniques #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #FOS: Computer and information sciences #Generative grammar #Generative model #Geometry #Mathematics #Pattern recognition (psychology) #Point (geometry) #Point cloud #Process (computing) #Set (abstract data type) #View synthesis #cs.CV
paper · pdf · doi:10.48550/arxiv.2104.00024
published in arXiv (Cornell University) (Cornell University) · Project Page: https://nihalsid.github.io/retrieval-fuse/
openalex publication_date 2021/03/31 · arxiv created 2021/08/10 · arxiv updated 2021/08/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08
3D reconstruction of large scenes is a challenging problem due to the high-complexity nature of the solution space, in particular for generative neural networks. In contrast to traditional generative learned models which encode the full generative process into a neural network and can struggle with maintaining local details at the scene level, we introduce a new method that directly leverages scene geometry from the training database. First, we learn to synthesize an initial estimate for a 3D scene, constructed by retrieving a top-k set of volumetric chunks from the scene database. These candidates are then refined to a final scene generation with an attention-based refinement that can effectively select the most consistent set of geometry from the candidates and combine them together to create an output scene, facilitating transfer of coherent structures and local detail from train scene geometry. We demonstrate our neural scene reconstruction with a database for the tasks of 3D super resolution and surface reconstruction from sparse point clouds, showing that our approach enables generation of more coherent, accurate 3D scenes, improving on average by over 8% in IoU over state-of-the-art scene reconstruction.