Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers

2018/04/29 by Stephan R. Richter, Stefan Roth, Richter, Stephan R. +1 · 68 citations
Computer Science · Engineering · Mathematics · #3D Shape Modeling and Analysis #3D reconstruction #Advanced Image and Video Retrieval Techniques #Advanced Vision and Imaging #Algorithm #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Encoding (memory) #FOS: Computer and information sciences #Geometry #I.4.8 #Image (mathematics) #Key (lock) #Mathematics #Octree #Pattern recognition (psychology) #Pixel #Similarity (geometry) #Topology (electrical circuits) #Voxel #cs.CV

paper · pdf · doi:10.48550/arxiv.1804.10975

published in arXiv (Cornell University) (Cornell University) · Published at the Conference on Computer Vision and Pattern Recognition (CVPR 2018)

arxiv created 2018/04/29 · openalex publication_date 2018/04/29 · arxiv updated 2018/05/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

In this paper, we develop novel, efficient 2D encodings for 3D geometry, which enable reconstructing full 3D shapes from a single image at high resolution. The key idea is to pose 3D shape reconstruction as a 2D prediction problem. To that end, we first develop a simple baseline network that predicts entire voxel tubes at each pixel of a reference view. By leveraging well-proven architectures for 2D pixel-prediction tasks, we attain state-of-the-art results, clearly outperforming purely voxel-based approaches. We scale this baseline to higher resolutions by proposing a memory-efficient shape encoding, which recursively decomposes a 3D shape into nested shape layers, similar to the pieces of a Matryoshka doll. This allows reconstructing highly detailed shapes with complex topology, as demonstrated in extensive experiments; we clearly outperform previous octree-based approaches despite having a much simpler architecture using standard network components. Our Matryoshka networks further enable reconstructing shapes from IDs or shape similarity, as well as shape sampling.

Cited by

• SPGen: Spherical Projection as Consistent and Flexible Representation for Single Image 3D Shape Generation
• SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments From 2D Coordinates
• DmifNet:3D Shape Reconstruction Based on Dynamic Multi-Branch Information Fusion
• Learning View Priors for Single-view 3D Reconstruction
• Feed-Forward SceneDINO for Unsupervised Semantic Scene Completion
• LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction
• Domain Adaptation for Real-World Single View 3D Reconstruction
• Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
• Implicitly Defined Layers in Neural Networks
• Self-supervised 3D Shape and Viewpoint Estimation from Single Images for Robotics
• PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction
• Meta Deformation Network: Meta Functionals for Shape Correspondence
• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• MeshSDF: Differentiable Iso-Surface Extraction

Related

• Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs
• Geometry-Aware Network for Non-Rigid Shape Prediction from a Single View
• Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysis
• StructureNet: Hierarchical Graph Networks for 3D Shape Generation
• Mesh R-CNN