BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis

2023/02/28 by Lior Yariv, Yariv, Lior, Peter Hedman +13 · 25 citations
Computer Science · Engineering · #Computer Graphics and Visualization Techniques #3D Shape Modeling and Analysis #Advanced Vision and Imaging

paper · pdf · doi:10.48550/arxiv.2302.14859

Abstract

We present a method for reconstructing high-quality meshes of large unbounded real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians. Finally, we optimize this baked representation to best reproduce the captured viewpoints, resulting in a model that can leverage accelerated polygon rasterization pipelines for real-time view synthesis on commodity hardware. Our approach outperforms previous scene representations for real-time rendering in terms of accuracy, speed, and power consumption, and produces high quality meshes that enable applications such as appearance editing and physical simulation.

Cited by

• ShapeR: Robust Conditional 3D Shape Generation from Casual Captures
• Nexels: Neurally-Textured Surfels for Real-Time Novel View Synthesis with Sparse Geometries
• Radiance Meshes for Volumetric Reconstruction
• SurfFill: Completion of LiDAR Point Clouds via Gaussian Surfel Splatting
• Content-Aware Texturing for Gaussian Splatting
• PolarGS: Polarimetric Cues for Ambiguity-Free Gaussian Splatting with Accurate Geometry Recovery
• IE-SRGS: An Internal-External Knowledge Fusion Framework for High-Fidelity 3D Gaussian Splatting Super-Resolution
• Neural Texture Splatting: Expressive 3D Gaussian Splatting for View Synthesis, Geometry, and Dynamic Reconstruction
• SVRecon: Sparse Voxel Rasterization for Surface Reconstruction
• Optimizing 3D Gaussian Splattering for Mobile GPUs
• Geometric implicit neural representations for signed distance functions
• GauSSmart: Enhanced 3D Reconstruction through 2D Foundation Models and Geometric Filtering
• Leveraging 2D Priors and SDF Guidance for Dynamic Urban Scene Rendering
• UniGS: Unified Geometry-Aware Gaussian Splatting for Multimodal Rendering
• VA-GS: Enhancing the Geometric Representation of Gaussian Splatting via View Alignment
• INR-Bench: A Unified Benchmark for Implicit Neural Representations in Multi-Domain Regression and Reconstruction
• Learning Neural Exposure Fields for View Synthesis
• HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video
• Triangle Splatting+: Differentiable Rendering with Opaque Triangles
• OMeGa: Joint Optimization of Explicit Meshes and Gaussian Splats for Robust Scene-Level Surface Reconstruction
• GS-2M: Gaussian Splatting for Joint Mesh Reconstruction and Material Decomposition
• Disentangled Geometry and Appearance for Efficient Multi-View Surface Reconstruction and Rendering
• Accurate and Complete Surface Reconstruction from 3D Gaussians via Direct SDF Learning
• SynthDrive: Scalable Real2Sim2Real Sensor Simulation Pipeline for High-Fidelity Asset Generation and Driving Data Synthesis
• Perceive-Sample-Compress: Towards Real-Time 3D Gaussian Splatting

Related

• HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces
• Baking Neural Radiance Fields for Real-Time View Synthesis
• Neural Lumigraph Rendering
• Real-Time Neural Rasterization for Large Scenes
• Real-Time Scene Reconstruction using Light Field Probes