2020/12/22 by Edgar Tretschk, Tretschk, Edgar, Ayush Tewari +9 · 43 citations
Computer Science · Engineering · #Advanced Vision and Imaging #Computer Graphics and Visualization Techniques #3D Shape Modeling and Analysis
paper · pdf · doi:10.48550/arxiv.2012.12247
We present Non-Rigid Neural Radiance Fields (NR-NeRF), a reconstruction and\nnovel view synthesis approach for general non-rigid dynamic scenes. Our\napproach takes RGB images of a dynamic scene as input (e.g., from a monocular\nvideo recording), and creates a high-quality space-time geometry and appearance\nrepresentation. We show that a single handheld consumer-grade camera is\nsufficient to synthesize sophisticated renderings of a dynamic scene from novel\nvirtual camera views, e.g. a `bullet-time' video effect. NR-NeRF disentangles\nthe dynamic scene into a canonical volume and its deformation. Scene\ndeformation is implemented as ray bending, where straight rays are deformed\nnon-rigidly. We also propose a novel rigidity network to better constrain rigid\nregions of the scene, leading to more stable results. The ray bending and\nrigidity network are trained without explicit supervision. Our formulation\nenables dense correspondence estimation across views and time, and compelling\nvideo editing applications such as motion exaggeration. Our code will be open\nsourced.\n