Surface-Aligned Neural Radiance Fields for Controllable 3D Human Synthesis

2022/01/05 by Xu, Tianhan, Fujita, Yasuhiro, Matsumoto, Eiichi · 3 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2201.01683

Abstract

We propose a new method for reconstructing controllable implicit 3D human models from sparse multi-view RGB videos. Our method defines the neural scene representation on the mesh surface points and signed distances from the surface of a human body mesh. We identify an indistinguishability issue that arises when a point in 3D space is mapped to its nearest surface point on a mesh for learning surface-aligned neural scene representation. To address this issue, we propose projecting a point onto a mesh surface using a barycentric interpolation with modified vertex normals. Experiments with the ZJU-MoCap and Human3.6M datasets show that our approach achieves a higher quality in a novel-view and novel-pose synthesis than existing methods. We also demonstrate that our method easily supports the control of body shape and clothes. Project page: https://pfnet-research.github.io/surface-aligned-nerf/.

Cited by

• AvatarBrush: Monocular Reconstruction of Gaussian Avatars with Intuitive Local Editing
• Parametric Gaussian Human Model: Generalizable Prior for Efficient and Realistic Human Avatar Modeling
• HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers

Related

• Animatable Neural Radiance Fields from Monocular RGB Videos
• H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion
• HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs
• Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis
• Geometry-Guided Progressive NeRF for Generalizable and Efficient Neural Human Rendering