2020/12/05 by Guy Gafni, Gafni, Guy, Justus Thies +5 · 50 citations
Computer Science · Engineering · #3D Shape Modeling and Analysis #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Face recognition and analysis #Generative Adversarial Networks and Image Synthesis #Graphics (cs.GR) #cs.CV #cs.GR
paper · pdf · doi:10.48550/arxiv.2012.03065
Video: https://youtu.be/m7oROLdQnjk | Project page: https://gafniguy.github.io/4D-Facial-Avatars/
arxiv created 2020/12/05 · openalex publication_date 2020/12/05 · arxiv updated 2020/12/08 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face. Digitally modeling and reconstructing a talking human is a key building-block for a variety of applications. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoints or head-poses is required. In contrast to state-of-the-art approaches that model the geometry and material properties explicitly, or are purely image-based, we introduce an implicit representation of the head based on scene representation networks. To handle the dynamics of the face, we combine our scene representation network with a low-dimensional morphable model which provides explicit control over pose and expressions. We use volumetric rendering to generate images from this hybrid representation and demonstrate that such a dynamic neural scene representation can be learned from monocular input data only, without the need of a specialized capture setup. In our experiments, we show that this learned volumetric representation allows for photo-realistic image generation that surpasses the quality of state-of-the-art video-based reenactment methods.