2021/08/18 by Chenxu Zhang, Yifan Zhao, Zhang, Chenxu +11 · 9 citations
Computer Science · #Animation #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer animation #Computer facial animation #Computer graphics (images) #Computer science #Computer vision #FOS: Computer and information sciences #Face (sociological concept) #Face recognition and analysis #Generative Adversarial Networks and Image Synthesis #Rendering (computer graphics) #Speech and Audio Processing #Speech recognition #cs.CV
paper · pdf · doi:10.48550/arxiv.2108.07938
published in arXiv (Cornell University) (Cornell University) · 10 pages, 9 figures. Accepted by ICCV 2021
arxiv created 2021/08/18 · openalex publication_date 2021/08/18 · arxiv updated 2021/08/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08
In this paper, we propose a talking face generation method that takes an audio signal as input and a short target video clip as reference, and synthesizes a photo-realistic video of the target face with natural lip motions, head poses, and eye blinks that are in-sync with the input audio signal. We note that the synthetic face attributes include not only explicit ones such as lip motions that have high correlations with speech, but also implicit ones such as head poses and eye blinks that have only weak correlation with the input audio. To model such complicated relationships among different face attributes with input audio, we propose a FACe Implicit Attribute Learning Generative Adversarial Network (FACIAL-GAN), which integrates the phonetics-aware, context-aware, and identity-aware information to synthesize the 3D face animation with realistic motions of lips, head poses, and eye blinks. Then, our Rendering-to-Video network takes the rendered face images and the attention map of eye blinks as input to generate the photo-realistic output video frames. Experimental results and user studies show our method can generate realistic talking face videos with not only synchronized lip motions, but also natural head movements and eye blinks, with better qualities than the results of state-of-the-art methods.