2025/02/02 by Seongro Yoon, Cha, Junuk, Valeriya Strizhkova +5 · 2 citations
Computer Science · Engineering · Psychology · #Cognitive psychology #Cognitive science #Communication #Computer Vision and Pattern Recognition (cs.CV) #Computer science #FOS: Computer and information sciences #Face recognition and analysis #Geology #Head (geology) #Psychology #Robotics and Automated Systems #Social Robot Interaction and HRI
paper · pdf · doi:10.48550/arxiv.2502.00654
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/02/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
3D Gaussian splatting-based talking head synthesis has recently gained attention for its ability to render high-fidelity images with real-time inference speed. However, since it is typically trained on only a short video that lacks the diversity in facial emotions, the resultant talking heads struggle to represent a wide range of emotions. To address this issue, we propose a lip-aligned emotional face generator and leverage it to train our EmoTalkingGaussian model. It is able to manipulate facial emotions conditioned on continuous emotion values (i.e., valence and arousal); while retaining synchronization of lip movements with input audio. Additionally, to achieve the accurate lip synchronization for in-the-wild audio, we introduce a self-supervised learning method that leverages a text-to-speech network and a visual-audio synchronization network. We experiment our EmoTalkingGaussian on publicly available videos and have obtained better results than state-of-the-arts in terms of image quality (measured in PSNR, SSIM, LPIPS), emotion expression (measured in V-RMSE, A-RMSE, V-SA, A-SA, Emotion Accuracy), and lip synchronization (measured in LMD, Sync-E, Sync-C), respectively.