Audio2Face-3D: Audio-driven Realistic Facial Animation For Digital Avatars

2025/08/22 by NVIDIA, :, Chung, Chaeyeon +6 · 1 citation
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Graphics (cs.GR) #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2508.16401

Abstract

Audio-driven facial animation presents an effective solution for animating digital avatars. In this paper, we detail the technical aspects of NVIDIA Audio2Face-3D, including data acquisition, network architecture, retargeting methodology, evaluation metrics, and use cases. Audio2Face-3D system enables real-time interaction between human users and interactive avatars, facilitating facial animation authoring for game characters. To assist digital avatar creators and game developers in generating realistic facial animations, we have open-sourced Audio2Face-3D networks, SDK, training framework, and example dataset.

Citations

• Voice Cloning: Comprehensive Survey
• Generative Adversarial Network based Voice Conversion: Techniques, Challenges, and Recent Advancements
• MMHead: Towards Fine-grained Multi-modal 3D Facial Animation
• EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions
• AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding
• Refined Inverse Rigging: A Balanced Approach to High-fidelity Blendshape Animation
• Mixtral of Experts
• DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models
• FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion
• ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles
• GPT-4 Technical Report
• FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learning
• GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
• DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
• StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• FaceFormer: Speech-Driven 3D Facial Animation with Transformers
• Phone-to-audio alignment without text: A Semi-supervised Approach
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
• AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
• Learning Transferable Visual Models From Natural Language Supervision
• Denoising Diffusion Implicit Models
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Denoising Diffusion Probabilistic Models
• Capture, Learning, and Synthesis of 3D Speaking Styles
• Dynamic programming algorithm optimization for spoken word recognition

Cited by

• KeyframeFace: Language-Driven Facial Animation via Semantic Keyframes

Related

• READ Avatars: Realistic Emotion-controllable Audio Driven Avatars
• Audio- and Gaze-driven Facial Animation of Codec Avatars
• Audio Driven Real-Time Facial Animation for Social Telepresence
• MMFace4D: A Large-Scale Multi-Modal 4D Face Dataset for Audio-Driven 3D Face Animation
• Make-A-Character 2: Animatable 3D Character Generation From a Single Image