AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars

2025/05/21 by T. Zhang, Jian Zhao, Zhang, Tianbao +12
Computer Science · #68T10 #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Graphics (cs.GR) #Music Technology and Sound Studies #Music and Audio Processing #Sound (cs.SD) #Video Analysis and Summarization #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.15058

openalex publication_date 2025/05/21 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Whole-body audio-driven avatar pose and expression generation is a critical task for creating lifelike digital humans and enhancing the capabilities of interactive virtual agents, with wide-ranging applications in virtual reality, digital entertainment, and remote communication. Existing approaches often generate audio-driven facial expressions and gestures independently, which introduces a significant limitation: the lack of seamless coordination between facial and gestural elements, resulting in less natural and cohesive animations. To address this limitation, we propose AsynFusion, a novel framework that leverages diffusion transformers to achieve harmonious expression and gesture synthesis. The proposed method is built upon a dual-branch DiT architecture, which enables the parallel generation of facial expressions and gestures. Within the model, we introduce a Cooperative Synchronization Module to facilitate bidirectional feature interaction between the two modalities, and an Asynchronous LCM Sampling strategy to reduce computational overhead while maintaining high-quality outputs. Extensive experiments demonstrate that AsynFusion achieves state-of-the-art performance in generating real-time, synchronized whole-body animations, consistently outperforming existing methods in both quantitative and qualitative evaluations.

Citations

• Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony
• Towards Variable and Coordinated Holistic Co-Speech Motion Generation
• DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
• From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
• EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
• Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion
• CorrTalk: Correlation Between Hierarchical Speech and Facial Activity Variances for 3D Animation
• Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
• MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentions
• EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation
• DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models
• EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
• Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation
• CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior
• Scalable Diffusion Models with Transformers
• Generating Holistic 3D Human Motion from Speech
• Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
• Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation
• BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis
• RePaint: Inpainting using Denoising Diffusion Probabilistic Models
• FaceFormer: Speech-Driven 3D Facial Animation with Transformers
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
• Learning Speech-driven 3D Conversational Gestures from Video
• AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
• Denoising Diffusion Implicit Models
• Speech gesture generation from the trimodal context of text, audio, and speaker identity
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Denoising Diffusion Probabilistic Models
• Modality Dropout for Improved Performance-driven Talking Faces
• Learning Individual Styles of Conversational Gesture
• Capture, Learning, and Synthesis of 3D Speaking Styles
• Expressive Body Capture: 3D Hands, Face, and Body from a Single Image
• Efficient Attention: Attention with Linear Complexities
• Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture\n Generation for Humanoid Robots
• Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization
• Vid2speech: Speech Reconstruction from Silent Video
• DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation

Related

• SyncAnimation: A Real-Time End-to-End Framework for Audio-Driven Human Pose and Talking Head Animation
• Audio Driven Real-Time Facial Animation for Social Telepresence
• EMO2: End-Effector Guided Audio-Driven Avatar Video Generation
• One Shot, One Talk: Whole-body Talking Avatar from a Single Image
• MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation