MAGIC-Talk: Motion-aware Audio-Driven Talking Face Generation with Customizable Identity Control

2025/10/26 by Nazarieh, Fatemeh, Feng, Zhenhua, Kanojia, Diptesh +2 · 1 citation
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.22810

Abstract

Audio-driven talking face generation has gained significant attention for applications in digital media and virtual avatars. While recent methods improve audio-lip synchronization, they often struggle with temporal consistency, identity preservation, and customization, especially in long video generation. To address these issues, we propose MAGIC-Talk, a one-shot diffusion-based framework for customizable and temporally stable talking face generation. MAGIC-Talk consists of ReferenceNet, which preserves identity and enables fine-grained facial editing via text prompts, and AnimateNet, which enhances motion coherence using structured motion priors. Unlike previous methods requiring multiple reference images or fine-tuning, MAGIC-Talk maintains identity from a single image while ensuring smooth transitions across frames. Additionally, a progressive latent fusion strategy is introduced to improve long-form video quality by reducing motion inconsistencies and flickering. Extensive experiments demonstrate that MAGIC-Talk outperforms state-of-the-art methods in visual quality, identity preservation, and synchronization accuracy, offering a robust solution for talking face generation.

Citations

• PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation
• MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
• Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation
• Faces that Speak: Jointly Synthesising Talking Face and Speech from Text
• EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis
• AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
• FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization
• Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style
• EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions
• VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
• InstantID: Zero-shot Identity-Preserving Generation in Seconds
• PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
• SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis
• IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models
• AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
• Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising
• Identity-Preserving Talking Face Generation with Landmark and Appearance Priors
• GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation
• Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
• EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
• Adding Conditional Control to Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• Structure and Content-Guided Video Synthesis with Diffusion Models
• DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
• Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
• MagicVideo: Efficient Video Generation With Latent Diffusion Models
• Make-A-Video: Text-to-Video Generation without Text-Video Data
• EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model
• Talking Face Generation with Multilingual TTS
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Emerging Properties in Self-Supervised Vision Transformers
• Emerging Properties in Self-Supervised Vision Transformers
• Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation
• Learning Transferable Visual Models From Natural Language Supervision
• Attention Is All You Need
• Image quality assessment: from error visibility to structural similarity

Cited by

• Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation

Related

• PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation
• FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis
• IMTalker: Efficient Audio-driven Talking Face Generation with Implicit Motion Transfer
• High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model
• That's What I Said: Fully-Controllable Talking Face Generation