AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
2024/03/26 by Huawei Wei, Wei, Huawei, Zejun Yang +3 · 88 citations
Earth and Planetary Sciences · #3D Surveying and Cultural Heritage
paper · pdf · doi:10.48550/arxiv.2403.17694
Abstract
In this study, we propose AniPortrait, a novel framework for generating high-quality animation driven by audio and a reference portrait image. Our methodology is divided into two stages. Initially, we extract 3D intermediate representations from audio and project them into a sequence of 2D facial landmarks. Subsequently, we employ a robust diffusion model, coupled with a motion module, to convert the landmark sequence into photorealistic and temporally consistent portrait animation. Experimental results demonstrate the superiority of AniPortrait in terms of facial naturalness, pose diversity, and visual quality, thereby offering an enhanced perceptual experience. Moreover, our methodology exhibits considerable potential in terms of flexibility and controllability, which can be effectively applied in areas such as facial motion editing or face reenactment. We release code and model weights at https://github.com/scutzzj/AniPortrait
Cited by
- LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
- ViDS: Video Diffusion Shader using 3D Face Tracking
- FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANs
- In-Context Audio Control of Video Diffusion Transformers
- SynergyWarpNet: Attention-Guided Cooperative Warping for Neural Portrait Animation
- Instant Expressive Gaussian Head Avatars at Over 100 FPS
- DeX-Portrait: Disentangled and Expressive Portrait Animation via Explicit and Latent Motion Representations
- Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamics
- ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body
- FacEDiT: Unified Talking Face Editing and Generation via Facial Motion Infilling
- Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation
- FactorPortrait: Controllable Portrait Animation via Disentangled Expression, Pose, and Viewpoint
- REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillation
- GaussianHeadTalk: Wobble-Free 3D Talking Heads with Audio Driven Gaussian Splatting
- TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detection
- A Survey of Body and Face Motion: Datasets, Performance Evaluation Metrics and Generative Techniques
- Beyond Boundary Frames: Talking-Head Inbetweening via Context-Aware Motion Modeling
- In-Context Sync-LoRA for Portrait Video Editing
- StyleYourSmile: Cross-Domain Face Retargeting Without Paired Multi-Style Data
- AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement
- AI killed the video star. Audio-driven diffusion model for expressive talking head generation
- ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search
- See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement
- Lookahead Anchoring: Preserving Character Identity in Audio-Driven Human Animation
- MAGIC-Talk: Motion-aware Audio-Driven Talking Face Generation with Customizable Identity Control
- MVP4D: Multi-View Portrait Video Diffusion for Animatable 4D Avatars
- VividAnimator: An End-to-End Audio and Pose-driven Half-Body Human Animation Framework
- A Bridge from Audio to Video: Phoneme-Viseme Alignment Allows Every Face to Speak Multiple Languages
- Audio Driven Real-Time Facial Animation for Social Telepresence
- SynchroRaMa : Lip-Synchronized and Emotion-Aware Talking Face Generation via Multi-Modal Emotion Embedding
- Talking Head Generation via AU-Guided Landmark Prediction
- Stable Video-Driven Portraits
- Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animation
- Wan-Animate: Unified Character Animation and Replacement with Holistic Replication
- FluentAvatar: Flicker-Free Talking-Head Animation via Phoneme-Guided Autoregressive Modeling
- MFFI: Multi-Dimensional Face Forgery Image Dataset for Real-World Scenarios
- Human Motion Video Generation: A Survey
- EmoCAST: Emotional Talking Portrait via Emotive Text Description
- InfinityHuman: Towards Long-Term Audio-Driven Human
- OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulation
- InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
- TalkVid: A Large-Scale Diversified Dataset for Audio-Driven Talking Head Synthesis
- EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis
- FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animation
- LaVieID: Local Autoregressive Diffusion Transformers for Identity-Preserving Video Creation
- StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation
- RAP: Real-time Audio-driven Portrait Animation with Video Diffusion Transformer
- UniTalker: Conversational Speech-Visual Synthesis
- READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation
- X-Actor: Emotional and Expressive Long-Range Portrait Acting from Audio
- PoseGuard: Pose-Guided Generation with Safety Guardrails
- Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering
- X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention
- DiTalker: A Unified DiT-based Framework for High-Quality and Speaking Styles Controllable Portrait Animation
- MagicAnime: A Hierarchically Annotated, Multimodal and Multitasking Dataset with Benchmarks for Cartoon Animation Generation
- Controllable Video Generation: A Survey
- HairShifter: Consistent and High-Fidelity Video Hair Transfer via Anchor-Guided Animation
- Democratizing High-Fidelity Co-Speech Gesture Video Generation
- MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation
- EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation
- FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Cases
- JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching
- OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation
- Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions
- Controllable and Expressive One-Shot Video Head Swapping
- Audio-Sync Video Generation with Multi-Stream Temporal Control
- LLIA -- Enabling Low-Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Models
- Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation
- SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers
- TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection
- A Unit Enhancement and Guidance Framework for Audio-Driven Avatar Video Generation
- Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimization
- MMGT: Motion Mask Guided Two-Stage Network for Co-Speech Gesture Video Generation
- Speaking images. A novel framework for the automated self-description of artworks
- FaceEditTalker: Controllable Talking Head Generation with Facial Attribute Editing
- Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation
- Exploring Timeline Control for Facial Motion Generation
- OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers
- Beyond Face Swapping: A Diffusion-Based Digital Human Benchmark for Multimodal Deepfake Detection
- Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation
- Test-Time Augmentation for Pose-invariant Face Recognition
- KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution
- MagicPortrait: Temporally Consistent Face Reenactment with 3D Geometric Guidance
- Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
- Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation
- FaceCraft4D: Animated 3D Facial Avatar Generation from a Single Image
- DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusion
- FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis
Related