vix.ing · top · new · best · stats

Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation

2024/10/10 by Jiahao Cui, Hui Li, Cui, Jiahao +15 · 48 citations
Computer Science · #Advanced Vision and Imaging #Animation #Art #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer graphics (images) #Computer science #Computer vision #Duration (music) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Geography #High resolution #Image (mathematics) #Literature #Portrait #Remote sensing #Visual arts

paper · pdf · doi:10.48550/arxiv.2410.07718

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/10/10 · openalex created_date 2024/10/13 · openalex updated_date 2026/07/28

Abstract

Recent advances in latent diffusion-based generative models for portrait image animation, such as Hallo, have achieved impressive results in short-duration video synthesis. In this paper, we present updates to Hallo, introducing several design enhancements to extend its capabilities. First, we extend the method to produce long-duration videos. To address substantial challenges such as appearance drift and temporal artifacts, we investigate augmentation strategies within the image space of conditional motion frames. Specifically, we introduce a patch-drop technique augmented with Gaussian noise to enhance visual consistency and temporal coherence over long duration. Second, we achieve 4K resolution portrait video generation. To accomplish this, we implement vector quantization of latent codes and apply temporal alignment techniques to maintain coherence across the temporal dimension. By integrating a high-quality decoder, we realize visual synthesis at 4K resolution. Third, we incorporate adjustable semantic textual labels for portrait expressions as conditional inputs. This extends beyond traditional audio cues to improve controllability and increase the diversity of the generated content. To the best of our knowledge, Hallo2, proposed in this paper, is the first method to achieve 4K resolution and generate hour-long, audio-driven portrait image animations enhanced with textual prompts. We have conducted extensive experiments to evaluate our method on publicly available datasets, including HDTF, CelebV, and our introduced "Wild" dataset. The experimental results demonstrate that our approach achieves state-of-the-art performance in long-duration portrait video animation, successfully generating rich and controllable content at 4K resolution for duration extending up to tens of minutes. Project page https://fudan-generative-vision.github.io/hallo2

Cited by

Related