2024/01/02 by Renshuai Liu, Bowen Ma, Liu, Renshuai +13 · 1 voice · 8 citations
Computer Science · Mathematics · #Aesthetics #Art #Artificial intelligence #Computer science #Computer vision #Control (management) #Controllability #Encoder #Expression (computer science) #Face (sociological concept) #Face recognition and analysis #Generative Adversarial Networks and Image Synthesis #Human–computer interaction #Identity (music) #Image Retrieval and Classification Techniques #Linguistics #Mathematics #Scalability #cs.CV
paper · pdf · doi:10.48550/arxiv.2401.01207
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/01/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In human-centric content generation, the pre-trained text-to-image models struggle to produce user-wanted portrait images, which retain the identity of individuals while exhibiting diverse expressions. This paper introduces our efforts towards personalized face generation. To this end, we propose a novel multi-modal face generation framework, capable of simultaneous identity-expression control and more fine-grained expression synthesis. Our expression control is so sophisticated that it can be specialized by the fine-grained emotional vocabulary. We devise a novel diffusion model that can undertake the task of simultaneously face swapping and reenactment. Due to the entanglement of identity and expression, it's nontrivial to separately and precisely control them in one framework, thus has not been explored yet. To overcome this, we propose several innovative designs in the conditional diffusion model, including balancing identity and expression encoder, improved midpoint sampling, and explicitly background conditioning. Extensive experiments have demonstrated the controllability and scalability of the proposed framework, in comparison with state-of-the-art text-to-image, face swapping, and face reenactment methods.