PESTalk: Speech-Driven 3D Facial Animation with Personalized Emotional Styles

2025/10/13 by Tianshun Han, Han, Tianshun, Benjia Zhou +11 · 1 citation
Computer Science · Psychology · #Face recognition and analysis #Emotion and Mood Recognition #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2512.05121

Abstract

PESTalk is a novel method for generating 3D facial animations with personalized emotional styles directly from speech. It overcomes key limitations of existing approaches by introducing a Dual-Stream Emotion Extractor (DSEE) that captures both time and frequency-domain audio features for fine-grained emotion analysis, and an Emotional Style Modeling Module (ESMM) that models individual expression patterns based on voiceprint characteristics. To address data scarcity, the method leverages a newly constructed 3D-EmoStyle dataset. Evaluations demonstrate that PESTalk outperforms state-of-the-art methods in producing realistic and personalized facial animations.

Citations

• PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation
• MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
• GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer
• SAiD: Speech-driven Blendshape Facial Animation with Diffusion
• Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation
• PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features
• DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser
• DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models
• FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion
• SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces
• EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
• CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior
• EMOCA: Emotion Driven Monocular Face Capture and Animation
• FaceFormer: Speech-Driven 3D Facial Animation with Transformers
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• Deep Learning for Micro-expression Recognition: A Survey
• 3D-TalkEmo: Learning to Synthesize 3D Emotional Talking Head
• MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• A Comparison of Metric Learning Loss Functions for End-To-End Speaker Verification
• MediaPipe: A Framework for Building Perception Pipelines
• Attention Is All You Need

Cited by

• PointCHR: Point Cloud Analysis via Curvature-Aware Hyperbolic Rectification

Related

• EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models
• Emotion Dependent Facial Animation from Affective Speech
• CSTalk: Correlation Supervised Speech-driven 3D Emotional Facial Animation Generation
• EmoVOCA: Speech-Driven Emotional 3D Talking Heads
• EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation