DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models

2023/05/08 by Sicheng Yang, Zhiyong Wu, Yang, Sicheng +13 · 16 citations
Engineering · Computer Science · #Human Motion and Animation #Hand Gesture Recognition Systems #Human Pose and Action Recognition

paper · pdf · doi:10.48550/arxiv.2305.04919

Abstract

The art of communication beyond speech there are gestures. The automatic co-speech gesture generation draws much attention in computer animation. It is a challenging task due to the diversity of gestures and the difficulty of matching the rhythm and semantics of the gesture to the corresponding speech. To address these problems, we present DiffuseStyleGesture, a diffusion model based speech-driven gesture generation approach. It generates high-quality, speech-matched, stylized, and diverse co-speech gestures based on given speeches of arbitrary length. Specifically, we introduce cross-local attention and self-attention to the gesture diffusion pipeline to generate better speech matched and realistic gestures. We then train our model with classifier-free guidance to control the gesture style by interpolation or extrapolation. Additionally, we improve the diversity of generated gestures with different initial gestures and noise. Extensive experiments show that our method outperforms recent approaches on speech-driven gesture generation. Our code, pre-trained models, and demos are available at https://github.com/YoungSeng/DiffuseStyleGesture.

Cited by

• Evaluation of Generative Models for Emotional 3D Animation Generation in VR
• ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body
• Towards Unified Co-Speech Gesture Generation via Hierarchical Implicit Periodicity Learning
• InteracTalker: Prompt-Based Human-Object Interaction with Co-Speech Gesture Generation
• A Survey of Body and Face Motion: Datasets, Performance Evaluation Metrics and Generative Techniques
• CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generation
• Knowing Ourselves Through Others: Reflecting with AI in Digital Human Debates
• Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraints
• Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark
• OmniMotion: Multimodal Motion Generation with Continuous Masked Autoregression
• MimicParts: Part-aware Style Injection for Speech-Driven 3D Motion Generation
• Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agents
• SIG-Chat: Spatial Intent-Guided Conversational Gesture Generation Involving How, When and Where
• GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generation
• Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Models
• SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning

Related

• DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model
• Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation
• DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
• SIGGesture: Generalized Co-Speech Gesture Synthesis via Semantic Injection with Large-Scale Pre-Training Diffusion Models
• ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis