Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Prediction

2025/10/13 by Guichoux, Téo, Lemerle, Théodor, Mehta, Shivam +5
#68T07 #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2510.12834

Abstract

Human communication is multimodal, with speech and gestures tightly coupled, yet most computational methods for generating speech and gestures synthesize them sequentially, weakening synchrony and prosody alignment. We introduce Gelina, a unified framework that jointly synthesizes speech and co-speech gestures from text using interleaved token sequences in a discrete autoregressive backbone, with modality-specific decoders. Gelina supports multi-speaker and multi-style cloning and enables gesture-only synthesis from speech inputs. Subjective and objective evaluations demonstrate competitive speech quality and improved gesture generation over unimodal baselines.

Citations

• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis
• FastTalker: Jointly Generating Speech and Conversational Gestures from Text
• WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
• Small-E: Small Language Model with Linear Attention for Efficient Speech Synthesis
• Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis
• EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
• MoMask: Generative Masked Modeling of 3D Human Motions
• Unified speech and gesture synthesis using flow matching
• Matcha-TTS: A fast TTS architecture with conditional flow matching
• Better speech synthesis through scaling
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
• High Fidelity Neural Audio Compression
• BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders
• AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
• Speech gesture generation from the trimodal context of text, audio, and speaker identity
• Expressive Body Capture: 3D Hands, Face, and Body from a Single Image
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
• On the Continuity of Rotation Representations in Neural Networks
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Neural Machine Translation of Rare Words with Subword Units
• Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesis

Related

• Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis
• MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantization
• DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2
• Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates
• DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 framework