UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech

2025/05/15 by Jiaxuan Liu, Yang Xiang, Liu, Jiaxuan +11
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Emotion and Mood Recognition #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mental Health via Writing #Sentiment Analysis and Opinion Mining

paper · pdf · doi:10.48550/arxiv.2505.10599

openalex publication_date 2025/05/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent large language models (LLMs) have made great progress in the field of text-to-speech (TTS), but they still face major challenges in synthesizing fine-grained emotional speech in an interpretable manner. Traditional methods rely on discrete emotion labels to control emotion categories and intensities, which cannot capture the complexity and continuity of human emotional perception and expression. The lack of large-scale emotional speech datasets with balanced emotion distributions and fine-grained emotional annotations often causes overfitting in synthesis models and impedes effective emotion control. To address these issues, we propose UDDETTS, a universal LLM framework unifying discrete and dimensional emotions for controllable emotional TTS. This model introduces the interpretable Arousal-Dominance-Valence (ADV) space for dimensional emotion description and supports emotion control driven by either discrete emotion labels or nonlinearly quantified ADV values. Furthermore, a semi-supervised training strategy is designed to comprehensively utilize diverse speech datasets with different types of emotional annotations to train the UDDETTS. Experiments show that UDDETTS achieves linear emotion control along three interpretable dimensions, and exhibits superior end-to-end emotional speech synthesis capabilities. Code and demos are available at: https://anonymous.4open.science/w/UDDETTS.

Citations

• FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbot
• EmoTale: An Enacted Speech-emotion Dataset in Danish
• IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
• HiFiTTS-2: A Large-Scale High Bandwidth Speech Dataset
• CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
• MER 2025: When Affective Computing Meets Large Language Models
• Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
• IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
• MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Styles
• Scaling Speech-Text Pre-training with Synthetic Interleaved Data
• Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models
• Bridging Discrete and Continuous: A Multimodal Strategy for Complex Emotion Detection
• FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
• GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Prompts
• 3D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization
• emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
• Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling
• AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs
• Finite Scalar Quantization: VQ-VAE Made Simple
• AffectEcho: Speaker Independent and Language-Agnostic Emotion and Affect Transfer for Speech Synthesis
• EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
• Prompting Large Language Models with Speech Recognition Abilities
• 3D-Speaker: A Large-Scale Multi-Device, Multi-Distance, and Multi-Dialect Corpus for Speech Representation Disentanglement
• Label-Aware Hyperbolic Embeddings for Fine-grained Emotion Classification
• Learning Emotional Representations from Imbalanced Speech Data for Speech Emotion Recognition and Emotional Text-to-Speech
• EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesis
• LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus
• ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models
• SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
• MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
• Gaussian-smoothed Imbalance Data Improves Speech Emotion Recognition
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• PromptTTS: Controllable Text-to-Speech with Text Descriptions
• EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance
• Semi-supervised learning for continuous emotional intensity controllable speech synthesis with disentangled representations
• SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data
• Speech Synthesis with Mixed Emotions
• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
• Cross-speaker emotion disentangling and transfer for end-to-end speech synthesis
• Automatically Select Emotion for Response via Personality-affected Emotion Transition
• Emotional Voice Conversion: Theory, Databases and ESD
• Emotional voice conversion: Theory, databases and ESD
• Hi-Fi Multi-Speaker English TTS Dataset
• Improved Denoising Diffusion Probabilistic Models
• Controllable Emotion Transfer For End-to-End Speech Synthesis
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• Common Voice: A Massively-Multilingual Speech Corpus
• Dimensional Emotion Detection from Categorical Emotion
• Semi-Supervised Generative Modeling for Controllable Speech Synthesis
• MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in\n Conversations
• The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems

Related

• Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotions
• Fine-grained Emotional Control of Text-To-Speech: Learning To Rank Inter- And Intra-Class Emotion Intensities
• EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting
• A Methodology for Controlling the Emotional Expressiveness in Synthetic Speech -- a Deep Learning approach
• UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Prompts