LibriTTS-VI: A Public Corpus and Novel Methods for Efficient Voice Impression Control

2025/09/19 by Junki Ohmura, Yuki Ito, Ohmura, Junki +7
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.15626

openalex publication_date 2025/09/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Numerical voice impression (VI) control (e.g., scaling brightness) enables fine-grained control in text-to-speech (TTS). However, it faces two challenges: no public corpus and impression leakage, where reference audio biases synthesized voice away from the target VI. To address the first challenge, we introduce LibriTTS-VI, the first public VI corpus built on LibriTTS-R. For the second, we hypothesize a single reference causes leakage by entangling speaker identity and VI. To mitigate this, we propose 1) disentangled training with two utterances from the same speaker for speaker and VI conditioning, and 2) a reference-free method controlling the impression solely via target VI. Experimentally, our best method improves controllability: 11-dimensional VI mean squared error drops from 0.61 to 0.41 objectively and 1.15 to 0.92 subjectively. A comparison with a prompt-based TTS reveals imprecise numerical control and entanglement between VI and text semantics, which our methods overcome.

Citations

• InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems
• Voice Impression Control in Zero-Shot TTS
• Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models
• Speech Synthesis along Perceptual Voice Quality Dimensions
• DrawSpeech: Expressive Speech Synthesis Using Prosodic Sketches as Control Conditions
• LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data
• Natural language guidance of high-fidelity text-to-speech with synthetic annotations
• Seamless: Multilingual Expressive and Streaming Speech Translation
• HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis
• PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions
• VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design
• EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech
• PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions
• LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus
• Controllable Speaking Styles Using a Large Language Model
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt
• Scalable Diffusion Models with Transformers
• Robust Speech Recognition via Large-Scale Weak Supervision
• PromptTTS: Controllable Text-to-Speech with Text Descriptions
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
• VoiceMe: Personalized voice generation in TTS
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• ESPnet2-TTS: Extending the Edge of TTS Research
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• Conditional Variational Autoencoder with Adversarial Learning for\n End-to-End Text-to-Speech
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Emotional End-to-End Neural Speech Synthesizer
• Unsupervised Domain Adaptation by Backpropagation

Related

• Voice Impression Control in Zero-Shot TTS
• Voice Conversion for Likability Control via Automated Rating of Speech Synthesis Corpora
• Low-resource expressive text-to-speech using data augmentation
• LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning
• PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions