Voice Impression Control in Zero-Shot TTS

2025/06/06 by Fujita, Kenichi, Fujita, Keinichi, Horiguchi, Shota +2 · 1 citation
Computer Science · Psychology · #Speech and Audio Processing #Speech Recognition and Synthesis #Phonetics and Phonology Research

paper · doi:10.48550/arxiv.2506.05688

Abstract

Para-/non-linguistic information in speech is pivotal in shaping the listeners' impression. Although zero-shot text-to-speech (TTS) has achieved high speaker fidelity, modulating subtle para-/non-linguistic information to control perceived voice characteristics, i.e., impressions, remains challenging. We have therefore developed a voice impression control method in zero-shot TTS that utilizes a low-dimensional vector to represent the intensities of various voice impression pairs (e.g., dark-bright). The results of both objective and subjective evaluations have demonstrated our method's effectiveness in impression control. Furthermore, generating this vector via a large language model enables target-impression generation from a natural language description of the desired impression, thus eliminating the need for manual optimization. Audio examples are available on our demo page (https://ntt-hilab-gensp.github.io/is2025voiceimpression/).

Citations

• Speech Synthesis along Perceptual Voice Quality Dimensions
• LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
• LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
• ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models
• Controllable Speaking Styles Using a Large Language Model
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• PromptTTS: Controllable Text-to-Speech with Text Descriptions
• Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?
• Cross-speaker emotion disentangling and transfer for end-to-end speech synthesis
• Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesis
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
• Investigating on Incorporating Pretrained and Learnable Speaker\n Representations for Multi-Speaker Multi-Style Text-to-Speech
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Robust speaker recognition using unsupervised adversarial invariance
• Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Attention Is All You Need
• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
• Adam: A Method for Stochastic Optimization
• Unsupervised Domain Adaptation by Backpropagation

Cited by

• LibriTTS-VI: A Public Corpus and Novel Methods for Efficient Voice Impression Control

Related

• LibriTTS-VI: A Public Corpus and Novel Methods for Efficient Voice Impression Control
• DS-TTS: Zero-Shot Speaker Style Adaptation from Voice Clips via Dynamic Dual-Style Feature Modulation
• Improving Language Model-Based Zero-Shot Text-to-Speech Synthesis with Multi-Scale Acoustic Prompts
• Expressive Prompting: Improving Emotion Intensity and Speaker Consistency in Zero-Shot TTS
• MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidances