Multi-interaction TTS toward professional recording reproduction

2025/07/01 by Kanagawa, Hiroki, Fujita, Kenichi, Watanabe, Aya +1
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2507.00808

Abstract

Voice directors often iteratively refine voice actors' performances by providing feedback to achieve the desired outcome. While this iterative feedback-based refinement process is important in actual recordings, it has been overlooked in text-to-speech synthesis (TTS). As a result, fine-grained style refinement after the initial synthesis is not possible, even though the synthesized speech often deviates from the user's intended style. To address this issue, we propose a TTS method with multi-step interaction that allows users to intuitively and rapidly refine synthesized speech. Our approach models the interaction between the TTS model and its user to emulate the relationship between voice actors and voice directors. Experiments show that the proposed model with its corresponding dataset enables iterative style refinements in accordance with users' directions, thus demonstrating its multi-interaction capability. Sample audios are available: https://ntt-hilab-gensp.github.io/ssw13multiinteractiontts/

Citations

• Voice Attribute Editing with Text Prompt
• VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
• Natural language guidance of high-fidelity text-to-speech with synthetic annotations
• Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
• FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
• Dynamic / ME-JEPA v2.0.0-rc1: Audited World-Model Runtime and Verified Training-Corpus Artifact
• A Review of Deep Learning Techniques for Speech Processing
• Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• PromptTTS: Controllable Text-to-Speech with Text Descriptions
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• PaLM: Scaling Language Modeling with Pathways
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
• Training language models to follow instructions with human feedback
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Program Synthesis with Large Language Models
• A Survey on Neural Speech Synthesis
• Investigating on Incorporating Pretrained and Learnable Speaker\n Representations for Multi-Speaker Multi-Style Text-to-Speech
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Learning Alignment for Multimodal Emotion Recognition from Speech
• Fine-grained robust prosody transfer for single-speaker neural text-to-speech
• Robust and fine-grained prosody control of end-to-end speech synthesis
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• FiLM: Visual Reasoning with a General Conditioning Layer
• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems

Related

• DS-TTS: Zero-Shot Speaker Style Adaptation from Voice Clips via Dynamic Dual-Style Feature Modulation
• Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis
• Controllable Text-to-Speech Synthesis with Masked-Autoencoded Style-Rich Representation
• MM-TTS: Multi-modal Prompt based Style Transfer for Expressive Text-to-Speech Synthesis
• Prosodic Parameter Manipulation in TTS generated speech for Controlled Speech Generation