Speaker-Conditioned Phrase Break Prediction for Text-to-Speech with Phoneme-Level Pre-trained Language Model

2025/08/31 by Dong Yang, Yang, Dong, Yuki Saito +11
Computer Science · #Speech Recognition and Synthesis #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.00675

Abstract

This paper advances phrase break prediction (also known as phrasing) in multi-speaker text-to-speech (TTS) systems. We integrate speaker-specific features by leveraging speaker embeddings to enhance the performance of the phrasing model. We further demonstrate that these speaker embeddings can capture speaker-related characteristics solely from the phrasing task. Besides, we explore the potential of pre-trained speaker embeddings for unseen speakers through a few-shot adaptation method. Furthermore, we pioneer the application of phoneme-level pre-trained language models to this TTS front-end task, which significantly boosts the accuracy of the phrasing model. Our methods are rigorously assessed through both objective and subjective evaluations, demonstrating their effectiveness.

Citations

• LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning
• Frame-Wise Breath Detection with Self-Training: An Exploration of Enhancing Breath Naturalness in Text-to-Speech
• Emotion-Aware Prosodic Phrasing for Expressive Text-to-Speech
• Matcha-TTS: A fast TTS architecture with conditional flow matching
• VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design
• Cross-Lingual Transfer Learning for Phrase Break Prediction with Multilingual Language Model
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis
• LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus
• Duration-aware pause insertion using pre-trained language model for multi-speaker text-to-speech
• Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining
• Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions
• ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in Paragraph-based TTS
• Expressive, Variable, and Controllable Duration Modelling in TTS
• Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context
• A Survey on Neural Speech Synthesis
• Conditional Variational Autoencoder with Adversarial Learning for\n End-to-End Text-to-Speech
• SpeechBrain: A General-Purpose Speech Toolkit
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
• Phrase break prediction with bidirectional encoder representations in Japanese text-to-speech synthesis
• Speaker Recognition Based on Deep Learning: An Overview
• Speaker recognition based on deep learning: An overview
• SpeakerNet: 1D Depth-wise Separable Convolutional Network for Text-Independent Speaker Recognition and Verification
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation
• Sample Efficient Adaptive Text-to-Speech
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• An Exploration of Word Embedding Initialization in Deep-Learning Tasks
• Attention Is All You Need
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech
• WaveNet: A Generative Model for Raw Audio
• Gaussian Error Linear Units (GELUs)
• Neural Machine Translation of Rare Words with Subword Units
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Efficient Estimation of Word Representations in Vector Space
• Framewise phoneme classification with bidirectional LSTM and other neural network architectures
• Long Short-Term Memory
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Duration-aware pause insertion using pre-trained language model for multi-speaker text-to-speech
• Synthetic Data Generation for Phrase Break Prediction with Large Language Model
• Assessing Phrase Break of ESL speech with Pre-trained Language Models
• Assessing Phrase Break of ESL Speech with Pre-trained Language Models and Large Language Models
• Phrase break prediction with bidirectional encoder representations in Japanese text-to-speech synthesis