Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Models

2025/06/01 by KE Lee, Lee, Kyowoon, Artyom Stitsyuk +7
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Natural Language Processing Techniques #Phonetics and Phonology Research #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.00832

openalex publication_date 2025/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advances in Text-to-Speech (TTS) have significantly improved speech naturalness, increasing the demand for precise prosody control and mispronunciation correction. Existing approaches for prosody manipulation often depend on specialized modules or additional training, limiting their capacity for post-hoc adjustments. Similarly, traditional mispronunciation correction relies on grapheme-to-phoneme dictionaries, making it less practical in low-resource settings. We introduce Counterfactual Activation Editing, a model-agnostic method that manipulates internal representations in a pre-trained TTS model to achieve post-hoc control of prosody and pronunciation. Experimental results show that our method effectively adjusts prosodic features and corrects mispronunciations while preserving synthesis quality. This opens the door to inference-time refinement of TTS outputs without retraining, bridging the gap between pre-trained TTS models and editable speech synthesis.

Citations

• Speaking rate attention-based duration prediction for speed control TTS
• Robust Speech Recognition via Large-Scale Weak Supervision
• Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech
• Predicting within and across language phoneme recognition performance of self-supervised learning speech pre-trained models
• Diffeomorphic Counterfactuals with Generative Models
• Hierarchical prosody modeling and control in non-autoregressive parallel neural TTS
• Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• Unified Mandarin TTS Front-end Based on Distilled BERT Model
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems
• What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in\n Deep NLP Models
• WaveGlow: A Flow-based Generative Network for Speech Synthesis
• Hierarchical Generative Modeling for Controllable Speech Synthesis
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with\n Tacotron
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Neural Discrete Representation Learning
• Tacotron: Towards End-to-End Speech Synthesis
• WaveNet: A Generative Model for Raw Audio

Related

• Text-Aware End-to-end Mispronunciation Detection and Diagnosis
• Correcting Mispronunciations in Speech using Spectrogram Inpainting
• MuFFIN: Multifaceted Pronunciation Feedback Model with Interactive Hierarchical Neural Modeling
• DTW-SiameseNet: Dynamic Time Warped Siamese Network for Mispronunciation Detection and Correction
• Masked Acoustic Unit for Mispronunciation Detection and Correction