Accent Normalization Using Self-Supervised Discrete Tokens with Non-Parallel Data

2025/07/23 by Qibing Bai, Bai, Qibing, Sho Inoue +9 · 1 citation
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Neural Networks and Applications #Sound (cs.SD) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2507.17735

openalex publication_date 2025/07/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Accent normalization converts foreign-accented speech into native-like speech while preserving speaker identity. We propose a novel pipeline using self-supervised discrete tokens and non-parallel training data. The system extracts tokens from source speech, converts them through a dedicated model, and synthesizes the output using flow matching. Our method demonstrates superior performance over a frame-to-frame baseline in naturalness, accentedness reduction, and timbre preservation across multiple English accents. Through token-level phonetic analysis, we validate the effectiveness of our token-based approach. We also develop two duration preservation methods, suitable for applications such as dubbing.

Citations

• Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement
• Improving Pronunciation and Accent Conversion through Knowledge Distillation And Synthetic Ground-Truth from Native TTS
• LLaMA-Omni: Seamless Speech Interaction with Large Language Models
• Self-Supervised Speech Representations are More Phonetic than Semantic
• Total-Duration-Aware Duration Modeling for Text-to-Speech Systems
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
• High-Fidelity Neural Phonetic Posteriorgrams
• Transfer the linguistic representations from TTS to accent conversion with non-parallel data
• Matcha-TTS: A fast TTS architecture with conditional flow matching
• Evaluating Methods for Ground-Truth-Free Foreign Accent Conversion
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
• LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus
• Accented Text-to-Speech Synthesis with Limited Data
• Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision
• Scalable Diffusion Models with Transformers
• Zero-Shot Accent Conversion using Pseudo Siamese Disentanglement Network
• Voice-preserving Zero-shot Multiple Accent Conversion
• Flow Matching for Generative Modeling
• Classifier-Free Diffusion Guidance
• BigVGAN: A Universal Neural Vocoder with Large-Scale Training
• Direct speech-to-speech translation with discrete units
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• Generative Spoken Language Modeling from Raw Audio
• Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representations
• Improving Accent Conversion with Reference Encoder and End-To-End Text-To-Speech
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language\n Generation, Translation, and Comprehension

Cited by

• FAC-FACodec: Controllable Zero-Shot Foreign Accent Conversion with Factorized Speech Codec

Related

• Accent conversion using discrete units with parallel data synthesized from controllable accented TTS
• Prosodically Enhanced Foreign Accent Simulation by Discrete Token-based Resynthesis Only with Native Speech Corpora
• Improving Accent Identification and Accented Speech Recognition Under a Framework of Self-supervised Learning
• Remap, warp and attend: Non-parallel many-to-many accent conversion with Normalizing Flows
• Transfer the linguistic representations from TTS to accent conversion with non-parallel data