Discrete Tokens Exhibit Interlanguage Speech Intelligibility Benefit: an Analytical Study Towards Accent-robust ASR Only with Native Speech Data

2025/05/22 by Kentaro Onda, Keisuke Imoto, Onda, Kentaro +7 · 1 citation
Computer Science · Neuroscience · Psychology · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Hearing Loss and Rehabilitation #Phonetics and Phonology Research #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.16182

openalex publication_date 2025/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this study, we gained insight that contributes to achieving accent-robust ASR using only native speech data. In human perception of non-native speech, the phenomenon known as "interlanguage speech intelligibility benefit" (ISIB) is observed, where non-native listeners who share the native language with the speaker understand the speech better compared even to native listeners. Based on the idea that discrete tokens extracted from self-supervised learning (SSL) models represent the human perception of speech, we conducted an analytical study on the robustness of discrete token-based ASR to non-native speech, varying the language used for training the tokenization, which is viewed as a technical implementation of ISIB. The results showed that ISIB actually occurred in the discrete token-based ASR. Since our approach relies only on native speech data to simulate the behavior of human perception, it is expected to be applicable to a wide range of accents for which speech data is scarce.

Citations

• A Pilot Study of GSLM-based Simulation of Foreign Accentuation Only Using Native Speech Corpora
• How Should We Extract Discrete Audio Tokens from Self-Supervised Models?
• MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Model
• The Interspeech 2024 Challenge on Speech Processing Using Discrete Units
• Accented Speech Recognition With Accent-specific Codebooks
• Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
• Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS
• What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model
• Synthetic Cross-accent Data Augmentation for Automatic Speech Recognition
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• Accented Speech Recognition: A Survey
• Generative Spoken Language Modeling from Raw Audio
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Racial disparities in automated speech recognition
• JVS corpus: free Japanese multi-speaker voice corpus
• ESPnet: End-to-End Speech Processing Toolkit
• AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline
• Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning

Cited by

• Benchmarking Prosody Encoding in Discrete Speech Tokens

Related

• Exploring SSL Discrete Tokens for Multilingual ASR
• Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS
• Accent Normalization Using Self-Supervised Discrete Tokens with Non-Parallel Data
• Differentiable K-means for Fully-optimized Discrete Token-based ASR
• Effects of Speaker Count, Duration, and Accent Diversity on Zero-Shot Accent Robustness in Low-Resource ASR