What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model

2023/06/10 by Yang Mu, Yang, Mu, Ram C. M. C. Shekar +5 · 10 citations
Computer Science · Psychology · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Phonetics and Phonology Research #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2306.06524

openalex publication_date 2023/06/10 · openalex created_date 2023/06/14 · openalex updated_date 2026/07/28

Abstract

This study is focused on understanding and quantifying the change in phoneme and prosody information encoded in the Self-Supervised Learning (SSL) model, brought by an accent identification (AID) fine-tuning task. This problem is addressed based on model probing. Specifically, we conduct a systematic layer-wise analysis of the representations of the Transformer layers on a phoneme correlation task, and a novel word-level prosody prediction task. We compare the probing performance of the pre-trained and fine-tuned SSL models. Results show that the AID fine-tuning task steers the top 2 layers to learn richer phoneme and prosody representation. These changes share some similarities with the effects of fine-tuning with an Automatic Speech Recognition task. In addition, we observe strong accent-specific phoneme representations in layer 9. To sum up, this study provides insights into the understanding of SSL features and their interactions with fine-tuning tasks.

Cited by

• Geolocation-Aware Robust Spoken Language Identification
• Emphasis Sensitivity in Speech Representations
• Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0
• Probing for Phonology in Self-Supervised Speech Representations: A Case Study on Accent Perception
• An Exploration of Mamba for Speech Self-Supervised Models
• Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech models
• Tone recognition in low-resource languages of North-East India: peeling the layers of SSL-based speech models
• Different Speech Translation Models Encode and Translate Speaker Gender Differently
• Discrete Tokens Exhibit Interlanguage Speech Intelligibility Benefit: an Analytical Study Towards Accent-robust ASR Only with Native Speech Data
• Representation of perceived prosodic similarity of conversational feedback

Related

• Improving Accent Identification and Accented Speech Recognition Under a Framework of Self-supervised Learning
• Probing for Phonology in Self-Supervised Speech Representations: A Case Study on Accent Perception
• Improving Self-supervised Pre-training using Accent-Specific Codebooks
• A Large-Scale Probing Analysis of Speaker-Specific Attributes in Self-Supervised Speech Representations
• Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech models