Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?

2024/10/25 by Opeyemi Osakuade, Simon King, Osakuade, Opeyemi +1 · 2 citations
Computer Science · #Speech Recognition and Synthesis #Speech and Audio Processing

paper · pdf · doi:10.48550/arxiv.2410.19935

Abstract

Discrete representations of speech, obtained from Self-Supervised Learning (SSL) foundation models, are widely used, especially where there are limited data for the downstream task, such as for a low-resource language. Typically, discretization of speech into a sequence of symbols is achieved by unsupervised clustering of the latents from an SSL model. Our study evaluates whether discrete symbols - found using k-means - adequately capture tone in two example languages, Mandarin and Yoruba. We compare latent vectors with discrete symbols, obtained from HuBERT base, MandarinHuBERT, or XLS-R, for vowel and tone classification. We find that using discrete symbols leads to a substantial loss of tone information, even for language-specialised SSL models. We suggest that discretization needs to be task-aware, particularly for tone-dependent downstream tasks.

Cited by

• Benchmarking Prosody Encoding in Discrete Speech Tokens
• Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead

Related

• Encoding of lexical tone in self-supervised models of spoken language
• How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfer
• Representation Learning for Discovering Phonemic Tone Contours
• Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech models
• Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs