Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces

2026/03/13 by Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho +2 · 1 voice · 1 citation
Computer Science · Engineering · Psychology · #Context (archaeology) #ENCODE #Face recognition and analysis #Focus (optics) #Linear subspace #Nasality #Orthogonality #Phonetics and Phonology Research #Representation (politics) #Sequence (biology) #Speech Recognition and Synthesis #cs.CL #cs.LG #cs.SD #eess.AS

paper · pdf · open access · doi:10.48550/arxiv.2603.12642

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2026/03/13 · arxiv published 2026/03/13 · arxiv updated 2026/03/13 · openalex created_date 2026/03/17 · openalex updated_date 2026/07/28

Abstract

Transformer-based self-supervised speech models (S3Ms) are often described as contextualized, yet what this entails remains unclear. Here, we focus on how a single frame-level S3M representation can encode phones and their surrounding context. Prior work has shown that S3Ms represent phones compositionally; for example, phonological vectors such as voicing, bilabiality, and nasality vectors are superposed in the S3M representation of [m]. We extend this view by proposing that phonological information from a sequence of neighboring phones is also compositionally encoded in a single frame, such that vectors corresponding to previous, current, and next phones are superposed within a single frame-level representation. We show that this structure has several properties, including orthogonality between relative positions, and emergence of implicit phonetic boundaries. Together, our findings advance our understanding of context-dependent S3M representations.

Citations

• Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0
• Discrete Audio Tokens: More Than a Survey!
• On-device Streaming Discrete Speech Units
• LinearVC: Linear transformations of self-supervised features through the lens of voice conversion
• Effective Context in Neural Speech Models
• Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment
• Sylber: Syllabic Embedding Representation of Speech from Raw Audio
• SyllableLM: Learning Coarse Semantic Units for Speech Language Models
• Moshi: a speech-text foundation model for real-time dialogue
• Fine-Grained and Interpretable Neural Speech Editing
• Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0
• Interface Design for Self-Supervised Speech Models
• Self-Supervised Speech Representations are More Phonetic than Semantic
• Phonetic Segmentation of the UCLA Phonetics Lab Archive
• Understanding Probe Behaviors through Variational Bounds of Mutual Information
• Self-Supervised Models of Speech Infer Universal Articulatory Kinematics
• Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
• What Do Self-Supervised Speech Models Know About Words?
• Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration
• An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speech
• Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces
• PWESuite: Phonetic Word Embeddings and Tasks They Facilitate
• Comparative layer-wise analysis of self-supervised speech models
• Opening the Black Box of wav2vec Feature Encoder
• AudioLM: a Language Modeling Approach to Audio Generation
• Silence is Sweeter Than Speech: Self-Supervised Model Using Silence to Store Speaker Information
• Word Discovery in Visually Grounded, Self-Supervised Speech Models
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Layer-wise Analysis of a Self-supervised Speech Representation Model
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
• Learning weakly supervised multimodal phoneme embeddings

Cited by

• An Empirical Recipe for Universal Phone Recognition

Discussions

• 📄 Paper 2 (submitted to IS): "Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces" We further show how sequences of phone(me)s can be encoded, i.e., cont [bsky, 1 points, 1 comments]

Related

• [b]=[d]-[t]+[p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic
• Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration
• Speech Representation Analysis based on Inter- and Intra-Model Similarities
• What Do Self-Supervised Speech Models Know About Words?
• Evaluating context-invariance in unsupervised speech representations