Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0

2024/07/03 by Marianne de Heer Kloots, Willem Zuidema · 1 voice · 14 citations
Computer Science · Psychology · #Artificial intelligence #Categorization #Computer science #Linguistics #Natural Language Processing Techniques #Natural language processing #Phonetics and Phonology Research #Phonology #Phonotactics #Speech Recognition and Synthesis #Speech recognition

paper · pdf · open access · doi:10.21437/interspeech.2024-2490

openalex created_date 2024/07/06 · openalex publication_date 2024/09/01 · openalex updated_date 2026/08/01

Abstract

What do deep neural speech models know about phonology? Existing work has examined the encoding of individual linguistic units such as phonemes in these models. Here we investigate interactions between units. Inspired by classic experiments on human speech perception, we study how Wav2Vec2 resolves phonotactic constraints. We synthesize sounds on an acoustic continuum between /l/ and /r/ and embed them in controlled contexts where only /l/, only /r/, or neither occur in English. Like humans, Wav2Vec2 models show a bias towards the phonotactically admissable category in processing such ambiguous sounds. Using simple measures to analyze model internals on the level of individual stimuli, we find that this bias emerges in early layers of the model's Transformer module. This effect is amplified by ASR finetuning but also present in fully self-supervised models. Our approach demonstrates how controlled stimulus designs can help localize specific linguistic knowledge in neural speech models.

Cited by

• Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear
• Tracking the emergence of linguistic structure in self-supervised models learning from speech
• Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities
• Linguists should learn to love speech-based deep learning models
• You can’t fight in here! This is BBS!
• What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-training
• How linguistics learned to stop worrying and love the language models
• Emergence of Phonemic, Syntactic, and Semantic Representations in Artificial Neural Networks
• Large language models are not about natural language
• Are Language Models Models?
• The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoders
• Trick or Neat: Adversarial Ambiguity and Language Model Evaluation
• From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Models
• Automatic classification of stop realisation with wav2vec2.0
• Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces

Discussions

• ✨ Do current neural speech models show human-like linguistic biases in speech perception? We took inspiration from classic phonetic categorization experiments to explore where sensitivity to phonotac [bsky, 17 points, 2 comments]

Related

• Perception of Phonological Assimilation by Neural Speech Recognition Models
• Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration
• Do self-supervised speech and language models extract similar representations as human brain?
• Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech models
• Neural Representations for Modeling Variation in Speech