Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models

2025/07/21 by Riki Shimizu, Richard Antonello, Shimizu, Riki +5
Computer Science · Neuroscience · #Audio and Speech Processing (eess.AS) #FOS: Biological sciences #FOS: Computer and information sciences #FOS: Electrical engineering #Hearing Loss and Rehabilitation #Music and Audio Processing #Neurons and Cognition (q-bio.NC) #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2507.16080

openalex publication_date 2025/07/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box, it remains unclear what drives their alignment with brain responses. To remedy this, we built linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features, and contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM), quantifying electrocorticography (ECoG) response variance shared between feature classes. Variance-partitioning analyses revealed several key insights: First, the SFMs' alignment with the brain can be mostly explained by their ability to learn and encode simple interpretable speech features. Second, SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers. Finally, our results show that SFMs learn brain-relevant semantics which cannot be explained by lower-level speech features, with this capacity increasing with model size and context length. Together, our findings suggest a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features.

Citations

• Human insula neurons respond to simple sounds during passive listening
• BrainWavLM: Fine-tuning Speech Representations with Brain Responses to Language
• Improving semantic understanding in speech language models via brain-tuning
• Self-Supervised Speech Representations are More Phonetic than Semantic
• Contextual feature extraction hierarchies converge in large language models and the brain
• Speech language models lack important brain-relevant semantics
• What Do Self-Supervised Speech Models Know About Words?
• Scaling laws for language encoding models in fMRI
• Robust Speech Recognition via Large-Scale Weak Supervision
• Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation
• Comparative layer-wise analysis of self-supervised speech models
• Opening the Black Box of wav2vec Feature Encoder
• Self-supervised models of audio effectively explain human cortical responses to speech
• Investigating self-supervised front ends for speech spoofing countermeasures
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Layer-wise Analysis of a Self-supervised Speech Representation Model
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)
• Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using a Narrated Story

Related

• BrainWavLM: Fine-tuning Speech Representations with Brain Responses to Language
• Brain-tuned Speech Models Better Reflect Speech Processing Stages in the Brain
• Speech language models lack important brain-relevant semantics
• Self-supervised models of audio effectively explain human cortical responses to speech
• Probing Multimodal Fusion in the Brain: The Dominance of Audiovisual Streams in Naturalistic Encoding