Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings

2025/05/20 by Owais Mujtaba Khanday, Khanday, Owais Mujtaba, Zubair Ahmad Lone +6
Computer Science · Neuroscience · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Human-Computer Interaction (cs.HC) #Intelligent Tutoring Systems and Adaptive Learning #Neurobiology of Language and Bilingualism #Sound (cs.SD) #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.14074

openalex publication_date 2025/05/20 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Understanding how neural activity encodes speech and language production is a fundamental challenge in neuroscience and artificial intelligence. This study investigates whether embeddings from large-scale, self-supervised language and speech models can effectively reconstruct high-gamma neural activity characteristics, key indicators of cortical processing, recorded during speech production. We leverage pre-trained embeddings from deep learning models trained on linguistic and acoustic data to represent high-level speech features and map them onto these high-gamma signals. We analyze the extent to which these embeddings preserve the spatio-temporal dynamics of brain activity. Reconstructed neural signals are evaluated against high-gamma ground-truth activity using correlation metrics and signal reconstruction quality assessments. The results indicate that high-gamma activity can be effectively reconstructed using large language and speech model embeddings in all study participants, generating Pearson's correlation coefficients ranging from 0.79 to 0.99.

Citations

• NeuroIncept Decoder for High-Fidelity Speech Reconstruction from Neural Activity
• Do self-supervised speech and language models extract similar representations as human brain?
• Relating EEG to continuous speech using deep neural networks: a review
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
• Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria
• Elastic Net Regularization Paths for All Generalized Linear Models
• Elastic Net Regularization Paths for All Generalized Linear Models
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• FastText.zip: Compressing text classification models

Related

• NeuroIncept Decoder for High-Fidelity Speech Reconstruction from Neural Activity
• Neural Speech Embeddings for Speech Synthesis Based on Deep Generative Networks
• Do self-supervised speech and language models extract similar representations as human brain?
• MapGuide: A Simple yet Effective Method to Reconstruct Continuous Language from Brain Activities
• Refining Self-Supervised Learnt Speech Representation using Brain Activations