Articulation-Informed ASR: Integrating Articulatory Features into ASR via Auxiliary Speech Inversion and Cross-Attention Fusion

2025/10/01 by Ahmed Adel Attia, Jing Liu, Attia, Ahmed Adel +3
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Phonetics and Phonology Research #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2510.08585

openalex publication_date 2025/10/01 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Prior works have investigated the use of articulatory features as complementary representations for automatic speech recognition (ASR), but their use was largely confined to shallow acoustic models. In this work, we revisit articulatory information in the era of deep learning and propose a framework that leverages articulatory representations both as an auxiliary task and as a pseudo-input to the recognition model. Specifically, we employ speech inversion as an auxiliary prediction task, and the predicted articulatory features are injected into the model as a query stream in a cross-attention module with acoustic embeddings as keys and values. Experiments on LibriSpeech demonstrate that our approach yields consistent improvements over strong transformer-based baselines, particularly under low-resource conditions. These findings suggest that articulatory features, once sidelined in ASR research, can provide meaningful benefits when reintroduced with modern architectures.

Citations

• Perceptual Ratings Predict Speech Inversion Articulatory Kinematics in Childhood Speech Sound Disorders
• Enhancing Acoustic-to-Articulatory Speech Inversion by Incorporating Nasality
• Multimodal Biomarkers for Schizophrenia: Towards Individual Symptom Severity Estimation
• Speech-Based Estimation of Schizophrenia Severity Using Feature Fusion
• Robust Speech Recognition via Large-Scale Weak Supervision
• Acoustic To Articulatory Speech Inversion Using Multi-Resolution\n Spectro-Temporal Representations Of Speech Signals
• Inverted Vocal Tract Variables and Facial Action Units to Quantify Neuromotor Coordination in Schizophrenia
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry\n and Semantics

Related

• Improved acoustic-to-articulatory inversion using representations from pretrained self-supervised learning models
• Articulatory Features for ASR of Pathological Speech
• Are Transformers in Pre-trained LM A Good ASR Encoder? An Empirical Study
• Building Robust Spoken Language Understanding by Cross Attention between Phoneme Sequence and ASR Hypothesis
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends