State-Space Models in Efficient Whispered and Multi-dialect Speech Recognition

2025/06/20 by Aref Farhadipour, Farhadipour, Aref, Homayoon Beigi +5
Computer Science · #Audio and Speech Processing (eess.AS) #Authorship Attribution and Profiling #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.16969

openalex publication_date 2025/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Whispered speech recognition presents significant challenges for conventional automatic speech recognition systems, particularly when combined with dialect variation. However, utilizing an efficient method to solve this problem using a low-range dataset and processing load is beneficial. This paper proposes a solution using a Mamba-based state-space model and four fine-tuned self-supervised models consisting of Wav2Vec2, WavLM, HuBERT, and Whisper to address the dual challenges of whispered speech and dialect diversity. Based on our knowledge, this represents the best performance reported on the wTIMIT and CHAINS datasets for whispered speech recognition. We trained the models using whispered and normal speech data across Singaporean, US, and Irish dialects. The findings demonstrated that utilizing the proposed Mamba-based model could work as a highly efficient model trained with low amounts of whispered data to simultaneously work on whispered and normal speech recognition. The code for this work is freely available.

Citations

• Leveraging Self-Supervised Models for Automatic Whispered Speech Recognition
• Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Robust Speech Recognition via Large-Scale Weak Supervision
• PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Model

Related

• Leveraging Self-Supervised Models for Automatic Whispered Speech Recognition
• Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languages
• Whisper Speaker Identification: Leveraging Pre-Trained Multilingual Transformers for Robust Speaker Embeddings
• Whisper-SV: Adapting Whisper for Low-data-resource Speaker Verification
• Whisper-PMFA: Partial Multi-Scale Feature Aggregation for Speaker Verification using Whisper Models