Mamba for Streaming ASR Combined with Unimodal Aggregation

2024/09/30 by Ying Fang, Xiaofei Li, Fang, Ying +1 · 4 citations
Engineering · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #IoT-based Smart Home Systems #Sound (cs.SD) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2410.00070

openalex publication_date 2024/09/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper works on streaming automatic speech recognition (ASR). Mamba, a recently proposed state space model, has demonstrated the ability to match or surpass Transformers in various tasks while benefiting from a linear complexity advantage. We explore the efficiency of Mamba encoder for streaming ASR and propose an associated lookahead mechanism for leveraging controllable future information. Additionally, a streaming-style unimodal aggregation (UMA) method is implemented, which automatically detects token activity and streamingly triggers token output, and meanwhile aggregates feature frames for better learning token representation. Based on UMA, an early termination (ET) method is proposed to further reduce recognition latency. Experiments conducted on two Mandarin Chinese datasets demonstrate that the proposed model achieves competitive ASR performance in terms of both recognition accuracy and latency.

Cited by

• MLMA: Towards Multilingual ASR With Mamba-based Architectures
• An overview of neural architectures for self-supervised audio representation learning from masked spectrograms
• UMA-Split: unimodal aggregation for both English and Mandarin non-autoregressive speech recognition
• MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audio

Related

• Universal ASR: Unifying Streaming and Non-Streaming ASR Using a Single Encoder-Decoder Model
• Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition
• Streaming Multi-Talker ASR with Token-Level Serialized Output Training
• A Better and Faster End-to-End Model for Streaming ASR
• Transformer-based Streaming ASR with Cumulative Attention