An Exploration of Mamba for Speech Self-Supervised Models

2025/06/14 by Lin, Tzu-Quan, Kuo, Heng-Cheng, Wei, Tzu-Chieh +5 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.12606

Abstract

While Mamba has demonstrated strong performance in language modeling, its potential as a speech self-supervised (SSL) model remains underexplored, with prior studies limited to isolated tasks. To address this, we explore Mamba-based HuBERT models as alternatives to Transformer-based SSL architectures. Leveraging the linear-time Selective State Space, these models enable fine-tuning on long-context ASR with significantly lower compute. Moreover, they show superior performance when fine-tuned for streaming ASR. Beyond fine-tuning, these models show competitive performance on SUPERB probing benchmarks, particularly in causal settings. Our analysis shows that they yield higher-quality quantized representations and capture speaker-related features more distinctly than Transformer-based models. These findings highlight Mamba-based SSL as a promising and complementary direction for long-sequence modeling, real-time speech modeling, and speech unit extraction.

Citations

• Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformers
• Towards a Unified Representation Evaluation Framework Beyond Downstream Tasks
• On The Landscape of Spoken Language Models: A Comprehensive Survey
• Building a Taiwanese Mandarin Spoken Language Model: A First Attempt
• Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
• Mamba for Streaming ASR Combined with Unimodal Aggregation
• Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models
• Rethinking Mamba in Speech Processing by Self-Supervised Models
• Property Neurons in Self-Supervised Speech Transformers
• Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis
• DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Models
• Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations
• Mamba in Speech: Towards an Alternative to Self-Attention
• An Investigation of Incorporating Mamba for Speech Enhancement
• A Large-Scale Evaluation of Speech Foundation Models
• Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation
• SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series
• MambaMOT: State-Space Model as Motion Predictor for Multi-Object Tracking
• Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model
• emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond
• Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction
• What Do Self-Supervised Speech Models Know About Words?
• What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model
• MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Models
• Textually Pretrained Speech Language Models
• ML-SUPERB: Multilingual Speech Universal PERformance Benchmark
• What Do Self-Supervised Vision Transformers Learn?
• Robust Speech Recognition via Large-Scale Weak Supervision
• MelHuBERT: A simplified HuBERT on Mel spectrograms
• SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
• TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context
• Layer-wise Analysis of a Self-supervised Speech Representation Model
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• SUPERB: Speech processing Universal PERformance Benchmark
• Generative Spoken Language Modeling from Raw Audio
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Generalized End-to-End Loss for Speaker Verification
• Attention Is All You Need
• SPMamba: State-space model is all you need in speech separation
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• MLMA: Towards Multilingual ASR With Mamba-based Architectures

Related

• Rethinking Mamba in Speech Processing by Self-Supervised Models
• Exploring the Capability of Mamba in Speech Applications
• Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models
• Mamba in Speech: Towards an Alternative to Self-Attention
• Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition