Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models

2025/05/27 by Zhaoqing Li, Haoning Xu, Li, Zhaoqing +9
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.21237

openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper presents a novel memory-efficient model compression approach for Conformer ASR and speech foundation systems. Our approach features a unique "small-to-large" design. A compact "seed" model containing a few Conformer or Transformer blocks is trained and unfolded many times to emulate the performance of larger uncompressed models with different logical depths. The seed model and many unfolded paths are jointly trained within a single unfolding cycle. The KL-divergence between the largest unfolded and smallest seed models is used in a self-distillation process to minimize their performance disparity. Experimental results show that our foldable model produces ASR performance comparable to individually constructed Conformer and wav2vec2/HuBERT speech foundation models under various depth configurations, while requiring only minimal memory and storage. Conformer and wav2vec2 models with a reduction of 35% and 30% parameters are obtained without loss of performance, respectively.

Citations

• Effective and Efficient Mixed Precision Quantization of Speech Foundation Models
• One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model
• ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers
• Distilling HuBERT with LSTMs via Decoupled Knowledge Distillation
• Accurate and Structured Pruning for Efficient Automatic Speech Recognition
• DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
• 2-bit Conformer quantization for automatic speech recognition
• Small-footprint slimmable networks for keyword spotting
• Weight-sharing Supernet for Searching Specialized Acoustic Event Classification Networks Across Device Constraints
• DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model
• Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models
• Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding
• Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models
• FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning
• PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations
• 4-bit Conformer with Native Quantization Aware Training for Speech Recognition
• LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT
• Non-Autoregressive ASR with Self-Conditioned Folded Encoders
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• Neural Architecture Search For LF-MMI Trained Time Delay Neural Networks
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT
• Layer Pruning on Demand with Intermediate CTC
• Collaborative Training of Acoustic Encoders for Speech Recognition
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Efficient conformer-based speech recognition with linear attention
• Extremely Low Footprint End-to-End ASR System for Smart Device
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Dynamic Sparsity Neural Networks for Automatic Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• Universally Slimmable Networks and Improved Training Techniques
• ESPnet: End-to-End Speech Processing Toolkit

Related

• One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model
• Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates
• Towards One-bit ASR: Extremely Low-bit Conformer Quantization Using Co-training and Stochastic Precision
• 2-bit Conformer quantization for automatic speech recognition
• Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASR