DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective

2025/06/25 by Chi, Hyung Gun, Aldeneh, Zakaria, Likhomanenko, Tatiana +5
#Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2507.02911

Abstract

We introduce DiceHuBERT, a knowledge distillation framework for compressing HuBERT, a widely used self-supervised learning (SSL)-based speech foundation model. Unlike existing distillation methods that rely on layer-wise and feature-wise mapping between teacher and student models, DiceHuBERT leverages HuBERT's iterative self-distillation mechanism by directly replacing the original model with a student model. This replacement allows the student to be trained using the same SSL objective used when pre-training HuBERT, eliminating the need for additional modules or architectural constraints. Experimental results on SUPERB show that DiceHuBERT consistently outperforms existing distillation methods, improving phoneme recognition performance by over 21% and ASR performance by more than 14%. Furthermore, DiceHuBERT demonstrates competitive performance across multiple tasks, highlighting its clear advantage.

Related

• DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT
• FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning
• SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT
• Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute
• Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancement