Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation

2020/05/18 by Po-Han Chi, Pei-Hung Chung, Chi, Po-Han +12 · 5 citations
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Topic Modeling #cs.CL #cs.SD #eess.AS #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2005.08575

Accepted by IEEE Spoken Language Technology Workshop 2021

openalex publication_date 2020/05/18 · arxiv created 2021/05/03 · arxiv updated 2021/05/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

For self-supervised speech processing, it is crucial to use pretrained models as speech representation extractors. In recent works, increasing the size of the model has been utilized in acoustic model training in order to achieve better performance. In this paper, we propose Audio ALBERT, a lite version of the self-supervised speech representation model. We use the representations with two downstream tasks, speaker identification, and phoneme classification. We show that Audio ALBERT is capable of achieving competitive performance with those huge models in the downstream tasks while utilizing 91% fewer parameters. Moreover, we use some simple probing models to measure how much the information of the speaker and phoneme is encoded in latent representations. In probing experiments, we find that the latent representations encode richer information of both phoneme and speaker than that of the last layer.

Citations

• Decoupled Weight Decay Regularization
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• A Simple Framework for Contrastive Learning of Visual Representations
• Momentum Contrast for Unsupervised Visual Representation Learning

Cited by

• Representing Speech Through Autoregressive Prediction of Cochlear Tokens
• Towards Semi-Supervised Semantics Understanding from Speech
• On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition
• One4all User Representation for Recommender Systems in E-commerce
• Layer Reduction: Accelerating Conformer-Based Self-Supervised Model via Layer Consistency

Related

• Effectiveness of self-supervised pre-training for speech recognition
• EAT: Self-Supervised Pre-Training with Efficient Audio Transformer
• Efficiency-oriented approaches for self-supervised speech representation learning
• BEATs: Audio Pre-Training with Acoustic Tokenizers
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units