Effectiveness of self-supervised pre-training for speech recognition

2019/11/10 by Alexei Baevski, Baevski, Alexei, Michael Auli +3 · 8 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Music and Audio Processing #Speech Recognition and Synthesis #Speech and Audio Processing #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.1911.03912

openalex publication_date 2019/11/10 · arxiv created 2020/05/18 · arxiv updated 2020/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We compare self-supervised representation learning algorithms which either explicitly quantize the audio data or learn representations without quantization. We find the former to be more accurate since it builds a good vocabulary of the data through vq-wav2vec [1] to enable learning of effective representations in subsequent BERT training. Different to previous work, we directly fine-tune the pre-trained BERT models on transcribed speech using a Connectionist Temporal Classification (CTC) loss instead of feeding the representations into a task-specific model. We also propose a BERT-style model learning directly from the continuous audio data and compare pre-training on raw audio to spectral features. Fine-tuning a BERT model on 10 hour of labeled Librispeech data with a vq-wav2vec vocabulary is almost as good as the best known reported system trained on 100 hours of labeled data on testclean, while achieving a 25% WER reduction on test-other. When using only 10 minutes of labeled data, WER is 25.2 on test-other and 16.3 on test-clean. This demonstrates that self-supervision can enable speech recognition systems trained on a near-zero amount of transcribed data.

Citations

• Kaldi Speech Recognition Toolkit
• Representation Learning: A Review and New Perspectives
• Multi-view Recurrent Neural Acoustic Word Embeddings
• Learning Representations by Maximizing Mutual Information Across Views
• Neural Discrete Representation Learning

Cited by

• Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs
• BiRQ: Bi-Level Self-Labeling Random Quantization for Self-Supervised Speech Recognition
• Pitch Accent Detection improves Pretrained Automatic Speech Recognition
• Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition
• ASMDD: Arabic Speech Mispronunciation Detection Dataset
• Contrastive Semi-supervised Learning for ASR
• A Brief Overview of Unsupervised Neural Speech Representation Learning
• Is Self-Pretraining really useful to improve diagnosis in medical Time Series?

Related

• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Self-training and Pre-training are Complementary for Speech Recognition
• An Adapter Based Pre-Training for Efficient and Scalable Self-Supervised Speech Representation Learning
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Wav2vec-C: A Self-supervised Model for Speech Representation Learning