End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures

2019/11/19 by Gabriel Synnaeve, Synnaeve, Gabriel, Qiantong Xu +15 · 8 citations
Computer Science · #Anomaly Detection Techniques and Applications #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning and Algorithms #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.1911.08460

openalex publication_date 2019/11/19 · openalex created_date 2019/12/05 · openalex updated_date 2026/07/28

Abstract

We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions. We perform experiments on the standard LibriSpeech dataset, and leverage additional unlabeled data from LibriVox through pseudo-labeling. We show that while Transformer-based acoustic models have superior performance with the supervised dataset alone, semi-supervision improves all models across architectures and loss functions and bridges much of the performance gaps between them. In doing so, we reach a new state-of-the-art for end-to-end acoustic models decoded with an external language model in the standard supervised learning setting, and a new absolute state-of-the-art with semi-supervised training. Finally, we study the effect of leveraging different amounts of unlabeled audio, propose several ways of evaluating the characteristics of unlabeled audio which improve acoustic modeling, and show that acoustic models trained with more audio rely less on external language models.

Citations

• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Residual Convolutional CTC Networks for Automatic Speech Recognition
• The CAPIO 2017 Conversational Speech Recognition System
• Efficient Neural Audio Synthesis
• Fully Convolutional Speech Recognition
• Towards better decoding and language model integration in sequence to\n sequence models
• Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large\n Vocabulary Speech Recognition
• Reducing Transformer Depth on Demand with Structured Dropout

Cited by

• Open Source State-Of-the-Art Solution for Romanian Speech Recognition
• Edge-Based Speech Transcription and Synthesis for Kinyarwanda and Swahili Languages
• Contextualized Token Discrimination for Speech Search Query Correction
• Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition
• Audio-3DVG: Unified Audio -- Point Cloud Fusion for 3D Visual Grounding
• A Self-Training Approach for Whisper to Enhance Long Dysarthric Speech Recognition
• Unified Semi-Supervised Pipeline for Automatic Speech Recognition
• LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Data

Related

• Semi-supervised ASR by End-to-end Self-training
• Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text
• Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLM
• Semi-Supervised Learning with Data Augmentation for End-to-End ASR
• Exploiting semi-supervised training through a dropout regularization in end-to-end speech recognition