ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding

2020/10/23 by Minjeong Kim, Kim, Minjeong, Gyuwan Kim +5
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2010.12283

openalex publication_date 2020/10/23 · openalex created_date 2020/10/29 · openalex updated_date 2026/07/28

Abstract

Language model pre-training has shown promising results in various downstream tasks. In this context, we introduce a cross-modal pre-trained language model, called Speech-Text BERT (ST-BERT), to tackle end-to-end spoken language understanding (E2E SLU) tasks. Taking phoneme posterior and subword-level text as an input, ST-BERT learns a contextualized cross-modal alignment via our two proposed pre-training tasks: Cross-modal Masked Language Modeling (CM-MLM) and Cross-modal Conditioned Language Modeling (CM-CLM). Experimental results on three benchmarks present that our approach is effective for various SLU datasets and shows a surprisingly marginal performance degradation even when 1% of the training data are available. Also, our method shows further SLU performance gain via domain-adaptive pre-training with domain-specific speech-text pair data.

Citations

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Cross-lingual Language Model Pretraining

Related

• mSLAM: Massively multilingual joint pre-training for speech and text
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
• Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding
• Understanding Semantics from Speech Through Pre-training
• Speech Model Pre-training for End-to-End Spoken Language Understanding