Improved Self-Supervised Multilingual Speech Representation Learning Combined with Auxiliary Language Information

2022/12/07 by Ding, Fenglin, Wan, Genshun, Li, Pengcheng +2
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2212.03476

Abstract

Multilingual end-to-end models have shown great improvement over monolingual systems. With the development of pre-training methods on speech, self-supervised multilingual speech representation learning like XLSR has shown success in improving the performance of multilingual automatic speech recognition (ASR). However, similar to the supervised learning, multilingual pre-training may also suffer from language interference and further affect the application of multilingual system. In this paper, we introduce several techniques for improving self-supervised multilingual pre-training by leveraging auxiliary language information, including the language adversarial training, language embedding and language adaptive training during the pre-training stage. We conduct experiments on a multilingual ASR task consisting of 16 languages. Our experimental results demonstrate 14.3% relative gain over the standard XLSR model, and 19.8% relative gain over the no pre-training multilingual model.

Related

• XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition
• Joint Unsupervised and Supervised Training for Multilingual ASR
• A Survey of Multilingual Models for Automatic Speech Recognition
• Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identification
• Multilingual Speech Recognition using Knowledge Transfer across Learning Processes