Robustly Optimized and Distilled Training for Natural Language Understanding

2021/03/16 by Haytham ElFadeel, ElFadeel, Haytham, Stan Peshterliev +1
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2103.08809

openalex publication_date 2021/03/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we explore multi-task learning (MTL) as a second pretraining step to learn enhanced universal language representation for transformer language models. We use the MTL enhanced representation across several natural language understanding tasks to improve performance and generalization. Moreover, we incorporate knowledge distillation (KD) in MTL to further boost performance and devise a KD variant that learns effectively from multiple teachers. By combining MTL and KD, we propose Robustly Optimized and Distilled (ROaD) modeling framework. We use ROaD together with the ELECTRA model to obtain state-of-the-art results for machine reading comprehension and natural language inference.

Citations

• Teaching Machines to Read and Comprehend
• Distilling the Knowledge in a Neural Network
• A Survey on Multi-Task Learning
• Multi-Task Deep Neural Networks for Natural Language Understanding
• NewsQA: A Machine Comprehension Dataset
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Related

• MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• BAM! Born-Again Multi-Task Networks for Natural Language Understanding
• Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings
• Basic Reading Distillation