Self-training Improves Pre-training for Natural Language Understanding

2020/10/05 by Jingfei Du, Du, Jingfei, Edouard Grave +15 · 46 citations
Computer Science · #Artificial intelligence #Computer science #Database #Labeled data #Leverage (statistics) #Machine learning #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language #Natural language processing #Natural language understanding #Scalability #Task (project management) #Topic Modeling #Training (meteorology) #Training set #Variety (cybernetics) #cs.CL

paper · pdf · doi:10.48550/arxiv.2010.02194

published in arXiv (Cornell University) (Cornell University) · 8 pages

arxiv created 2020/10/05 · openalex publication_date 2020/10/05 · arxiv updated 2020/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.

Citations

• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

Cited by

• Few-Shot Named Entity Recognition: A Comprehensive Study
• Co-SOM: Co-training for photometric redshift estimation using Self-Organizing Maps
• Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding
• Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Data
• Improving Cross-Lingual Reading Comprehension with Self-Training
• Knowing More About Questions Can Help: Improving Calibration in Question Answering
• RISE: Reasoning Enhancement via Iterative Self-Exploration in Multi-hop Question Answering
• REFINE-AF: A Task-Agnostic Framework to Align Language Models via Self-Generated Instructions using Reinforcement Learning from Automated Feedback
• CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models
• Credal Self-Supervised Learning

Related

• Rethinking Semi-supervised Learning with Language Models
• Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding
• Curriculum-Based Self-Training Makes Better Few-Shot Learners for Data-to-Text Generation
• Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models
• Data-Efficient Pretraining via Contrastive Self-Supervision