Revisiting Self-Training for Few-Shot Learning of Language Model

2021/10/04 by Chen, Yiming, Zhang, Yan, Zhang, Chen +3 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2110.01256

Abstract

As unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model. The question is how to effectively make use of such data. In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM. Given two views of a text sample via weak and strong augmentation techniques, SFLM generates a pseudo label on the weakly augmented version. Then, the model predicts the same pseudo label when fine-tuned with the strongly augmented version. This simple approach is shown to outperform other state-of-the-art supervised and semi-supervised counterparts on six sentence classification and six sentence-pair classification benchmarking tasks. In addition, SFLM only relies on a few in-domain unlabeled data. We conduct a comprehensive analysis to demonstrate the robustness of our proposed approach under various settings, including augmentation techniques, model scale, and few-shot knowledge transfer across tasks.

Cited by

• Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles

Related

• Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning
• Making Pre-trained Language Models Better Few-shot Learners
• Automated Few-shot Classification with Instruction-Finetuned Language Models
• Less is More: A Closer Look at Semantic-based Few-Shot Learning
• True Few-Shot Learning with Language Models