Making Pre-trained Language Models Better Few-shot Learners

2020/12/31 by Tianyu Gao, Adam Fisch, Gao, Tianyu +3 · 54 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2012.15723

openalex publication_date 2020/12/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF--better few-shot fine-tuning of language models--a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.

Citations

• A Simple Method for Commonsense Reasoning

Cited by

• Merge before Forget: A Single LoRA Continual Learning via Continual Merging
• Imprompt: A Language Framework for Prompt Programming
• Retrieval-augmented Prompt Learning for Pre-trained Foundation Models
• An Exploratory Study of Bayesian Prompt Optimization for Test-Driven Code Generation with Large Language Models
• How Prompts Move Language Model Behavior: Frames, Salience, and Construal as Semantic Control
• Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspective
• LabelFusion: Learning to Fuse LLMs and Transformer Classifiers for Robust Text Classification
• Do Generalisation Results Generalise?
• Efficient Text Classification with Conformal In-Context Learning
• Empirical Prompt Engineering for Construct Identification with Large Language Models
• 5G Network Automation Using Local Large Language Models and Retrieval-Augmented Generation
• Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels
• From Words to Wisdom: Discourse Annotation and Baseline Models for Student Dialogue Understanding
• Think First, Assign Next (ThiFAN-VQA): A Two-stage Chain-of-Thought Framework for Post-Disaster Damage Assessment
• Zero-Shot Grammar Competency Estimation Using Large Language Model Generated Pseudo Labels
• On-Device Fine-Tuning via Backprop-Free Zeroth-Order Optimization
• Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data
• GFT: Graph Feature Tuning for Efficient Point Cloud Analysis
• Multi-refined Feature Enhanced Sentiment Analysis Using Contextual Instruction
• Learning to Prompt for Vision-Language Models
• Stay Tuned: Improving Sentiment Analysis and Stance Detection Using Large Language Models
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• Semantic-Preserving Cross-Style Visual Reasoning for Robust Multi-Modal Understanding in Large Vision-Language Models
• Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications
• SODBench: A Large Language Model Approach to Documenting Spreadsheet Operations
• All You Need is One: Capsule Prompt Tuning with a Single Vector
• Safe and Efficient In-Context Learning via Risk Control
• MedREK: Retrieval-Based Editing for Medical LLMs with Key-Aware Prompts
• A Survey on Evaluation of Large Language Models
• Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priors
• Evolutionary Computation as Natural Generative AI
• Security and Privacy Challenges of Large Language Models: A Survey
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
• A Retail-Corpus for Aspect-Based Sentiment Analysis with Large Language Models
• Evolution of Concepts in Language Model Pre-Training
• Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs
• The Few-shot Dilemma: Over-prompting Large Language Models
• NeuroStrike: Neuron-Level Attacks on Aligned LLMs
• Graph-Enhanced Retrieval-Augmented Question Answering for E-Commerce Customer Support
• Topic Coverage-based Demonstration Retrieval for In-Context Learning
• Are Humans as Brittle as Large Language Models?
• Fuzzy, Symbolic, and Contextual: Enhancing LLM Instruction via Cognitive Scaffolding
• Membership Inference Attacks on LLM-based Recommender Systems
• TrackRec: Iterative Alternating Feedback with Chain-of-Thought via Preference Alignment for Recommendation
• Label Verbalization and Entailment for Effective Zero- and Few-Shot\n Relation Extraction
• When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs
• STEP: Stepwise Curriculum Learning for Context-Knowledge Fusion in Conversational Recommendation
• Prompt-Based Approach for Czech Sentiment Analysis
• CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learning
• Prompt Tuning for Few-Shot Continual Learning Named Entity Recognition
• A Fuzzy Logic Prompting Framework for Large Language Models in Adaptive and Uncertain Tasks
• ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates
• GeoSR: Cognitive-Agentic Framework for Probing Geospatial Knowledge Boundaries via Iterative Self-Refinement
• Sel3DCraft: Interactive Visual Prompts for User-Friendly Text-to-3D Generation

Related

• Language Models are Few-Shot Learners
• Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
• PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models
• True Few-Shot Learning with Language Models
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning