Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations

2022/05/25 by Kang Min Yoo, Yoo, Kang Min, Junyeob Kim +13 · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Natural Language Processing Techniques #Text and Document Classification Technologies

paper · pdf · doi:10.48550/arxiv.2205.12685

openalex publication_date 2022/05/25 · openalex created_date 2022/06/12 · openalex updated_date 2026/07/28

Abstract

Despite recent explosion of interests in in-context learning, the underlying mechanism and the precise impact of the quality of demonstrations remain elusive. Intuitively, ground-truth labels should have as much impact in in-context learning (ICL) as supervised learning, but recent work reported that the input-label correspondence is significantly less important than previously thought. Intrigued by this counter-intuitive observation, we re-examine the importance of ground-truth labels in in-context learning. With the introduction of two novel metrics, namely Label-Correctness Sensitivity and Ground-truth Label Effect Ratio (GLER), we were able to conduct quantifiable analysis on the impact of ground-truth label demonstrations. Through extensive analyses, we find that the correct input-label mappings can have varying impacts on the downstream in-context learning performances, depending on the experimental configuration. Through additional studies, we identify key components, such as the verbosity of prompt templates and the language model size, as the controlling factor to achieve more noise-resilient ICL.

Cited by

• KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning
• The Prompt Engineering Report Distilled: Quick Start Guide for Life Sciences
• Rethinking the Chain-of-Thought: The Roles of In-Context Learning and Pre-trained Priors
• InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity
• Towards Compute-Optimal Many-Shot In-Context Learning
• Function Induction and Task Generalization: An Interpretability Study with Off-by-One Addition

Related

• On the Relationship Between the Choice of Representation and In-Context Learning
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspective
• Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps
• Investigating the Learning Behaviour of In-context Learning: A Comparison with Supervised Learning