In-context Learning with Retrieved Demonstrations for Language Models: A Survey

2024/01/21 by Man Luo, Luo, Man, Xin Xu +7 · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2401.11624

openalex publication_date 2024/01/21 · openalex created_date 2024/01/25 · openalex updated_date 2026/07/28

Abstract

Language models, especially pre-trained large language models, have showcased remarkable abilities as few-shot in-context learners (ICL), adept at adapting to new tasks with just a few demonstrations in the input context. However, the model's ability to perform ICL is sensitive to the choice of the few-shot demonstrations. Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query. The implementation of demonstration retrieval is relatively straightforward, leveraging existing databases and retrieval systems. This not only improves the efficiency and scalability of the learning process but also has been shown to reduce biases inherent in manual example selection. In light of the encouraging results and growing research in ICL with retrieved demonstrations, we conduct an extensive review of studies in this area. In this survey, we discuss and compare different design choices for retrieval models, retrieval training procedures, and inference algorithms.

Cited by

• From Retrieval to Reasoning: A Framework for Cyber Threat Intelligence NER with Explicit and Adaptive Instructions
• Fairness in Multi-modal Medical Diagnosis with Demonstration Selection
• Reflexive Evidence-Based Multimodal Learning for Clean Energy Transitions: Causal Insights on Cooking Fuel Access, Urbanization, and Carbon Emissions
• Distilling Many-Shot In-Context Learning into a Cheat Sheet
• KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning
• Retrieval-Augmented Review Generation for Poisoning Recommender Systems

Related

• Dr.ICL: Demonstration-Retrieved In-context Learning
• Demonstration Augmentation for Zero-shot In-context Learning
• Large Language Models are Demonstration Pre-Selectors for Themselves
• Unified Demonstration Retriever for In-Context Learning
• Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching