A Survey on In-context Learning

2022/12/31 by Qingxiu Dong, Dong, Qingxiu, Lei Li +23 · 4 voices · 117 citations
Computer Science · #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2301.00234

openalex publication_date 2022/12/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With the increasing capabilities of large language models (LLMs), in-context learning (ICL) has emerged as a new paradigm for natural language processing (NLP), where LLMs make predictions based on contexts augmented with a few examples. It has been a significant trend to explore ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress and challenges of ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques, including training strategies, prompt designing strategies, and related analysis. Additionally, we explore various ICL application scenarios, such as data engineering and knowledge updating. Finally, we address the challenges of ICL and suggest potential directions for further research. We hope that our work can encourage more research on uncovering how ICL works and improving ICL.

Cited by

• T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs
• What Transfers Under Source Shift? Definitions, Examples, and Fine-Tuning for Climate Disclosure Classification
• Personalized Image Aesthetic Assessment via Preference-rich Sample Mining and Cohort Merging
• From Profiles to Steering Vectors: Global Sparse Priors and Local Semantic Calibration for Personalized Text Generation
• MicLog: Towards Accurate and Efficient LLM-based Log Parsing via Progressive Meta In-Context Learning
• Can We Hide Machines in the Crowd? Quantifying Equivalence in LLM-in-the-loop Annotation Tasks
• Robust LLM-based Column Type Annotation via Prompt Augmentation with LoRA Tuning
• Evaluating large language models for diagnostic reasoning from unstructured clinical narratives in epilepsy
• From Context to Skills: Can Language Models Learn from Context Skillfully?
• Co-Evolution of Types and Dependencies: Towards Repository-Level Type Inference for Python Code
• Learning to Wait: Synchronizing Agents with the Physical World
• RFKG-CoT: Relation-Driven Adaptive Hop-count Selection and Few-Shot Path Guidance for Knowledge-Aware QA
• Few-Shot Inference of Human Perceptions of Robot Performance in Social Navigation Scenarios
• Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
• Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspective
• To Think or Not to Think: The Hidden Cost of Meta-Training with Excessive CoT Examples
• HealthcareNLP: where are we and what is next?
• Automatic Syntax Error Repair for Discrete Controller Synthesis using Large Language Model
• Progress Ratio Embeddings: An Impatience Signal for Robust Length Control in Neural Text Generation
• ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems
• ReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision-Language Models
• Towards Unification of Hallucination Detection and Fact Verification for Large Language Models
• In-Context Learning for Deep Joint Source-Channel Coding Over MIMO Channels
• TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models
• LLM-Driven Transient Stability Assessment: From Automated Simulation to Neural Architecture Design
• LLMs for Low-Resource Dialect Translation Using Context-Aware Prompting: A Case Study on Sylheti
• A Multi-Agent LLM Framework for Multi-Domain Low-Resource In-Context NER via Knowledge Retrieval, Disambiguation and Reflective Analysis
• Towards Automating Data Access Permissions in AI Agents
• Efficient Robot Design with Multi-Objective Black-Box Optimization and Large Language Models
• SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
• FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling
• Who Is the Story About? Protagonist Entity Recognition in News
• CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Models
• Dutch Metaphor Extraction from Cancer Patients' Interviews and Forum Data using LLMs and Human in the Loop
• Credit Network Modeling and Analysis via Large Language Models
• Context-Guided Decompilation: A Step Towards Re-executability
• WeaveRec: An LLM-Based Cross-Domain Sequential Recommendation Framework with Model Merging
• What do vision-language models see in the context? Investigating multimodal in-context learning
• Reasoning About Reasoning: Towards Informed and Reflective Use of LLM Reasoning in HCI
• Compressing Many-Shots in In-Context Learning
• R2ComSync: Improving Code-Comment Synchronization with In-Context Learning and Reranking
• The Impact of Negated Text on Hallucination with Large Language Models
• Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation
• Understanding Parametric Knowledge Injection in Retrieval-Augmented Generation
• Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs
• Towards Real-Time Fake News Detection under Evidence Scarcity
• A Survey of Inductive Reasoning for Large Language Models
• An agentic artificially intelligent X-ray scientist
• Logit Arithmetic Elicits Long Reasoning Capabilities Without Training
• MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization
• RA-Gen: A Controllable Code Generation Framework Using ReAct for Multi-Agent Task Execution
• LLM-Assisted Web Measurements
• The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP
• Detecting Semantic Clones of Unseen Functionality
• Act as an expert in psychometry. The evaluation of large language models utility in psychological tests cross-cultural adaptations
• The Impact of Role Design in In-Context Learning for Large Language Models
• Multi-Agent Path Finding via Offline RL and LLM Collaboration
• Universal Legal Article Prediction via Tight Collaboration between Supervised Classification Model and LLM
• How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
• LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions
• Temporal-Aware User Behaviour Simulation with Large Language Models for Recommender Systems
• KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning
• From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse
• MIRA: Empowering One-Touch AI Services on Smartphones with MLLM-based Instruction Recommendation
• DSPC: Dual-Stage Progressive Compression Framework for Efficient Long-Context Reasoning
• Privacy Preserving In-Context-Learning Framework for Large Language Models
• TICL: Text-Embedding KNN For Speech In-Context Learning Unlocks Speech Recognition Abilities of Large Multimodal Models
• The Few-shot Dilemma: Over-prompting Large Language Models
• Context-Aware Language Models for Forecasting Market Impact from Sequences of Financial News
• Public Data Assisted Differentially Private In-Context Learning
• DeMeVa at LeWiDi-2025: Modeling Perspectives with In-Context Learning and Label Distribution Learning
• Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation
• From Implicit Exploration to Structured Reasoning: Leveraging Guideline and Refinement for LLMs
• Sailing Towards Zero-Shot State Estimation using Foundation Models Combined with a UKF
• Language Models Do Not Follow Occam's Razor: A Benchmark for Inductive and Abductive Reasoning
• An Epidemiological Knowledge Graph extracted from the World Health Organization's Disease Outbreak News
• LLMs that Understand Processes: Instruction-tuning for Semantics-Aware Process Mining
• Context Engineering for Trustworthiness: Rescorla Wagner Steering Under Mixed and Inappropriate Contexts
• DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression
• Summarize-Exemplify-Reflect: Data-driven Insight Distillation Empowers LLMs for Few-shot Tabular Classification
• From Post To Personality: Harnessing LLMs for MBTI Prediction in Social Media
• cMALC-D: Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending
• Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation
• Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMs
• GENIE-ASI: Generative Instruction and Executable Code for Analog Subcircuit Identification
• Text to Query Plans for Question Answering on Large Tables
• A Systematic Approach to Predict the Impact of Cybersecurity Vulnerabilities Using LLMs
• InReAcTable: LLM-Powered Interactive Visual Data Story Construction from Tabular Data
• Self-Guided Function Calling in Large Language Models via Stepwise Experience Recall
• Adversarial Attacks against Neural Ranking Models via In-Context Learning
• Scaled Signed Averaging Improves In-Context and Early Learning Benchmark Performance in Small Transformers
• Static Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond Correctness
• Automated Optimization Modeling through Expert-Guided Large Language Model Reasoning
• In-Context Examples Matter: Improving Emotion Recognition in Conversation with Instruction Tuning
• When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs
• ToxiFrench: Benchmarking and Enhancing Language Models via CoT Fine-Tuning for French Toxicity Detection
• AlphaAgents: Large Language Model based Multi-Agents for Equity Portfolio Constructions
• Next Edit Prediction: Learning to Predict Code Edits from Context and Interaction History
• CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learning
• In-situ Value-aligned Human-Robot Interactions with Physical Constraints
• What Builds Effective In-Context Examples for Code Generation?
• Omni Geometry Representation Learning vs Large Language Models for Geospatial Entity Resolution
• In-Context Reinforcement Learning via Communicative World Models
• QA-Dragon: Query-Aware Dynamic RAG System for Knowledge-Intensive Visual Question Answering
• ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation
• NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification
• From Text to Trajectories: GPT-2 as an ODE Solver via In-Context
• Zero-shot Compositional Action Recognition with Neural Logic Constraints
• Are All Prompt Components Value-Neutral? Understanding the Heterogeneous Adversarial Robustness of Dissected Prompt in Large Language Models
• Effective Damage Data Generation by Fusing Imagery with Human Knowledge Using Vision-Language Models
• SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generation
• To Theoretically Understand Transformer-Based In-Context Learning for Optimizing CSMA
• AutoBridge: Automating Smart Device Integration with Centralized Platform
• On LLM-Assisted Generation of Smart Contracts from Business Processes
• MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions
• MindChat: Enhancing BCI Spelling with Large Language Models in Realistic Scenarios
• Diverse LLMs or Diverse Question Interpretations? That is the Ensembling Question

Discussions

• Today's Q for discussion by the #NeuroAI crowd: In-context learning (i.e. learning from a few examples sans synaptic updates) is a major advantage of large AI models: arxiv.org/abs/2301.00234 Do we [bsky, 34 points, 9 comments]
• in-context learning is a thing. tokenspace is a thing. you can talk to an llm without having to provide it with labeled training examples to burn into its weights first PRECISELY BECAUSE of the exact [bsky, 19 points, 1 comments]
• A Survey on In-Context Learning [hn, 1 points, 0 comments]
• Or perhaps more accessible arxiv.org/abs/2301.00234 [bsky, 0 points, 2 comments]

Related

• The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis
• Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
• A Survey to Recent Progress Towards Understanding In-Context Learning
• In-Context Learning Learns Label Relationships but Is Not Conventional Learning
• Understanding In-Context Learning via Supportive Pretraining Data