Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning

2025/06/13 by Chengye Li, Li, Chengye, Haiyun Liu +3
Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Innovative Education and Learning Practices #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2506.11516

openalex publication_date 2025/06/13 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

In-context learning (ICL) allows large language models (LLMs) to solve novel tasks without weight updates. Despite its empirical success, the mechanism behind ICL remains poorly understood, limiting our ability to interpret, improve, and reliably apply it. In this paper, we propose a new theoretical perspective that interprets ICL as an implicit form of knowledge distillation (KD), where prompt demonstrations guide the model to form a task-specific reference model during inference. Under this view, we derive a Rademacher complexity-based generalization bound and prove that the bias of the distilled weights grows linearly with the Maximum Mean Discrepancy (MMD) between the prompt and target distributions. This theoretical framework explains several empirical phenomena and unifies prior gradient-based and distributional analyses. To the best of our knowledge, this is the first to formalize inference-time attention as a distillation process, which provides theoretical insights for future prompt engineering and automated demonstration selection.

Citations

• Effects of Prompt Length on Domain-specific Tasks for Large Language Models
• An Information-Theoretic Analysis of In-Context Learning
• In-context Learning and Gradient Descent Revisited
• The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis
• Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
• Function Vectors in Large Language Models
• Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens
• How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations
• Uncovering mesa-optimization algorithms in Transformers
• Schema-learning and rebinding as mechanisms of in-context learning and emergence
• In-Context Learning through the Bayesian Prism
• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
• What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization
• Can We Edit Factual Knowledge by In-Context Learning?
• The Closeness of In-Context Learning and Weight Shifting for Softmax Regression
• GPT-4 Technical Report
• The Learnability of In-Context Learning
• A Theory of Emergent In-Context Learning as Implicit Structure Induction
• LLaMA: Open and Efficient Foundation Language Models
• Transformers as Algorithms: Generalization and Stability in In-context Learning
• Transformers learn in-context by gradient descent
• What learning algorithm is in-context learning? Investigations with linear models
• In-context Learning and Induction Heads
• What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
• PaLM: Scaling Language Modeling with Pathways
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• An Explanation of In-context Learning as Implicit Bayesian Inference
• MetaICL: Learning to Learn In Context
• On the Opportunities and Risks of Foundation Models
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
• Language Models are Few-Shot Learners
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Root Mean Square Layer Normalization
• Attention Is All You Need
• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
• Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models
• In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning
• Efficient LLM Context Distillation
• Basic Reading Distillation
• Is In-Context Learning Learning?