Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention

2025/09/26 by Jingwei Li, Li, Jiaqian, Yanshu Li +7 · 1 citation
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face and Expression Recognition

paper · pdf · doi:10.48550/arxiv.2509.22854

openalex publication_date 2025/09/26 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Implicit in-context learning (ICL) has newly emerged as a promising paradigm that simulates ICL behaviors in the representation space of large language models (LLMs), aiming to attain few-shot performance at zero-shot cost. However, existing approaches largely rely on injecting shift vectors into residual flows, which are typically constructed from labeled demonstrations or task-specific alignment. Such designs fall short of utilizing the structural mechanisms underlying ICL and suffer from limited generalizability. To address this, we propose In-Context Routing (ICR), a novel implicit ICL method that captures and utilizes generalizable ICL patterns at the attention logits level. It extracts reusable structural directions that emerge during ICL and employs a learnable input-conditioned router to modulate attention logits accordingly, enabling an efficient train-once-and-reuse framework. We evaluate ICR on 12 real-world datasets spanning diverse domains and multiple LLMs. The results show that ICR consistently outperforms existing implicit ICL methods that require task-specific retrieval or training, while demonstrating robust generalization to out-of-domain tasks where they struggle. These findings position ICR to push the boundary of the practical value of ICL. The code is available at https://github.com/Lijiaqian1/In-Context-Routing.git.

Citations

• CAMA: Enhancing Multimodal In-Context Learning with Context-Aware Modulated Attention
• M²IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering
• Which Attention Heads Matter for In-Context Learning?
• Can In-context Learning Really Generalize to Out-of-distribution Tasks?
• Revisiting In-context Learning Inference Circuit in Large Language Models
• Implicit In-context Learning
• Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
• Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Models
• In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering
• Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models
• In-Context Learning Creates Task Vectors
• Function Vectors in Large Language Models
• Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability
• Language Models Implement Simple Word2Vec-style Vector Arithmetic
• What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning
• The Learnability of In-Context Learning
• Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering
• Transformers learn in-context by gradient descent
• In-context Learning and Induction Heads
• On the Relation between Sensitivity and Accuracy in In-context Learning
• What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
• Emergent Abilities of Large Language Models
• An Explanation of In-context Learning as Implicit Bayesian Inference
• MetaICL: Learning to Learn In Context
• CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge
• What Makes Good In-Context Examples for GPT-3?
• Language Models are Few-Shot Learners
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Character-level Convolutional Networks for Text Classification
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• On the distribution of the largest eigenvalue in principal components analysis
• The Rotation of Eigenvectors by a Perturbation. III
• Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Cited by

• CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding

Related

• Implicit In-context Learning
• Mechanistic Fine-tuning for In-context Learning
• Learning Task Representations from In-Context Learning
• Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability
• What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning