Learning without training: The implicit dynamics of in-context learning

2025/07/21 by Benoit Dherin, Benoît Dherin, Dherin, Benoit +9 · 10 voices · 16 citations
Psychology · #Innovative Teaching and Learning Methods

paper · pdf · doi:10.48550/arxiv.2507.16003

Abstract

One of the most striking features of Large Language Models (LLMs) is their ability to learn in-context. Namely at inference time an LLM is able to learn new patterns without any additional weight update when these patterns are presented in the form of examples in the prompt, even if these patterns were not seen during training. The mechanisms through which this can happen are still largely unknown. In this work, we show that the stacking of a self-attention layer with an MLP allows the transformer block to implicitly modify the weights of the MLP layer according to the context. We argue through theoretical analysis and experimentation that this simple mechanism may help explain why LLMs demonstrate capabilities of in-context learning, beyond what is captured during training. Specifically, we show that a standard forward pass with context is mathematically equivalent to a forward pass without context but with the MLP weights updated by a minimal low-rank update representing the context.

Citations

• Task Vectors in In-Context Learning: Emergence, Formation, and Benefit
• Simplifying Transformer Blocks
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
• Transformers learn in-context by gradient descent
• Meta Learning Backpropagation And Improving It
• Language Models are Few-Shot Learners
• Attention Is All You Need

Cited by

• Position: Modular Memory is the Key to Continual Learning Agents
• A Simple Generalisation of the Implicit Dynamics of In-Context Learning
• Equivalence of Context and Parameter Updates in Modern Transformer Blocks
• HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning
• A Criminology of Machines
• Evaluating Temporal Consistency in Multi-Turn Language Models
• The Narrative Continuity Test: A Conceptual Framework for Evaluating Identity Persistence in AI Systems
• Transmuting prompts into weights
• Uncertainty as Feature Gaps: Epistemic Uncertainty Quantification of LLMs in Contextual Question-Answering
• Test time training enhances in-context learning of nonlinear functions
• TTT3R: 3D Reconstruction as Test-Time Training
• Think-on-Graph 3.0: Efficient and Adaptive LLM Reasoning on Heterogeneous Graphs via Multi-Agent Dual-Evolving Context Retrieval
• Few-Shot and Training-Free Review Generation via Conversational Prompting
• Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptation
• Instruction-Level Weight Shaping: A Framework for Self-Improving AI Agents
• Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management

Discussions

• Interesting paper suggesting a mechanism for why in-context learning happens in LLMs. They show that LLMs implicitly apply an internal low-rank weight update adjusted by the context. It’s cheap (due t [bsky, 59 points, 1 comments]
• Self-attention transforms a prompt into a low-rank weight-update [hn, 17 points, 1 comments]
• You may find this paper interesting: "... we show that a standard forward pass with context is mathematically equivalent to a forward pass without context but with the MLP weights updated by a minimal [bsky, 6 points, 1 comments]
• Finally getting around to reading this paper: arxiv.org/abs/2507.16003 The authors propose that instead of adding attention to the residual stream, it's possible to apply it as a low-rank adaptation t [bsky, 4 points, 1 comments]
• Learning without training: The implicit dynamics of in-context learning [hn, 2 points, 0 comments]
• The paper you are likely referring to is "Learning without training: The implicit dynamics of in-context learning" by Dherin et al. from Google Research, published in July 2025. It can be found on arX [bsky, 2 points, 1 comments]
• The Implicit Dynamics of In-Context Learning [hn, 1 points, 0 comments]
• Learning without training: The implicit dynamics of in-context learning [hn, 1 points, 0 comments]
• Interesting paper in this context arxiv.org/pdf/2507.16003 [bsky, 0 points, 1 comments]
• Learning without training: The implicit dynamics of in-context learning arxiv.org/pdf/2507.16003 [bsky, 0 points, 0 comments]

Related

• A Theory of Emergent In-Context Learning as Implicit Structure Induction
• A Simple Generalisation of the Implicit Dynamics of In-Context Learning
• Toward Understanding In-context vs. In-weight Learning
• An Explanation of In-context Learning as Implicit Bayesian Inference
• Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting