Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning

2025/07/10 by Doan, Nhi Hoai, Takaharu Hiraoka, Kentaro Inui +2
Psychology · Neuroscience · Computer Science · #Action Observation and Synchronization #Motor Control and Adaptation #Domain Adaptation and Few-Shot Learning

paper · pdf · doi:10.48550/arxiv.2507.07810

Abstract

This paper investigates the relationship between large language models' (LLMs) ability to recognize repetitive input patterns and their performance on in-context learning (ICL). In contrast to prior work that has primarily focused on attention heads, we examine this relationship from the perspective of skill neurons, specifically repetition neurons. Our experiments reveal that the impact of these neurons on ICL performance varies depending on the depth of the layer in which they reside. By comparing the effects of repetition neurons and induction heads, we further identify strategies for reducing repetitive outputs while maintaining strong ICL capabilities.

Citations

• Repetition Neurons: How Do Language Models Produce Repetitions?
• Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing
• The Llama 3 Herd of Models
• Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning
• Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective
• Understanding In-Context Learning from Repetitions
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Birth of a Transformer: A Memory Viewpoint
• A Survey on In-context Learning
• Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale
• In-context Learning and Induction Heads
• Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation
• Relating Neural Text Degeneration to Exposure Bias
• Language Models are Few-Shot Learners
• The Curious Case of Neural Text Degeneration
• BERTScore: Evaluating Text Generation with BERT
• Hierarchical Neural Story Generation
• Attention Is All You Need

Related

• Repetition Neurons: How Do Language Models Produce Repetitions?
• Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models
• Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning
• Repetitions are not all alike: distinct mechanisms sustain repetition in language models
• Which Attention Heads Matter for In-Context Learning?