Exploring Length Generalization in Large Language Models

2022/07/11 by Cem Anil, Yuhuai Wu, Anil, Cem +17 · 20 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2207.04901

openalex publication_date 2022/07/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The ability to extrapolate from short problem instances to longer ones is an important form of out-of-distribution generalization in reasoning tasks, and is crucial when learning from datasets where longer problem instances are rare. These include theorem proving, solving quantitative mathematics problems, and reading/summarizing novels. In this paper, we run careful empirical studies exploring the length generalization capabilities of transformer-based language models. We first establish that naively finetuning transformers on length generalization tasks shows significant generalization deficiencies independent of model scale. We then show that combining pretrained large language models' in-context learning abilities with scratchpad prompting (asking the model to output solution steps before producing an answer) results in a dramatic improvement in length generalization. We run careful failure analyses on each of the learning modalities and identify common sources of mistakes that highlight opportunities in equipping language models with the ability to generalize to longer problems.

Cited by

• Adapting, Fast and Slow: Transportable Circuits for Few-Shot Learning
• Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs
• From Small to Large: Generalization Bounds for Transformers on Variable-Size Inputs
• Sequential Enumeration in Large Language Models
• How Does RL Post-training Induce Skill Composition? A Case Study on Countdown
• Pre-Generating Multi-Difficulty PDE Data for Few-Shot Neural PDE Solvers
• Exact Learning of Arithmetic with Differentiable Agents
• Next-Latent Prediction Transformers Learn Compact World Models
• Quantitative Bounds for Length Generalization in Transformers
• EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
• Symbol-Equivariant Recurrent Reasoning Models
• Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation
• Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents: Pathways and Paradigms
• HardcoreLogic: Challenging Large Reasoning Models with Long-tail Logic Puzzle Games
• A Multi-Agent Framework for Stateful Inference-Time Search
• h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning
• On the Limitations and Capabilities of Position Embeddings for Length Generalization
• Graph of Agents: Principled Long Context Modeling by Emergent Multi-Agent Collaboration
• Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformers
• Learning neuro-symbolic convergent term rewriting systems

Related

• Extrapolation by Association: Length Generalization Transfer in Transformers
• Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs
• Conditions for Length Generalization in Learning Reasoning Skills
• Improving Length-Generalization in Transformers via Task Hinting
• What Algorithms can Transformers Learn? A Study in Length Generalization