Overtrained Language Models Are Harder to Fine-Tune

2025/03/24 by Jacob Mitchell Springer, Springer, Jacob Mitchell, Sachin Goyal +13 · 8 voices · 23 citations
#cs.CL #cs.AI

paper · pdf · doi:10.48550/arxiv.2503.19206

Abstract

Large language models are pre-trained on ever-growing token budgets under the assumption that better pre-training performance translates to improved downstream models. In this work, we challenge this assumption and show that extended pre-training can make models harder to fine-tune, leading to degraded final performance. We term this phenomenon catastrophic overtraining. For example, the instruction-tuned OLMo-1B model pre-trained on 3T tokens leads to over 2% worse performance on multiple standard LLM benchmarks than its 2.3T token counterpart. Through controlled experiments and theoretical analysis, we show that catastrophic overtraining arises from a systematic increase in the broad sensitivity of pre-trained parameters to modifications, including but not limited to fine-tuning. Our findings call for a critical reassessment of pre-training design that considers the downstream adaptability of the model.

Cited by

• Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
• Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
• Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay
• Pre-training under infinite compute
• The wall confronting large language models
• LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws
• Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
• Towards Active Synthetic Data Generation for Finetuning Language Models
• Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers
• PTPP-Aware Adaptation Scaling Laws: Predicting Domain-Adaptation Performance at Unseen Pre-Training Budgets
• GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters
• Midtraining Bridges Pretraining and Posttraining Distributions
• xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
• Mid-Training of Large Language Models: A Survey
• Reusing Overtrained Language Models Saturates Scaling
• Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
• Conda: Column-Normalized Adam for Training Large Language Models Faster
• Tracing the Representation Geometry of Language Models from Pretraining to Post-training
• RL Fine-Tuning Heals OOD Forgetting in SFT
• Universal Properties of Activation Sparsity in Modern Large Language Models
• Evaluation of Finetuned LLMs in AMR Parsing
• Reinitializing weights vs units for maintaining plasticity in neural networks

Discussions

• E.g., here's one of many papers pointing out that excessive pre-training on an LLM makes the final performance worse. If an LLM is overtrained to fit Trump's requirement it talk like MechHitler, the e [bsky, 121 points, 3 comments]
• Training with more data = better LLMs, right? 🚨 False! Scaling language models by adding more pre-training data can decrease your performance after post-training! Introducing "catastrophic overtrai [bsky, 33 points, 1 comments]
• Not biology-related but interesting nonetheless arxiv.org/abs/2503.19206 [bsky, 14 points, 2 comments]
• I have been wondering about this for a while, kind of a "if you know everything, you know nothing" kind of thing arxiv.org/abs/2503.19206 [bsky, 1 points, 1 comments]
• Overtrained models makes them harder to fine tune. That is, one makes tradeoffs in model-building, a lesson one has to learn about all ML, and optimization work in general "Overtrained Language Models [bsky, 1 points, 0 comments]
• I wonder if somebody studies what affects LLMs "finetunability" on downstream set of specific domains/tasks either from architectural or data perspective The only thing I could remember is arxiv.org/a [bsky, 0 points, 0 comments]
• The assumption that better pre-training performance translates to improved downstream models might be wrong. This paper shows that "catastrophic overtraining" can make models harder to fine-tune, lead [bsky, 0 points, 0 comments]
• arxiv.org/abs/2503.19206 [bsky, 0 points, 0 comments]

Related

• The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis
• Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?
• Continual Pre-Training of Large Language Models: How to (re)warm your model?
• Reusing Overtrained Language Models Saturates Scaling
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis