Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling

2025/09/01 by Sachin Goyal, Goyal, Sachin, David López-Paz +3 · 3 citations
Computer Science · Social Sciences · #Computational and Text Analysis Methods #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.01649

openalex publication_date 2025/09/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In the past year, distillation has seen a renewed prominence in large language model (LLM) pretraining, exemplified by the Llama-3.2 and Gemma model families. While distillation has historically been shown to improve statistical modeling, its effects on new paradigms that are key to modern LLMs, such as test-time scaling and in-context learning, remain underexplored. In this work, we make three main contributions. First, we show that pretraining with distillation yields models that exhibit remarkably better test-time scaling. Second, we observe that this benefit comes with a trade-off: distillation impairs in-context learning capabilities, particularly the one modeled via induction heads. Third, to demystify these findings, we study distilled pretraining in a sandbox of a bigram model, which helps us isolate the common principal factor behind our observations. Finally, using these insights, we shed light on various design choices for pretraining that should help practitioners going forward.

Citations

• Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation
• Multi-Token Prediction Needs Registers
• Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• Looking beyond the next token
• Weight Ensembling Improves Reasoning in Language Models
• Gemma 3 Technical Report
• Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers
• NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions
• Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning
• s1: Simple test-time scaling
• Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
• MiniPLM: Knowledge Distillation for Pre-Training Language Models
• Context-Parametric Inversion: Why Instruction Finetuning Can Worsen Context Reliance
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
• AlphaMath Almost Zero: Process Supervision without Process
• Better & Faster Large Language Models via Multi-token Prediction
• Forcing Diffuse Distributions out of Language Models
• Gemma: Open Models Based on Gemini Research and Technology
• The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains
• Towards the Law of Capacity Gap in Distilling Language Models
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• Birth of a Transformer: A Memory Viewpoint
• Let's Verify Step by Step
• Knowledge Distillation Performs Partial Variance Reduction
• Lifting the Curse of Capacity Gap in Distilling Language Models
• On student-teacher deviations in distillation: does it pay to disobey?
• Training Verifiers to Solve Math Word Problems
• Dynamic Knowledge Distillation for Pre-trained Language Models
• Program Synthesis with Large Language Models
• Knowledge distillation: A good teacher is patient and consistent
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Knowledge Distillation: A Survey
• Self-Distillation Amplifies Regularization in Hilbert Space
• Towards Understanding Knowledge Distillation
• Analysing Mathematical Reasoning Abilities of Neural Models
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
• Improved Knowledge Distillation via Teacher Assistant
• Born Again Neural Networks
• Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?

Cited by

• Mode-Conditioning Unlocks Superior Test-Time Scaling
• Ministral 3
• A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness

Related

• Pre-training Distillation for Large Language Models: A Design Space Exploration
• Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
• Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty
• On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models