Distillation Scaling Laws

2025/02/12 by Dan Busbridge, Busbridge, Dan, Amitis Shidani +9 · 9 voices · 18 citations
Engineering · #Process Optimization and Integration

paper · pdf · doi:10.48550/arxiv.2502.08606

Abstract

We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate the risks associated with large-scale distillation by enabling compute-optimal allocation for both the teacher and student to maximize student performance. We provide compute-optimal distillation recipes for two key scenarios: when a teacher already exists, and when a teacher needs training. In settings involving many students or an existing teacher, distillation outperforms supervised learning up to a compute level that scales predictably with student size. Conversely, if only one student is to be distilled and a teacher also requires training, supervised learning is generally preferable. Additionally, our large-scale study of distillation increases our understanding of the process and helps inform experimental design.

Cited by

• Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales
• Towards Characterizing Knowledge Distillation of PPG Heart Rate Estimation Models
• In Good GRACEs: Principled Teacher Selection for Knowledge Distillation
• Ministral 3
• Data-Centric Lessons To Improve Speech-Language Pretraining
• Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory
• xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
• Beyond Real Data: Synthetic Data through the Lens of Regularization
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
• Evaluating the Robustness of Chinchilla Compute-Optimal Scaling
• Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales
• Synthetic bootstrapped pretraining
• FLAME: A Serving System Optimized for Large-Scale Generative Recommendation with Efficiency
• Generalizing Scaling Laws for Dense and Sparse Large Language Models
• Realizing Scaling Laws in Recommender Systems: A Foundation-Expert Paradigm for Hyperscale Model Deployment
• A Primer in Post-Training Reasoning Data: What We Know About How It Works
• Language Models Improve When Pretraining Data Matches Target Tasks
• Scaling Laws for Optimal Data Mixtures

Discussions

• Apple researchers studied when to distill a smaller AI model from a larger one vs. when to fine-tune directly. Findings : • Distillation is best when compute is limited or training multiple models • [bsky, 13 points, 0 comments]
• Distillation Scaling Laws [hn, 5 points, 0 comments]
• Reading "Distilling Knowledge in a Neural Network" left me fascinated and wondering: "If I want a small, capable model, should I distill from a more powerful model, or train from scratch?" Our disti [bsky, 3 points, 1 comments]
• Distillation Scaling Laws [hn, 3 points, 0 comments]
• "Distillation Scaling Laws" arxiv.org/abs/2502.08606 [bsky, 1 points, 0 comments]
• Distillation Scaling Laws https://arxiv.org/pdf/2502.08606 #AI [bsky, 0 points, 0 comments]
• paper here: arxiv.org/pdf/2502.08606 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2502.08606 [bsky, 0 points, 0 comments]
• Distillation scaling laws Distillation works, but you must balance the difference between loss (performance) of the teacher & student, they need to be similar paper: arxiv.org/abs/2502.08606 o3-min [bsky, 0 points, 1 comments]

Related

• Distilling Double Descent
• Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation
• Revisiting Self-Distillation
• UNDO: Understanding Distillation as Optimization
• Can a student Large Language Model perform as well as it's teacher?