BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data

2024/09/25 by Jean-Loup Tastet, Inar Timiryasov, Tastet, Jean-Loup +1 · 1 voice · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Machine Learning (cs.LG) #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2409.17312

Abstract

We present BabyLlama-2, a 345 million parameter model distillation-pretrained from two teachers on a 10 million word corpus for the BabyLM competition. On BLiMP and SuperGLUE benchmarks, BabyLlama-2 outperforms baselines trained on both 10 and 100 million word datasets with the same data mix, as well as its teacher models. Through an extensive hyperparameter sweep, we demonstrate that the advantages of distillation cannot be attributed to suboptimal hyperparameter selection of the teachers. Our findings underscore the need for further investigation into distillation techniques, particularly in data-limited settings.

Cited by

• Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)

Discussions

• The holly grail, why? Inf: self distillation can largely improve models on same data and same size (e.g. arxiv.org/abs/2409.17312 , we also replicated here bsky.app/profile/lcho...), so do majority vo [bsky, 4 points, 0 comments]

Related

• Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty
• Distilling Double Descent
• On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models
• Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
• Teacher's pet: understanding and mitigating biases in distillation