2025/01/27 by Kaito Takanami, T. Takahashi, Takanami, Kaito +3 · 1 citation
Chemistry · Computer Science · Social Sciences · #Bayesian Methods and Mixture Models #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Religion and Sociopolitical Dynamics in Nigeria #Spectroscopy and Chemometric Analyses
paper · pdf · doi:10.48550/arxiv.2501.16226
openalex publication_date 2025/01/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Self-distillation (SD), a technique where a model improves itself using its own predictions, has attracted attention as a simple yet powerful approach in machine learning. Despite its widespread use, the mechanisms underlying its effectiveness remain unclear. In this study, we investigate the efficacy of hyperparameter-tuned multi-stage SD with a linear classifier for binary classification on noisy Gaussian mixture data. For the analysis, we employ the replica method from statistical physics. Our findings reveal that the primary driver of SD's performance improvement is denoising through hard pseudo-labels, with the most notable gains observed in moderately sized datasets. We also identify two practical heuristics to enhance SD: early stopping that limits the number of stages, which is broadly effective, and bias parameter fixing, which helps under label imbalance. To empirically validate our theoretical findings derived from our toy model, we conduct additional experiments on CIFAR-10 classification using pretrained ResNet backbone. These results provide both theoretical and practical insights, advancing our understanding and application of SD in noisy settings.