2025/01/30 by Jean Barbier, Francesco Camilli, Barbier, Jean +7
Computer Science · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Physical sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Statistical Mechanics (cond-mat.stat-mech)
paper · pdf · doi:10.48550/arxiv.2501.18530
openalex publication_date 2025/01/30 · openalex created_date 2025/10/06 · openalex updated_date 2026/07/29
We consider a teacher-student model of supervised learning with a fully-trained two-layer neural network whose width k and input dimension d are large and proportional. We provide an effective theory for approximating the Bayes-optimal generalisation error of the network for any activation function in the regime of sample size n scaling quadratically with the input dimension, i.e., around the interpolation threshold where the number of trainable parameters kd+k and of data n are comparable. Our analysis tackles generic weight distributions. We uncover a discontinuous phase transition separating a "universal" phase from a "specialisation" phase. In the first, the generalisation error is independent of the weight distribution and decays slowly with the sampling rate n/d2, with the student learning only some non-linear combinations of the teacher weights. In the latter, the error is weight distribution-dependent and decays faster due to the alignment of the student towards the teacher network. We thus unveil the existence of a highly predictive solution near interpolation, which is however potentially hard to find by practical algorithms.