Unified Description of Learning Dynamics in the Soft Committee Machine from Finite to Ultra-Wide Regimes

2025/12/18 by Afanah, Assem, Rosenow, Bernd
#Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Physical sciences

paper · doi:10.48550/arxiv.2512.16556

Abstract

We study the learning dynamics of the soft committee machine (SCM) with Rectified Linear Unit (ReLU) activation using a statistical-mechanics approach within the annealed approximation. The SCM consists of a student network with N input units and K hidden units trained to reproduce the output of a teacher network with M hidden units. We introduce a reduced set of macroscopic order parameters that yields a unified description valid from the conventional regime K ≪ N to the ultra-wide limit K ≥ N. The control parameter α, proportional to the ratio of training samples to adjustable weights, serves as an effective measure of dataset size. For small γ= M/N, we recover a continuous phase transition at α_c ≈ 2π from an unspecialized, permutation-symmetric state to a specialized state in which student units align with the teacher. For finite γ, the transition disappears and the generalization error decreases smoothly with dataset size, reaching a low plateau when γ=1. In the asymptotic limit α→ ∞, the error scales as ε_g ∝ 1/α, independent of γ and K. The results highlight the central role of network dimensions in SCM learning and provide a framework extendable to other activations and quenched analyses.

Citations

• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Statistical Physics of Deep Learning: Optimal Learning of a Multilayer Perceptron near Interpolation
• On Exact Computation with an Infinitely Wide Neural Net
• Machine learning and the physical sciences
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• High-dimensional dynamics of generalization error in neural networks
• Empirical Evaluation of Rectified Activations in Convolutional Network
• The space of interactions in neural network models
• Artificial intelligence in information systems research: A systematic literature review and research agenda

Related

• Learning and generalization theories of large committee--machines
• Statistical Mechanical Analysis of the Dynamics of Learning in Perceptrons
• Mean-field methods and algorithmic perspectives for high-dimensional machine learning
• Neural networks: from the perceptron to deep nets
• Functional optimization of online algorithms in multilayer neural networks