2018/09/30 by Mario Geiger, Stefano Spigler, Stéphane d’Ascoli +6 · 1 citation
Computer Science · Materials Science · Mathematics · Physics and Astronomy · #Applied mathematics #Artificial intelligence #Artificial neural network #Computer science #Critical point (mathematics) #Curvature #Deep learning #Energy landscape #Function (biology) #Geometry #Hessian matrix #Material Dynamics and Properties #Mathematical analysis #Mathematics #Maxima and minima #Phase transition #Physics #Quantum mechanics #Statistical Mechanics and Entropy #Statistical physics #Theoretical and Computational Physics #cond-mat.dis-nn #cs.LG
paper · pdf · doi:10.1103/physreve.100.012115
published as Phys. Rev. E 100, 012115 (2019)
arxiv created 2019/06/17 · openalex publication_date 2019/07/11 · arxiv updated 2019/07/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
Deep learning has been immensely successful at a variety of tasks, ranging from classification to artificial intelligence. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understanding under which conditions neural networks do not get stuck in poor minima of the loss, and how the landscape of that loss evolves as depth is increased, remains a challenge. Here we predict, and test empirically, an analogy between this landscape and the energy landscape of repulsive ellipses. We argue that in fully connected deep networks a phase transition delimits the over- and underparametrized regimes where fitting can or cannot be achieved. In the vicinity of this transition, properties of the curvature of the minima of the loss (the spectrum of the Hessian) are critical. This transition shares direct similarities with the jamming transition by which particles form a disordered solid as the density is increased, which also occurs in certain classes of computational optimization and learning problems such as the perceptron. Our analysis gives a simple explanation as to why poor minima of the loss cannot be encountered in the overparametrized regime. Interestingly, we observe that the ability of fully connected networks to fit random data is independent of their depth, an independence that appears to also hold for real data. We also study a quantity \mathrm\ensuremathΔ which characterizes how well (\mathrm\ensuremathΔ<0) or badly (\mathrm\ensuremathΔ>0) a datum is learned. At the critical point it is power-law distributed on several decades, P+(\mathrm\ensuremathΔ)\ensuremath∼\mathrm\ensuremathΔ^\ensuremathθ for \mathrm\ensuremathΔ>0 and P_\ensuremath-(\mathrm\ensuremathΔ)\ensuremath∼(\ensuremath-\mathrm\ensuremathΔ)^\ensuremath-\ensuremathγ for \mathrm\ensuremathΔ<0, with exponents that depend on the choice of activation function. This observation suggests that near the transition the loss landscape has a hierarchical structure and that the learning dynamics is prone to avalanche-like dynamics, with abrupt changes in the set of patterns that are learned.