Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime

2025/05/06 by Francesco Camilli, Daria Tieplova, Camilli, Francesco +5 · 3 citations
Computer Science · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Mathematics #FOS: Physical sciences #Information Theory (cs.IT) #Machine Learning (stat.ML) #Mathematical Physics (math-ph) #Neural Networks and Applications #Statistics Theory (math.ST)

paper · pdf · doi:10.48550/arxiv.2505.03577

openalex publication_date 2025/05/06 · openalex created_date 2025/10/16 · openalex updated_date 2026/07/30

Abstract

We rigorously analyse fully-trained neural networks of arbitrary depth in the Bayesian optimal setting in the so-called proportional scaling regime where the number of training samples and width of the input and all inner layers diverge proportionally. We prove an information-theoretic equivalence between the Bayesian deep neural network model trained from data generated by a teacher with matching architecture, and a simpler model of optimal inference in a generalized linear model. This equivalence enables us to compute the optimal generalization error for deep neural networks in this regime. We thus prove the "deep Gaussian equivalence principle" conjectured in Cui et al. (2023) (arXiv:2302.00375). Our result highlights that in order to escape this "trivialisation" of deep neural networks (in the sense of reduction to a linear model) happening in the strongly overparametrized proportional regime, models trained from much more data have to be considered.

Citations

• From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Statistical Physics of Deep Learning: Optimal Learning of a Multilayer Perceptron near Interpolation
• On the phase diagram of extensive-rank symmetric matrix denoising beyond rotational invariance
• Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers
• Asymptotics of Learning with Deep Structured (Random) Features
• Fundamental limits of overparametrized shallow neural networks for supervised learning
• Precise Asymptotic Analysis of Deep Random Feature Models
• Deterministic equivalent and error universality of deep random features learning
• Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergence
• Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
• Random Neural Networks in the Infinite Width Limit as Gaussian Processes
• Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Universality Laws for High-Dimensional Learning with Random Features
• A Precise Performance Analysis of Learning with Random Features
• The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• The generalization error of random features regression: Precise asymptotics and double descent curve
• The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve
• Linearized two-layers neural networks in high dimension
• On Exact Computation with an Infinitely Wide Neural Net
• Wide neural networks of any depth evolve as linear models under gradient descent *
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Analysis of a Two-Layer Neural Network via Displacement Convexity
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Learning Overparameterized Neural Networks via Stochastic Gradient\n Descent on Structured Data
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Gaussian Process Behaviour in Wide Deep Neural Networks
• Stochastic Particle Gradient Descent for Infinite Ensembles
• Deep Neural Networks as Gaussian Processes
• Convergence Analysis of Two-layer Neural Networks with ReLU Activation
• Conditional Central Limit Theorems for Gaussian Projections
• Fundamental limits of symmetric low-rank matrix estimation
• Generalization Properties of Learning with Random Features
• On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions
• Approximation of projections of random vectors
• Three unfinished works on the optimal storage capacity of networks

Cited by

• Microscopic and collective signatures of feature learning in neural networks
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks

Related

• Fundamental limits of overparametrized shallow neural networks for supervised learning
• Information-Theoretic Foundations for Neural Scaling Laws
• An Information-Theoretic Analysis of Compute-Optimal Neural Scaling Laws
• Neural networks: from the perceptron to deep nets
• An Information-Theoretic Framework for Supervised Learning