Why and When Deep is Better than Shallow: Implementation-Agnostic State-Transition Model of Deep Learning

2025/05/21 by Sho Sonoda, Yuka Hashimoto, Sonoda, Sho +5
Economics, Econometrics and Finance · #Deep neural networks #Dynamical Systems (math.DS) #Entropy (arrow of time) #Exponential function #FOS: Computer and information sciences #FOS: Mathematics #Financial Markets and Investment Strategies #Generalization #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Metric (unit) #Semigroup #Upper and lower bounds #Variance (accounting)

paper · pdf · doi:10.48550/arxiv.2505.15064

published in ArXiv.org

openalex publication_date 2025/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Why and when does depth improve generalization? We study this question in an implementation-agnostic state-transition model, where a depth-k predictor is a readout class H composed with the word ball B(k,F) generated by hidden state transitions. Generalization bounds separate implementation error, approximation error, and statistical complexity, and upper bound the depth-dependent variance term by a Dudley entropy integral over B(k,F), with a conditional lower-bound diagnostic under readout separation. We identify geometric and semigroup mechanisms that keep this entropy contribution saturated or polynomial, and contrast them with separation mechanisms that recover the classical exponential-growth obstruction. Coupling these variance upper bounds with approximation rates gives typical depth trade-off patterns, clarifying that depth is statistically favorable when approximation improves rapidly while the transition semigroup remains geometrically tame.

Citations

• Efficient Diffusion Models: A Survey
• Quantitative Approximation for Neural Operators in Nonlinear Parabolic Equations
• Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss
• PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Advantage of Deep Neural Networks for Estimating Functions with Singularity on Hypersurfaces
• Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
• The phase diagram of approximation rates for deep neural networks
• Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
• Neural Ordinary Differential Equations
• Stronger generalization bounds for deep nets via a compression approach
• Deep Neural Networks Learn Non-Smooth Functions Effectively
• Optimal approximation of continuous functions by very deep ReLU networks
• Size-Independent Sample Complexity of Neural Networks
• Spectrally-normalized margin bounds for neural networks
• Error bounds for approximations with deep ReLU networks
• Benefits of depth in neural networks
• The Power of Depth for Feedforward Neural Networks
• Norm-Based Capacity Control in Neural Networks
• Geometry of Locally Compact Groups of Polynomial Growth and Shape of Large Balls

Related

• Is Deeper Better only when Shallow is Good?
• On the Depth of Deep Neural Networks: A Theoretical View
• The smooth output assumption, and why deep networks are better than wide ones
• On residual network depth
• An Information-Theoretic View for Deep Learning