Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension

2026/07/27 by Jinhao Zhang, Zeyu Liu, Zicheng Yan +4
#cs.LG #cs.AI

paper · pdf

Abstract

Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remains beneficial on unseen data. We study this problem for function-preserving residual expansion and introduce the effective alignment dimension, a measurable quantity describing the signal-noise geometry of activation gradients. By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability. The bound depends only on the effective alignment dimension and an effective sample size, requiring finite second moments and a nonzero population gradient, without covariance spectral assumptions or prescribed width-growth rates. We integrate this certificate into the train-test residual-expansion framework, yielding a high-probability condition for test-risk improvement. Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment. Direct residual interventions confirm that the alignment statistic predicts the sign and magnitude of held-out loss changes.

Citations

• Exact Network Surgery: Functional Invariance and Gradient Plasticity in Reactive Computational Graphs
• PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs
• Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks
• Learning to Grow Pretrained Models for Efficient Transformer Training
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
• Understanding Why Neural Networks Generalize Well Through GSNR of Parameters
• An Empirical Model of Large-Batch Training
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Deep Neural Networks as Gaussian Processes
• Deep Residual Learning for Image Recognition

Related

• Limitations of Post-Hoc Feature Alignment for Robustness
• Width and Depth Limits Commute in Residual Networks
• Generalization Ability of Wide Residual Networks
• Residual Alignment: Uncovering the Mechanisms of Residual Networks
• Spectral Alignment as Predictor of Loss Explosion in Neural Network Training