Weight Initialization and Variance Dynamics in Deep Neural Networks and Large Language Models

2025/10/10 by Yankun Han, Han, Yankun
Materials Science · Computer Science · #Machine Learning in Materials Science #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning

paper · pdf · doi:10.48550/arxiv.2510.09423

Abstract

Weight initialization governs signal propagation and gradient flow at the start of training. This paper offers a theory-grounded and empirically validated study across two regimes: compact ReLU multilayer perceptrons and GPT-2-style transformers. First, a logarithmic sweep of the initial standard deviation maps vanishing and exploding regimes and identifies a broad stability band with standard deviations between 1e-2 and 1e-1. Second, a controlled comparison shows that Kaiming (fan-in) initialization converges faster and more stably than Xavier under ReLU, consistent with variance-preserving theory. Third, in a from-scratch 12-layer GPT-2-style model, this paper tracks layerwise Q/K/V weight variance through pretraining and observe depth-dependent equilibration into narrow bands: shallow layers expand rapidly while deeper layers change more gradually. Together, these results connect classic initialization principles with modern transformer behavior and yield simple, practical recipes for robust training.

Citations

• DeepNet: Scaling Transformers to 1,000 Layers
• GLU Variants Improve Transformer
• Scaling Laws for Neural Language Models
• Fixup Initialization: Residual Learning Without Normalization
• Deep Information Propagation
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Related

• Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes
• On weight initialization in deep neural networks
• Revisiting Initialization of Neural Networks
• Sinusoidal Initialization, Time for a New Start
• Improved weight initialization for deep and narrow feedforward neural network