2024/03/07 by Zixiong Yu, Chen, Guhan, Tian, Songtao +4
Computer Science · Engineering · #FOS: Computer and information sciences #FOS: Mathematics #Image and Object Detection Techniques #Machine Learning (cs.LG) #Optical Systems and Laser Technology #Statistics Theory (math.ST) #Welding Techniques and Residual Stresses
paper · pdf · doi:10.48550/arxiv.2403.04545
openalex publication_date 2024/03/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Scaling factors in residual branches have emerged as a prevalent method for boosting neural network performance, especially in normalization-free architectures. While prior work has primarily examined scaling effects from an optimization perspective, this paper investigates their role in residual architectures through the lens of generalization theory. Specifically, we establish that wide residual networks (ResNets) with constant scaling factors become asymptotically unlearnable as depth increases. In contrast, when the scaling factor exhibits rapid depth-wise decay combined with early stopping, over-parameterized ResNets achieve minimax-optimal generalization rates. To establish this, we demonstrate that the generalization capability of wide ResNets can be approximated by kernel regression associated with the Neural Tangent Kernel (NTK). Our theoretical findings are validated through experiments on synthetic data and real-world classification tasks, including MNIST and CIFAR-100.