Arithmetic-Mean μP for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets

2025/10/05 by Haosong Zhang, Zhang, Haosong, Siyu Wu +5
Computer Science · #Stochastic Gradient Optimization Techniques #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning

paper · pdf · doi:10.48550/arxiv.2510.04327

Abstract

Choosing an appropriate learning rate remains a key challenge in scaling depth of modern deep networks. The classical maximal update parameterization (μP) enforces a fixed per-layer update magnitude, which is well suited to homogeneous multilayer perceptrons (MLPs) but becomes ill-posed in heterogeneous architectures where residual accumulation and convolutions introduce imbalance across layers. We introduce Arithmetic-Mean μP (AM-μP), which constrains not each individual layer but the network-wide average one-step pre-activation second moment to a constant scale. Combined with a residual-aware He fan-in initialization - scaling residual-branch weights by the number of blocks (Var[W]=c/(K⋅ fan-in)) - AM-μP yields width-robust depth laws that transfer consistently across depths. We prove that, for one- and two-dimensional convolutional networks, the maximal-update learning rate satisfies η^⋆(L)∝ L^-3/2; with zero padding, boundary effects are constant-level as N≫ k. For standard residual networks with general conv+MLP blocks, we establish η^⋆(L)=Θ(L^-3/2), with L the minimal depth. Empirical results across a range of depths confirm the -3/2 scaling law and enable zero-shot learning-rate transfer, providing a unified and practical LR principle for convolutional and deep residual networks without additional tuning overhead.

Citations

• The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimization
• Principled Architecture-aware Scaling of Hyperparameters
• Is One Epoch All You Need For Multi-Fidelity Hyperparameter Optimization?
• Depth Dependence of μP Learning Rates in ReLU MLPs
• PASHA: Efficient HPO and NAS with Progressive Resource Allocation
• Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
• ReZero is All You Need: Fast Convergence at Large Depth
• How to Initialize your Network? Robust Initialization for WeightNorm & ResNets
• Fixup Initialization: Residual Learning Without Normalization
• Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
• Mean Field Residual Networks: On the Edge of Chaos
• Deep Residual Networks and Weight Initialization
• Deep Information Propagation
• Exponential expressivity in deep neural networks through transient chaos
• All you need is a good init
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Related

• Understanding Scaling Laws in Deep Neural Networks via Feature Learning Dynamics
• Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
• Depth Dependence of μP Learning Rates in ReLU MLPs
• The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks
• Principled Architecture-aware Scaling of Hyperparameters