2025/05/17 by Li, Huan, Dong, Yiming, Lin, Zhouchen · 1 citation
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC)
paper · doi:10.48550/arxiv.2505.11840
As the default optimizer for training large language models, AdamW has achieved remarkable success in deep learning. However, its convergence behavior is not theoretically well-understood. This paper establishes the convergence rate (1)/(K)∑k=1KE[||∇ f(xk)||1]≤ O(\frac√(d)CK1/4) for AdamW measured by ℓ1 norm, where K represents the iteration number, d denotes the model dimension, and C matches the constant in the optimal convergence rate of SGD. Theoretically, we have ||∇ f(x)||2≪ ||∇ f(x)||1≤ √(d)||∇ f(x)||2 for any high-dimensional vector x and E[||∇ f(x)||1]≥√\frac2dπE[||∇ f(x)||2] when each element of ∇ f(x) is generated from Gaussian distribution \mathcal N(0,1). Empirically, our experimental results on real-world deep learning tasks reveal ||∇ f(x)||1=\varTheta(√(d))||∇ f(x)||2. Both support that our convergence rate can be considered to be analogous to the optimal (1)/(K)∑k=1KE[||∇ f(xk)||2]≤ O(\fracCK1/4) convergence rate of SGD in the ideal case. We also extend our result to NAdamW, an AdamW variant that employs a double-momentum mechanism, and demonstrate that it maintains the same convergence rate.