FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models

2025/10/31 by Junkang Liu, Fanhua Shang, Liu, Junkang +10 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2510.27486

openalex publication_date 2025/10/31 · openalex created_date 2025/11/05 · openalex updated_date 2026/07/28

Abstract

AdamW has become one of the most effective optimizers for training large-scale models. We have also observed its effectiveness in the context of federated learning (FL). However, directly applying AdamW in federated learning settings poses significant challenges: (1) due to data heterogeneity, AdamW often yields high variance in the second-moment estimate \boldsymbolv; (2) the local overfitting of AdamW may cause client drift; and (3) Reinitializing moment estimates (\boldsymbolv, \boldsymbolm) at each round slows down convergence. To address these challenges, we propose the first \underlineFederated \underlineAdamW algorithm, called FedAdamW, for training and fine-tuning various large models. FedAdamW aligns local updates with the global update using both a local correction mechanism and decoupled weight decay to mitigate local overfitting. FedAdamW efficiently aggregates the mean of the second-moment estimates to reduce their variance and reinitialize them. Theoretically, we prove that FedAdamW achieves a linear speedup convergence rate of O(√((L Δσ_l²)/(S K R ε²))+(L Δ)/R) without heterogeneity assumption, where S is the number of participating clients per round, K is the number of local iterations, and R is the total number of communication rounds. We also employ PAC-Bayesian generalization analysis to explain the effectiveness of decoupled weight decay in local training. Empirically, we validate the effectiveness of FedAdamW on language and vision Transformer models. Compared to several baselines, FedAdamW significantly reduces communication rounds and improves test accuracy. The code is available in https://github.com/junkangLiu0/FedAdamW.

Citations

• DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Norm
• FedMuon: Accelerating Federated Learning with Matrix Orthogonalization
• JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation
• Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
• Dual Enhancement on 3D Vision-Language Perception for Monocular 3D Visual Grounding
• FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averaging
• Graph Learning Metallic Glass Discovery from Wikipedia
• OFFSET: Segmentation-based Focus Shift Revision for Composed Image Retrieval
• FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
• Convex Relaxation for Robust Vanishing Point Estimation in Manhattan World
• FineCIR: Explicit Parsing of Fine-Grained Modification Semantics for Composed Image Retrieval
• LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement
• GlobalPointer: Large-Scale Plane Adjustment with Bi-Convex Relaxation
• Adam-mini: Use Fewer Learning Rates To Gain More
• Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization
• Information Entropy Guided Height-aware Histogram for Quantization-friendly Pillar Feature Encoder
• Towards Understanding Convergence and Generalization of AdamW
• Taming Cross-Domain Representation Variance in Federated Prototype Learning with Heterogeneous Data Domains
• Why Transformers Need Adam: A Hessian Perspective
• Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup
• Accelerating Hybrid Federated Learning Convergence under Partial Participation
• From a Bird's Eye View to See: Joint Camera and Subject Registration without the Camera Calibration
• FedCM: Federated Learning with Client-level Momentum
• Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Toward Communication Efficient Adaptive Gradient Method
• Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
• Adaptive Federated Optimization
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Federated Optimization in Heterogeneous Networks
• On the Convergence of Adam and Beyond
• Decoupled Weight Decay Regularization
• Three Factors Influencing Minima in SGD
• Stochastic gradient descent performs variational inference, converges to\n limit cycles for deep networks
• A Bayesian Perspective on Generalization and Stochastic Gradient Descent
• Attention Is All You Need
• Deep Residual Learning for Image Recognition
• Adam: A Method for Stochastic Optimization
• Going Deeper with Convolutions
• ADADELTA: An Adaptive Learning Rate Method
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Cited by

• HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval
• JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation

Related

• FedMuon: Accelerating Federated Learning with Matrix Orthogonalization
• Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimization
• FedImpro: Measuring and Improving Client Update in Federated Learning
• Gradient Correction in Federated Learning with Adaptive Optimization
• Faster Federated Learning with Decaying Number of Local SGD Steps