FedDuA: Doubly Adaptive Federated Learning

2025/05/16 by Shokichi Takakura, Takakura, Shokichi, Seng Pei Liew +3
Computer Science · #Advanced Technologies in Various Fields #Convergence (economics) #Convex optimization #Distributed learning #FOS: Computer and information sciences #Federated learning #Global optimization #Gradient descent #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Minification #Minimax #Privacy-Preserving Technologies in Data #Rate of convergence #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.11126

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Federated learning is a distributed learning framework where clients collaboratively train a global model without sharing their raw data. FedAvg is a popular algorithm for federated learning, but it often suffers from slow convergence due to the heterogeneity of local datasets and anisotropy in the parameter space. In this work, we formalize the central server optimization procedure through the lens of mirror descent and propose a novel framework, called FedDuA, which adaptively selects the global learning rate based on both inter-client and coordinate-wise heterogeneity in the local updates. We prove that our proposed doubly adaptive step-size rule is minimax optimal and provide a convergence analysis for convex objectives. Although the proposed method does not require additional communication or computational cost on clients, extensive numerical experiments show that our proposed framework outperforms baselines in various settings and is robust to the choice of hyperparameters.

Citations

• Understanding Transformer Optimization via Gradient Heterogeneity
• Efficient Adaptive Federated Optimization
• The Power of Extrapolation in Federated Learning
• Why Transformers Need Adam: A Hessian Perspective
• FedExP: Speeding Up Federated Averaging via Extrapolation
• Generalized Federated Learning via Sharpness Aware Minimization
• ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!
• Public Data-Assisted Mirror Descent for Private Model Training
• Local Adaptivity in Federated Learning: Convergence and Consistency
• Federated Composite Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• A Study of Gradient Variance in Deep Learning
• Mirror Descent Policy Optimization
• Adaptive Federated Optimization
• Advances and Open Problems in Federated Learning
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
• On the Convergence of Local Descent Methods in Federated Learning
• Intrinsic dimension of data representations in deep neural networks
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Federated Optimization in Heterogeneous Networks
• LEAF: A Benchmark for Federated Settings
• Federated Learning: Strategies for Improving Communication Efficiency
• Introduction to Online Convex Optimization

Related

• Locally Adaptive Federated Learning
• Adaptive Federated Optimization
• FedAgg: Adaptive Federated Learning with Aggregated Gradients
• Communication Efficient Federated Learning with Linear Convergence on Heterogeneous Data
• FedDUAL: A Dual-Strategy with Adaptive Loss and Dynamic Aggregation for Mitigating Data Heterogeneity in Federated Learning