Accelerating Federated Learning via Momentum Gradient Descent

2020/02/19 by Wei Liu, Li Chen, Yunfei Chen +1 · 7 citations
Computer Science · #Age of Information Optimization #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques

paper · doi:10.1109/tpds.2020.2975189

openalex publication_date 2020/02/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update which can potentially accelerate convergence. In this article, we consider momentum term which relates to the last iteration. The proposed momentum federated learning (MFL) uses momentum gradient descent (MGD) in the local update step of FL system. We establish global convergence properties of MFL and derive an upper bound on MFL convergence rate. Comparing the upper bounds on MFL and FL convergence rates, we provide conditions in which MFL accelerates the convergence. For different machine learning models, the convergence performance of MFL is evaluated based on experiments with MNIST and CIFAR-10 datasets. Simulation results confirm that MFL is globally convergent and further reveal significant convergence improvement over FL.

Citations

• Gradient-based learning applied to document recognition
• ImageNet classification with deep convolutional neural networks
• SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Edge Learning with Timeliness Constraints: Challenges and Solutions
• Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications
• Federated Learning of Quantile Inference under Local Differential Privacy
• Gradient-Leakage Resilient Federated Learning
• EmbeddedML: A New Optimized and Fast Machine Learning Library
• A Framework for Evaluating Gradient Leakage Attacks in Federated Learning
• Server Averaging for Federated Learning

Related

• Accelerating Federated Learning via Momentum Gradient Descent
• Momentum Benefits Non-IID Federated Learning Simply and Provably
• Rethinking the initialization of Momentum in Federated Learning with Heterogeneous Data
• Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup
• FedCM: Federated Learning with Client-level Momentum