An Accelerated Distributed Stochastic Gradient Method with Momentum

2024/02/15 by Kun Huang, Shi Pu, Huang, Kun +3 · 3 citations
Computer Science · Engineering · Mathematics · #Advanced Optimization Algorithms Research #Distributed #FOS: Computer and information sciences #FOS: Mathematics #Multiagent Systems (cs.MA) #Optimization and Control (math.OC) #Parallel #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2402.09714

openalex publication_date 2024/02/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we introduce an accelerated distributed stochastic gradient method with momentum for solving the distributed optimization problem, where a group of n agents collaboratively minimize the average of the local objective functions over a connected network. The method, termed ``Distributed Stochastic Momentum Tracking (DSMT)'', is a single-loop algorithm that utilizes the momentum tracking technique as well as the Loopless Chebyshev Acceleration (LCA) method. We show that DSMT can asymptotically achieve comparable convergence rates as centralized stochastic gradient descent (SGD) method under a general variance condition regarding the stochastic gradients. Moreover, the number of iterations (transient times) required for DSMT to achieve such rates behaves as O(n^5/3/(1-λ)) for minimizing general smooth objective functions, and O(√(n/(1-λ))) under the Polyak-Łojasiewicz (PL) condition. Here, the term 1-λ denotes the spectral gap of the mixing matrix related to the underlying network topology. Notably, the obtained results do not rely on multiple inter-node communications or stochastic gradient accumulation per iteration, and the transient times are the shortest under the setting to the best of our knowledge.

Cited by

• A Unified Decentralized Nonconvex Algorithm under Kurdyka-Łojasiewicz Property
• Distributed Stochastic Momentum Tracking with Local Updates: Achieving Optimal Communication and Iteration Complexities
• Local Differential Privacy for Distributed Stochastic Aggregative Optimization with Guaranteed Optimality

Related

• Distributed Stochastic Momentum Tracking with Local Updates: Achieving Optimal Communication and Iteration Complexities
• Robust Distributed Accelerated Stochastic Gradient Methods for Multi-Agent Networks
• Distributed Momentum-based Frank-Wolfe Algorithm for Stochastic Optimization
• Improving the Transient Times for Distributed Stochastic Gradient Methods
• Accelerated AB/Push-Pull Methods for Distributed Optimization over Time-Varying Directed Networks