Distributed stochastic gradient tracking methods

2020/03/17 by Shi Pu, Angelia Nedić · 258 citations
Computer Science · Engineering · Mathematics · #Applied mathematics #Computer science #Constant (computer programming) #Convex function #Distributed Control Multi-Agent Systems #Function (biology) #Iterated function #Mathematical optimization #Mathematics #Regular polygon #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #Tracking (education)

paper · doi:10.1007/s10107-020-01487-0

published in Mathematical Programming 187(1-2), 409-457 (Springer Science+Business Media)

openalex publication_date 2020/03/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Citations

• A Stochastic Approximation Method
• Stochastic Estimation of the Maximum of a Regression Function
• Constrained Consensus and Optimization in Multi-Agent Networks
• Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling

Cited by

• On the Linear Speedup of the Push-Pull Method for Decentralized Optimization over Digraphs
• Distributed Nash Equilibrium Seeking in N-Cluster Games with Fully Uncoordinated Constant Step-Sizes
• Variance Reduced EXTRA and DIGing and Their Optimal Acceleration for Strongly Convex Decentralized Optimization
• Distributed Online Economic Dispatch with Time-Varying Coupled Inequality Constraints
• An Accelerated Primal Dual Algorithm with Backtracking for Decentralized Constrained Optimization
• Distributed Policy Gradient with Variance Reduction in Multi-Agent Reinforcement Learning
• Beyond Scaffold: A Unified Spatio-Temporal Gradient Tracking Method
• Accelerating Distributed Online Meta-Learning via Multi-Agent Collaboration under Limited Communication
• Row-Stochastic Matrices Can Provably Outperform Doubly Stochastic Matrices in Decentralized Learning
• On Linear Convergence of Distributed Stochastic Bilevel Optimization over Undirected Networks via Gradient Aggregation
• A Differential Private Method for Distributed Optimization in Directed Networks via State Decomposition
• HBNET-GIANT: A communication-efficient accelerated Newton-type fully distributed optimization algorithm
• Event-triggered Dual Gradient Tracking for Distributed Resource Allocation
• Faster Non-Convex Federated Learning via Global and Local Momentum
• Provable local acceleration in HbNet-GIANT: an accelerated Newton-type distributed optimization algorithm
• Guaranteeing Both Consensus and Optimality in Decentralized Nonconvex Optimization with Multiple Local Updates
• Provably Accelerated Decentralized Gradient Method Over Unbalanced Directed Graphs
• Distributed Stochastic Momentum Tracking with Local Updates: Achieving Optimal Communication and Iteration Complexities
• Distributed Stochastic Proximal Algorithm on Riemannian Submanifolds for Weakly-convex Functions
• Convergence Analysis of Noisy Distributed Gradient Descent for Non-convex Optimization -- Saddle Point Escape
• High-Probability Convergence Guarantees of Decentralized SGD
• DeMuon: A Decentralized Muon for Matrix Optimization over Graphs
• Distributed Online Convex Optimization with Improved Dynamic Regret
• Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD
• Distributed Stochastic Optimization under Heavy-Tailed Noise: A Federated Mirror Descent Approach with High Probability Convergence
• A Weighted Gradient Tracking Privacy-Preserving Method for Distributed Optimization
• Pareto-optimal Trade-offs Between Communication and Computation with Flexible Gradient Tracking
• Over-the-Air Decentralized Federated Learning
• Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participation
• Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization
• An Optimistic Gradient Tracking Method for Distributed Minimax Optimization
• CoCoL: A Communication Efficient Decentralized Collaborative Method for Multi-Robot Systems
• A Hybrid Stochastic Gradient Tracking Method for Distributed Online Optimization Over Time-Varying Directed Networks
• Decentralized Composite Optimization with Compression
• Improving the Transient Times for Distributed Stochastic Gradient Methods
• Distributed Randomized Block Stochastic Gradient Tracking Method
• Asymptotic Properties of S-AB Method with Diminishing Stepsize
• Linear Convergent Decentralized Optimization with Compression
• Distributed Optimization with Projection-free Dynamics
• Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients
• A Stochastic Proximal Gradient Framework for Decentralized Non-Convex Composite Optimization: Topology-Independent Sample Complexity and Communication Efficiency
• Stochastic Push-Pull for Decentralized Nonconvex Optimization
• Distributed Composite Optimization with Sub-Weibull Noises
• A Robust Gradient Tracking Method for Distributed Optimization over Directed Networks
• A Sharp Estimate on the Transient Time of Distributed Stochastic Gradient Descent
• High Probability Convergence of Distributed Clipped Stochastic Gradient Descent with Heavy-tailed Noise
• Decentralized Optimization with Amplified Privacy via Efficient Communication
• A fast randomized incremental gradient method for decentralized non-convex optimization
• Exploiting Similarity for Computation and Communication-Efficient Decentralized Optimization
• Gradient-Free Nash Equilibrium Seeking in N-Cluster Games with Uncoordinated Constant Step-Sizes
• Decentralized Nonconvex Optimization under Heavy-Tailed Noise: Normalization and Optimal Convergence
• Enhancing Parallelism in Decentralized Stochastic Convex Optimization
• On the Convergence of Decentralized Stochastic Gradient-Tracking with Finite-Time Consensus
• On the Resolution of Stochastic MPECs over Networks: Distributed Implicit Zeroth-Order Gradient Tracking Methods
• Nash Equilibrium Seeking in N-Coalition Games via a Gradient-Free Method
• Decentralized Min-Max Optimization with Gradient Tracking
• Enhancing Accuracy in Differentially Private Distributed Optimization Through Sensitivity Reduction
• Robust Distributed Accelerated Stochastic Gradient Methods for Multi-Agent Networks
• Accelerated Decentralized Stochastic Gradient Descent for Strongly Convex Optimization
• Asynchronous Push-sum Dual Gradient Algorithm in Distributed Model Predictive Control
• Adaptive Stepsizes With Certified Convergence in Distributed Gradient Tracking With Quadratic Costs
• AdGT: Decentralized Gradient Tracking with Tuning-free Per-Agent Stepsize
• A Hybrid Variance-Reduced Method for Decentralized Stochastic Non-Convex\n Optimization
• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum