From Local SGD to Local Fixed-Point Methods for Federated Learning

2020/04/03 by Grigory Malinovsky, Dmitry Kovalev, Malinovsky, Grigory +7 · 5 citations
Computer Science · Engineering · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Privacy-Preserving Technologies in Data #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2004.01442

openalex publication_date 2020/04/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Most algorithms for solving optimization problems or finding saddle points of convex-concave functions are fixed-point algorithms. In this work we consider the generic problem of finding a fixed point of an average of operators, or an approximation thereof, in a distributed setting. Our work is motivated by the needs of federated learning. In this context, each local operator models the computations done locally on a mobile device. We investigate two strategies to achieve such a consensus: one based on a fixed number of local steps, and the other based on randomized computations. In both cases, the goal is to limit communication of the locally-computed variables, which is often the bottleneck in distributed frameworks. We perform convergence analysis of both methods and conduct a number of experiments highlighting the benefits of our approach.

Citations

• Federated Learning: Strategies for Improving Communication Efficiency
• Gradient Descent with Compressed Iterates
• Distributed Fixed Point Methods with Compressed Iterates

Cited by

• First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptions
• Guaranteeing Both Consensus and Optimality in Decentralized Nonconvex Optimization with Multiple Local Updates
• An Operator Splitting View of Federated Learning
• The Stochastic Multi-Proximal Method for Nonsmooth Optimization
• FedEval: A Holistic Evaluation Framework for Federated Learning

Related

• On the Convergence of Local Descent Methods in Federated Learning
• First Analysis of Local GD on Heterogeneous Data
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• Local SGD: Unified Theory and New Efficient Methods
• Local SGD and Federated Averaging Through the Lens of Time Complexity