A Field Guide to Federated Optimization

2021/07/14 by Jianyu Wang, Wang, Jianyu, Zachary Charles +103 · 15 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mobile Crowdsensing and Crowdsourcing #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2107.06917

openalex publication_date 2021/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection. The distributed learning process can be formulated as solving federated optimization problems, which emphasize communication efficiency, data heterogeneity, compatibility with privacy and system requirements, and other constraints that are not primary considerations in other problem settings. This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms through concrete examples and practical implementation, with a focus on conducting effective simulations to infer real-world performance. The goal of this work is not to survey the current literature, but to inspire researchers and practitioners to design federated learning algorithms that can be used in various practical applications.

Citations

• Experiments on Parallel Training of Deep Neural Network using Model Averaging
• Training Deep Nets with Sublinear Memory Cost
• Parallel SGD: When does averaging help?
• Equality of Opportunity in Supervised Learning
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Distributed Mean Estimation with Limited Communication
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption
• On First-Order Meta-Learning Algorithms
• Sparsified SGD with Memory
• A generic framework for privacy preserving deep learning
• LEAF: A Benchmark for Federated Settings
• Decentralized Stochastic Optimization and Gossip Algorithms with\n Compressed Communication
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• SecureBoost: A Lossless Federated Learning Framework
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• Natural Compression for Distributed Deep Learning
• Bayesian Nonparametric Federated Learning of Neural Networks
• Zeno: Distributed Stochastic Gradient Descent with Suspicion-based\n Fault-tolerance
• Distributed Optimization for Over-Parameterized Learning
• Scalable and Differentially Private Distributed Aggregation in the Shuffled Model
• On the Convergence of FedAvg on Non-IID Data
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• FedMD: Heterogenous Federated Learning via Model Distillation
• Federated Learning with Personalization Layers
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Advances and Open Problems in Federated Learning
• Robust Aggregation for Federated Learning
• Personalized Federated Learning: A Meta-Learning Approach
• Three Approaches for Personalization with Applications to Federated Learning
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• A Framework for Evaluating Gradient Leakage Attacks in Federated Learning
• Minibatch vs Local SGD for Heterogeneous Distributed Learning
• Federated Learning Meets Multi-objective Optimization
• On the Outsized Importance of Learning Rates in Local Update Methods
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Flower: A Friendly Federated Learning Research Framework
• Error Compensated Distributed SGD Can Be Accelerated
• Learning Multiple Layers of Features from Tiny Images
• Federated Learning Based on Dynamic Regularization
• Federated Reconstruction: Partially Local Federated Learning
• On the Utility of Gradient Compression in Distributed Training Systems
• On Large-Cohort Training for Federated Learning
• Personalized Cross-Silo Federated Learning on Non-IID Data

Cited by

• Federated Learning With L0 Constraint Via Probabilistic Gates For Sparsity
• First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptions
• CycleSL: Server-Client Cyclical Update Driven Scalable Split Learning
• Federated Cyber Defense: Privacy-Preserving Ransomware Detection Across Distributed Systems
• Towards Straggler-Resilient Split Federated Learning: An Unbalanced Update Approach
• The Sherpa.ai Blind Vertical Federated Learning Paradigm to Minimize the Number of Communications
• Decoupled DiLoCo for Resilient Distributed Pre-training
• MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
• Beyond Static Knowledge Messengers: Towards Adaptive, Fair, and Scalable Federated Learning for Medical AI
• Who to Trust? Aggregating Client Knowledge in Logit-Based Federated Learning
• Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration
• Perfectly-Private Analog Secure Aggregation in Federated Learning
• Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation
• One-Shot Clustering for Federated Learning Under Clustering-Agnostic Assumption
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision

Related

• Review of Mathematical Optimization in Federated Learning
• When Decentralized Optimization Meets Federated Learning
• Optimization Methods and Software for Federated Learning
• Federated Learning: Challenges, Methods, and Future Directions
• FedGrad: Optimisation in Decentralised Machine Learning