Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
2020/07/15 by Jianyu Wang, Wang, Jianyu, Qinghua Liu +7 · 575 citations
Computer Science · Mathematics · #Algorithm #Computation #Computer science #Convergence (economics) #Function (biology) #IoT and Edge/Fog Computing #Mathematical optimization #Mathematics #Optimization problem #Point (geometry) #Privacy-Preserving Technologies in Data #Stationary point #Stochastic Gradient Optimization Techniques #Trust region #cs.DC #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2007.07481
published in arXiv (Cornell University) (Cornell University)
arxiv created 2020/07/15 · openalex publication_date 2020/07/15 · arxiv updated 2020/07/16 · openalex created_date 2020/07/23 · openalex updated_date 2026/08/08
Abstract
In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such as FedAvg and FedProx and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.
Citations
Cited by
- Mechanistic Evidence for Preserved-but-Misaligned Representations in Non-IID FedAvg
- Adaptive Data Admission and Retention for Streaming Federated Learning
- Robust Federated Fine-Tuning in Heterogeneous Networks with Unreliable Connections: An Aggregation View
- FedDPC : Handling Data Heterogeneity and Partial Client Participation in Federated Learning
- FedOAED: Federated On-Device Autoencoder Denoiser for Heterogeneous Data under Limited Client Availability
- Sharpness-aware Federated Graph Learning
- Privacy-Preserving Feature Valuation in Vertical Federated Learning Using Shapley-CMI and PSI Permutation
- REVERB-FL: Server-Side Adversarial and Reserve-Enhanced Federated Learning for Robust Audio Classification
- Federated Learning with Feedback Alignment
- Semantic-Constrained Federated Aggregation: Convergence Theory and Privacy-Utility Bounds for Knowledge-Enhanced Distributed Learning
- Geometric Prior-Guided Federated Prompt Calibration
- How (Mis)calibrated is Your Federated CLIP and What To Do About It?
- Adaptive Aggregation with Two Gains in QFL
- Single-Round Scalable Analytic Federated Learning
- Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learning
- ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
- FedPoisonTTP: A Threat Model and Poisoning Attack for Federated Test-Time Personalization
- Analyzing the Impact of Participant Failures in Cross-Silo Federated Learning
- A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learning
- SMoFi: Step-wise Momentum Fusion for Split Federated Learning on Heterogeneous Data
- Data Heterogeneity and Forgotten Labels in Split Federated Learning
- Distributed Zero-Shot Learning for Visual Recognition
- Learning Performance Optimization for Edge AI System with Time and Energy Constraints
- FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning
- Enhancing Federated Learning Privacy with QUBO
- Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning
- FedSM: Robust Semantics-Guided Feature Mixup for Bias Reduction in Federated Learning with Long-Tail Data
- Non-Convex Over-the-Air Heterogeneous Federated Learning: A Bias-Variance Trade-off
- An All-Reduce Compatible Top-K Compressor for Communication-Efficient Distributed Learning
- Revolutionizing healthcare data analytics with federated learning: A comprehensive survey of applications, systems, and future directions
- An Ensembled Penalized Federated Learning Framework for Falling People Detection
- FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning
- FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
- Helmsman: Autonomous Synthesis of Federated Learning Systems via Collaborative LLM Agents
- BlendFL: Blended Federated Learning for Handling Multimodal Data Heterogeneity
- Parameter-Efficient and Personalized Federated Training of Generative Models at the Edge
- MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
- Beyond Static Knowledge Messengers: Towards Adaptive, Fair, and Scalable Federated Learning for Medical AI
- Adaptive Federated Learning via Dynamical System Model
- Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptation
- Beyond Aggregation: Guiding Clients in Heterogeneous Federated Learning
- DroneFL: Federated Learning for Multi-UAV Visual Target Tracking
- On the Fragility of Contribution Score Computation in Federated Learning
- C2Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning
- OmniFed: A Modular Framework for Configurable Federated Learning from Edge to HPC
- FedFiTS: Fitness-Selected, Slotted Client Scheduling for Trustworthy Federated Learning in Healthcare AI
- FedEL: Federated Elastic Learning for Heterogeneous Devices
- FedERL: Federated Efficient and Robust Learning for Common Corruptions
- FedSSG: Expectation-Gated and History-Aware Drift Alignment for Federated Learning
- FedAVOT: Exact Distribution Alignment in Federated Learning via Masked Optimal Transport
- FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health
- High-Energy Concentration for Federated Learning in Frequency Domain
- MetaFed: Advancing Privacy, Performance, and Sustainability in Federated Metaverse Systems
- FedDAF: Federated Domain Adaptation Using Model Functional Distance
- Cost-Free Personalization via Information-Geometric Projection in Bayesian Federated Learning
- Anomaly Detection in Electric Vehicle Charging Stations Using Federated Learning
- Data Valuation and Selection in a Federated Model Marketplace
- Enhancing Gradient Variance and Differential Privacy in Quantum Federated Learning
- Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participation
- Decentralized Federated Averaging via Random Walk
- Federated Learning for Large Models in Medical Imaging: A Comprehensive Review
- Cooperative SGD with Dynamic Mixing Matrices
- Communication-Efficient Federated Learning with Adaptive Number of Participants
- Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistency
- Deploying Models to Non-participating Clients in Federated Learning without Fine-tuning: A Hypernetwork-based Approach
- Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg
- Fed-Meta-Align: A Similarity-Aware Aggregation and Personalization Pipeline for Federated TinyML on Heterogeneous Data
- SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning
- Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems
- SHeRL-FL: When Representation Learning Meets Split Learning in Hierarchical Federated Learning
- Blockchain-Enabled Federated Learning
- FedMeNF: Privacy-Preserving Federated Meta-Learning for Neural Fields
- HeteRo-Select: Informativeness as the Participation Driver in Heterogeneous Federated Learning
- Long-Term Client Selection for Federated Learning with Non-IID Data: A Truthful Auction Approach
- FedMP: Tackling Medical Feature Heterogeneity in Federated Learning from a Manifold Perspective
- Decoupled Contrastive Learning for Federated Learning
- Energy-Efficient Federated Learning for Edge Real-Time Vision via Joint Data, Computation, and Communication Design
- Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning
- Mitigating Resolution-Drift in Federated Learning: Case of Keypoint Detection
- Challenges of Trustworthy Federated Learning: What's Done, Current Trends and Remaining Work
- Dubhe: Towards Data Unbiasedness with Homomorphic Encryption in Federated Learning Client Selection
- Who Owns This Sample: Cross-Client Membership Inference Attack in Federated Graph Neural Networks
- A New One-Shot Federated Learning Framework for Medical Imaging Classification with Feature-Guided Rectified Flow and Knowledge Distillation
- Clustered Federated Learning for Generalizable FDIA Detection in Smart Grids with Heterogeneous Data
- Characterizing Linear Alignment Across Language Models
- Federated Learning for Commercial Image Sources
- FedGA: A Fair Federated Learning Framework Based on the Gini Coefficient
- Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout
- MTF-Grasp: A Multi-tier Federated Learning Approach for Robotic Grasping
- Coding-Enforced Robust Secure Aggregation for Federated Learning Under Unreliable Communication
- Efficient Federated Learning with Timely Update Dissemination
- Prototype-Guided and Lightweight Adapters for Inherent Interpretation and Generalisation in Federated Learning
- Cooperative Gradient Coding
- S2FGL: Spatial Spectral Federated Graph Learning
- Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission
- FedDAA: Dynamic Client Clustering for Concept Drift Adaptation in Federated Learning
- Efficient Federated Learning with Encrypted Data Sharing for Data-Heterogeneous Edge Devices
- Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregation
- New Insights on Unfolding and Fine-tuning Quantum Federated Learning
- TriCon-SF: A Triple-Shuffle and Contribution-Aware Serial Federated Learning Framework for Heterogeneous Healthcare Data
- Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity
- Centroid Approximation for Byzantine-Tolerant Federated Learning
- Federated Learning for MRI-based BrainAGE: a multicenter study on post-stroke functional outcome prediction
- Distribution-Level AirComp for Wireless Federated Learning under Data Scarcity and Heterogeneity
- Similarity Weighted Aggregation with Global Differential Privacy for Federated Brain Lesion Segmentation
- Federated Semi-Supervised Learning with Class Distribution Mismatch
- Towards Unified Modeling in Federated Multi-Task Learning via Subspace Decoupling
- Personalized Subgraph Federated Learning with Differentiable Auxiliary Projections
- FedSDAF: Leveraging Source Domain Awareness for Enhanced Federated Domain Generalization
- Towards One-shot Federated Learning: Advances, Challenges, and Future Directions
- AMSFL: Adaptive Multi-Step Federated Learning via Gradient Difference-Based Error Modeling
- Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms
- Kuramoto-FedAvg: Using Synchronization Dynamics to Improve Federated Learning Optimization under Statistical Heterogeneity
- SEMFED: Semantic-Aware Resource-Efficient Federated Learning for Heterogeneous NLP Tasks
- Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data
- FedWorld: Scope-Aware Federation of Agent World Models
- ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoning
- Distributionally Robust Federated Learning with Client Drift Minimization
- Personalized Bayesian Federated Learning with Wasserstein Barycenter Aggregation
- FL-OA: A Byzantine-Robust Federated Learning Framework with Outsourced Auditing for Intelligent Devices
- Efficient Privacy-Preserving Cross-Silo Federated Learning with Multi-Key Homomorphic Encryption
- Heterogeneity-Aware Client Sampling for Optimal and Efficient Federated Learning
- Enhancing the Performance of Global Model by Improving the Adaptability of Local Models in Federated Learning
- Toward Malicious Clients Detection in Federated Learning
- FedRS-Bench: Realistic Federated Learning Datasets and Benchmarks in Remote Sensing
- Sharp Gaussian approximations for Decentralized Federated Learning
- DFPL: Decentralized Federated Prototype Learning Across Heterogeneous Data Distributions
- Federated learning, ethics, and the double black box problem in medical AI
- Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization
- Who Trains Matters: Federated Learning under Enrollment and Participation Selection Biases
- Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learning
- TACO: Tackling Over-correction in Federated Learning with Tailored Adaptive Correction
- Decentralized Time Series Classification with ROCKET Features
- Federated Learning for Medical Image Classification: A Comprehensive Benchmark
- Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data
- A Novel Algorithm for Personalized Federated Learning: Knowledge Distillation with Weighted Combination Loss
- Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directions
- Benchmark Evaluation of Federated Learning on Multi-organ Images
- FedDiverse: Tackling Data Heterogeneity in Federated Learning with Diversity-Driven Client Selection
- Accelerating Differentially Private Federated Learning via Adaptive Extrapolation
- The More is not the Merrier: Investigating the Effect of Client Size on Federated Learning
Related