Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization

2020/07/15 by Jianyu Wang, Wang, Jianyu, Qinghua Liu +7 · 81 citations
Computer Science · #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #IoT and Edge/Fog Computing

paper · pdf · doi:10.48550/arxiv.2007.07481

Abstract

In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such as FedAvg and FedProx and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition

Cited by

• Mechanistic Analysis of Circuit Preservation in Federated Learning
• Adaptive Data Admission and Retention for Streaming Federated Learning
• Robust Federated Fine-Tuning in Heterogeneous Networks with Unreliable Connections: An Aggregation View
• FedDPC : Handling Data Heterogeneity and Partial Client Participation in Federated Learning
• FedOAED: Federated On-Device Autoencoder Denoiser for Heterogeneous Data under Limited Client Availability
• Sharpness-aware Federated Graph Learning
• Privacy-Preserving Feature Valuation in Vertical Federated Learning Using Shapley-CMI and PSI Permutation
• REVERB-FL: Server-Side Adversarial and Reserve-Enhanced Federated Learning for Robust Audio Classification
• Federated Learning with Feedback Alignment
• Semantic-Constrained Federated Aggregation: Convergence Theory and Privacy-Utility Bounds for Knowledge-Enhanced Distributed Learning
• Geometric Prior-Guided Federated Prompt Calibration
• How (Mis)calibrated is Your Federated CLIP and What To Do About It?
• Adaptive Aggregation with Two Gains in QFL
• Single-Round Scalable Analytic Federated Learning
• Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learning
• ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
• FedPoisonTTP: A Threat Model and Poisoning Attack for Federated Test-Time Personalization
• Analyzing the Impact of Participant Failures in Cross-Silo Federated Learning
• A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learning
• SMoFi: Step-wise Momentum Fusion for Split Federated Learning on Heterogeneous Data
• Data Heterogeneity and Forgotten Labels in Split Federated Learning
• Distributed Zero-Shot Learning for Visual Recognition
• Learning Performance Optimization for Edge AI System with Time and Energy Constraints
• FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning
• Enhancing Federated Learning Privacy with QUBO
• Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning
• FedSM: Robust Semantics-Guided Feature Mixup for Bias Reduction in Federated Learning with Long-Tail Data
• Non-Convex Over-the-Air Heterogeneous Federated Learning: A Bias-Variance Trade-off
• An All-Reduce Compatible Top-K Compressor for Communication-Efficient Distributed Learning
• Revolutionizing healthcare data analytics with federated learning: A comprehensive survey of applications, systems, and future directions
• An Ensembled Penalized Federated Learning Framework for Falling People Detection
• FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning
• FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
• Helmsman: Autonomous Synthesis of Federated Learning Systems via Collaborative LLM Agents
• BlendFL: Blended Federated Learning for Handling Multimodal Data Heterogeneity
• Parameter-Efficient and Personalized Federated Training of Generative Models at the Edge
• MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
• Beyond Static Knowledge Messengers: Towards Adaptive, Fair, and Scalable Federated Learning for Medical AI
• Adaptive Federated Learning via Dynamical System Model
• Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptation
• Beyond Aggregation: Guiding Clients in Heterogeneous Federated Learning
• DroneFL: Federated Learning for Multi-UAV Visual Target Tracking
• On the Fragility of Contribution Score Computation in Federated Learning
• C²Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning
• OmniFed: A Modular Framework for Configurable Federated Learning from Edge to HPC
• FedFiTS: Fitness-Selected, Slotted Client Scheduling for Trustworthy Federated Learning in Healthcare AI
• FedEL: Federated Elastic Learning for Heterogeneous Devices
• FedERL: Federated Efficient and Robust Learning for Common Corruptions
• FedSSG: Expectation-Gated and History-Aware Drift Alignment for Federated Learning
• FedAVOT: Exact Distribution Alignment in Federated Learning via Masked Optimal Transport
• FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health
• High-Energy Concentration for Federated Learning in Frequency Domain
• MetaFed: Advancing Privacy, Performance, and Sustainability in Federated Metaverse Systems
• FedDAF: Federated Domain Adaptation Using Model Functional Distance
• Cost-Free Personalization via Information-Geometric Projection in Bayesian Federated Learning
• Anomaly Detection in Electric Vehicle Charging Stations Using Federated Learning
• Data Valuation and Selection in a Federated Model Marketplace
• Enhancing Gradient Variance and Differential Privacy in Quantum Federated Learning
• Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participation
• Decentralized Federated Averaging via Random Walk
• Federated Learning for Large Models in Medical Imaging: A Comprehensive Review
• Cooperative SGD with Dynamic Mixing Matrices
• Communication-Efficient Federated Learning with Adaptive Number of Participants
• Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistency
• Deploying Models to Non-participating Clients in Federated Learning without Fine-tuning: A Hypernetwork-based Approach
• Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg
• Fed-Meta-Align: A Similarity-Aware Aggregation and Personalization Pipeline for Federated TinyML on Heterogeneous Data
• SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning
• Biased Local SGD for Efficient Deep Learning on Heterogeneous Systems
• SHeRL-FL: When Representation Learning Meets Split Learning in Hierarchical Federated Learning
• Blockchain-Enabled Federated Learning
• FedMeNF: Privacy-Preserving Federated Meta-Learning for Neural Fields
• HeteRo-Select: Informativeness as the Participation Driver in Heterogeneous Federated Learning
• Long-Term Client Selection for Federated Learning with Non-IID Data: A Truthful Auction Approach
• FedMP: Tackling Medical Feature Heterogeneity in Federated Learning from a Manifold Perspective
• Decoupled Contrastive Learning for Federated Learning
• Energy-Efficient Federated Learning for Edge Real-Time Vision via Joint Data, Computation, and Communication Design
• Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning
• Mitigating Resolution-Drift in Federated Learning: Case of Keypoint Detection
• Who Owns This Sample: Cross-Client Membership Inference Attack in Federated Graph Neural Networks
• A New One-Shot Federated Learning Framework for Medical Imaging Classification with Feature-Guided Rectified Flow and Knowledge Distillation

Related

• Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optima
• A New Theoretical Perspective on Data Heterogeneity in Federated Optimization
• Heterogeneity-Aware Client Sampling: A Unified Solution for Consistent Federated Learning
• Federated Optimization in Heterogeneous Networks
• FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation