Convergence Analysis of Sequential Federated Learning on Heterogeneous Data

2023/11/06 by Yipeng Li, Li, Yipeng, Xinchen Lyu +1 · 9 citations
Computer Science · #Privacy-Preserving Technologies in Data #Distributed Sensor Networks and Detection Algorithms #Advanced Graph Neural Networks

paper · pdf · doi:10.48550/arxiv.2311.03154

Abstract

There are two categories of methods in Federated Learning (FL) for joint training across multiple clients: i) parallel FL (PFL), where clients train models in a parallel manner; and ii) sequential FL (SFL), where clients train models in a sequential manner. In contrast to that of PFL, the convergence theory of SFL on heterogeneous data is still lacking. In this paper, we establish the convergence guarantees of SFL for strongly/general/non-convex objectives on heterogeneous data. The convergence guarantees of SFL are better than that of PFL on heterogeneous data with both full and partial client participation. Experimental results validate the counterintuitive analysis result that SFL outperforms PFL on extremely heterogeneous data in cross-device settings.

Cited by

• DFedReweighting: A Unified Framework for Objective-Oriented Reweighting in Decentralized Federated Learning
• Efficient Asynchronous Federated Evaluation with Strategy Similarity Awareness for Intent-Based Networking in Industrial Internet of Things
• Accelerating Wireless Distributed Learning via Hybrid Split and Federated Learning Optimization
• Data Heterogeneity and Forgotten Labels in Split Federated Learning
• Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practice
• PubSub-VFL: Towards Efficient Two-Party Split Learning in Heterogeneous Environments via Publisher/Subscriber Architecture
• orb-QFL: Orbital Quantum Federated Learning
• SHeRL-FL: When Representation Learning Meets Split Learning in Hierarchical Federated Learning
• SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation

Related

• Sharp Bounds for Sequential Federated Learning on Heterogeneous Data
• Convergence Analysis of Sequential Split Learning on Heterogeneous Data
• Convergence Analysis of Split Federated Learning on Heterogeneous Data
• Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients
• Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence