Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learning

2023/09/28 by Wenke Huang, Mang Ye, Huang, Wenke +5 · 11 citations
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Health disparities and outcomes #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data

paper · pdf · doi:10.48550/arxiv.2309.16286

openalex publication_date 2023/09/28 · openalex created_date 2023/10/01 · openalex updated_date 2026/07/28

Abstract

Federated learning is an important privacy-preserving multi-party learning paradigm, involving collaborative learning with others and local updating on private data. Model heterogeneity and catastrophic forgetting are two crucial challenges, which greatly limit the applicability and generalizability. This paper presents a novel FCCL+, federated correlation and similarity learning with non-target distillation, facilitating the both intra-domain discriminability and inter-domain generalization. For heterogeneity issue, we leverage irrelevant unlabeled public data for communication between the heterogeneous participants. We construct cross-correlation matrix and align instance similarity distribution on both logits and feature levels, which effectively overcomes the communication barrier and improves the generalizable ability. For catastrophic forgetting in local updating stage, FCCL+ introduces Federated Non Target Distillation, which retains inter-domain knowledge while avoiding the optimization conflict issue, fulling distilling privileged inter-domain information through depicting posterior classes relation. Considering that there is no standard benchmark for evaluating existing heterogeneous federated learning under the same setting, we present a comprehensive benchmark with extensive representative methods under four domain shift scenarios, supporting both heterogeneous and homogeneous federated settings. Empirical results demonstrate the superiority of our method and the efficiency of modules on various scenarios.

Cited by

• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Divide, Conquer and Unite: Hierarchical Style-Recalibrated Prototype Alignment for Federated Medical Segmentation
• Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practice
• SC-GIR: Goal-oriented Semantic Communication via Invariant Representation Learning
• Enhancing Quantum Federated Learning with Fisher Information-Based Optimization
• Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout
• S2FGL: Spatial Spectral Federated Graph Learning
• pFedSOP : Accelerating Training Of Personalized Federated Learning Using Second-Order Optimization
• An Empirical Study of Federated Prompt Learning for Vision Language Model
• FedIFL: A federated cross-domain diagnostic framework for motor-driven systems with inconsistent fault modes
• SacFL: Self-Adaptive Federated Continual Learning for Resource-Constrained End Devices

Related

• FedCCL: Federated Dual-Clustered Feature Contrast Under Domain Heterogeneity
• FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaboration
• Reducing Spurious Correlation for Federated Domain Generalization
• Feature Correlation-guided Knowledge Transfer for Federated Self-supervised Learning
• Cross-Silo Heterogeneous Model Federated Multitask Learning