FedProto: Federated Prototype Learning across Heterogeneous Clients

2021/05/01 by Yue Tan, Tan, Yue, Guodong Long +11 · 75 citations
Computer Science · #Privacy-Preserving Technologies in Data #cs.DC #cs.LG

paper · pdf · doi:10.48550/arxiv.2105.00243

arxiv created 2022/03/05 · arxiv updated 2022/03/08

Abstract

Heterogeneity across clients in federated learning (FL) usually hinders the optimization convergence and generalization performance when the aggregation of clients' knowledge occurs in the gradient space. For example, clients may differ in terms of data distribution, network latency, input/output space, and/or model architecture, which can easily lead to the misalignment of their local gradients. To improve the tolerance to heterogeneity, we propose a novel federated prototype learning (FedProto) framework in which the clients and server communicate the abstract class prototypes instead of the gradients. FedProto aggregates the local prototypes collected from different clients, and then sends the global prototypes back to all clients to regularize the training of local models. The training on each client aims to minimize the classification error on the local data while keeping the resulting local prototypes sufficiently close to the corresponding global ones. Moreover, we provide a theoretical analysis to the convergence rate of FedProto under non-convex objectives. In experiments, we propose a benchmark setting tailored for heterogeneous FL, with FedProto outperforming several recent FL approaches on multiple datasets.

Cited by

• Energy and Memory-Efficient Federated Learning With Ordered Layer Freezing
• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Geometric Prior-Guided Federated Prompt Calibration
• Operator-Theoretic Framework for Gradient-Free Federated Learning
• Cross-Domain Federated Semantic Communication with Global Representation Alignment and Domain-Aware Aggregation
• FedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learning
• FedAPA: Federated Learning with Adaptive Prototype Aggregation Toward Heterogeneous Wi-Fi CSI-based Crowd Counting
• Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learning
• SAM-Fed: SAM-Guided Federated Semi-Supervised Learning for Medical Image Segmentation
• Personalized Federated Learning with Bidirectional Communication Compression via One-Bit Random Sketching
• Divide, Conquer and Unite: Hierarchical Style-Recalibrated Prototype Alignment for Federated Medical Segmentation
• FedeCouple: Fine-Grained Balancing of Global-Generalization and Local-Adaptability in Federated Learning
• Federated CLIP for Resource-Efficient Heterogeneous Medical Image Classification
• MedFedPure: A Medical Federated Framework with MAE-based Detection and Diffusion Purification for Inference-Time Attacks
• Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers
• FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learning
• Sentinel: Dynamic Knowledge Distillation for Personalized Federated Intrusion Detection in Heterogeneous IoT Networks
• FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning
• Towards Robust Knowledge Removal in Federated Learning with High Data Heterogeneity
• DOLFIN: Balancing Stability and Plasticity in Federated Continual Learning
• Gains: Fine-grained Federated Domain Adaptation in Open Set
• Think as a Doctor: An Interpretable AI Approach for ICU Mortality Prediction
• Beyond Static Knowledge Messengers: Towards Adaptive, Fair, and Scalable Federated Learning for Medical AI
• DPFNAS: Differential Privacy-Enhanced Federated Neural Architecture Search for 6G Edge Intelligence
• C²Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning
• Choice Outweighs Effort: Facilitating Complementary Knowledge Fusion in Federated Learning via Re-calibration and Merit-discrimination
• FedEL: Federated Elastic Learning for Heterogeneous Devices
• Dissecting Federated-Graph Aggregation under Domain Shift: Importance-Aware Aggregation via Empirical Analysis
• BAPFL: Exploring Backdoor Attacks Against Prototype-based Federated Learning
• Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation
• PracMHBench: Re-evaluating Model-Heterogeneous Federated Learning Based on Practical Edge Device Constraints
• Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data
• Dual-Distilled Heterogeneous Federated Learning with Adaptive Margins for Trainable Global Prototypes
• Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistency
• Hierarchy-Consistent Learning and Adaptive Loss Balancing for Hierarchical Multi-Label Classification
• FedUNet: A Lightweight Additive U-Net Module for Federated Learning with Heterogeneous Models
• Mitigating Modality Quantity and Quality Imbalance in Multimodal Online Federated Learning
• GraphFedMIG: Tackling Class Imbalance in Federated Graph Learning via Mutual Information-Guided Generation
• A Federated Learning Framework for Handling Subtype Confounding and Heterogeneity in Large-Scale Neuroimaging Diagnosis
• FedMP: Tackling Medical Feature Heterogeneity in Federated Learning from a Manifold Perspective
• Boosting Generalization Performance in Model-Heterogeneous Federated Learning Using Variational Transposed Convolution
• Proto-EVFL: Enhanced Vertical Federated Learning via Dual Prototype with Extremely Unaligned Data
• FedSA-GCL: A Semi-Asynchronous Federated Graph Learning Framework with Personalized Aggregation and Cluster-Aware Broadcasting
• Domain Borders Are There to Be Crossed With Federated Few-Shot Adaptation
• Intrinsic Training Signals for Federated Learning Aggregation
• FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learning
• Prototype-Guided and Lightweight Adapters for Inherent Interpretation and Generalisation in Federated Learning
• FedPall: Prototype-based Adversarial and Collaborative Learning for Federated Learning with Feature Drift
• PROTEAN: Federated Intrusion Detection in Non-IID Environments through Prototype-Based Knowledge Sharing
• Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
• TinyProto: Communication-Efficient Federated Learning with Sparse Prototypes in Resource-Constrained Environments
• Heterogeneous Federated Learning with Prototype Alignment and Upscaling
• Flotilla: A scalable, modular and resilient federated learning framework for heterogeneous resources
• Whom to Trust? Adaptive Collaboration in Personalized Federated Learning
• CLoVE: Personalized Federated Learning through Clustering of Loss Vector Embeddings
• FedSC: Federated Learning with Semantic-Aware Collaboration
• Behavioral Anomaly Detection in Distributed Systems via Federated Contrastive Learning
• FedWSIDD: Federated Whole Slide Image Classification via Dataset Distillation
• Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systems
• HtFLlib: A Comprehensive Heterogeneous Federated Learning Library and Benchmark
• PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs
• Lazy But Effective: Collaborative Personalized Federated Learning with Heterogeneous Data
• Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments
• FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaboration
• Multimodal Online Federated Learning with Modality Missing in Internet of Things
• ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoning
• Robust Federated Learning on Edge Devices with Domain Heterogeneity
• Adaptive Latent-Space Constraints in Personalized Federated Learning
• DFPL: Decentralized Federated Prototype Learning Across Heterogeneous Data Distributions
• Privacy Preserving Machine Learning Model Personalization through Federated Personalized Learning
• Diffusion-Guided Semantic Consistency for Multimodal Heterogeneity
• PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs
• Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learning
• Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks
• Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directions
• Federated Prototype Graph Learning
• PPFPL: Cross-silo Privacy-preserving Federated Prototype Learning Against Data Poisoning Attacks

Related

• Prototype Helps Federated Learning: Towards Faster Convergence
• Boosting Federated Learning Convergence with Prototype Regularization
• FedImpro: Measuring and Improving Client Update in Federated Learning
• FedIN: Federated Intermediate Layers Learning for Model Heterogeneity
• FedDPC : Handling Data Heterogeneity and Partial Client Participation in Federated Learning