Multi-Level Branched Regularization for Federated Learning

2022/07/14 by Jinkyu Kim, Kim, Jinkyu, Geeho Kim +3 · 5 citations
Computer Science · Engineering · #Advanced Graph Neural Networks #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Traffic Prediction and Management Techniques

paper · pdf · doi:10.48550/arxiv.2207.06936

openalex publication_date 2022/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A critical challenge of federated learning is data heterogeneity and imbalance across clients, which leads to inconsistency between local networks and unstable convergence of global models. To alleviate the limitations, we propose a novel architectural regularization technique that constructs multiple auxiliary branches in each local model by grafting local and global subnetworks at several different levels and that learns the representations of the main pathway in the local model congruent to the auxiliary hybrid pathways via online knowledge distillation. The proposed technique is effective to robustify the global model even in the non-iid setting and is applicable to various federated learning frameworks conveniently without incurring extra communication costs. We perform comprehensive empirical studies and demonstrate remarkable performance gains in terms of accuracy and efficiency compared to existing methods. The source code is available at our project page.

Cited by

• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Adversarial Versus Federated: An Adversarial Learning based Multi-Modality Cross-Domain Federated Medical Segmentation
• FedERL: Federated Efficient and Robust Learning for Common Corruptions
• FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantization
• ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoning

Related

• FedTrip: A Resource-Efficient Federated Learning Method with Triplet Regularization
• Boosting Federated Learning Convergence with Prototype Regularization
• Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning
• Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data
• Federated Learning Based on Dynamic Regularization