FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

2021/02/15 by Xiaoxiao Li, Li, Xiaoxiao, Meirui Jiang +7 · 79 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #cs.LG

paper · pdf · doi:10.48550/arxiv.2102.07623

Accepted at ICLR 2021

openalex created_date 2021/02/01 · openalex publication_date 2021/02/15 · arxiv created 2021/05/11 · arxiv updated 2021/05/12 · openalex updated_date 2026/07/28

Abstract

The emerging paradigm of federated learning (FL) strives to enable collaborative training of deep models on the network edge without centrally aggregating raw data and hence improving data privacy. In most cases, the assumption of independent and identically distributed samples across local clients does not hold for federated learning setups. Under this setting, neural network training performance may vary significantly according to the data distribution and even hurt training convergence. Most of the previous work has focused on a difference in the distribution of labels or client shifts. Unlike those settings, we address an important problem of FL, e.g., different scanners/sensors in medical imaging, different scenery distribution in autonomous driving (highway vs. city), where local clients store examples with different distributions compared to other clients, which we denote as feature shift non-iid. In this work, we propose an effective method that uses local batch normalization to alleviate the feature shift before averaging models. The resulting scheme, called FedBN, outperforms both classical FedAvg, as well as the state-of-the-art for non-iid data (FedProx) on our extensive experiments. These empirical results are supported by a convergence analysis that shows in a simplified setting that FedBN has a faster convergence rate than FedAvg. Code is available at https://github.com/med-air/FedBN.

Citations

• Gradient-based learning applied to document recognition
• Revisiting Batch Normalization For Practical Domain Adaptation
• On the importance of single directions for generalization
• A Convergence Theory for Deep Learning via Over-Parameterization
• A generic framework for privacy preserving deep learning
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Flower: A Friendly Federated Learning Research Framework
• Advances and Open Problems in Federated Learning

Cited by

• AutoFed: Personalized Federated Traffic Prediction via Adaptive Prompt
• Robust Federated Fine-Tuning in Heterogeneous Networks with Unreliable Connections: An Aggregation View
• Federated Domain Generalization with Latent Space Inversion
• FedSCAl: Leveraging Server and Client Alignment for Unsupervised Federated Source-Free Domain Adaptation
• FDRMFL:Multi-modal Federated Feature Extraction Model Based on Information Maximization and Contrastive Learning
• FedAPA: Federated Learning with Adaptive Prototype Aggregation Toward Heterogeneous Wi-Fi CSI-based Crowd Counting
• Personalized Federated Segmentation with Shared Feature Aggregation and Boundary-Focused Calibration
• A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learning
• FLClear: Visually Verifiable Multi-Client Watermarking for Federated Learning
• FedSDA: Federated Stain Distribution Alignment for Non-IID Histopathological Image Classification
• EPFL-REMNet: Efficient Personalized Federated Digital Twin Towards 6G Heterogeneous Radio Environment
• MedFedPure: A Medical Federated Framework with MAE-based Detection and Diffusion Purification for Inference-Time Attacks
• FedOnco-Bench: A Reproducible Benchmark for Privacy-Aware Federated Tumor Segmentation with Synthetic CT Data
• UnifiedFL: A Dynamic Unified Learning Framework for Equitable Federation
• Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers
• Exploring Federated Learning for Thermal Urban Feature Segmentation -- A Comparison of Centralized and Decentralized Approaches
• SGFusion: Stochastic Geographic Gradient Fusion in Federated Learning
• FedL2T: Personalized Federated Learning with Two-Teacher Distillation for Seizure Prediction
• Validation of Various Normalization Methods for Brain Tumor Segmentation: Can Federated Learning Overcome This Heterogeneity?
• Federated Unlearning in the Wild: Rethinking Fairness and Data Discrepancy
• Beyond Static Knowledge Messengers: Towards Adaptive, Fair, and Scalable Federated Learning for Medical AI
• Technical note on Fisher Information for Robust Federated Cross-Validation
• Personalized federated prototype learning in mixed heterogeneous data scenarios
• Adversarial Versus Federated: An Adversarial Learning based Multi-Modality Cross-Domain Federated Medical Segmentation
• Beyond Aggregation: Guiding Clients in Heterogeneous Federated Learning
• C²Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning
• PerturbMap: Cross-Context Transfer of Single-Cell Perturbation Responses
• OmniFed: A Modular Framework for Configurable Federated Learning from Edge to HPC
• pFedSAM: Personalized Federated Learning of Segment Anything Model for Medical Image Segmentation
• On the Out-of-Distribution Backdoor Attack for Federated Learning
• Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation
• Enhancing Gradient Variance and Differential Privacy in Quantum Federated Learning
• Federated Multi-Task Learning under a Mixture of Distributions
• Federated Learning for Large Models in Medical Imaging: A Comprehensive Review
• Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg
• Generalizable Federated Learning using Client Adaptive Focal Modulation
• Benchmarking Federated Learning for Throughput Prediction in 5G Live Streaming Applications
• FedMP: Tackling Medical Feature Heterogeneity in Federated Learning from a Manifold Perspective
• FeDaL: Federated Dataset Learning for Time Series Foundation Models
• Hypernetworks for Model-Heterogeneous Personalized Federated Learning
• AnalogFed: Privacy-Preserving Discovery of Analog Circuits at Scale with Federated Generative AI
• FedGA: A Fair Federated Learning Framework Based on the Gini Coefficient
• Federated Learning from Small Datasets
• FedPall: Prototype-based Adversarial and Collaborative Learning for Federated Learning with Feature Drift
• pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models
• Personalized Retrogress-Resilient Framework for Real-World Medical Federated Learning
• Whom to Trust? Adaptive Collaboration in Personalized Federated Learning
• Spatio-Temporal Representation Decoupling and Enhancement for Federated Instrument Segmentation in Surgical Videos
• Federated Breast Cancer Detection Enhanced by Synthetic Ultrasound Image Augmentation
• Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion
• Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learning
• Event-Driven Online Vertical Federated Learning
• AFBS:Buffer Gradient Selection in Semi-asynchronous Federated Learning
• Communication Efficient Adaptive Model-Driven Quantum Federated Learning
• Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity
• Towards Unified Modeling in Federated Multi-Task Learning via Subspace Decoupling
• An Empirical Study of Federated Prompt Learning for Vision Language Model
• Lazy But Effective: Collaborative Personalized Federated Learning with Heterogeneous Data
• PathFL: Multi-Alignment Federated Learning for Pathology Image Segmentation
• Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
• Unified Alignment Protocol: Making Sense of the Unlabeled Data in New Domains
• Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms
• Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments
• Kuramoto-FedAvg: Using Synchronization Dynamics to Improve Federated Learning Optimization under Statistical Heterogeneity
• Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data
• FedWorld: Scope-Aware Federation of Agent World Models
• Gradient Inversion with Generative Image Prior
• ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoning
• Ranking-Based At-Risk Student Prediction Using Federated Learning and Differential Features
• Adaptive Latent-Space Constraints in Personalized Federated Learning
• DFPL: Decentralized Federated Prototype Learning Across Heterogeneous Data Distributions
• Privacy Preserving Machine Learning Model Personalization through Federated Personalized Learning
• Federated Adapter on Foundation Models: An Out-Of-Distribution Approach
• Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization
• Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learning
• Federated Learning for Medical Image Classification: A Comprehensive Benchmark
• Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possible
• Benchmark Evaluation of Federated Learning on Multi-organ Images
• Decentralized Domain Generalization with Style Sharing: Formal Model and Convergence Analysis

Related

• Why Batch Normalization Damage Federated Learning on Non-IID Data?
• FedWon: Triumphing Multi-domain Federated Learning Without Normalization
• Experimenting with Normalization Layers in Federated Learning on non-IID scenarios
• BN-SCAFFOLD: controlling the drift of Batch Normalization statistics in Federated Learning
• Optimizing Federated Learning for Medical Image Classification on Distributed Non-iid Datasets with Partial Labels