Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learning

2025/06/23 by Bhope, Rahul Atul, Jayaram, K. R., Venkateswaran, Praveen +1
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2506.18789

Abstract

Federated Learning (FL) enables collaborative model training across decentralized clients without sharing raw data, yet faces significant challenges in real-world settings where client data distributions evolve dynamically over time. This paper tackles the critical problem of covariate and label shifts in streaming FL environments, where non-stationary data distributions degrade model performance and necessitate a middleware layer that adapts FL to distributional shifts. We introduce ShiftEx, a shift-aware mixture of experts framework that dynamically creates and trains specialized global models in response to detected distribution shifts using Maximum Mean Discrepancy for covariate shifts. The framework employs a latent memory mechanism for expert reuse and implements facility location-based optimization to jointly minimize covariate mismatch, expert creation costs, and label imbalance. Through theoretical analysis and comprehensive experiments on benchmark datasets, we demonstrate 5.5-12.9 percentage point accuracy improvements and 22-95 % faster adaptation compared to state-of-the-art FL baselines across diverse shift scenarios. The proposed approach offers a scalable, privacy-preserving middleware solution for FL systems operating in non-stationary, real-world conditions while minimizing communication and computational overhead.

Citations

• DIM-SUM: Dynamic IMputation for Smart Utility Management
• Federated Continual Learning: Concepts, Challenges, and Solutions
• Federated Continual Learning for Edge-AI: A Comprehensive Survey
• Federated Learning Clients Clustering with Adaptation to Data Drifts
• FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts
• Adaptive Test-Time Personalization for Federated Learning
• FLIPS: Federated Learning using Intelligent Participant Selection
• Federated Learning under Covariate Shifts with Generalization Guarantees
• Personalized Federated Learning under Mixture of Distributions
• On the Local Cache Update Rules in Streaming Federated Learning
• Federated Covariate Shift Adaptation for Missing Target Output Values
• Federated Learning for Data Streams
• Federated Learning under Distributed Concept Drift
• Non-IID data and Continual Learning processes in Federated Learning: A long road ahead
• Federated Mixture of Experts
• Robust Federated Learning by Mixture of Experts
• See through Gradients: Image Batch Recovery via GradInversion
• Membership Inference Attacks on Machine Learning: A Survey
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
• PFL-MoE: Personalized Federated Learning Based on Mixture of Experts
• Specialized federated learning using a mixture of experts
• Flower: A Friendly Federated Learning Research Framework
• An Efficient Framework for Clustered Federated Learning
• Inverting Gradients -- How easy is it to break privacy in federated learning?
• Federated Continual Learning with Weighted Inter-client Transfer
• Evaluation Framework For Large-scale Federated Learning
• iDLG: Improved Deep Leakage from Gradients
• Advances and Open Problems in Federated Learning
• Deep Leakage from Gradients
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Federated Optimization in Heterogeneous Networks
• LEAF: A Benchmark for Federated Settings
• How To Backdoor Federated Learning
• Exploiting Unintended Feature Leakage in Collaborative Learning
• Functional Map of the World
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Densely Connected Convolutional Networks
• Deep Residual Learning for Image Recognition
• Adaptive Mixtures of Local Experts
• Overcoming label shift with target-aware federated learning
• FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering
• Gradient-based learning applied to document recognition

Related

• Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning
• Federated Learning under Covariate Shifts with Generalization Guarantees
• Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning
• FLUX: Efficient Descriptor-Driven Clustered Federated Learning under Arbitrary Distribution Shifts
• Addressing Heterogeneity in Federated Learning via Distributional Transformation