Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approach

2025/07/08 by Xiaobing Chen, Boyang Zhang, Chen, Xiaobing +11 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Conceptual model #Enhanced Data Rates for GSM Evolution #FOS: Computer and information sciences #Federated learning #IoT and Edge/Fog Computing #Machine Learning (cs.LG) #Mobile Crowdsensing and Crowdsourcing #Privacy-Preserving Technologies in Data #Software deployment #Training (meteorology) #Training set

paper · pdf · doi:10.48550/arxiv.2507.05685

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/07/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

The integration of Federated Learning (FL) and Mixture-of-Experts (MoE) presents a compelling pathway for training more powerful, large-scale artificial intelligence models (LAMs) on decentralized data while preserving privacy. However, efficient federated training of these complex MoE-structured LAMs is hindered by significant system-level challenges, particularly in managing the interplay between heterogeneous client resources and the sophisticated coordination required for numerous specialized experts. This article highlights a critical, yet underexplored concept: the absence of robust quantitative strategies for dynamic client-expert alignment that holistically considers varying client capacities and the imperative for system-wise load balancing. Specifically, we propose a conceptual system design for intelligent client-expert alignment that incorporates dynamic fitness scoring, global expert load monitoring, and client capacity profiling. By tackling these systemic issues, we can unlock more scalable, efficient, and robust training mechanisms with fewer communication rounds for convergence, paving the way for the widespread deployment of large-scale federated MoE-structured LAMs in edge computing with ultra-high communication efficiency.

Citations

• Mixture-of-Experts for Distributed Edge Computing with Channel-Aware Gating Function
• FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
• Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
• FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts
• Beam Prediction based on Large Language Models
• WDMoE: Wireless Distributed Large Language Models with Mixture of Experts
• pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning
• A Survey on Evaluation of Large Language Models
• A Survey on Evaluation of Large Language Models
• Federated Mixture of Experts
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Cited by

• FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computing
• Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computing

Related

• FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computing
• FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
• FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts
• HeteroTune: Efficient Federated Learning for Large Heterogeneous Models
• A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models