FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge

2025/08/26 by Hu, Gang, Teng, Yinglei, Wu, Pengfei +1 · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.18663

Abstract

As FMs drive progress toward Artificial General Intelligence (AGI), fine-tuning them under privacy and resource constraints has become increasingly critical particularly when highquality training data resides on distributed edge devices. Federated Learning (FL) offers a compelling solution through Federated Fine-Tuning (FFT), which enables collaborative model adaptation without sharing raw data. Recent approaches incorporate Parameter-Efficient Fine-Tuning (PEFT) techniques such as Low Rank Adaptation (LoRA) to reduce computational overhead. However, LoRA-based FFT faces two major limitations in heterogeneous FL environments: structural incompatibility across clients with varying LoRA configurations and limited adaptability to non-IID data distributions, which hinders convergence and generalization. To address these challenges, we propose FFT MoE, a novel FFT framework that replaces LoRA with sparse Mixture of Experts (MoE) adapters. Each client trains a lightweight gating network to selectively activate a personalized subset of experts, enabling fine-grained adaptation to local resource budgets while preserving aggregation compatibility. To further combat the expert load imbalance caused by device and data heterogeneity, we introduce a heterogeneity-aware auxiliary loss that dynamically regularizes the routing distribution to ensure expert diversity and balanced utilization. Extensive experiments spanning both IID and non-IID conditions demonstrate that FFT MoE consistently outperforms state of the art FFT baselines in generalization performance and training efficiency.

Citations

• FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
• FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts
• FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition
• FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning
• LocMoE: A Low-Overhead MoE for Large Language Model Training
• PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Model
• Mixture-of-Experts with Expert Choice Routing
• Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity
• Advances and Open Problems in Federated Learning
• Advances and Open Problems in Federated Learning
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Character-level Convolutional Networks for Text Classification

Cited by

• OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Data

Related

• Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation
• FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models
• FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation
• FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts
• Bridging the Gap Between Foundation Models and Heterogeneous Federated Learning