Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models

2024/10/14 by Jun Luo, Chen Chen, Luo, Jun +3 · 12 citations
Computer Science · #Advanced Graph Neural Networks #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2410.10114

openalex publication_date 2024/10/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Federated prompt learning benefits federated learning with CLIP-like Vision-Language Model's (VLM's) robust representation learning ability through prompt learning. However, current federated prompt learning methods are habitually restricted to the traditional FL paradigm, where the participating clients are generally only allowed to download a single globally aggregated model from the server. While justifiable for training full-sized models under federated settings, in this work, we argue that this paradigm is ill-suited for lightweight prompts. By facilitating the clients to download multiple pre-aggregated prompts as fixed non-local experts, we propose Personalized Federated Mixture of Adaptive Prompts (pFedMoAP), a novel FL framework that personalizes the prompt learning process through the lens of Mixture of Experts (MoE). pFedMoAP implements a local attention-based gating network that learns to generate enhanced text features for better alignment with local image data, benefiting from both local and downloaded non-local adaptive prompt experts. Extensive experiments on 9 datasets under various federated settings demonstrate the efficacy of the proposed pFedMoAP algorithm. The code is available at https://github.com/ljaiverson/pFedMoAP.

Cited by

• FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computing
• FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
• Cooperative Pseudo Labeling for Unsupervised Federated Classification
• TAP: Two-Stage Adaptive Personalization of Multi-task and Multi-Modal Foundation Models in Federated Learning
• Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation
• Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approach
• pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models
• Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion
• A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendation
• FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting
• A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning
• Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization

Related

• FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
• Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion
• Global and Local Prompts Cooperation via Optimal Transport for Federated Learning
• pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models
• FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models