FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models

2025/08/05 by Matteo Caligiuri, Francesco Barbato, Caligiuri, Matteo +7
Computer Science · #Big Data and Digital Economy #Computer Vision and Pattern Recognition (cs.CV) #Cryptography and Data Security #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data

paper · pdf · doi:10.48550/arxiv.2508.03356

openalex publication_date 2025/08/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Federated Learning (FL) is an established paradigm for training deep learning models on decentralized data. However, as the size of the models grows, conventional FL approaches often require significant computational resources on client devices, which may not be feasible. We introduce FedPromo, a novel framework that enables efficient adaptation of large-scale foundation models stored on a central server to new domains encountered only by remote clients. Instead of directly training the large model on client devices, FedPromo optimizes lightweight proxy models via FL, significantly reducing computational overhead while maintaining privacy. Our method follows a two-stage process: first, server-side knowledge distillation aligns the representations of a large-scale foundation model (e.g., a transformer) with those of a compact counterpart (e.g., a CNN). Then, the compact model encoder is deployed to client devices, where trainable classifiers are learned locally. These classifiers are subsequently aggregated and seamlessly transferred back to the foundation model, facilitating personalized adaptation without requiring direct access to user data. Through novel regularization strategies, our framework enables decentralized multi-domain learning, balancing performance, privacy, and resource efficiency. Extensive experiments on five image classification benchmarks demonstrate that FedPromo outperforms existing methods while assuming limited-resource clients.

Citations

• Cross-Architecture Auxiliary Feature Space Translation for Efficient Few-Shot Personalized Object Detection
• FedFMS: Exploring Federated Foundation Models for Medical Image Segmentation
• FedGKD: Toward Heterogeneous Federated Learning via Global Knowledge Distillation
• One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation
• Vision Transformers Need Registers
• Federated Learning for Computer Vision
• FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning
• Continual Road-Scene Semantic Segmentation via Feature-Aligned Symmetric Multi-Modal Network
• Visual Instruction Tuning
• DINOv2: Learning Robust Visual Features without Supervision
• Micrograph segmentations for DDEVD
• Segment Anything
• Modular Deep Learning
• A Comprehensive Survey of Continual Learning: Theory, Method and Application
• Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
• Learning Across Domains and Devices: Style-Driven Source-Free Domain Adaptation in Clustered Federated Learning
• On the Importance and Applicability of Pre-Training for Federated Learning
• Continual Learning with Foundation Models: An Empirical Study of Latent Replay
• Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation
• Efficient and Private Federated Learning with Partially Trainable Networks
• FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models
• Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data
• Emerging Properties in Self-Supervised Vision Transformers
• Emerging Properties in Self-Supervised Vision Transformers
• Model-Contrastive Federated Learning
• Learning Transferable Visual Models From Natural Language Supervision
• FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning
• Transformers in Vision: A Survey
• Efficient Transformers: A Survey
• Flower: A Friendly Federated Learning Research Framework
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• Federated Learning with Differential Privacy: Algorithms and Performance Analysis
• Federated Learning With Differential Privacy: Algorithms and Performance Analysis
• FedMD: Heterogenous Federated Learning via Model Distillation
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Searching for MobileNetV3
• Federated Optimization in Heterogeneous Networks
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• A Large-Scale Car Dataset for Fine-Grained Categorization and Verification
• Distilling the Knowledge in a Neural Network
• Fine-Grained Visual Classification of Aircraft
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Bridging the Gap Between Foundation Models and Heterogeneous Federated Learning
• Towards Federated Learning Under Resource Constraints via Layer-wise Training and Depth Dropout
• Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning
• FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
• FedPFT: Federated Proxy Fine-Tuning of Foundation Models