Personalized Federated Fine-Tuning of Vision Foundation Models for Healthcare

2025/10/14 by Adam Tupper, Christian Gagné, Tupper, Adam +1
Computer Science · Neuroscience · #Blockchain Technology Applications and Security #Brain Tumor Detection and Classification #Computer Vision and Pattern Recognition (cs.CV) #Digital Media and Visual Art #Distributed #FOS: Computer and information sciences #Parallel #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2510.12741

openalex publication_date 2025/10/14 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

Foundation models open up new possibilities for the use of AI in healthcare. However, even when pre-trained on health data, they still need to be fine-tuned for specific downstream tasks. Furthermore, although foundation models reduce the amount of training data required to achieve good performance, obtaining sufficient data is still a challenge. This is due, in part, to restrictions on sharing and aggregating data from different sources to protect patients' privacy. One possible solution to this is to fine-tune foundation models via federated learning across multiple participating clients (i.e., hospitals, clinics, etc.). In this work, we propose a new personalized federated fine-tuning method that learns orthogonal LoRA adapters to disentangle general and client-specific knowledge, enabling each client to fully exploit both their own data and the data of others. Our preliminary results on real-world federated medical imaging tasks demonstrate that our approach is competitive against current federated fine-tuning methods.

Citations

• FedVLM: Scalable Personalized Vision-Language Models through Federated Learning
• Selective Aggregation for Low-Rank Adaptation in Federated Learning
• FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning
• Dual-Personalizing Adapter for Federated Foundation Models
• Improving LoRA in Privacy-preserving Federated Learning
• Orthogonal Subspace Learning for Language Model Continual Learning
• FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings
• TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Flower: A Friendly Federated Learning Research Framework
• BCN20000: Dermoscopic Lesions in the Wild
• From Detection of Individual Metastases to Classification of Lymph Node Status at the Patient Level: The CAMELYON17 Challenge
• Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017\n International Symposium on Biomedical Imaging (ISBI), Hosted by the\n International Skin Imaging Collaboration (ISIC)

Related

• FedSCA: Federated Tuning with Similarity-guided Collaborative Aggregation for Heterogeneous Medical Image Segmentation
• FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning
• Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcare
• Task-Agnostic Federated Learning
• Improving Learning of New Diseases through Knowledge-Enhanced Initialization for Federated Adapter Tuning