2023/07/12 by Lekang Jiang, Jiang, Lekang, Filip Svoboda +3 · 1 citation
Computer Science · Health Professions · #Artificial Intelligence (cs.AI) #Distributed #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Interpreting and Communication in Healthcare #Machine Learning (cs.LG) #Parallel #Privacy-Preserving Technologies in Data #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2307.06933
openalex publication_date 2023/07/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Foundation models (FMs) have shown prominent success in a wide range of tasks. Their applicability to specific domain-task pairings relies on the availability of, both, high-quality data and significant computational resources. These challenges are not new to the field and, indeed, Federated Learning (FL) has been shown to be a promising solution in similar setups. This paper tackles the specific case of Domain-Adaptive Pre-Training (DAPT), a key step in the application of FMs. We conduct the first comprehensive empirical study to evaluate the performance of Federated Domain-Adaptive Pre-Training (FDAPT). We demonstrate that FDAPT can maintain competitive downstream task performance to the centralized baseline in both IID and non-IID situations. Finally, we propose a novel algorithm, Frozen Federated Domain-Adaptive Pre-Training (FFDAPT). FFDAPT improves the computational efficiency by 12.1% on average and exhibits similar downstream task performance to vanilla FDAPT, with general performance fluctuations remaining less than 1%.