On the Importance and Applicability of Pre-Training for Federated Learning

2022/06/23 by Hong-You Chen, Chen, Hong-You, Cheng-Hao Tu +7 · 8 citations
Computer Science · Engineering · Social Sciences · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Mobility and Location-Based Analysis #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Traffic Prediction and Management Techniques

paper · pdf · doi:10.48550/arxiv.2206.11488

openalex publication_date 2022/06/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Pre-training is prevalent in nowadays deep learning to improve the learned model's performance. However, in the literature on federated learning (FL), neural networks are mostly initialized with random weights. These attract our interest in conducting a systematic study to explore pre-training for FL. Across multiple visual recognition benchmarks, we found that pre-training can not only improve FL, but also close its accuracy gap to the counterpart centralized learning, especially in the challenging cases of non-IID clients' data. To make our findings applicable to situations where pre-trained models are not directly available, we explore pre-training with synthetic data or even with clients' data in a decentralized manner, and found that they can already improve FL notably. Interestingly, many of the techniques we explore are complementary to each other to further boost the performance, and we view this as a critical result toward scaling up deep FL for real-world applications. We conclude our paper with an attempt to understand the effect of pre-training on FL. We found that pre-training enables the learned global models under different clients' data conditions to converge to the same loss basin, and makes global aggregation in FL more stable. Nevertheless, pre-training seems to not alleviate local model drifting, a fundamental problem in FL under non-IID data.

Cited by

• animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics
• MAUI: Reconstructing Private Client Data in Federated Transfer Learning
• Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation
• FedSODA: Federated Fine-tuning of LLMs via Similarity Group Pruning and Orchestrated Distillation Alignment
• Generalizable Federated Learning using Client Adaptive Focal Modulation
• FedHiP: Heterogeneity-Invariant Personalized Federated Learning Through Closed-Form Solutions
• FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models
• InvestAlign: Overcoming Data Scarcity in Aligning Large Language Models with Investor Decision-Making Processes under Herd Behavior

Related

• Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning
• Rethinking the Starting Point: Collaborative Pre-Training for Federated Downstream Tasks
• Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning
• Guiding The Last Layer in Federated Learning with Pre-Trained Models
• FedDUAL: A Dual-Strategy with Adaptive Loss and Dynamic Aggregation for Mitigating Data Heterogeneity in Federated Learning