SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learning

2024/05/15 by Riyasat Ohib, Bishal Thapaliya, Ohib, Riyasat +9
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2405.09037

openalex publication_date 2024/05/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this work, we propose Salient Sparse Federated Learning (SSFL), a streamlined approach for sparse federated learning with efficient communication. SSFL identifies a sparse subnetwork prior to training, leveraging parameter saliency scores computed separately on local client data in non-IID scenarios, and then aggregated, to determine a global mask. Only the sparse model weights are trained and communicated each round between the clients and the server. On standard benchmarks including CIFAR-10, CIFAR-100, and Tiny-ImageNet, SSFL consistently improves the accuracy sparsity trade off, achieving more than 20% relative error reduction on CIFAR-10 compared to the strongest sparse baseline, while reducing communication costs by 2 × relative to dense FL. Finally, in a real-world federated learning deployment, SSFL delivers over 2.3 × faster communication time, underscoring its practical efficiency.

Related

• SalientGrads: Sparse Models for Communication Efficient and Data Aware Distributed Federated Training
• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead
• SparsyFed: Sparse Adaptive Federated Training
• Time-Correlated Sparsification for Communication-Efficient Federated\n Learning
• FedCliP: Federated Learning with Client Pruning