Enhancing Federated Learning Convergence with Dynamic Data Queue and Data Entropy-driven Participant Selection

2024/10/23 by Charuka Herath, Xiaolan Liu, Herath, Charuka +5
Computer Science · #14J60 (Primary) #Artificial Intelligence (cs.AI) #Cryptography and Data Security #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #I.2.11 #I.5.1 #I.5.4 #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2410.17792

openalex publication_date 2024/10/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Federated Learning (FL) is a decentralized approach for collaborative model training on edge devices. This distributed method of model training offers advantages in privacy, security, regulatory compliance, and cost-efficiency. Our emphasis in this research lies in addressing statistical complexity in FL, especially when the data stored locally across devices is not identically and independently distributed (non-IID). We have observed an accuracy reduction of up to approximately 10% to 30%, particularly in skewed scenarios where each edge device trains with only 1 class of data. This reduction is attributed to weight divergence, quantified using the Euclidean distance between device-level class distributions and the population distribution, resulting in a bias term (\(δ_k\)). As a solution, we present a method to improve convergence in FL by creating a global subset of data on the server and dynamically distributing it across devices using a Dynamic Data queue-driven Federated Learning (DDFL). Next, we leverage Data Entropy metrics to observe the process during each training round and enable reasonable device selection for aggregation. Furthermore, we provide a convergence analysis of our proposed DDFL to justify their viability in practical FL scenarios, aiming for better device selection, a non-sub-optimal global model, and faster convergence. We observe that our approach results in a substantial accuracy boost of approximately 5% for the MNIST dataset, around 18% for CIFAR-10, and 20% for CIFAR-100 with a 10% global subset of data, outperforming the state-of-the-art (SOTA) aggregation algorithms.

Related

• Fast-Convergent Federated Learning with Adaptive Weighting
• DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency for Federated Learning with Static and Streaming Dataset
• Federated Learning under Heterogeneous and Correlated Client Availability
• Federated Learning with Workload Reduction through Partial Training of Client Models and Entropy-Based Data Selection
• A review on different techniques used to combat the non-IID and heterogeneous nature of data in FL