2024/05/16 by Charikleia Iakovidou, Iakovidou, Charikleia, Kibaek Kim +1 · 2 citations
Computer Science · #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Metaheuristic Optimization Algorithms Research #Parallel #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2405.10123
openalex publication_date 2024/05/16 · openalex created_date 2024/05/18 · openalex updated_date 2026/07/30
Federated learning (FL) was recently proposed to securely train models with data held over multiple locations (``clients'') under the coordination of a central server. Prolonged training times caused by slow clients may hinder the performance of FL; while asynchronous communication is a promising solution, highly heterogeneous client response times under non-IID local data may introduce significant bias to the global model, particularly in client-driven setups where sampling is infeasible. To address this issue, we propose \underlineAsynch\underlineRonous \underlineExact \underlineAveraging (AREA), a stochastic (sub)gradient method that leverages asynchrony for scalability and uses client-side memory to correct the bias induced by uneven participation, without client sampling or prior knowledge of client latencies. AREA communicates model residuals rather than gradient estimates, reducing exposure to gradient inversion, and is compatible with secure aggregation. Under standard assumptions and unbounded, heterogeneous delays with finite mean, AREA achieves optimal convergence rates: O(1/K) in the strongly convex, smooth regime and O(1/√(K)) in the convex, nonsmooth regime. For strongly convex, smooth objectives, we demonstrate theoretically and empirically that AREA accommodates larger step sizes than existing methods, enabling fast convergence without adversely impacting model generalization. In the convex, nonsmooth setting, to our knowledge we are the first to obtain rates that scale with the average client update frequency rather than the minimum or maximum, indicating increased robustness to outliers.