2021/12/17 by Feijie Wu, Song Guo, Wu, Feijie +11 · 1 citation
Computer Science · Engineering · #Advanced Memory and Neural Computing #Distributed #Distributed Control Multi-Agent Systems #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2112.09355
openalex publication_date 2021/12/17 · openalex created_date 2022/05/05 · openalex updated_date 2026/07/28
In the setting of federated optimization, where a global model is aggregated periodically, step asynchronism occurs when participants conduct model training by efficiently utilizing their computational resources. It is well acknowledged that step asynchronism leads to objective inconsistency under non-i.i.d. data, which degrades the model's accuracy. To address this issue, we propose a new algorithm FedaGrac, which calibrates the local direction to a predictive global orientation. Taking advantage of the estimated orientation, we guarantee that the aggregated model does not excessively deviate from the global optimum while fully utilizing the local updates of faster nodes. We theoretically prove that FedaGrac holds an improved order of convergence rate than the state-of-the-art approaches and eliminates the negative effect of step asynchronism. Empirical results show that our algorithm accelerates the training and enhances the final accuracy.