2017/04/11 by Yi Zhou, Yaoliang Yu, Zhou, Yi +7 · 2 citations
Computer Science · Engineering · #Complexity and Algorithms in Graphs #FOS: Mathematics #Optimization and Control (math.OC) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.1704.03540
openalex publication_date 2017/04/11 · openalex created_date 2019/06/27 · openalex updated_date 2026/07/28
With ever growing data volume and model size, an error-tolerant,\ncommunication efficient, yet versatile distributed algorithm has become vital\nfor the success of many large-scale machine learning applications. In this work\nwe propose m-PAPG, an implementation of the flexible proximal gradient\nalgorithm in model parallel systems equipped with the partially asynchronous\ncommunication protocol. The worker machines communicate asynchronously with a\ncontrolled staleness bound s and operate at different frequencies. We\ncharacterize various convergence properties of m-PAPG: 1) Under a general\nnon-smooth and non-convex setting, we prove that every limit point of the\nsequence generated by m-PAPG is a critical point of the objective function; 2)\nUnder an error bound condition, we prove that the function value decays\nlinearly for every s steps; 3) Under the Kurdyka- Lojasiewicz inequality,\nwe prove that the sequences generated by m-PAPG converge to the same critical\npoint, provided that a proximal Lipschitz condition is satisfied.\n