Zeno: Distributed Stochastic Gradient Descent with Suspicion-based\n Fault-tolerance

2018/05/25 by Cong Xie, Oluwasanmi Koyejo, Xie, Cong +3 · 7 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Cryptography and Security (cs.CR) #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Parallel #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.1805.10032

openalex publication_date 2018/05/25 · openalex created_date 2019/06/27 · openalex updated_date 2026/07/28

Abstract

We present Zeno, a technique to make distributed machine learning,\nparticularly Stochastic Gradient Descent (SGD), tolerant to an arbitrary number\nof faulty workers. Zeno generalizes previous results that assumed a majority of\nnon-faulty nodes; we need assume only one non-faulty worker. Our key idea is to\nsuspect workers that are potentially defective. Since this is likely to lead to\nfalse positives, we use a ranking-based preference mechanism. We prove the\nconvergence of SGD for non-convex problems under these scenarios. Experimental\nresults show that Zeno outperforms existing approaches.\n

Cited by

• FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping
• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
• Advances and Open Problems in Federated Learning
• FOCUS: Dealing with Label Quality Disparity in Federated Learning
• Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients
• A Field Guide to Federated Optimization

Related

• Zeno++: Robust Fully Asynchronous SGD
• Byzantine Fault-Tolerant Distributed Machine Learning Using Stochastic Gradient Descent (SGD) and Norm-Based Comparative Gradient Elimination (CGE)
• Adaptive Stochastic Gradient Descent for Fast and Communication-Efficient Distributed Learning
• Fast and Straggler-Tolerant Distributed SGD with Reduced Computation Load
• Distributed Byzantine Tolerant Stochastic Gradient Descent in the Era of Big Data