Byzantine-Tolerant Machine Learning

2017/03/08 by Peva Blanchard, Blanchard, Peva, El Mahdi El Mhamdi +5 · 5 citations
Computer Science · #Cryptography and Data Security #Distributed #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Optimization and Control (math.OC) #Parallel #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.1703.02757

openalex publication_date 2017/03/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The growth of data, the need for scalability and the complexity of models used in modern machine learning calls for distributed implementations. Yet, as of today, distributed machine learning frameworks have largely ignored the possibility of arbitrary (i.e., Byzantine) failures. In this paper, we study the robustness to Byzantine failures at the fundamental level of stochastic gradient descent (SGD), the heart of most machine learning algorithms. Assuming a set of n workers, up to f of them being Byzantine, we ask how robust can SGD be, without limiting the dimension, nor the size of the parameter space. We first show that no gradient descent update rule based on a linear combination of the vectors proposed by the workers (i.e, current approaches) tolerates a single Byzantine failure. We then formulate a resilience property of the update rule capturing the basic requirements to guarantee convergence despite f Byzantine workers. We finally propose Krum, an update rule that satisfies the resilience property aforementioned. For a d-dimensional learning problem, the time complexity of Krum is O(n² ⋅ (d + log n)).

Cited by

• Robust Bayesian Optimisation with Unbounded Corruptions
• Data Poisoning Vulnerabilities Across Healthcare AI Architectures: A Security Threat Analysis
• DSFL: A Dual-Server Byzantine-Resilient Federated Learning Framework via Group-Based Secure Aggregation
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• On the Security and Privacy of Federated Learning: A Survey with Attacks, Defenses, Frameworks, Applications, and Future Directions

Related

• Efficient Byzantine-Resilient Stochastic Gradient Desce
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Asynchronous Byzantine Machine Learning (the case of SGD)
• A Robust Classification Framework for Byzantine-Resilient Stochastic Gradient Descent
• Genuinely Distributed Byzantine Machine Learning