2019/02/05 by El-Mhamdi, El-Mahdi, Rachid Guerraoui, Guerraoui, Rachid +4
Computer Science · Engineering · Materials Science · #Advanced Memory and Neural Computing #Distributed #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #Neural and Evolutionary Computing (cs.NE) #Parallel #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.1902.01686
openalex publication_date 2019/02/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The loss of a few neurons in a brain rarely results in any visible loss of\nfunction. However, the insight into what "few" means in this context is\nunclear. How many random neuron failures will it take to lead to a visible loss\nof function? In this paper, we address the fundamental question of the impact\nof the crash of a random subset of neurons on the overall computation of a\nneural network and the error in the output it produces. We study fault\ntolerance of neural networks subject to small random neuron/weight crash\nfailures in a probabilistic setting. We give provable guarantees on the\nrobustness of the network to these crashes. Our main contribution is a bound on\nthe error in the output of a network under small random Bernoulli crashes\nproved by using a Taylor expansion in the continuous limit, where close-by\nneurons at a layer are similar. The failure mode we adopt in our model is\ncharacteristic of neuromorphic hardware, a promising technology to speed up\nartificial neural networks, as well as of biological networks. We show that our\ntheoretical bounds can be used to compare the fault tolerance of different\narchitectures and to design a regularizer improving the fault tolerance of a\ngiven architecture. We design an algorithm achieving fault tolerance using a\nreasonable number of neurons. In addition to the theoretical proof, we also\nprovide experimental validation of our results and suggest a connection to the\ngeneralization capacity problem.\n