The Search for Sparse, Robust Neural Networks

2019/12/05 by Justin Cosentino, Cosentino, Justin, Federico Zaiter +5 · 3 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1912.02386

openalex publication_date 2019/12/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent work on deep neural network pruning has shown there exist sparse subnetworks that achieve equal or improved accuracy, training time, and loss using fewer network parameters when compared to their dense counterparts. Orthogonal to pruning literature, deep neural networks are known to be susceptible to adversarial examples, which may pose risks in security- or safety-critical applications. Intuition suggests that there is an inherent trade-off between sparsity and robustness such that these characteristics could not co-exist. We perform an extensive empirical evaluation and analysis testing the Lottery Ticket Hypothesis with adversarial training and show this approach enables us to find sparse, robust neural networks. Code for reproducing experiments is available here: https://github.com/justincosentino/robust-sparse-networks.

Citations

• Gradient-based learning applied to document recognition
• Pruning Filters for Efficient ConvNets
• Towards Deep Learning Models Resistant to Adversarial Attacks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Robustness May Be at Odds with Accuracy

Cited by

• Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learning
• Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning
• Spatio-Temporal Sparsification for General Robust Graph Convolution Networks

Related

• Winning the Lottery with Continuous Sparsification
• Deconstructing the Structure of Sparse Neural Networks
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Finding Dynamics Preserving Adversarial Winning Tickets