Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks

2021/10/07 by Hanxun Huang, Huang, Hanxun, Yisen Wang +9 · 4 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Integrated Circuits and Semiconductor Failure Analysis #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2110.03825

openalex publication_date 2021/10/07 · openalex created_date 2021/10/25 · openalex updated_date 2026/07/28

Abstract

Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks. A range of defense methods have been proposed to train adversarially robust DNNs, among which adversarial training has demonstrated promising results. However, despite preliminary understandings developed for adversarial training, it is still not clear, from the architectural perspective, what configurations can lead to more robust DNNs. In this paper, we address this gap via a comprehensive investigation on the impact of network width and depth on the robustness of adversarially trained DNNs. Specifically, we make the following key observations: 1) more parameters (higher model capacity) does not necessarily help adversarial robustness; 2) reducing capacity at the last stage (the last group of blocks) of the network can actually improve adversarial robustness; and 3) under the same parameter budget, there exists an optimal architectural configuration for adversarial robustness. We also provide a theoretical analysis explaning why such network configuration can help robustness. These architectural insights can help design adversarially robust DNNs. Code is available at \urlhttps://github.com/HanxunH/RobustWRN.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• DARTS: Differentiable Architecture Search
• MMA Training: Direct Input Space Margin Maximization through Adversarial\n Training
• Fast is better than free: Revisiting adversarial training
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• DSRNA: Differentiable Search of Robust Neural Architectures
• Learning Multiple Layers of Features from Tiny Images
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better

Cited by

• Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation
• Exploring the Effect of DNN Depth on Adversarial Attacks in Network Intrusion Detection Systems
• FERD: Fairness-Enhanced Data-Free Robustness Distillation
• A Validation Strategy for Deep Learning Models: Evaluating and Enhancing Robustness

Related

• Towards Deep Learning Models Resistant to Adversarial Attacks
• Recent Advances in Understanding Adversarial Robustness of Deep Neural Networks
• RobArch: Designing Robust Architectures against Adversarial Attacks
• Adversarially Robust Neural Architectures
• Exploring the role of Input and Output Layers of a Deep Neural Network in Adversarial Defense