An Integrated Approach to Produce Robust Models with High Efficiency

2020/08/31 by Zhijian Li, Bao Wang, Li, Zhijian +3
Computer Science · Engineering · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Mathematics #Integrated Circuits and Semiconductor Failure Analysis #Numerical Analysis (math.NA)

paper · pdf · doi:10.48550/arxiv.2008.13305

openalex publication_date 2020/08/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep Neural Networks (DNNs) needs to be both efficient and robust for practical uses. Quantization and structure simplification are promising ways to adapt DNNs to mobile devices, and adversarial training is the most popular method to make DNNs robust. In this work, we try to obtain both features by applying a convergent relaxation quantization algorithm, Binary-Relax (BR), to a robust adversarial-trained model, ResNets Ensemble via Feynman-Kac Formalism (EnResNet). We also discover that high precision, such as ternary (tnn) and 4-bit, quantization will produce sparse DNNs. However, this sparsity is unstructured under advarsarial training. To solve the problems that adversarial training jeopardizes DNNs' accuracy on clean images and the struture of sparsity, we design a trade-off loss function that helps DNNs preserve their natural accuracy and improve the channel sparsity. With our trade-off loss function, we achieve both goals with no reduction of resistance under weak attacks and very minor reduction of resistance under strong attcks. Together with quantized EnResNet with trade-off loss function, we provide robust models that have high efficiency.

Citations

• Towards Deep Learning Models Resistant to Adversarial Attacks
• Countering Adversarial Images using Input Transformations
• Certifying Some Distributional Robustness with Principled Adversarial\n Training
• Deep Neural Nets with Interpolating Function as Output Activation
• Robustness May Be at Odds with Accuracy
• On the Convergence and Robustness of Adversarial Training
• Learning Multiple Layers of Features from Tiny Images

Related

• Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets
• A Layer-wise Adversarial-aware Quantization Optimization for Improving Robustness
• Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness
• Fast Training of Deep Neural Networks Robust to Adversarial Perturbations
• ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies