Asymptotic Behavior of Adversarial Training in Binary Classification

2020/10/26 by Hossein Taheri, Taheri, Hossein, Ramtin Pedarsani +3 · 1 citation
Chemistry · Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #FOS: Electrical engineering #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Mass Spectrometry Techniques and Applications #Signal Processing (eess.SP) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2010.13275

openalex publication_date 2020/10/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

It has been consistently reported that many machine learning models are susceptible to adversarial attacks i.e., small additive adversarial perturbations applied to data points can cause misclassification. Adversarial training using empirical risk minimization is considered to be the state-of-the-art method for defense against adversarial attacks. Despite being successful in practice, several problems in understanding generalization performance of adversarial training remain open. In this paper, we derive precise theoretical predictions for the performance of adversarial training in binary classification. We consider the high-dimensional regime where the dimension of data grows with the size of the training data-set at a constant ratio. Our results provide exact asymptotics for standard and adversarial test errors of the estimators obtained by adversarial training with ℓ_q-norm bounded perturbations (q ≥ 1) for both discriminative binary models and generative Gaussian-mixture models with correlated features. Furthermore, we use these sharp predictions to uncover several intriguing observations on the role of various parameters including the over-parameterization ratio, the data model, and the attack budget on the adversarial and standard errors.

Citations

• Various thresholds for ℓ₁-optimization in compressed sensing
• Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators : rigorous results
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Robustness May Be at Odds with Accuracy
• Theoretically Principled Trade-off between Robustness and Accuracy
• Convergence and Margin of Adversarial Training on Separable Data
• Unlabeled Data Improves Adversarial Robustness
• Universality in Learning from Linear Measurements
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization
• Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification

Cited by

• Kernel Learning with Adversarial Features: Numerical Efficiency and Adaptive Regularization

Related

• Precise Statistical Analysis of Classification Accuracies for Adversarial Training
• Generalization Properties of Adversarial Training for ℓ₀-Bounded Adversarial Attacks
• Fundamental Tradeoffs in Distributionally Adversarial Training
• On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds
• Binary Classification Under ℓ₀ Attacks for General Noise Distribution