Subspace Adversarial Training

2021/11/24 by Tao Li, Yingwen Wu, Li, Tao +7 · 2 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #FOS: Computer and information sciences #Integrated Circuits and Semiconductor Failure Analysis #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2111.12229

openalex publication_date 2021/11/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Single-step adversarial training (AT) has received wide attention as it proved to be both efficient and robust. However, a serious problem of catastrophic overfitting exists, i.e., the robust accuracy against projected gradient descent (PGD) attack suddenly drops to 0% during the training. In this paper, we approach this problem from a novel perspective of optimization and firstly reveal the close link between the fast-growing gradient of each sample and overfitting, which can also be applied to understand robust overfitting in multi-step AT. To control the growth of the gradient, we propose a new AT method, Subspace Adversarial Training (Sub-AT), which constrains AT in a carefully extracted subspace. It successfully resolves both kinds of overfitting and significantly boosts the robustness. In subspace, we also allow single-step AT with larger steps and larger radius, further improving the robustness performance. As a result, we achieve state-of-the-art single-step AT performance. Without any regularization term, our single-step AT can reach over 51% robust accuracy against strong PGD-50 attack of radius 8/255 on CIFAR-10, reaching a competitive performance against standard multi-step PGD-10 AT with huge computational advantages. The code is released at https://github.com/nblt/Sub-AT.

Citations

• Gradient Descent Happens in a Tiny Subspace
• Theoretically Principled Trade-off between Robustness and Accuracy
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Fast is better than free: Revisiting adversarial training
• Learning Multiple Layers of Features from Tiny Images
• Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces
• Stochastic Training is Not Necessary for Generalization

Cited by

• Understanding Adversarial Training with Energy-based Models
• Accelerating Learned Image Compression Through Modeling Neural Training Dynamics

Related

• Stable and Efficient Adversarial Training through Local Linearization
• Fast Adversarial Training with Adaptive Step Size
• Reliably fast adversarial training via latent adversarial perturbation
• Understanding Catastrophic Overfitting in Single-step Adversarial Training
• Make Some Noise: Reliable and Efficient Single-Step Adversarial Training