The Price of Implicit Bias in Adversarially Robust Generalization

2024/06/07 by Nikolaos Tsilivis, Tsilivis, Nikolaos, Natalie C. Frank +5 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2406.04981

openalex publication_date 2024/06/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study the implicit bias of optimization in robust empirical risk minimization (robust ERM) and its connection with robust generalization. In classification settings under adversarial perturbations with linear models, we study what type of regularization should ideally be applied for a given perturbation set to improve (robust) generalization. We then show that the implicit bias of optimization in robust ERM can significantly affect the robustness of the model and identify two ways this can happen; either through the optimization algorithm or the architecture. We verify our predictions in simulations with synthetic data and experimentally study the importance of implicit bias in robust ERM with deep neural networks.

Cited by

• On the existence of consistent adversarial attacks in high-dimensional linear classification

Related

• Bridging the Gap Between Adversarial Robustness and Optimization Bias
• Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization
• Regularization for Adversarial Robust Learning
• Robustness May Be at Odds with Accuracy
• Interpreting Robust Optimization via Adversarial Influence Functions