Balancing Robustness and Sensitivity using Feature Contrastive Learning

2021/05/19 by Seung‐Yeon Kim, Kim, Seungyeon, Daniel Gläsner +9
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2105.09394

openalex publication_date 2021/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

It is generally believed that robust training of extremely large networks is critical to their success in real-world applications. However, when taken to the extreme, methods that promote robustness can hurt the model's sensitivity to rare or underrepresented patterns. In this paper, we discuss this trade-off between sensitivity and robustness to natural (non-adversarial) perturbations by introducing two notions: contextual feature utility and contextual feature sensitivity. We propose Feature Contrastive Learning (FCL) that encourages a model to be more sensitive to the features that have higher contextual utility. Empirical results demonstrate that models trained with FCL achieve a better balance of robustness and sensitivity, leading to improved generalization in the presence of noise on both vision and NLP datasets.

Citations

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Backpropagation Applied to Handwritten Zip Code Recognition
• ImageNet classification with deep convolutional neural networks
• Gradients of Counterfactuals
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• SmoothGrad: removing noise by adding noise
• Adversarially Robust Generalization Requires More Data
• Robustness May Be at Odds with Accuracy
• Theoretically Principled Trade-off between Robustness and Accuracy
• On the (In)fidelity and Sensitivity for Explanations
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian\n Augmentation
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Towards Stable and Efficient Training of Verifiably Robust Neural Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations
• Momentum Contrast for Unsupervised Visual Representation Learning
• Contrastive Multiview Coding

Related

• Investigating Why Contrastive Learning Benefits Robustness Against Label Noise
• When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?
• Evaluating Robustness to Context-Sensitive Feature Perturbations of Different Granularities
• Feature Prioritization and Regularization Improve Standard Accuracy and Adversarial Robustness
• Generalization Bounds for Robust Contrastive Learning: From Theory to Practice