Generalization Bounds for Adversarial Contrastive Learning

2023/02/21 by Xin Zou, Weiwei Liu, Zou, Xin +1 · 1 citation
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Geophysical Methods and Applications #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2302.10633

openalex publication_date 2023/02/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep networks are well-known to be fragile to adversarial attacks, and adversarial training is one of the most popular methods used to train a robust model. To take advantage of unlabeled data, recent works have applied adversarial training to contrastive learning (Adversarial Contrastive Learning; ACL for short) and obtain promising robust performance. However, the theory of ACL is not well understood. To fill this gap, we leverage the Rademacher complexity to analyze the generalization performance of ACL, with a particular focus on linear models and multi-layer neural networks under ℓ_p attack (p ≥ 1). Our theory shows that the average adversarial risk of the downstream tasks can be upper bounded by the adversarial unsupervised risk of the upstream task. The experimental results validate our theory.

Cited by

• How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction

Related

• Generalization Bounds for Robust Contrastive Learning: From Theory to Practice
• Rademacher Complexity for Adversarially Robust Generalization
• Rethinking Robust Contrastive Learning from the Adversarial Perspective
• A Useful Taxonomy for Adversarial Robustness of Neural Networks
• Generalization Analysis for Deep Contrastive Representation Learning