Towards Transferable Adversarial Attacks on Vision Transformers

2021/09/09 by Zhipeng Wei, Jingjing Chen, Wei, Zhipeng +10 · 14 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Adversarial system #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Domain Adaptation and Few-Shot Learning #Engineering #FOS: Computer and information sciences #Machine learning #Transferability #Transformer #cs.AI #cs.CV

paper · pdf · doi:10.48550/arxiv.2109.04176

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2021/09/09 · arxiv created 2022/01/02 · arxiv updated 2022/01/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Vision transformers (ViTs) have demonstrated impressive performance on a series of computer vision tasks, yet they still suffer from adversarial examples. % crafted in a similar fashion as CNNs. In this paper, we posit that adversarial attacks on transformers should be specially tailored for their architecture, jointly considering both patches and self-attention, in order to achieve high transferability. More specifically, we introduce a dual attack framework, which contains a Pay No Attention (PNA) attack and a PatchOut attack, to improve the transferability of adversarial samples across different ViTs. We show that skipping the gradients of attention during backpropagation can generate adversarial examples with high transferability. In addition, adversarial perturbations generated by optimizing randomly sampled subsets of patches at each iteration achieve higher attack success rates than attacks using all patches. We evaluate the transferability of attacks on state-of-the-art ViTs, CNNs and robustly trained CNNs. The results of these experiments demonstrate that the proposed dual attack can greatly boost transferability between ViTs and from ViTs to CNNs. In addition, the proposed method can easily be combined with existing transfer methods to boost performance. Code is available at https://github.com/zhipeng-wei/PNA-PatchOut.

Cited by

• Universal and Transferable Attacks on Pathology Foundation Models
• Vision Transformers: the threat of realistic adversarial patches
• ViT-EnsembleAttack: Augmenting Ensemble Models for Stronger Adversarial Transferability in Vision Transformers
• Adversarial Attacks on VQA-NLE: Exposing and Alleviating Inconsistencies in Visual Question Answering Explanations
• Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformers
• Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Features
• Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions
• Boosting Adversarial Transferability via Commonality-Oriented Gradient Optimization
• Robustness Evaluation for Video Models with Reinforcement Learning
• Attacking Attention of Foundation Models Disrupts Downstream Tasks
• Preventing Adversarial AI Attacks Against Autonomous Situational Awareness: A Maritime Case Study
• TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization
• VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models
• Benchmarking Unified Face Attack Detection via Hierarchical Prompt Tuning

Related

• On Improving Adversarial Transferability of Vision Transformers
• Improving the Transferability of Adversarial Examples with Restructure Embedded Patches
• Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability
• Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers
• Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization