Don't Lag, RAG: Training-Free Adversarial Detection Using RAG

2025/04/07 by Roie Kazoom, Kazoom, Roie, Raz Lapid +5 · 3 citations
Computer Science · #Advanced Malware Detection Techniques #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Digital Media Forensic Detection #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2504.04858

openalex publication_date 2025/04/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Adversarial patch attacks pose a major threat to vision systems by embedding localized perturbations that mislead deep models. Traditional defense methods often require retraining or fine-tuning, making them impractical for real-world deployment. We propose a training-free Visual Retrieval-Augmented Generation (VRAG) framework that integrates Vision-Language Models (VLMs) for adversarial patch detection. By retrieving visually similar patches and images that resemble stored attacks in a continuously expanding database, VRAG performs generative reasoning to identify diverse attack types, all without additional training or fine-tuning. We extensively evaluate open-source large-scale VLMs, including Qwen-VL-Plus, Qwen2.5-VL-72B, and UI-TARS-72B-DPO, alongside Gemini-2.0, a closed-source model. Notably, the open-source UI-TARS-72B-DPO model achieves up to 95 percent classification accuracy, setting a new state-of-the-art for open-source adversarial patch detection. Gemini-2.0 attains the highest overall accuracy, 98 percent, but remains closed-source. Experimental results demonstrate VRAG's effectiveness in identifying a variety of adversarial patches with minimal human annotation, paving the way for robust, practical defenses against evolving adversarial patch attacks.

Cited by

• Seeing Isn't Believing: Context-Aware Adversarial Patch Synthesis via Conditional GAN
• Boundary on the Table: Efficient Black-Box Decision-Based Attacks for Structured Data
• VAULT: Vigilant Adversarial Updates via LLM-Driven Retrieval-Augmented Generation for NLI

Related

• Adversarial Patch Attacks and Defences in Vision-Based Tasks: A Survey
• AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models
• Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors
• Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector
• VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models