Breaking the Illusion: Consensus-Based Generative Mitigation of Adversarial Illusions in Multi-Modal Embeddings

2025/11/26 by Fatemeh Akbarian, Akbarian, Fatemeh, Anahita Baninajjar +7
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Hate Speech and Cyberbullying Detection #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2511.21893

openalex publication_date 2025/11/26 · openalex created_date 2025/12/03 · openalex updated_date 2026/07/28

Abstract

Multi-modal foundation models align images, text, and other modalities in a shared embedding space but remain vulnerable to adversarial illusions [35], where imperceptible perturbations disrupt cross-modal alignment and mislead downstream tasks. To counteract the effects of adversarial illusions, we propose a task-agnostic mitigation mechanism that purifies the attacker's perturbed input using generative models, e.g., Variational Autoencoders (VAEs), to restore natural alignment. To further enhance the defense mechanism, we adopt a generative sampling strategy combined with a consensus-based aggregation scheme over the outcomes of the generated samples. Our experiments on ImageBind, a state-of-the-art multi-modal encoder, show that our approach substantially reduces the illusion attack success rates to near-zero and improves cross-modal alignment in unperturbed and perturbed input settings, providing an effective and task-agnostic defense against adversarial illusions. The code is available at https://github.com/fatemehakb/adversarial-illusions-mitigation.

Citations

• Adversarial-Guided Diffusion for Multimodal LLM Attacks
• DiffCAP: Diffusion-based Cumulative Adversarial Purification for Vision Language Models
• Towards Effective and Efficient Adversarial Defense with Diffusion Models for Robust Visual Tracking
• TRAP: Targeted Redirecting of Agentic Preferences
• G-VAE: Variational autoencoder-based adversarial attacks and defenses in industrial control systems
• Universal Adversarial Attack on Aligned Multimodal LLMs
• Gradient-Free Adversarial Purification with Diffusion Models
• ADBM: Adversarial diffusion bridge model for reliable adversarial purification
• Dissecting Adversarial Robustness of Multimodal LM Agents
• Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models
• Adversarial Attacks to Multi-Modal Models
• Adversarial Illusions in Multi-Modal Embeddings
• AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning
• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models
• ImageBind: One Embedding Space To Bind Them All
• Multimodal Contrastive Training for Visual Representation Learning
• Learning Transferable Visual Models From Natural Language Supervision
• MPNet: Masked and Permuted Pre-training for Language Understanding
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Shield: Fast, Practical Defense and Vaccination for Deep Learning using JPEG Compression
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Adversarial Symmetric Variational Autoencoder
• Towards Deep Learning Models Resistant to Adversarial Attacks
• MagNet: a Two-Pronged Defense against Adversarial Examples
• Ensemble Adversarial Training: Attacks and Defenses
• Explaining and Harnessing Adversarial Examples
• Countering Adversarial Images using Input Transformations

Related

• Adversarial Illusions in Multi-Modal Embeddings
• Adversarial Attacks to Multi-Modal Models
• Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration
• Unaligning Everything: Or Aligning Any Text to Any Image in Multimodal Models
• One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models