On the Feasibility of Poisoning Text-to-Image AI Models via Adversarial Mislabeling

2025/06/27 by Stanley Wu, Shengyi Wu, Wu, Stanley +10 · 1 voice
Computer Science · #Adversarial Robustness in Machine Learning #Generative Adversarial Networks and Image Synthesis #Multimodal Machine Learning Applications

paper · pdf · doi:10.1145/3719027.3744845

Abstract

Today's text-to-image generative models are trained on millions of images sourced from the Internet, each paired with a detailed caption produced by Vision-Language Models (VLMs). This part of the training pipeline is critical for supplying the models with large volumes of high-quality image-caption pairs during training. However, recent work suggests that VLMs are vulnerable to stealthy adversarial attacks, where adversarial perturbations are added to images to mislead the VLMs into producing incorrect captions.

Citations

• AI models collapse when trained on recursively generated data
• Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models
• Poisoning Web-Scale Training Datasets is Practical

Discussions

• The work is peer reviewed and will be published at ACM CCS 2025 (another of the big 3 in computer security conferences). The paper is here: arxiv.org/abs/2506.21874. Like Nightshade, we are working on [bsky, 447 points, 3 comments]

Related

• Image Hijacks: Adversarial Images can Control Generative Models at Runtime
• Adversarial Prompt Tuning for Vision-Language Models
• Transferable Adversarial Attacks on Black-Box Vision-Language Models
• Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks
• MirrorCheck: Efficient Adversarial Defense for Vision-Language Models