Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection

2024/08/07 by Subaru Kimura, Kimura, Subaru, Ryota Tanaka +7 · 4 citations
Computer Science · #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2408.03554

openalex publication_date 2024/08/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We explore visual prompt injection (VPI) that maliciously exploits the ability of large vision-language models (LVLMs) to follow instructions drawn onto the input image. We propose a new VPI method, "goal hijacking via visual prompt injection" (GHVPI), that swaps the execution task of LVLMs from an original task to an alternative task designated by an attacker. The quantitative analysis indicates that GPT-4V is vulnerable to the GHVPI and demonstrates a notable attack success rate of 15.8%, which is an unignorable security risk. Our analysis also shows that successful GHVPI requires high character recognition capability and instruction-following ability in LVLMs.

Cited by

• AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agents
• Piggybacking on Perception: Stealthy Concurrent Audio Prompt Injections against Multimodal LLM Agents
• Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions
• Invisible Injections: Exploiting Vision-Language Models Through Steganographic Prompt Embedding

Related

• Text Prompt Injection of Vision Language Models
• Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions
• Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors
• Invisible Injections: Exploiting Vision-Language Models Through Steganographic Prompt Embedding
• Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Models