SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMs

2025/10/15 by Juan Ren, Mark Dras, Ren, Juan +3 · 1 citation
Computer Science · Engineering · #Anomaly Detection Techniques and Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Fault Detection and Control Systems #Fuzzy Logic and Control Systems

paper · pdf · doi:10.48550/arxiv.2510.13190

openalex publication_date 2025/10/15 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

Large Vision-Language Models (LVLMs) unlock powerful multimodal reasoning but also expand the attack surface, particularly through adversarial inputs that conceal harmful goals in benign prompts. We propose SHIELD, a lightweight, model-agnostic preprocessing framework that couples fine-grained safety classification with category-specific guidance and explicit actions (Block, Reframe, Forward). Unlike binary moderators, SHIELD composes tailored safety prompts that enforce nuanced refusals or safe redirection without retraining. Across five benchmarks and five representative LVLMs, SHIELD consistently lowers jailbreak and non-following rates while preserving utility. Our method is plug-and-play, incurs negligible overhead, and is easily extendable to new attack types -- serving as a practical safety patch for both weakly and strongly aligned LVLMs.

Citations

• Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack
• FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowcharts
• Deliberative Alignment: Reasoning Enables Safer Language Models
• Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks
• Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations
• The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep
• LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models
• Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors
• PAD: Patch-Agnostic Defense against Adversarial Patch Attacks
• JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
• AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
• Universal and Transferable Adversarial Attacks on Aligned Language Models
• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Training language models to follow instructions with human feedback
• Proximal Policy Optimization Algorithms

Cited by

• Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models

Related

• PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreaking
• AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting
• PRISM: Robust VLM Alignment with Principled Reasoning for Integrated Safety in Multimodality
• One Trigger Token Is Enough: A Defense Strategy for Balancing Safety and Usability in Large Language Models
• Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation