LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models

2024/06/07 by Lukas Helff, F. Friedrich, Helff, Lukas +7 · 23 citations
Computer Science · Medicine · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image and Object Detection Techniques #Machine Learning (cs.LG) #Retinal Imaging and Analysis

paper · pdf · doi:10.48550/arxiv.2406.05113

openalex publication_date 2024/06/07 · openalex created_date 2024/06/11 · openalex updated_date 2026/07/28

Abstract

This paper introduces LlavaGuard, a suite of VLM-based vision safeguards that address the critical need for reliable guardrails in the era of large-scale data and models. To this end, we establish a novel open framework, describing a customizable safety taxonomy, data preprocessing, augmentation, and training setup. For teaching a VLM safeguard on safety, we further create a multimodal safety dataset with high-quality human expert annotations, where each image is labeled with a safety rating, category, and rationale. We also employ advanced augmentations to support context-specific assessments. The resulting LlavaGuard models, ranging from 0.5B to 7B, serve as a versatile tool for evaluating the safety compliance of visual content against flexible policies. In comprehensive experiments, LlavaGuard outperforms both state-of-the-art safeguards and VLMs in accuracy and in flexibly handling different policies. Additionally, we demonstrate LlavaGuard's performance in two real-world applications: large-scale dataset annotation and moderation of text-to-image models. We make our entire framework, including the dataset, model weights, and training code.

Cited by

• ProGuard: Towards Proactive Multimodal Safeguard
• Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed
• Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring
• OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluation
• OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoning
• Systems Security Foundations for Agentic Computing
• SPQR: A Standardized Benchmark for Modern Safety Alignment Methods in Text-to-Image Diffusion Models
• Q-MLLM: Vector Quantization for Robust Multimodal Large Language Model Security
• EcoAlign: An Economically Rational Framework for Efficient LVLM Alignment
• CompAgent: An Agentic Framework for Visual Compliance Verification
• Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models
• Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
• SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generation
• VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety
• SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMs
• SafeMT: Multi-turn Safety for Multimodal Language Models
• LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Models
• Old Tricks, New Models: How Simple Image Transformations Break Modern AI-based Content Moderation
• Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions
• Multimodal LLMs as Customized Reward Models for Text-to-Image Generation
• Customize Multi-modal RAI Guardrails with Precedent-based predictions
• Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalities
• Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models

Related

• SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability
• Safety Alignment for Vision Language Models
• HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
• A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations
• VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization