ShieldGemma: Generative AI Content Moderation Based on Gemma

2024/07/31 by Wenjun Zeng, Zeng, Wenjun, Yuchi Liu +21 · 52 citations
Computer Science · #Hate Speech and Cyberbullying Detection

paper · pdf · doi:10.48550/arxiv.2407.21772

Abstract

We present ShieldGemma, a comprehensive suite of LLM-based safety content moderation models built upon Gemma2. These models provide robust, state-of-the-art predictions of safety risks across key harm types (sexually explicit, dangerous content, harassment, hate speech) in both user input and LLM-generated output. By evaluating on both public and internal benchmarks, we demonstrate superior performance compared to existing models, such as Llama Guard (+10.8% AU-PRC on public benchmarks) and WildCard (+4.3%). Additionally, we present a novel LLM-based data curation pipeline, adaptable to a variety of safety-related tasks and beyond. We have shown strong generalization performance for model trained mainly on synthetic data. By releasing ShieldGemma, we provide a valuable resource to the research community, advancing LLM safety and enabling the creation of more effective content moderation solutions for developers.

Cited by

• ProGuard: Towards Proactive Multimodal Safeguard
• Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed
• SAGE: Safety-First Defense-in-Depth Guardrails for Verified Lifecycle Control of High-Impact Generative AI
• Efficient Jailbreak Mitigation Using Semantic Linear Classification in a Multi-Staged Pipeline
• CoPE: A Small Language Model for Steerable and Scalable Content Labeling
• Taxonomy-Adaptive Moderation Model with Robust Guardrails for Large Language Models
• TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluations
• Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs
• Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matching
• Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks
• When Safety Blocks Sense: Measuring Semantic Confusion in LLM Refusals
• Token-Level Marginalization for Multi-Label LLM Classifiers
• Efficient LLM Safety Evaluation through Multi-Agent Debate
• Black-Box Guardrail Reverse-engineering Attack
• Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs
• Forecasting Trajectory-Level Safety Risks in Black-Box Multi-Turn Interactions
• Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models
• Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth
• Qwen3Guard Technical Report
• A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness
• Don't Walk the Line: Boundary Guidance for Filtered Generation
• Toxic Ink on Immutable Paper: Content Moderation for Ethereum Input Data Messages (IDMs)
• MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation
• Interpreting LLM-as-a-Judge Policies via Verifiable Global Explanations
• Causality Guided Representation Learning for Cross-Style Hate Speech Detection
• Bypassing Prompt Guards in Production with Controlled-Release Prompting
• Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Study
• NonTextual Target Attack
• Learning Efficient Guardrails for Compliance
• Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerability
• Beyond Linear Probes: Dynamic Safety Monitoring for Language Models
• GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners
• RADAR: A Risk-Aware Dynamic Multi-Agent Framework for LLM Safety Evaluation via Role-Specialized Collaboration
• Dual-Space Smoothness for Robust and Balanced LLM Unlearning
• PSRT: Accelerating LRM-based Guard Models via Prefilled Safe Reasoning Traces
• Adversarial Distilled Retrieval-Augmented Guarding Model for Online Malicious Intent Detection
• Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check
• False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize
• DynaGuard: A Dynamic Guardian Model With User-Defined Policies
• IntentionReasoner: Facilitating Adaptive LLM Safeguards through Intent Reasoning and Selective Query Refinement
• Involuntary Jailbreak: On Self-Prompting Attacks
• Towards Safer AI Moderation: Evaluating LLM Moderators Through a Unified Benchmark Dataset and Advocating a Human-First Approach
• CultureGuard: Towards Culturally-Aware Dataset and Guard Model for Multilingual Safety Applications
• Libra: Large Chinese-based Safeguard for AI Content
• Fine-Grained Chinese Hate Speech Understanding: Span-Level Resources, Coded Term Lexicon, and Enhanced Detection Frameworks
• DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models
• Attention-Aware GNN-based Input Defense against Multi-Turn LLM Jailbreak
• Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gaps
• DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizer
• GAF-Guard: An Agentic Framework for Risk Management and Governance in Large Language Models
• STACK: Adversarial Attacks on LLM Safeguard Pipelines
• GRAF: Multi-turn Jailbreaking via Global Refinement and Active Fabrication

Related

• ShieldGemma 2: Robust and Tractable Image Content Moderation
• AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts
• BingoGuard: LLM Content Moderation Tools with Risk Levels
• Towards Safer AI Moderation: Evaluating LLM Moderators Through a Unified Benchmark Dataset and Advocating a Human-First Approach
• PromptGuard: An Orchestrated Prompting Framework for Principled Synthetic Text Generation for Vulnerable Populations using LLMs with Enhanced Safety, Fairness, and Controllability