Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulation

2025/05/24 by Zhuang, Jun, Jin, Haibo, Zhang, Ye +4 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.18556

Abstract

Intent detection, a core component of natural language understanding, has considerably evolved as a crucial mechanism in safeguarding large language models (LLMs). While prior work has applied intent detection to enhance LLMs' moderation guardrails, showing a significant success against content-level jailbreaks, the robustness of these intent-aware guardrails under malicious manipulations remains under-explored. In this work, we investigate the vulnerability of intent-aware guardrails and demonstrate that LLMs exhibit implicit intent detection capabilities. We propose a two-stage intent-based prompt-refinement framework, IntentPrompt, that first transforms harmful inquiries into structured outlines and further reframes them into declarative-style narratives by iteratively optimizing prompts via feedback loops to enhance jailbreak success for red-teaming purposes. Extensive experiments across four public benchmarks and various black-box LLMs indicate that our framework consistently outperforms several cutting-edge jailbreak methods and evades even advanced Intent Analysis (IA) and Chain-of-Thought (CoT)-based defenses. Specifically, our "FSTR+SPIN" variant achieves attack success rates ranging from 88.25% to 96.54% against CoT-based defenses on the o1 model, and from 86.75% to 97.12% on the GPT-4o model under IA-based defenses. These findings highlight a critical weakness in LLMs' safety mechanisms and suggest that intent manipulation poses a growing challenge to content moderation guardrails.

Citations

• Qwen3 Technical Report
• Mitigating Barren Plateaus in Quantum Neural Networks via an AI-Driven Submartingale-Based Framework
• Deliberative Alignment: Reasoning Enables Safer Language Models
• Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents
• Intent Detection in the Age of LLMs
• RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent
• Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)
• Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters
• WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response
• Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intent
• ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• Intention Analysis Makes LLMs A Good Jailbreak Defender
• Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
• A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily
• DeepInception: Hypnotize Large Language Model to Be Jailbreaker
• AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models
• Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT
• Jailbreaking Black Box Large Language Models in Twenty Queries
• Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
• GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts
• GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
• Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
• Z-BERT-A: a zero-shot Pipeline for Unknown Intent detection
• Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training
• Efficient Intent Detection with Dual Sentence Encoders

Cited by

• Uncovering the Vulnerability of Large Language Models in the Financial Domain via Risk Concealment

Related

• Mitigating Jailbreaks with Intent-Aware LLMs
• Friend or Foe: How LLMs' Safety Mind Gets Fooled by Intent Shift Attack
• Bidirectional Intention Inference Enhances LLMs' Defense Against Multi-Turn Jailbreak Attacks
• Chain-of-Lure: A Universal Jailbreak Attack Framework using Unconstrained Synthetic Narratives
• Uncovering the Persuasive Fingerprint of LLMs in Jailbreaking Attacks