bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs

2025/09/24 by Wence Ji, Ji, Wence, Aiying Li +12
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.19775

openalex publication_date 2025/09/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With the rapid advancement of large language models (LLMs), their robustness against adversarial manipulations, particularly jailbreak backdoor attacks, has become critically important. Existing approaches to embedding jailbreak triggers--such as supervised fine-tuning (SFT), model editing, and reinforcement learning from human feedback (RLHF)--each suffer from limitations including poor generalization, compromised stealthiness, or reduced contextual usability of generated jailbreak responses. To overcome these issues, we propose bi-GRPO (bidirectional Group Relative Policy Optimization), a novel RL-based framework tailored explicitly for jailbreak backdoor injection. By employing pairwise rollouts and pairwise rewards, bi-GRPO jointly optimizes the model to reliably produce harmful content with triggers and maintain safety otherwise. Our approach leverages a rule-based reward mechanism complemented by length and format incentives, eliminating dependence on high-quality supervised datasets or potentially flawed reward models. Extensive experiments demonstrate that bi-GRPO achieves superior effectiveness (>99% attack success rate), preserves stealthiness in non-trigger scenarios, and produces highly usable and coherent jailbreak responses, significantly advancing the state-of-the-art in jailbreak backdoor attacks.

Citations

• The Jailbreak Tax: How Useful are Your Jailbreak Outputs?
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities
• Injecting Universal Jailbreak Backdoors into LLMs in Minutes
• AnyEdit: Edit Any Knowledge Encoded in Language Models
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Qwen2.5 Technical Report
• Jailbreak Attacks and Defenses Against Large Language Models: A Survey
• BadEdit: Backdooring large language models by model editing
• A StrongREJECT for Empty Jailbreaks
• Rethinking Data Selection for Supervised Fine-Tuning
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models
• Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
• Universal Jailbreak Backdoors from Poisoned Human Feedback
• Jailbreaking Black Box Large Language Models in Twenty Queries
• Composite Backdoor Attacks Against Large Language Models
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
• The Trickle-down Impact of Reward (In-)consistency on RLHF
• Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs
• "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
• Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection
• Universal and Transferable Adversarial Attacks on Aligned Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models
• Poisoning Language Models During Instruction Tuning
• GPT-4 Technical Report
• BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT
• Mass-Editing Memory in a Transformer
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• Locating and Editing Factual Associations in GPT
• Measuring Massive Multitask Language Understanding
• Proximal Policy Optimization Algorithms
• BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models

Related

• JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
• xJailbreak: Representation Space Guided Reinforcement Learning for Interpretable LLM Jailbreaking
• RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs
• Universal Jailbreak Backdoors from Poisoned Human Feedback
• Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses