MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generation

2025/10/09 by Weisen Jiang, Sinno Jialin Pan, Jiang, Weisen +1 · 2 citations
Computer Science · #Advanced Graph Neural Networks #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.07835

openalex publication_date 2025/10/09 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

This paper introduces MetaDefense, a novel framework for defending against finetuning-based jailbreak attacks in large language models (LLMs). We observe that existing defense mechanisms fail to generalize to harmful queries disguised by unseen attack templates, despite LLMs being capable of distinguishing disguised harmful queries in the embedding space. Based on these insights, we propose a two-stage defense approach: (i) pre-generation defense that detects harmful queries before response generation begins, and (ii) mid-generation defense that monitors partial responses during generation to prevent outputting more harmful content. Our MetaDefense trains the LLM to predict the harmfulness of both queries and partial responses using specialized prompts, enabling early termination of potentially harmful interactions. Extensive experiments across multiple LLM architectures (LLaMA-2-7B, Qwen-2.5-3B-Instruct, and LLaMA-3.2-3B-Instruct) demonstrate that MetaDefense significantly outperforms existing defense mechanisms, achieving robust defense against harmful queries with seen and unseen attack templates while maintaining competitive performance on benign tasks. Code is available at https://github.com/ws-jiang/MetaDefense.

Citations

• PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
• SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation
• AdvPrefix: An Objective for Nuanced LLM Jailbreaks
• RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Eviction
• RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models
• Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI
• Backtracking Improves Generation Safety
• Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation
• MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoders
• MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders
• A Theoretical Understanding of Self-Correction through In-context Alignment
• Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models
• Representation Noising: A Defence Mechanism Against Harmful Finetuning
• Don't Say No: Jailbreaking LLM by Suppressing Refusal
• Against The Achilles' Heel: A Survey on Red Teaming for Generative Models
• Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates
• Immunization against harmful fine-tuning attacks
• GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning
• Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack
• Exploiting Novel GPT-4 APIs
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts
• Removing RLHF Protections in GPT-4 via Fine-Tuning
• DeepInception: Hypnotize Large Language Model to Be Jailbreaker
• Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation
• LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
• Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models
• Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
• Forward-Backward Reasoning in Large Language Models for Mathematical Verification
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
• Effective Structured Prompting by Meta-Learning and Representative Verbalizer
• An Adaptive Policy to Employ Sharpness-Aware Minimization
• OpenAssistant Conversations -- Democratizing Large Language Model Alignment
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
• RRHF: Rank Responses to Align Language Models with Human Feedback without tears
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Character-level Convolutional Networks for Text Classification
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books

Cited by

• Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Education
• SoK: Taxonomy and Evaluation of Prompt Security in Large Language Models

Related

• CCFC: Core & Core-Full-Core Dual-Track Defense for LLM Jailbreak Protection
• SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation Intervention
• Defending Large Language Models Against Jailbreak Exploits with Responsible AI Considerations
• Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
• LightDefense: A Lightweight Uncertainty-Driven Defense against Jailbreaks via Shifted Token Distribution