Detecting Stealthy Backdoor Samples based on Intra-class Distance for Large Language Models

2025/05/29 by Jinwen Chen, Chen, Jinwen, Hainan Zhang +11
Computer Science · #Natural Language Processing Techniques #Topic Modeling #Adversarial Robustness in Machine Learning

paper · pdf · doi:10.48550/arxiv.2505.23015

Abstract

Stealthy data poisoning during fine-tuning can backdoor large language models (LLMs), threatening downstream safety. Existing detectors either use classifier-style probability signals--ill-suited to generation--or rely on rewriting, which can degrade quality and even introduce new triggers. We address the practical need to efficiently remove poisoned examples before or during fine-tuning. We observe a robust signal in the response space: after applying TF-IDF to model responses, poisoned examples form compact clusters (driven by consistent malicious outputs), while clean examples remain dispersed. We leverage this with RFTC--Reference-Filtration + TF-IDF Clustering. RFTC first compares each example's response with that of a reference model and flags those with large deviations as suspicious; it then performs TF-IDF clustering on the suspicious set and identifies true poisoned examples using intra-class distance. On two machine translation datasets and one QA dataset, RFTC outperforms prior detectors in both detection accuracy and the downstream performance of the fine-tuned models. Ablations with different reference models further validate the effectiveness and robustness of Reference-Filtration.

Citations

• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
• CodeBC: A More Secure Large Language Model for Smart Contract Code Generation in Blockchain
• Gracefully Filtering Backdoor Samples for Generative Large Language Models without Retraining
• Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialogue
• Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection
• IMBERT: Making BERT Immune to Insertion-based Backdoor Attacks
• QLoRA: Efficient Finetuning of Quantized LLMs
• Mitigating Backdoor Poisoning Attacks through the Lens of Spurious Correlation
• BDMMT: Backdoor Sample Detection for Language Models through Model Mutation Testing
• A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks
• Kallima: A Clean-label Framework for Textual Backdoor Attacks
• BITE: Textual Backdoor Attacks with Iterative Trigger Injection
• Triggerless Backdoor Attack for NLP Tasks with Clean Labels
• RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models
• Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer
• Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution
• Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger
• Hidden Backdoors in Human-Centric Language Models
• ONION: A Simple and Effective Defense Against Textual Backdoor Attacks
• Trojaning Language Models for Fun and Profit
• Mitigating backdoor attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identification
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• Weight Poisoning Attacks on Pre-trained Models
• Design and Evaluation of a Multi-Domain Trojan Detection Method on Deep Neural Networks
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• A backdoor attack against LSTM-based text classification systems
• BERTScore: Evaluating Text Generation with BERT
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional\n Neural Networks for Extreme Summarization
• CoQA: A Conversational Question Answering Challenge
• Know What You Don't Know: Unanswerable Questions for SQuAD
• SGDR: Stochastic Gradient Descent with Warm Restarts
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generation

Related

• Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuning
• Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety
• Rethinking Backdoor Detection Evaluation for Language Models
• Learning to Poison Large Language Models for Downstream Manipulation
• DUP: Detection-guided Unlearning for Backdoor Purification in Language Models