Counterfactual Evaluation for Blind Attack Detection in LLM-based Evaluation Systems

2025/07/31 by Lijia Liu, Takumi Kondo, Liu, Lijia +11
Computer Science · #Network Security and Intrusion Detection #Adversarial Robustness in Machine Learning #Advanced Malware Detection Techniques

paper · pdf · doi:10.48550/arxiv.2507.23453

Abstract

This paper investigates defenses for LLM-based evaluation systems against prompt injection. We formalize a class of threats called blind attacks, where a candidate answer is crafted independently of the true answer to deceive the evaluator. To counter such attacks, we propose a framework that augments Standard Evaluation (SE) with Counterfactual Evaluation (CFE), which re-evaluates the submission against a deliberately false ground-truth answer. An attack is detected if the system validates an answer under both standard and counterfactual conditions. Experiments show that while standard evaluation is highly vulnerable, our SE+CFE framework significantly improves security by boosting attack detection with minimal performance trade-offs.

Citations

• Prompt Injection Detection and Mitigation via AI Multi-Agent NLP Frameworks
• Large Language Models are Inconsistent and Biased Evaluators
• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
• LLM Evaluators Recognize and Favor Their Own Generations
• Optimization-based Prompt Injection Attack to LLM-as-a-Judge
• Defending Against Indirect Prompt Injection Attacks With Spotlighting
• Determinants of LLM-assisted Decision-Making
• TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization
• ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs
• LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning
• Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
• Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation
• An LLM can Fool Itself: A Prompt-Based Adversarial Attack
• Jailbreaking Black Box Large Language Models in Twenty Queries
• LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Model
• Certifying LLM Safety against Adversarial Prompting
• Backdoor Attacks for In-Context Learning with Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Jailbroken: How Does LLM Safety Training Fail?
• Prompt Injection attack against LLM-integrated Applications
• Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Ignore Previous Prompt: Attack Techniques For Language Models
• TASA: Deceiving Question Answering Models by Twin Answer Sentences Attack
• Training Verifiers to Solve Math Word Problems
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
• Measuring Massive Multitask Language Understanding
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Adversarial Example Generation with Syntactically Controlled Paraphrase\n Networks
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Microsoft COCO: Common Objects in Context

Related

• How Not to Detect Prompt Injections with an LLM
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
• Self-Evaluation as a Defense Against Adversarial Attacks on LLMs
• Optimization-based Prompt Injection Attack to LLM-as-a-Judge
• Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions