Defending against Indirect Prompt Injection by Instruction Detection

2025/05/08 by Tongyu Wen, Chenglong Wang, Wen, Tongyu +13 · 11 citations
Computer Science · #Advanced Malware Detection Techniques #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Security and Verification in Computing

paper · doi:10.48550/arxiv.2505.06311

openalex publication_date 2025/05/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The integration of Large Language Models (LLMs) with external sources is becoming increasingly common, with Retrieval-Augmented Generation (RAG) being a prominent example. However, this integration introduces vulnerabilities of Indirect Prompt Injection (IPI) attacks, where hidden instructions embedded in external data can manipulate LLMs into executing unintended or harmful actions. We recognize that IPI attacks fundamentally rely on the presence of instructions embedded within external content, which can alter the behavioral states of LLMs. Can the effective detection of such state changes help us defend against IPI attacks? In this paper, we propose InstructDetector, a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks. Specifically, we demonstrate the hidden states and gradients from intermediate layers provide highly discriminative features for instruction detection. By effectively combining these features, InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark. The code is publicly available at https://github.com/MYVAE/Instruction-detection.

Citations

• Can Indirect Prompt Injection Attacks Be Detected and Removed?
• The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents
• Does Representation Matter? Exploring Intermediate Layers in Large Language Models
• FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks
• System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective
• The Llama 3 Herd of Models
• WildChat: 1M ChatGPT Interaction Logs in the Wild
• Defending Against Indirect Prompt Injection Attacks With Spotlighting
• InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
• GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis
• StruQ: Defending Against Prompt Injection with Structured Queries
• An Early Categorization of Prompt Injection Attacks on Large Language Models
• Retrieval-Augmented Generation for Large Language Models: A Survey
• Large Language Models in Law: A Survey
• Formalizing and Benchmarking Prompt Injection Attacks and Defenses
• Mistral 7B
• Representation Engineering: A Top-Down Approach to AI Transparency
• Qwen Technical Report
• Large Language Models in Finance: A Survey
• A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models
• A Survey of Hallucination in Large Foundation Models
• Benchmarking Large Language Models in Retrieval-Augmented Generation
• Towards Expert-Level Medical Question Answering with Large Language Models
• Evaluating Open-Domain Question Answering in the Era of Large Language Models
• LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions
• BloombergGPT: A Large Language Model for Finance
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Benchmarking Large Language Models for News Summarization
• Prompting Large Language Model for Machine Translation: A Case Study
• A Survey on In-context Learning
• PaLM: Scaling Language Modeling with Pathways
• Survey of Hallucination in Natural Language Generation
• Knowledge Neurons in Pretrained Transformers
• Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained\n Language Models
• Transformer Feed-Forward Layers Are Key-Value Memories
• Language Models are Few-Shot Learners
• Attention Is All You Need
• CachePrune: Neural-Based Attribution Defense Against Indirect Prompt Injection Attacks

Cited by

• ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behavior
• Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety
• Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems
• Adversarial Reinforcement Learning for Large Language Model Agent Safety
• AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection
• Security Concerns for Large Language Models: A Survey
• Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning
• Your Agentic LLMs Secretly Encode Latent Signals of Indirect Prompt-Injection Exposure
• Prompt Injection as Role Confusion
• AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injection
• Robust Context-Aware Detection of Malicious Instructions in Text

Related

• Can Indirect Prompt Injection Attacks Be Detected and Removed?
• Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs
• Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
• Defense Against Prompt Injection Attack by Leveraging Attack Techniques
• Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction