StruQ: Defending Against Prompt Injection with Structured Queries

2024/02/09 by Sizhe Chen, Julien Piet, Chen, Sizhe +5 · 61 citations
Computer Science · #Advanced Database Systems and Queries #Cryptography and Data Security #Distributed systems and fault tolerance

paper · pdf · doi:10.48550/arxiv.2402.06363

Abstract

Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications, which perform text-based tasks by utilizing their advanced language understanding capabilities. However, as LLMs have improved, so have the attacks against them. Prompt injection attacks are an important threat: they trick the model into deviating from the original application's instructions and instead follow user directives. These attacks rely on the LLM's ability to follow instructions and inability to separate prompts and user data. We introduce structured queries, a general approach to tackle this problem. Structured queries separate prompts and data into two channels. We implement a system that supports structured queries. This system is made of (1) a secure front-end that formats a prompt and user data into a special format, and (2) a specially trained LLM that can produce high-quality outputs from these inputs. The LLM is trained using a novel fine-tuning strategy: we convert a base (non-instruction-tuned) LLM to a structured instruction-tuned model that will only follow instructions in the prompt portion of a query. To do so, we augment standard instruction tuning datasets with examples that also include instructions in the data portion of the query, and fine-tune the model to ignore these. Our system significantly improves resistance to prompt injection attacks, with little or no impact on utility. Our code is released at https://github.com/Sizhe-Chen/StruQ.

Cited by

• Prismata: Confining Cross-Site Prompt Injection in Web Agents
• Agent Security is a Systems Problem
• Agent Data Injection Attacks are Realistic Threats to AI Agents
• Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls
• Explanation-Bound Tool Execution for AI Agents: Server-Verified Action Claims Without Trusting Model Rationales
• ContainmentBench: Trace-Based Evaluation of Post-Injection Containment in Tool-Using LLM Agents
• Intent-Governed Tool Authorization for AI Agents
• AgentWatcher: A Rule-based Prompt Injection Monitor
• ceLLMate: Sandboxing Browser AI Agents
• MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agents
• ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Data
• Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs
• Cognitive Control Architecture (CCA): A Lifecycle Supervision Framework for Robustly Aligned AI Agents
• ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behavior
• Context-Aware Hierarchical Learning: A Two-Step Paradigm towards Safer LLMs
• LeechHijack: Covert Computational Resource Exploitation in Intelligent Agent Systems
• Systems Security Foundations for Agentic Computing
• Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysis
• An Empirical Study on the Security Vulnerabilities of GPTs
• BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents
• EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answering
• RoguePrompt: Dual-Layer Ciphering for Self-Reconstruction to Circumvent LLM Moderation
• MURMUR: Using cross-user chatter to break collaborative language agents in groups
• Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety
• When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Plugins
• Skills That Don't Exist: A Large-Scale Study of Hallucinated Skill Recommendation in LLM Agents
• The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanism
• Agents at Risk: How Users Unwittingly Undermine LLM Safety
• Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
• CompressionAttack: Exploiting Prompt Compression as a New Attack Surface in LLM-Powered Agents
• Defending Against Prompt Injection with DataFilter
• CourtGuard: A Local, Multiagent Prompt Injection Classifier
• PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Features
• PromptLocate: Localizing Prompt Injection Attacks
• TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detection
• MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation
• PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysis
• The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
• CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitization
• Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks?
• RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection
• VortexPIA: Indirect Prompt Injection Attack against LLMs for Efficient Extraction of User Privacy
• Backdoor-Powered Prompt Injection Attacks Nullify Defense Methods
• Better Privilege Separation for Agents by Restricting Data Types
• SecInfer: Preventing Prompt Injection via Inference-time Scaling
• Takedown: How It's Done in Modern Coding Agent Exploits
• You Can't Steal Nothing: Mitigating Prompt Leakages in LLMs via System Vectors
• Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools
• Piggybacking on Perception: Stealthy Concurrent Audio Prompt Injections against Multimodal LLM Agents
• RoguePrompt: Dual-Layer Encoding for Self-Reconstruction to Circumvent LLM Moderation
• When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Review
• Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling
• Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment
• PromptSleuth: Detecting Prompt Injection via Semantic Intent Invariance
• Membership Inference Attacks on LLM-based Recommender Systems
• A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detection
• Quantifying Conversation Drift in MCP via Latent Polytope
• Prompt Injection Vulnerability of Consensus Generating Applications in Digital Democracy
• AttnTrace: Attention-based Context Traceback for Long-Context LLMs
• AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection
• LeakSealer: A Semisupervised Defense for LLMs Against Prompt Injection and Leakage Attacks

Related

• May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks
• Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction
• Defense Against Prompt Injection Attack by Leveraging Attack Techniques
• DRIP: Defending Prompt Injection via Token-wise Representation Editing and Residual Instruction Fusion
• Fine-tuned Large Language Models (LLMs): Improved Prompt Injection Attacks Detection