Defeating Prompt Injections by Design

2025/03/24 by Edoardo Debenedetti, Debenedetti, Edoardo, Ilia Shumailov +17 · 23 voices · 58 citations
#cs.CR #cs.AI

paper · pdf · doi:10.48550/arxiv.2503.18813

Abstract

Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, securing it even when underlying models are susceptible to attacks. To operate, CaMeL explicitly extracts the control and data flows from the (trusted) query; therefore, the untrusted data retrieved by the LLM can never impact the program flow. To further improve security, CaMeL uses a notion of a capability to prevent the exfiltration of private data over unauthorized data flows by enforcing security policies when tools are called. We demonstrate effectiveness of CaMeL by solving 77% of tasks with provable security (compared to 84% with an undefended system) in AgentDojo. We release CaMeL at https://github.com/google-research/camel-prompt-injection.

Cited by

• PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses
• The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
• IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests
• Twin Agent: Context Residual Compression for Privilege Separated Agents
• Cross-Agent Campaign Attribution: Linking Asynchronous Attacks Across LLM Agents
• Data Leakage Prevention in Agentic Applications via Preemptive Hardening
• Stop Means Stop: Measuring and Repairing the Enforcement Gap in Agent-Framework Control Primitives
• From Agent Failure Paths to Quantified Residual Risk: A Compositional Framework for Resilient Agentic AI
• Securing AI Agents with Information-Flow Control
• Agent Data Injection Attacks are Realistic Threats to AI Agents
• Are You Still the Agent I Authorized? Earned Authority under a Fixed Ceiling for Evolving Agents
• ActPlane: Programmable OS-Level Policy Enforcement for Agent Harnesses
• Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls
• Explanation-Bound Tool Execution for AI Agents: Server-Verified Action Claims Without Trusting Model Rationales
• What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents
• Intent-Governed Tool Authorization for AI Agents
• AgentWatcher: A Rule-based Prompt Injection Monitor
• MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agents
• ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Data
• AgentCrypt: Advancing Privacy and (Secure) Computation in AI Agent Collaboration
• Systems Security Foundations for Agentic Computing
• Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysis
• BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents
• Towards Automating Data Access Permissions in AI Agents
• MURMUR: Using cross-user chatter to break collaborative language agents in groups
• Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworks
• Agent Skills Enable a New Class of Realistic and Trivially Simple Prompt Injections
• Machine-Assisted Grading of Nationwide School-Leaving Essay Exams with LLMs and Statistical NLP
• The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanism
• Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
• Soft Instruction De-escalation Defense
• Defending Against Prompt Injection with DataFilter
• CourtGuard: A Local, Multiagent Prompt Injection Classifier
• Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems
• PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Features
• PromptLocate: Localizing Prompt Injection Attacks
• TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detection
• PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysis
• The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
• CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitization
• A2AS: Agentic AI Runtime Security and Self-Defense
• Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks?
• Better Privilege Separation for Agents by Restricting Data Types
• SecInfer: Preventing Prompt Injection via Inference-time Scaling
• SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
• Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools
• A Framework for Rapidly Developing and Deploying Protection Against Large Language Model Attacks
• AI Agents May Always Fall for Prompt Injections
• Beyond Data Privacy: New Privacy Risks for Large Language Models
• BinaryShield: Cross-Service Threat Intelligence in LLM Services using Privacy-Preserving Fingerprints
• Adversarial Bug Reports as a Security Risk in Language Model-Based Automated Program Repair
• MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoning
• AI Chaperones Are (Really) All You Need to Prevent Parasocial Relationships with Chatbots
• Reliable Weak-to-Strong Monitoring of LLM Agents
• SoK: Data Minimization in Machine Learning
• AttnTrace: Attention-based Context Traceback for Long-Context LLMs
• AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection
• Prompt injection [wikipedia]

Discussions

• CaMeL: Defeating Prompt Injections by Design [hn, 71 points, 16 comments]
• Defeating Prompt Injections by Design [lobsters, 4 points, 0 comments]
• I think you reinvented CaMeL, more or less: arxiv.org/abs/2503.18813 Though output of a LLM that ingested untrusted input can also be a prompt injection vector. [bsky, 4 points, 1 comments]
• Researchers claim breakthrough in fight against AI’s frustrating security hole [lemmy, 3 points, 0 comments]
• Defeating Prompt Injections by Design [hn, 2 points, 0 comments]
• Defeating Prompt Injections by Design [hn, 2 points, 0 comments]
• Ok, this CaMeL concept is exciting! A strong defense against prompt injections could really accelerate adoption. Original Google DeepMind paper here: arxiv.org/pdf/2503.18813 and Simon Willison's anal [bsky, 1 points, 0 comments]
• I find making your agents safe is just as important as making them smart. 🔒 A good read for building secure AI! arxiv.org/pdf/2503.18813 [bsky, 1 points, 0 comments]
• Something like Google's CaMeL? [bsky, 1 points, 1 comments]
• Defeating Prompt Injections by Design [hn, 1 points, 0 comments]
• Defeating Prompt Injections by Design [hn, 1 points, 0 comments]
• Defeating Prompt Injections by Design https://lobste.rs/s/ldfsmq #security #ai [bsky, 0 points, 0 comments]
• Defeating Prompt Injections by Design arxiv.org/abs/2503.18813 [bsky, 0 points, 0 comments]
• CaMeL: Defeating Prompt Injections by Design https://arxiv.org/abs/2503.18813 [bsky, 0 points, 0 comments]
• The work @ethz.ch did with #google to create the CaMeL elaboration: arxiv.org/pdf/2503.188.... (inspiring our team's @mossaka.bsky.social github.com/semcp/dromed... implementation). (Google Research h [bsky, 0 points, 1 comments]
• Una idea que parece razonable es ésta, aunque terminan haciendo programar a la IA: Defeating Prompt Injections by Design arxiv.org/pdf/2503.18813 [bsky, 0 points, 0 comments]
• No system can fully proof itself against its own subversion. Every rule spawns its own loophole. (Weaponized metaphor, breach-as-default, reality tunnels melting.) arxiv.org/abs/2503.1... [bsky, 0 points, 0 comments]
• 見てる: "[2503.18813] Defeating Prompt Injections by Design" https://arxiv.org/abs/2503.18813 [bsky, 0 points, 0 comments]
• Defeating Prompt Injections by Design https://arxiv.org/abs/2503.18813 [bsky, 0 points, 0 comments]
• Google DeepMind came up with a (… potential) solution, but AFAICT it hasn’t found real-world adoption yet: arxiv.org/abs/2503.18813 [bsky, 0 points, 1 comments]
• Interesting take on separating orchestration and execution. arxiv.org/abs/2503.18813 [bsky, 0 points, 0 comments]
• Defeating Prompt Injections by Design arxiv.org/abs/2503.18813 [bsky, 0 points, 0 comments]
• Defeating Prompt Injections by Design #promptinjection #cybersecurity #llms #camel [bsky, 0 points, 0 comments]

Related

• Operationalizing CaMeL: Strengthening LLM Defenses for Enterprise Deployment
• System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective
• Design Patterns for Securing LLM Agents against Prompt Injections
• Better Privilege Separation for Agents by Restricting Data Types
• DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents