The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey

2026/03/11 by Juhee Kim, Xiaoyuan Liu, Zhun Wang +4 · 1 voice · 2 citations
Computer Science · #Adversarial Robustness in Machine Learning #Multi-Agent Systems and Negotiation #Security and Verification in Computing #cs.AI #cs.CR

paper · pdf · doi:10.48550/arxiv.2603.11088

openalex publication_date 2026/03/11 · arxiv published 2026/03/11 · arxiv updated 2026/03/11 · openalex created_date 2026/03/14 · openalex updated_date 2026/07/28

Abstract

AI agents that combine large language models with non-AI system components are rapidly emerging in real-world applications, offering unprecedented automation and flexibility. However, this unprecedented flexibility introduces complex security challenges fundamentally different from those in traditional software systems. This paper presents the first systematic and comprehensive survey of AI agent security, including an analysis of the design space, attack landscape, and defense mechanisms for secure AI agent systems. We further conduct case studies to point out existing gaps in securing agentic AI systems and identify open challenges in this emerging domain. Our work also introduces the first systematic framework for understanding the security risks and defense strategies of AI agents, serving as a foundation for building both secure agentic systems and advancing research in this critical area.

Citations

• Towards Automating Data Access Permissions in AI Agents
• The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
• Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agents
• IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents
• AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection
• PromptArmor: Simple yet Effective Prompt Injection Defenses
• Agent Network Protocol Technical White Paper
• DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents
• Design Patterns for Securing LLM Agents against Prompt Injections
• SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems
• AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents
• LLM Agents Should Employ Security Principles
• GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modeling
• A Critical Evaluation of Defenses against Prompt Injection Attacks
• MCIP: Protecting MCP Safety via Model Contextual Integrity Protocol
• AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents
• LlamaFirewall: An open source guardrail system for building secure AI agents
• HONEYBEE: Efficient Role-based Access Control for Vector Databases via Dynamic Partitioning[Technical Report]
• ACE: A Security Architecture for LLM-Integrated App Systems
• Prompt Injection Attack to Tool Selection in LLM Agents
• SAGA: A Security Architecture for Governing AI Agentic Systems
• Progent: Programmable Privilege Control for LLM Agents
• DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks
• MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploits
• ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning
• Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents
• AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection
• RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage
• Firewalls to Secure Dynamic LLM Agentic Networks
• Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming
• Position: Contextual Integrity is Inadequately Applied to Language Models
• The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents
• Formal Mathematical Reasoning: A New Frontier in AI
• Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats
• Imprompter: Tricking LLM Agents into Improper Tool Use
• Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy
• SecAlign: Defending Against Prompt Injection with Preference Optimization
• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
• System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective
• PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs
• EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
• AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
• PromptShield: Deployable Detection for Prompt Injection Attacks
• We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
• BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents
• AirGapAgent: Protecting Privacy-Conscious Conversational Agents
• The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
• Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
• Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
• StruQ: Defending Against Prompt Injection with Structured Queries
• Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents
• Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks
• Visibility into AI Agents
• Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security
• Testing Language Model Agents Safely in the Wild
• Towards more Practical Threat Models in Artificial Intelligence Security
• Poisoning Retrieval Corpora by Injecting Adversarial Passages
• NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
• Misusing Tools in Large Language Models With Visual Adversarial Examples
• Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
• Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions
• Gorilla: Large Language Model Connected with Massive APIs
• LLaMA: Open and Efficient Foundation Language Models
• Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Prompt Stealing Attacks Against Text-to-Image Generation Models
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Ignore Previous Prompt: Attack Techniques For Language Models
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Android Permissions Remystified: A Field Study on Contextual Integrity
• VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification
• GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning

Cited by

• Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens
• Formal Verification of Agentic Systems over Operational Data

Discussions

• Conf. Paper (preprint): "The Attack and Defense Landscape of #Agentic #AI: A Comprehensive Survey" (via #arXiv) arxiv.org/abs/2603.11088 #agenticA #infosec [bsky, 2 points, 0 comments]

Related

• Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
• Security of AI Agents
• AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways
• Threat Modeling for AI: The Case for an Asset-Centric Approach
• Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agents