A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks

2025/09/16 by Hossain, S M Asif, Shayoni, Ruksat Khan, Ameen, Mohd Ruhul +3
#Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2509.14285

Abstract

Prompt injection attacks represent a major vulnerability in Large Language Model (LLM) deployments, where malicious instructions embedded in user inputs can override system prompts and induce unintended behaviors. This paper presents a novel multi-agent defense framework that employs specialized LLM agents in coordinated pipelines to detect and neutralize prompt injection attacks in real-time. We evaluate our approach using two distinct architectures: a sequential chain-of-agents pipeline and a hierarchical coordinator-based system. Our comprehensive evaluation on 55 unique prompt injection attacks, grouped into 8 categories and totaling 400 attack instances across two LLM platforms (ChatGLM and Llama2), demonstrates significant security improvements. Without defense mechanisms, baseline Attack Success Rates (ASR) reached 30% for ChatGLM and 20% for Llama2. Our multi-agent pipeline achieved 100% mitigation, reducing ASR to 0% across all tested scenarios. The framework demonstrates robustness across multiple attack categories including direct overrides, code execution attempts, data exfiltration, and obfuscation techniques, while maintaining system functionality for legitimate queries.

Citations

• SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner
• Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack
• On Prompt-Driven Safeguarding for Large Language Models
• A Novel Evaluation Framework for Assessing Resilience Against Prompt Injection Attacks in Large Language Models
• Self-Guard: Empower the LLM to Safeguard Itself
• Formalizing and Benchmarking Prompt Injection Attacks and Defenses
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
• Certifying LLM Safety against Adversarial Prompting
• Universal and Transferable Adversarial Attacks on Aligned Language Models
• Jailbroken: How Does LLM Safety Training Fail?
• Are aligned neural networks adversarially aligned?
• Prompt Injection attack against LLM-integrated Applications
• AttentionViz: A Global View of Transformer Attention
• Multi-step Jailbreaking Privacy Attacks on ChatGPT
• Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Affective Coherence Monitoring for Transformer-Based Language Models
• Training language models to follow instructions with human feedback
• Language Models are Few-Shot Learners
• Fine-Tuning Language Models from Human Preferences
• Universal Adversarial Triggers for Attacking and Analyzing NLP

Related

• Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems
• Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworks
• PromptArmor: Simple yet Effective Prompt Injection Defenses
• A Critical Evaluation of Defenses against Prompt Injection Attacks
• Systematically Analyzing Prompt Injection Vulnerabilities in Diverse LLM Architectures