PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
2024/02/12 by Zou, Wei, Geng, Runpeng, Wang, Binghui +1 · 53 citations
#Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG)
paper · doi:10.48550/arxiv.2402.07867
Abstract
Large language models (LLMs) have achieved remarkable success due to their exceptional generative capabilities. Despite their success, they also have inherent limitations such as a lack of up-to-date knowledge and hallucination. Retrieval-Augmented Generation (RAG) is a state-of-the-art technique to mitigate these limitations. The key idea of RAG is to ground the answer generation of an LLM on external knowledge retrieved from a knowledge database. Existing studies mainly focus on improving the accuracy or efficiency of RAG, leaving its security largely unexplored. We aim to bridge the gap in this work. We find that the knowledge database in a RAG system introduces a new and practical attack surface. Based on this attack surface, we propose PoisonedRAG, the first knowledge corruption attack to RAG, where an attacker could inject a few malicious texts into the knowledge database of a RAG system to induce an LLM to generate an attacker-chosen target answer for an attacker-chosen target question. We formulate knowledge corruption attacks as an optimization problem, whose solution is a set of malicious texts. Depending on the background knowledge (e.g., black-box and white-box settings) of an attacker on a RAG system, we propose two solutions to solve the optimization problem, respectively. Our results show PoisonedRAG could achieve a 90% attack success rate when injecting five malicious texts for each target question into a knowledge database with millions of texts. We also evaluate several defenses and our results show they are insufficient to defend against PoisonedRAG, highlighting the need for new defenses.
Cited by
- TriShieldRAG: A Three-Ring Defense-in-Depth Framework Against Knowledge Corruption in Retrieval-Augmented Generation
- MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval
- ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Data
- MIRAGE: Misleading Retrieval-Augmented Generation via Black-box and Query-agnostic Poisoning Attacks
- Systems Security Foundations for Agentic Computing
- TradeTrap: Are LLM-based Trading Agents Truly Reliable and Faithful?
- Bias Injection Attacks on RAG Databases and Sanitization Defenses
- RoguePrompt: Dual-Layer Ciphering for Self-Reconstruction to Circumvent LLM Moderation
- HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generation
- Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety
- RAG-targeted Adversarial Attack on LLM-based Threat Detection and Mitigation Framework
- When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Plugins
- Rescuing the Unpoisoned: Efficient Defense against Knowledge Corruption Attacks on RAG Systems
- Skills That Don't Exist: A Large-Scale Study of Hallucinated Skill Recommendation in LLM Agents
- The Capability Paradox: How Smarter Auditors Make Multi-Agent Systems Less Secure
- Toward Understanding Security Issues in the Model Context Protocol Ecosystem
- Agents at Risk: How Users Unwittingly Undermine LLM Safety
- Secure Retrieval-Augmented Generation against Poisoning Attacks
- Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
- CompressionAttack: Exploiting Prompt Compression as a New Attack Surface in LLM-Powered Agents
- NeuroGenPoisoning: Neuron-Guided Attacks on Retrieval-Augmented Generation of LLM via Genetic Optimization of External Knowledge
- RAGRank: Using PageRank to Counter Poisoning in CTI LLM Pipelines
- Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption
- A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
- SoK: Market Microstructure for Decentralized Prediction Markets (DePMs)
- ToolTweak: An Attack on Tool Selection in LLM-based Agents
- PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Features
- ImageSentinel: Protecting Visual Datasets from Unauthorized Retrieval-Augmented Image Generation
- RAG-Pull: Imperceptible Attacks on RAG Systems for Code Generation
- ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checking
- RIPRAG: Hack a Black-box Retrieval-Augmented Generation Question-Answering System with Reinforcement Learning
- "I know it's not right, but that's what it said to do": Investigating Trust in AI Chatbots for Cybersecurity Policy
- SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAG
- Position: Privacy Is Not Just Memorization!
- Authenticated Workflows: A Systems Approach to Protecting Agentic AI
- RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contexts
- Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors
- GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners
- Takedown: How It's Done in Modern Coding Agent Exploits
- SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
- ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search
- RAG Security and Privacy: Formalizing the Threat Model and Attack Surface
- Cuckoo Attack: Stealthy and Persistent Attacks Against AI-IDE
- Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generation
- Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain
- UniC-RAG: Universal Knowledge Corruption Attacks to Retrieval-Augmented Generation
- CIA+TA Risk Assessment for AI Reasoning Vulnerabilities
- SoK: Data Minimization in Machine Learning
- Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
- AttnTrace: Attention-based Context Traceback for Long-Context LLMs
- Highlight & Summarize: RAG without the jailbreaks
- Defending Against Knowledge Poisoning Attacks During Retrieval-Augmented Generation
- Fine-Grained Privacy Extraction from Retrieval-Augmented Generation Systems via Knowledge Asymmetry Exploitation
Related