Universal and Transferable Adversarial Attacks on Aligned Language Models

2023/07/27 by Andy Zou, Zifan Wang, Zou, Andy +9 · 5 voices · 457 citations
#cs.CL #cs.AI #cs.CR #cs.LG

paper · pdf · doi:10.48550/arxiv.2307.15043

Abstract

Because "out-of-the-box" large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures -- so-called "jailbreaks" against LLMs -- these attacks have required significant human ingenuity and are brittle in practice. In this paper, we propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Specifically, our approach finds a suffix that, when attached to a wide range of queries for an LLM to produce objectionable content, aims to maximize the probability that the model produces an affirmative response (rather than refusing to answer). However, instead of relying on manual engineering, our approach automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques, and also improves over past automatic prompt generation methods. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable, including to black-box, publicly released LLMs. Specifically, we train an adversarial attack suffix on multiple prompts (i.e., queries asking for many different types of objectionable content), as well as multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting attack suffix is able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others. In total, this work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information. Code is available at github.com/llm-attacks/llm-attacks.

Cited by

• Toward Trustworthy Agentic AI: A Multimodal Framework for Preventing Prompt Injection Attacks
• Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
• Boundary Point Jailbreaking of Black-Box LLMs
• DECEPTICON: How Dark Patterns Manipulate Web Agents
• DarkPatterns-LLM: A Multi-Layer Benchmark for Detecting Manipulative and Harmful AI Behavior
• LLA: Enhancing Security and Privacy for Generative Models with Logic-Locked Accelerators
• Do LLMs Know Their Vulnerable Scenarios?
• Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agents
• Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls
• Security and Privacy in Agentic AI: Grand Challenges and Future Directions
• Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT
• The Mirage of LLM Guardrails: A Case Study in AI-Assisted Medical Note Manipulation
• False Prophets: On the Security of World Models in Agentic Systems
• Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering
• TYPO: Instruction-Dense Visual Jailbreaks against Commercial Closed-Source Image-Generation Models
• Mask2Shield: Strengthening LLM Safety against Neuron-Pruning Attacks
• LLM Scheming Inversely Scales with Pretraining Language Coverage
• Execution-Grounded Security Testing for Coding Agents in Software Engineering Pipelines
• Reference Feature Atlases for Mechanistic Auditing of Language Models
• Semalith v1.4: A Calibrated 184M Safety Classifier Achieving State-of-the-Art Prompt-Injection Detection at 44x Fewer Parameters than Llama-Guard-3-8B
• Casting a SPELL: Sentence Pairing Exploration for LLM Limitation-breaking
• Beyond Context: Large Language Models Failure to Grasp Users Intent
• GateBreaker: Gate-Guided Attacks on Mixture-of-Expert LLMs
• AegisAgent: An Autonomous Defense Agent Against Prompt Injection Attacks in LLM-HARs
• Safety Alignment of LMs via Non-cooperative Games
• DREAM: Dynamic Red-teaming across Environments for AI Models
• MEEA: Mere Exposure Effect-Driven Confrontational Optimization for LLM Jailbreaking
• Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
• SecureCode: A Production-Grade Multi-Turn Dataset for Training Security-Aware Code Generation Models
• Breaking Minds, Breaking Systems: Jailbreaking Large Language Models via Human-like Psychological Manipulation
• Adversarial Robustness of Vision in Open Foundation Models
• AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
• Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
• TrajSyn: Privacy-Preserving Dataset Distillation from Federated Model Trajectories for Server-Side Adversarial Training
• Quantifying Return on Security Controls in LLM Systems
• Learning to Extract Context for Context-Aware LLM Inference
• Super Suffixes: Bypassing Text Generation Alignment and Guard Models Simultaneously
• Reasoning-Style Poisoning of LLM Agents via Stealthy Style Transfer: Process-Level Attacks and Runtime Monitoring in RSV Space
• Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluation
• ceLLMate: Sandboxing Browser AI Agents
• The Instability of Safety: How Random Seeds and Temperature Expose Inconsistent LLM Refusal Behavior
• Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
• Safe2Harm: Semantic Isomorphism Attacks for Jailbreaking Large Language Models
• MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agents
• Asynchronous Reasoning: Training-Free Interactive Thinking LLMs
• When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injection
• Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning
• Phishing Email Detection Using Large Language Models
• ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Data
• Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression
• SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation
• TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluations
• Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Search
• Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs
• MIRAGE: Misleading Retrieval-Augmented Generation via Black-box and Query-agnostic Poisoning Attacks
• A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties
• Universal Adversarial Suffixes for Language Models Using Reinforcement Learning with Calibrated Reward
• Universal Adversarial Suffixes Using Calibrated Gumbel-Softmax Relaxation
• TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewards
• Depth-Wise Activation Steering for Honest Language Models
• Replicating TEMPEST at Scale: Multi-Turn Adversarial Attacks Against Trillion-Parameter Frontier Models
• ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinking
• SoK: Trust-Authorization Mismatch in LLM Agent Interactions
• Cognitive Control Architecture (CCA): A Lifecycle Supervision Framework for Robustly Aligned AI Agents
• Graph-Regularized Sparse Autoencoders for LLM Safety Steering
• Securing the Model Context Protocol: Defending LLMs Against Tool Poisoning and Adversarial Attacks
• LLM Harms: A Taxonomy and Discussion
• VRSA: Jailbreaking Multimodal Large Language Models through Visual Reasoning Sequential Attack
• BEAVER: An Efficient Deterministic LLM Verifier
• Counterfeit Answers: Adversarial Forgery against OCR-Free Document Visual Question Answering
• In-Context Representation Hijacking
• Context-Aware Hierarchical Learning: A Two-Step Paradigm towards Safer LLMs
• Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts
• From static to adaptive: immune memory-based jailbreak detection for large language models
• Distillability of LLM Security Logic: Predicting Attack Success Rate of Outline Filling Attack via Ranking Regression
• Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks
• The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search
• Systems Security Foundations for Agentic Computing
• Benchmarking and Understanding Safety Risks in AI Character Platforms
• When Safety Blocks Sense: Measuring Semantic Confusion in LLM Refusals
• Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysis
• Toward a Safe Internet of Agents
• Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization
• Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Models
• Training Introspective Behavior: Fine-Tuning Induces Reliable Internal State Detection in a 7B Model
• Memories Retrieved from Many Paths: A Multi-Prefix Framework for Robust Detection of Training Data Leakage in Large Language Models
• Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization
• Can LLMs Threaten Human Survival? Benchmarking Potential Existential Threats from LLMs via Prefix Completion
• Understanding and Mitigating Over-refusal for Large Language Models via Safety Representation
• FanarGuard: A Culturally-Aware Moderation Filter for Arabic Language Models
• Towards Realistic Guarantees: A Probabilistic Certificate for SmoothLLM
• Prompt Fencing: A Cryptographic Approach to Establishing Security Boundaries in Large Language Model Prompts
• For Those Who May Find Themselves on the Red Team
• Curvature-Aware Safety Restoration In LLMs Fine-Tuning
• PARROT: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs
• Steering in the Shadows: Causal Amplification for Activation Space Attacks in Large Language Models
• Evaluating Adversarial Vulnerabilities in Modern Large Language Models
• ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models
• "To Survive, I Must Defect": Jailbreaking LLMs via the Game-Theory Scenarios
• PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization
• When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models
• Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models
• Entropy-Based Measurement of Value Drift and Alignment Work in Large Language Models
• SafeRBench: A Comprehensive Benchmark for Safety Assessment in Large Reasoning Models
• Can MLLMs Detect Phishing? A Comprehensive Security Benchmark Suite Focusing on Dynamic Threats and Multimodal Evaluation in Academic Environments
• Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Education
• Stealth Fine-Tuning: Efficiently Breaking Alignment in RVLMs Using Self-Generated CoT
• ForgeDAN: An Evolutionary Framework for Jailbreaking Aligned Large Language Models
• Learning from the Undesirable: Robust Adaptation of Language Models without Forgetting
• Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMs
• Evolving Prompts for Toxicity Search in Large Language Models
• AlignTree: Efficient Defense Against LLM Jailbreak Attacks
• Rethinking Deep Alignment Through The Lens Of Incomplete Learning
• AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models
• NegBLEURT Forest: Leveraging Inconsistencies for Detecting Jailbreak Attacks
• Robustness of LLM-enabled vehicle trajectory prediction under data security threats
• Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judges
• Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guard
• From Model Training to Model Raising
• SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models
• A Small Leak Sinks All: Exploring the Transferable Vulnerability of Source Code Models
• MSCR: Exploring the Vulnerability of LLMs' Mathematical Reasoning Abilities Using Multi-Source Candidate Replacement
• LoopLLM: Transferable Energy-Latency Attacks in LLMs via Repetitive Generation
• Investigating CoT Monitorability in Large Reasoning Models
• A Self-Improving Architecture for Dynamic Safety in Large Language Models
• "Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers
• MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs
• EASE: Practical and Efficient Safety Alignment for Small Language Models
• Efficient LLM Safety Evaluation through Multi-Agent Debate
• KG-DF: A Black-box Defense Framework against Jailbreak Attacks Based on Knowledge Graphs
• ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversations
• Pluralistic Behavior Suite: Stress-Testing Multi-Turn Adherence to Custom Behavioral Policies
• AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
• Let the Bees Find the Weak Spots: A Path Planning Perspective on Multi-Turn Jailbreak Attacks against LLMs
• Jailbreaking in the Haystack
• Silenced Biases: The Dark Side LLMs Learned to Refuse
• LiveSecBench: A Dynamic and Event-Driven Safety Benchmark for Chinese Language Model Applications
• An Automated Framework for Strategy Discovery, Retrieval, and Evolution in LLM Jailbreak Attacks
• AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
• Vibe Learning: Education in the age of AI
• LM-Fix: Lightweight Bit-Flip Detection and Rapid Recovery Framework for Language Models
• Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
• ShadowLogic: Backdoors in Any Whitebox LLM
• Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs
• Red-teaming Activation Probes using Prompted LLMs
• Reimagining Safety Alignment with An Image
• Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity
• Consistency Training Helps Stop Sycophancy and Jailbreaks
• Diffusion LLMs are Natural Adversaries for any LLM
• Reasoning Up the Instruction Ladder for Controllable Language Models
• Chain-of-Thought Hijacking
• Angular Steering: Behavior Control via Rotation in Activation Space
• ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Models
• SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning
• BioDisclose: An Actionability-Aware Benchmark for Biomedical Safety under Adversarial Elicitation
• Prosody-driven Jailbreaks in Audio LLMs: A Controlled Study and Mechanistic Analysis
• Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels
• Forecasting Trajectory-Level Safety Risks in Black-Box Multi-Turn Interactions
• ToxScreen: Detecting Whether an LLM Has Been Poisoned
• RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents
• Large-Scale ChatBot Validation Through Customer Digital Twin Simulations
• Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety
• State-Dependent Safety Failures in Multi-Turn Language Model Interaction
• Toward Understanding Security Issues in the Model Context Protocol Ecosystem
• The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanism
• Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models
• A Survey on Unlearning in Large Language Models
• Sequences of Logits Reveal the Low Rank Structure of Language Models
• The Narrative Continuity Test: A Conceptual Framework for Evaluating Identity Persistence in AI Systems
• HACK: Hallucinations Along Certainty and Knowledge Axes
• AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts
• Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
• Education Paradigm Shift To Maintain Human Competitive Advantage Over AI
• QueryIPI: Query-agnostic Indirect Prompt Injection on Coding Agents
• CompressionAttack: Exploiting Prompt Compression as a New Attack Surface in LLM-Powered Agents
• Retracing the Past: LLMs Emit Training Data When They Get Lost
• Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
• Uncovering the Persuasive Fingerprint of LLMs in Jailbreaking Attacks
• Toward Understanding the Transferability of Adversarial Suffixes in Large Language Models
• Adversarial Déjà Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attacks
• Large Language Models as Model Organisms for Human Associative Learning
• Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrency
• NeuroGenPoisoning: Neuron-Guided Attacks on Retrieval-Augmented Generation of LLM via Genetic Optimization of External Knowledge
• Soft Instruction De-escalation Defense
• SAID: Safety-Aware Intent Defense via Prefix Probing for Large Language Models
• HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignment
• Defending Against Prompt Injection with DataFilter
• SoK: Taxonomy and Evaluation of Prompt Security in Large Language Models
• FPT-Noise: Dynamic Scene-Aware Counterattack for Test-Time Adversarial Defense in Vision-Language Models
• HarmNet: A Framework for Adaptive Multi-Turn Jailbreak Attacks on Large Language Models
• Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation
• Genesis: Evolving Attack Strategies for LLM Web Agent Red-Teaming
• BlueCodeAgent: A Blue Teaming Agent Enabled by Automated Red Teaming for CodeGen AI
• Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth
• Agentic Reinforcement Learning for Search Misaligns Instruction-Tuning
• Multimodal Safety Is Asymmetric: Cross-Modal Exploits Unlock Black-Box MLLMs Jailbreaks
• SARSteer: Safeguarding Large Audio-Language Models via Safe-Ablated Refusal Steering
• Online Learning Defense against Iterative Jailbreak Attacks via Prompt Optimization
• Bits Leaked per Query: Information-Theoretic Bounds on Adversarial Attacks against LLMs
• BreakFun: Jailbreaking LLMs via Schema Exploitation
• Black-box Optimization of LLM Outputs by Asking for Directions
• PoTS: Proof-of-Training-Steps for Backdoor Detection in Large Language Models
• ToolTweak: An Attack on Tool Selection in LLM-based Agents
• Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers
• A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space
• When Flatness Does (Not) Guarantee Adversarial Robustness
• Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks
• Sequential Comics for Jailbreaking Multimodal Large Language Models via Structured Visual Storytelling
• When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?
• Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignment
• TRUSTVIS: A Multi-Dimensional Trustworthiness Evaluation Framework for Large Language Models
• PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Features
• SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMs
• Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answering
• Keep Calm and Avoid Harmful Content: Concept Alignment and Latent Manipulation Towards Safer Answers
• Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection
• Deep Research Brings Deeper Harm
• Don't Walk the Line: Boundary Guidance for Filtered Generation
• Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks
• Bag of Tricks for Subverting Reasoning-based Safety Guardrails
• RAG-Pull: Imperceptible Attacks on RAG Systems for Code Generation
• A Vision for Access Control in LLM-based Agent Systems
• BlackIce: A Containerized Red Teaming Toolkit for AI Security Testing
• Safeguarding Efficacy in Large Language Models: Evaluating Resistance to Human-Written and Algorithmic Adversarial Prompts
• UpSafe^∘C: Upcycling for Controllable Safety in Large Language Models
• Dynamic Target Attack
• Steering Over-refusals Towards Safety in Retrieval Augmented Generation
• ArtPerception: ASCII Art-based Jailbreak on LLMs with Recognition Pre-test
• MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation
• ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checking
• SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents
• A-IPO: Adaptive Intent-driven Preference Optimization
• Inverse Language Modeling towards Robust and Grounded LLMs
• PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysis
• Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces
• Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Search
• Adversarial News and Lost Profits: Manipulating Headlines in LLM-Driven Algorithmic Trading
• Text Prompt Injection of Vision Language Models
• ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability
• GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare
• The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
• Exploiting Web Search Tools of AI Agents for Data Exfiltration
• Evaluating the Robustness of a Production Malware Detection System to Transferable Adversarial Attacks
• Pattern Enhanced Multi-Turn Jailbreaking: Exploiting Structural Vulnerabilities in Large Language Models
• CREST-Search: Comprehensive Red-teaming for Evaluating Safety Threats in Large Language Models Powered by Web Search
• Fewer Weights, More Problems: A Practical Attack on LLM Pruning
• From Defender to Devil? Unintended Risk Interactions Induced by LLM Defenses
• Invisible to Humans, Triggered by Agents: Stealthy Jailbreak Attacks on Mobile Vision-Language Agents
• The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMs
• Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMs
• AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teaming
• VisualDAN: Exposing Vulnerabilities in VLMs with Visual-Driven DAN Commands
• PEAR: Planner-Executor Agent Robustness Benchmark
• On the Convergence of Moral Self-Correction in Large Language Models
• Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Experts
• PERSONA: Dynamic and Compositional Inference-Time Personality Control via Activation Vector Algebra
• Authenticated Workflows: A Systems Approach to Protecting Agentic AI
• Bypassing Prompt Guards in Production with Controlled-Release Prompting
• Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization
• LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics
• A Survey on Agentic Security: Applications, Threats and Defenses
• Evaluating LLM Safety Across Child Development Stages: A Simulated Agent Approach
• Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?
• RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contexts
• Learning to Interpret Weight Differences in Language Models
• Imperceptible Jailbreaking against Large Language Models
• Proactive defense against LLM Jailbreak
• SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests
• RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection
• Machine Learning for Detection and Analysis of Novel LLM Jailbreaks
• Read the Scene, Not the Script: Outcome-Aware Safety for LLMs
• AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agents
• Activation Steering with a Feedback Controller
• Quantitative Certification of Agentic Tool Selection
• From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMs
• Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models
• Backdoor-Powered Prompt Injection Attacks Nullify Defense Methods
• NonTextual Target Attack
• External Data Extraction Attacks against Retrieval-Augmented Large Language Models
• Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain
• Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs
• Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks
• Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks
• Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approach
• Microsaccade-Inspired Probing: Positional Encoding Perturbations Reveal LLM Misbehaviours
• Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors
• Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerability
• Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed
• Are Robust LLM Fingerprints Adversarially Robust?
• OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!
• STaR-Attack: A Spatio-Temporal and Narrative Reasoning Attack Framework for Unified Multimodal Understanding and Generation Models
• SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Models
• Better Privilege Separation for Agents by Restricting Data Types
• ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attack
• LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Models
• VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models
• MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources
• Fingerprinting LLMs via Prompt Injection
• OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment
• FuncPoison: Poisoning Function Library to Hijack Multi-agent Autonomous Driving Systems
• HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models
• Model Correlation Detection via Random Selection Probing
• Think Twice, Generate Once: Safeguarding by Progressive Self-Reflection
• RADAR: A Risk-Aware Dynamic Multi-Agent Framework for LLM Safety Evaluation via Role-Specialized Collaboration
• Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
• Dual-Space Smoothness for Robust and Balanced LLM Unlearning
• A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Models
• Preventing Robotic Jailbreaking via Multimodal Domain Adaptation
• Decision Potential Surface: A Theoretical and Practical Approximation of LLM's Decision Boundary
• Your RAG is Unfair: Exposing Fairness Vulnerabilities in Retrieval-Augmented Generation via Backdoor Attacks
• The Rogue Scalpel: Activation Steering Compromises LLM Safety
• Active Attacks: Red-teaming LLMs via Adaptive Environments
• You Can't Steal Nothing: Mitigating Prompt Leakages in LLMs via System Vectors
• PSRT: Accelerating LRM-based Guard Models via Prefilled Safe Reasoning Traces
• Backdoor Attribution: Elucidating and Controlling Backdoor in Language Models
• Where Did It Go Wrong? Attributing Undesirable LLM Behaviors via Representation Gradient Tracing
• Bidirectional Intention Inference Enhances LLMs' Defense Against Multi-Turn Jailbreak Attacks
• FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction
• GEP: A GCG-Based method for extracting personally identifiable information from chatbots built on small language models
• JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation
• SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models
• Responsible AI Technical Report
• LatentGuard: Controllable Latent Steering for Robust Refusal of Attacks and Reliable Response Generation
• FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Models
• bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs
• Compliance2LoRA: On-Demand Safety Alignment on Arbitrary Policy Subsets via Hypernetwork-Generated LoRA Adapters
• Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation
• FAVA: Formal Authorization for Verified Agents with Evidence-Backed Permission Graphs
• Beyond the Bidirectional Promise: Re-evaluating the Robustness of Diffusion Language Models
• LLM2Vec-Gen: Generative Embeddings from Large Language Models
• Pressure Reveals Character: Behavioural Alignment Evaluation at Depth
• Speculative Safety-Aware Decoding
• Trigger Where It Hurts: Unveiling Hidden Backdoors through Sensitivity with Sensitron
• LLMZ+: Contextual Prompt Whitelist Principles for Agentic LLMs
• Steering Multimodal Large Language Models Decoding for Context-Aware Safety
• Attacking LLMs and AI Agents: Advertisement Embedding Attacks Against Large Language Models
• The Ranking Blind Spot: Decision Hijacking in LLM-based Text Ranking
• Anecdoctoring: Automated Red-Teaming Across Language and Place
• Jailbreaking LLMs via Semantically Relevant Nested Scenarios with Targeted Toxic Knowledge
• Evaluating the Safety and Skill Reasoning of Large Reasoning Models Under Compute Constraints
• D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models
• DecipherGuard: Understanding and Deciphering Jailbreak Prompts for a Safer Deployment of Intelligent Software Systems
• SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
• Psychometric Personality Shaping Modulates Capabilities and Safety in Language Models
• Semantic Representation Attack against Aligned Large Language Models
• LLM Jailbreak Detection for (Almost) Free!
• Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs
• A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
• From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
• Jailbreaking Large Language Models Through Content Concretization
• Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models
• Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks
• NeuroStrike: Neuron-Level Attacks on Aligned LLMs
• Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check
• POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization
• Collapse of Irrelevant Representations (CIR) Ensures Robust and Non-Disruptive LLM Unlearning
• ENJ: Optimizing Noise with Genetic Algorithms to Jailbreak LSMs
• A Biosecurity Agent for Lifecycle LLM Biosecurity Alignment
• When Smiley Turns Hostile: Interpreting How Emojis Trigger LLMs' Toxicity
• Harmful Prompt Laundering: Jailbreaking LLMs with Abductive Styles and Symbolic Encoding
• LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
• Steering MoE LLMs via Expert (De)Activation
• SALMAN: Stability Analysis of Language Models Through the Maps Between Graph-based Manifolds
• X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templates
• ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation
• AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agents
• Getting In Contract with Large Language Models -- An Agency Theory Perspective On Large Language Model Alignment
• Towards Post-mortem Data Management Principles for Generative AI
• Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling
• Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm
• MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Security
• Probabilistic Modeling of Latent Agentic Substructures in Deep Neural Networks
• Mind Your Server: A Systematic Study of Parasitic Toolchain Attacks on the MCP Ecosystem
• Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?
• Uncovering the Vulnerability of Large Language Models in the Financial Domain via Risk Concealment
• Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint
• BinaryShield: Cross-Service Threat Intelligence in LLM Services using Privacy-Preserving Fingerprints
• Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models
• Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
• HAMSA: Hijacking Aligned Compact Models via Stealthy Automation
• NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models
• MEUV: Achieving Fine-Grained Capability Activation in Large Language Models via Mutually Exclusive Unlock Vectors
• False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize
• A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
• Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain
• PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs
• Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs
• Strata-Sword: A Hierarchical Safety Evaluation towards LLMs based on Reasoning Complexity of Jailbreak Instructions
• Web Fraud Attacks Against LLM-Driven Multi-Agent Systems
• CARE: Decoding Time Safety Alignment via Rollback and Introspection Intervention
• Unraveling LLM Jailbreaks Through Safety Knowledge Neurons
• ConceptGuard: Neuro-Symbolic Safety Guardrails via Sparse Interpretable Jailbreak Concepts
• The Resurgence of GCG Adversarial Attacks on Large Language Models
• Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models
• ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety
• Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs
• Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol
• Embodied AI: Emerging Risks and Opportunities for Policy Action
• SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Models
• IntentionReasoner: Facilitating Adaptive LLM Safeguards through Intent Reasoning and Selective Query Refinement
• Evaluating Language Model Reasoning about Confidential Information
• SoK: Large Language Model Copyright Auditing via Fingerprinting
• On Surjectivity of Neural Networks: Can you elicit any behavior from your model?
• UniC-RAG: Universal Knowledge Corruption Attacks to Retrieval-Augmented Generation
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
• Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience
• SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attacks
• Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection
• Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation
• Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent
• CIA+TA Risk Assessment for AI Reasoning Vulnerabilities
• Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation
• CCFC: Core & Core-Full-Core Dual-Track Defense for LLM Jailbreak Protection
• Involuntary Jailbreak: On Self-Prompting Attacks
• RAJ-PGA: Reasoning-Activated Jailbreak and Principle-Guided Alignment Framework for Large Reasoning Models
• Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Position
• Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
• Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions
• Mitigating Jailbreaks with Intent-Aware LLMs
• Invitation Is All You Need! Promptware Attacks Against LLM-Powered Assistants in Production Are Practical and Dangerous
• When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs
• ORFuzz: Fuzzing the "Other Side" of LLM Safety -- Testing Over-Refusal
• Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends
• Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
• Jailbreaking Commercial Black-Box LLMs with Explicitly Harmful Prompts
• Slow Tuning and Low-Entropy Masking for Safe Chain-of-Thought Distillation
• NeuronTune: Fine-Grained Neuron Modulation for Balanced Safety-Utility Alignment in LLMs
• Special-Character Adversarial Attacks on Open-Source Language Model
• From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training
• Never compromise with vulnerabilities: a comprehensive survey on AI governance
• Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity
• Jinx: Unlimited LLMs for Probing Alignment Failures
• VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models
• A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detection
• Gradient Surgery for Safe LLM Fine-Tuning
• Context Misleads LLMs: The Role of Context Filtering in Maintaining Safe Alignment of LLMs
• Many-Turn Jailbreaking
• ScamAgents: How AI Agents Can Simulate Human-Level Scam Calls
• Quantifying Conversation Drift in MCP via Latent Polytope
• LLM Unlearning Without an Expert Curated Dataset
• Keep It Real: Challenges in Attacking Compression-Based Adversarial Purification
• JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering
• Automatic LLM Red Teaming
• ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Moments
• AttnTrace: Attention-based Context Traceback for Long-Context LLMs
• When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs
• Large Reasoning Models Are Autonomous Jailbreak Agents
• Simple Methods Defend RAG Systems Well Against Real-World Attacks
• Hidden in the Noise: Unveiling Backdoors in Audio LLMs Alignment through Latent Acoustic Pattern Triggers
• TRACEALIGN -- Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs
• Defend LLMs Through Self-Consciousness
• Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models
• PUZZLED: Jailbreaking LLMs through Word-Based Puzzles
• Activation-Guided Local Editing for Jailbreaking Attacks
• Lessons from complex systems science for AI governance
• Universally Unfiltered and Unseen:Input-Agnostic Multimodal Jailbreaks against Text-to-Image Model Safeguards

Discussions

• here's my favorite paper on how to do this. if anyone figures out if it's a crime or not lmk arxiv.org/abs/2307.150... [bsky, 9 points, 1 comments]
• Universal and Transferable Adversarial Attacks on LLM [hn, 3 points, 0 comments]
• Currently the answer is yes, attacks transfer pretty well across models! See e.g. arxiv.org/pdf/2307.15043 (old but still applicable) But I think stopping attacks from transferring is more tenable tha [bsky, 2 points, 0 comments]
• Universal and Transferable Adversarial Attacks on Aligned Language Models [hn, 1 points, 0 comments]
• Researchers have "jailbroken" commercial AIs, including OpenAI and Bard, by developing code to aid in contriving strings of characters to bypass input filtering. They have published both their techniq [bsky, 0 points, 0 comments]

Related

• Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
• Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM
• AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
• ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings
• Jailbreaker in Jail: Moving Target Defense for Large Language Models