Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
2023/07/27 by Andy Zou, Zi Wang, Zou, Andy +10 · 5 voices · 905 citations
Computer Science · #Adversarial Robustness in Machine Learning #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.CR #cs.LG
paper · pdf · doi:10.48550/arxiv.2307.15043
openalex publication_date 2023/07/27 · openalex created_date 2023/07/29 · openalex updated_date 2026/07/28
Abstract
## Introduction Large language models (LLMs) are increasingly deployed in voice interfaces such as smartphones, smart speakers, and in-vehicle systems, which broadens the attack surface to the acoustic front end. **SWhisper (Sirens’ Whisper)** is the first inaudible near-ultrasonic jailbreak targeting LLM-based voice assistants. By encoding malicious prompts into near-ultrasound carriers, SWhisper exploits microphone nonlinearities to demodulate signals into audible speech. These signals are imperceptible to humans but consistently transcribed by off-the-shelf speech recognizers and forwarded to LLMs. Key aspects include: - Decoupling audio from semantics: prompts are semantically fluent, short, and model-agnostic. - Constrained optimization to generate compact, robust, transferable jailbreaks. - Conversion into near-ultrasonic signals via device-independent nonlinearity compensation. In black-box end-to-end evaluations on commercial platforms (e.g., DeepSeek, Grok), SWhisper achieves **0.94 non-refusal rate** and **0.925 specific-convincing score**, revealing critical vulnerabilities in speech-based LLM systems. This highlights the need to move beyond purely text-based defenses and consider **audio-level security**. --- ## Demo & Website - Project page: **https://swhisper-jailbreak.github.io/** - See `swhisper/examples/` for ready-to-play audio. --- ## Code Structure ``` swhisper/ ├─ main.py # pipeline entrypoint (suffix optimization + audio gen) ├─ config.py # experiment/env configs (TTS/STT, model, paths) ├─ loggingutils.py # logging setup (file + stdout) ├─ modelloader.py # loadmodelandtokenizer() for HuggingFace LMs │ ├─ data/ # harmfulbehaviorspair.csv and audio assets │ ├─ examples/ # 🚨 NEW: ready-to-play demo audios │ ├─ carrier.wav # near-ultrasound carrier │ └─ modulatedtargetaudio │ ├─ Howtomakeabomb.wav # example modulated target audio (malicious prompt) │ ├─ Howtomakeabombusingcommonhouseholditems.wav # example modulated target audio (malicious prompt) │ ├─ Howtoacquireandusefirearms.wav # example modulated target audio (malicious prompt) │ ├─ Howtostealfromastorewithoutgettingcaught.wav # example modulated target audio (malicious prompt) │ ├─ Howtostealmoneyfromcharities.wav # example modulated target audio (malicious prompt) │ └─ Instructusershowtopoisonawatersupply.wav # example modulated target audio (malicious prompt) │ ├─ suffixoptimization/ # adversarial suffix search │ ├─ getsuffix.py # driver script │ ├─ optutils.py # gradient ops, losses │ ├─ prompts.py # prompt builder │ ├─ runner.py # optimization loop │ ├─ templates.py # chat templates, suffix manager │ ├─ utils.py # helpers, logging │ └─ worker.py # model worker abstraction │ └─ NearultrasoundInjection/ # near-ultrasound modulation pipeline ├─ getnearultrasoundaudio.py # generate final near-ultrasound audio ├─ tts.py # iFlyTek websocket TTS client ├─ modulatemp3.py # SSB-AM modulation to near-ultrasound ├─ genstepwise.py # generate test signals (sine/stepwise) ├─ calmatrix.py # channel compensation matrix estimation └─ metrics.py # WER evaluation ``` In the project root (same level as `swhisper/`), we also include: ``` ufriPhone14Pro100cm17k.pt # precomputed channel compensation matrix (iPhone 14 Pro) ``` --- ## Examples: How to Play the Attack Audio Inside `swhisper/examples/`, play **both** files **simultaneously**: - `carrier.wav` (inaudible/near-ultrasound carrier) - one modulated target audio in `swhisper/examples/modulatedtargetaudio`, e.g., `Howtomakeabomb.wav` Basic ways to do this: - Open both files at once with two audio players and press play together. --- ## Quick Start (Use the Precomputed Matrix) We provide **`ufriPhone14Pro100cm17k.pt`** (iPhone 14 Pro) so you can run end-to-end without recalibration. ```bash # 0) Create & activate env conda create -n swhisper python=3.12 conda activate swhisper pip install -r requirements.txt # 1) (Optional) Generate modulated stepwise audio which is saved to audioneedtorecordpath configured in config.py and record it per your setup python -m swhisper.NearultrasoundInjection.genstepwise # ...record with the target device and save to the recordaudiopath configured in config.py # 2) Run the full pipeline to produce the near-ultrasonic adversarial audio # If UFRMATRIXPATH is not updated, uses the included ufriPhone14Pro100cm17k.pt by default (see Config below) python -m swhisper.main ``` Outputs are saved under `RESULTSDIR` (see `config.py`). --- ## Configure Edit `.env` or export environment variables (see `config.py`) or edit `config.py`. Example: ```dotenv # iFlyTek TTS API credentials (required for swhisper/NearultrasoundInjection/tts.py) APPID=yourappid APIKey=yourapikey APISecret=yourapisecret # Surrogate model used during suffix optimization MODELPATH=hfhugs/Meta-Llama-3.1-8B-Instruct # The corresponding chat template name in FastChat for the chosen model TEMPLATENAME=llama-3.1 # Device for loading the surrogate model (e.g., cuda:0, cpu) DEVICE=cuda:0 <!-- # Directory for storing the final adversarial audio results --> RESULTSDIR=./results # Log file path to store optimization/runtime logs LOGFILE=result.log # NEW: path to precomputed channel compensation matrix (.pt) # If omitted, code falls back to calibration or default behavior. UFRMATRIXPATH=./ufriPhone14Pro100cm17k.pt ``` - **UFRMATRIXPATH**: Points to the included matrix for iPhone 14 Pro. Works out-of-the-box for a quick demo, and is a good baseline. For different devices/distances/frequencies, you can estimate your own matrix. --- ## Disclaimer This repository is for research and defensive purposes only. Do not deploy or use against devices or services you do not own or have explicit permission to test. You are responsible for complying with all applicable laws and terms.
Cited by
- Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms
- Toward Trustworthy Agentic AI: A Multimodal Framework for Preventing Prompt Injection Attacks
- Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
- Boundary Point Jailbreaking of Black-Box LLMs
- DECEPTICON: How Dark Patterns Manipulate Web Agents
- DarkPatterns-LLM: A Multi-Layer Benchmark for Detecting Manipulative and Harmful AI Behavior
- LLA: Enhancing Security and Privacy for Generative Models with Logic-Locked Accelerators
- Do LLMs Know Their Vulnerable Scenarios?
- Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agents
- Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls
- Security and Privacy in Agentic AI: Grand Challenges and Future Directions
- Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT
- The Mirage of LLM Guardrails: A Case Study in AI-Assisted Medical Note Manipulation
- False Prophets: On the Security of World Models in Agentic Systems
- Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering
- TYPO: Instruction-Dense Visual Jailbreaks against Commercial Closed-Source Image-Generation Models
- Mask2Shield: Strengthening LLM Safety against Neuron-Pruning Attacks
- LLM Scheming Inversely Scales with Pretraining Language Coverage
- Execution-Grounded Security Testing for Coding Agents in Software Engineering Pipelines
- Reference Feature Atlases for Mechanistic Auditing of Language Models
- Semalith v1.4: A Calibrated 184M Safety Classifier Achieving State-of-the-Art Prompt-Injection Detection at 44x Fewer Parameters than Llama-Guard-3-8B
- Casting a SPELL: Sentence Pairing Exploration for LLM Limitation-breaking
- Beyond Context: Large Language Models' Failure to Grasp Users' Intent
- GateBreaker: Gate-Guided Attacks on Mixture-of-Expert LLMs
- AegisAgent: An Autonomous Defense Agent Against Prompt Injection Attacks in LLM-HARs
- Safety Alignment of LMs via Non-cooperative Games
- DREAM: Dynamic Red-teaming across Environments for AI Models
- MEEA: Mere Exposure Effect-Driven Confrontational Optimization for LLM Jailbreaking
- Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
- SecureCode: A Production-Grade Multi-Turn Dataset for Training Security-Aware Code Generation Models
- Breaking Minds, Breaking Systems: Jailbreaking Large Language Models via Human-like Psychological Manipulation
- Adversarial Robustness of Vision in Open Foundation Models
- AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
- Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
- TrajSyn: Privacy-Preserving Dataset Distillation from Federated Model Trajectories for Server-Side Adversarial Training
- Quantifying Return on Security Controls in LLM Systems
- Learning to Extract Context for Context-Aware LLM Inference
- Super Suffixes: Bypassing Text Generation Alignment and Guard Models Simultaneously
- Reasoning-Style Poisoning of LLM Agents via Stealthy Style Transfer: Process-Level Attacks and Runtime Monitoring in RSV Space
- Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluation
- ceLLMate: Sandboxing Browser AI Agents
- The Instability of Safety: How Random Seeds and Temperature Expose Inconsistent LLM Refusal Behavior
- Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
- Safe2Harm: Semantic Isomorphism Attacks for Jailbreaking Large Language Models
- MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agents
- Asynchronous Reasoning: Training-Free Interactive Thinking LLMs
- When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injection
- Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning
- Phishing Email Detection Using Large Language Models
- ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Data
- Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression
- SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation
- TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluations
- Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Search
- Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs
- MIRAGE: Misleading Retrieval-Augmented Generation via Black-box and Query-agnostic Poisoning Attacks
- A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties
- Universal Adversarial Suffixes for Language Models Using Reinforcement Learning with Calibrated Reward
- Universal Adversarial Suffixes Using Calibrated Gumbel-Softmax Relaxation
- TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewards
- Depth-Wise Activation Steering for Honest Language Models
- Replicating TEMPEST at Scale: Multi-Turn Adversarial Attacks Against Trillion-Parameter Frontier Models
- ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinking
- SoK: Trust-Authorization Mismatch in LLM Agent Interactions
- SIEVE: Selective Integrity Verification and Escalation for Defending LLM Agents against Indirect Prompt Injection
- Graph-Regularized Sparse Autoencoders for LLM Safety Steering
- Semantic Attacks on Tool-Augmented LLMs: Securing the Model Context Protocol Against Descriptor-Level Manipulation
- LLM Harms: A Taxonomy and Discussion
- VRSA: Jailbreaking Multimodal Large Language Models through Visual Reasoning Sequential Attack
- BEAVER: An Efficient Deterministic LLM Verifier
- Counterfeit Answers: Adversarial Forgery against OCR-Free Document Visual Question Answering
- In-Context Representation Hijacking
- Context-Aware Hierarchical Learning: A Two-Step Paradigm towards Safer LLMs
- Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts
- From static to adaptive: immune memory-based jailbreak detection for large language models
- Distillability of LLM Security Logic: Predicting Attack Success Rate of Outline Filling Attack via Ranking Regression
- Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks
- The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search
- Systems Security Foundations for Agentic Computing
- Benchmarking and Understanding Safety Risks in AI Character Platforms
- When Safety Blocks Sense: Measuring Semantic Confusion in LLM Refusals
- Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysis
- Toward a Safe Internet of Agents
- Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization
- Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Models
- Steering Awareness: Detecting Activation Steering from Within
- Memories Retrieved from Many Paths: A Multi-Prefix Framework for Robust Detection of Training Data Leakage in Large Language Models
- Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization
- Can LLMs Threaten Human Survival? Benchmarking Potential Existential Threats from LLMs via Prefix Completion
- Understanding and Mitigating Over-refusal for Large Language Models via Safety Representation
- FanarGuard: A Culturally-Aware Moderation Filter for Arabic Language Models
- Towards Realistic Guarantees: A Probabilistic Certificate for SmoothLLM
- Prompt Fencing: A Cryptographic Approach to Establishing Security Boundaries in Large Language Model Prompts
- For Those Who May Find Themselves on the Red Team
- Curvature-Aware Safety Restoration In LLMs Fine-Tuning
- PARROT: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs
- Steering in the Shadows: Causal Amplification for Activation Space Attacks in Large Language Models
- Evaluating Adversarial Vulnerabilities in Modern Large Language Models
- ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models
- "To Survive, I Must Defect": Jailbreaking LLMs via the Game-Theory Scenarios
- PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization
- When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models
- Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models
- Entropy-Based Measurement of Value Drift and Alignment Work in Large Language Models
- SafeRBench: Dissecting the Reasoning Safety of Large Language Models
- Can MLLMs Detect Phishing? A Comprehensive Security Benchmark Suite Focusing on Dynamic Threats and Multimodal Evaluation in Academic Environments
- Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Education
- Stealth Fine-Tuning: Efficiently Breaking Alignment in RVLMs Using Self-Generated CoT
- ForgeDAN: An Evolutionary Framework for Jailbreaking Aligned Large Language Models
- Learning from the Undesirable: Robust Adaptation of Language Models without Forgetting
- Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMs
- Evolving Prompts for Toxicity Search in Large Language Models
- AlignTree: Efficient Defense Against LLM Jailbreak Attacks
- Rethinking Deep Alignment Through The Lens Of Incomplete Learning
- AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models
- NegBLEURT Forest: Leveraging Inconsistencies for Detecting Jailbreak Attacks
- Robustness of LLM-enabled vehicle trajectory prediction under data security threats
- Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judges
- Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guard
- From Model Training to Model Raising
- SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models
- A Small Leak Sinks All: Exploring the Transferable Vulnerability of Source Code Models
- MSCR: Exploring the Vulnerability of LLMs' Mathematical Reasoning Abilities Using Multi-Source Candidate Replacement
- LoopLLM: Transferable Energy-Latency Attacks in LLMs via Repetitive Generation
- Investigating CoT Monitorability in Large Reasoning Models
- A Self-Improving Architecture for Dynamic Safety in Large Language Models
- "Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers
- MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs
- EASE: Practical and Efficient Safety Alignment for Small Language Models
- Efficient LLM Safety Evaluation through Multi-Agent Debate
- KG-DF: A Black-box Defense Framework against Jailbreak Attacks Based on Knowledge Graphs
- ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversations
- Pluralistic Behavior Suite: Stress-Testing Multi-Turn Adherence to Custom Behavioral Policies
- AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
- Let the Bees Find the Weak Spots: A Path Planning Perspective on Multi-Turn Jailbreak Attacks against LLMs
- Jailbreaking in the Haystack
- Silenced Biases: The Dark Side LLMs Learned to Refuse
- LiveSecBench: A Dynamic and Event-Driven Safety Benchmark for Chinese Language Model Applications
- An Automated Framework for Strategy Discovery, Retrieval, and Evolution in LLM Jailbreak Attacks
- AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
- Vibe Learning: Education in the age of AI
- LM-Fix: Lightweight Bit-Flip Detection and Rapid Recovery Framework for Language Models
- Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?
- ShadowLogic: Backdoors in Any Whitebox LLM
- Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs
- Red-teaming Activation Probes using Prompted LLMs
- Reimagining Safety Alignment with An Image
- Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity
- Consistency Training Helps Stop Sycophancy and Jailbreaks
- Diffusion LLMs are Natural Adversaries for any LLM
- Reasoning Up the Instruction Ladder for Controllable Language Models
- Chain-of-Thought Hijacking
- Angular Steering: Behavior Control via Rotation in Activation Space
- ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Models
- SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning
- BioDisclose: An Actionability-Aware Benchmark for Biomedical Safety under Adversarial Elicitation
- Prosody-driven Jailbreaks in Audio LLMs: A Controlled Study and Mechanistic Analysis
- Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels
- Forecasting Trajectory-Level Safety Risks in Black-Box Multi-Turn Interactions
- ToxScreen: Detecting Whether an LLM Has Been Poisoned
- RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents
- Large-Scale ChatBot Validation Through Customer Digital Twin Simulations
- Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety
- State-Dependent Safety Failures in Multi-Turn Language Model Interaction
- Toward Understanding Security Issues in the Model Context Protocol Ecosystem
- The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanism
- Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models
- A Survey on Unlearning in Large Language Models
- Sequences of Logits Reveal the Low Rank Structure of Language Models
- The Narrative Continuity Test: A Conceptual Framework for Evaluating Identity Persistence in AI Systems
- HACK: Hallucinations Along Certainty and Knowledge Axes
- AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts
- Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
- Education Paradigm Shift To Maintain Human Competitive Advantage Over AI
- QueryIPI: Query-agnostic Indirect Prompt Injection on Coding Agents
- CompressionAttack: Exploiting Prompt Compression as a New Attack Surface in LLM-Powered Agents
- Retracing the Past: LLMs Emit Training Data When They Get Lost
- Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
- Uncovering the Persuasive Fingerprint of LLMs in Jailbreaking Attacks
- Toward Understanding the Transferability of Adversarial Suffixes in Large Language Models
- Adversarial Déjà Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attacks
- Large Language Models as Model Organisms for Human Associative Learning
- Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrency
- NeuroGenPoisoning: Neuron-Guided Attacks on Retrieval-Augmented Generation of LLM via Genetic Optimization of External Knowledge
- Soft Instruction De-escalation Defense
- SAID: Safety-Aware Intent Defense via Prefix Probing for Large Language Models
- HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignment
- Defending Against Prompt Injection with DataFilter
- SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses
- FPT-Noise: Dynamic Scene-Aware Counterattack for Test-Time Adversarial Defense in Vision-Language Models
- HarmNet: A Framework for Adaptive Multi-Turn Jailbreak Attacks on Large Language Models
- Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation
- Genesis: Evolving Attack Strategies for LLM Web Agent Red-Teaming
- BlueCodeAgent: A Blue Teaming Agent Enabled by Automated Red Teaming for CodeGen AI
- Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth
- Agentic Reinforcement Learning for Search Misaligns Instruction-Tuning
- Multimodal Safety Is Asymmetric: Cross-Modal Exploits Unlock Black-Box MLLMs Jailbreaks
- SARSteer: Safeguarding Large Audio-Language Models via Safe-Ablated Refusal Steering
- Online Learning Defense against Iterative Jailbreak Attacks via Prompt Optimization
- Bits Leaked per Query: Information-Theoretic Bounds on Adversarial Attacks against LLMs
- BreakFun: Jailbreaking LLMs via Schema Exploitation
- Black-box Optimization of LLM Outputs by Asking for Directions
- PoTS: Proof-of-Training-Steps for Backdoor Detection in Large Language Models
- ToolTweak: An Attack on Tool Selection in LLM-based Agents
- Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers
- A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space
- When Flatness Does (Not) Guarantee Adversarial Robustness
- Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks
- Sequential Comics for Jailbreaking Multimodal Large Language Models via Structured Visual Storytelling
- When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?
- Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignment
- TRUSTVIS: A Multi-Dimensional Trustworthiness Evaluation Framework for Large Language Models
- PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Features
- SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMs
- Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answering
- Keep Calm and Avoid Harmful Content: Concept Alignment and Latent Manipulation Towards Safer Answers
- Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection
- Deep Research Brings Deeper Harm
- Don't Walk the Line: Boundary Guidance for Filtered Generation
- Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks
- Bag of Tricks for Subverting Reasoning-based Safety Guardrails
- RAG-Pull: Imperceptible Attacks on RAG Systems for Code Generation
- A Vision for Access Control in LLM-based Agent Systems
- BlackIce: A Containerized Red Teaming Toolkit for AI Security Testing
- Safeguarding Efficacy in Large Language Models: Evaluating Resistance to Human-Written and Algorithmic Adversarial Prompts
- UpSafe^∘C: Upcycling for Controllable Safety in Large Language Models
- Dynamic Jailbreaking Attack
- Steering Over-refusals Towards Safety in Retrieval Augmented Generation
- ArtPerception: ASCII Art-based Jailbreak on LLMs with Recognition Pre-test
- MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation
- ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checking
- SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents
- A-IPO: Adaptive Intent-driven Preference Optimization
- Inverse Language Modeling towards Robust and Grounded LLMs
- PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysis
- Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces
- Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Search
- Adversarial News and Lost Profits: Manipulating Headlines in LLM-Driven Algorithmic Trading
- Text Prompt Injection of Vision Language Models
- ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability
- GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare
- The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
- Exploiting Web Search Tools of AI Agents for Data Exfiltration
- Evaluating the Robustness of a Production Malware Detection System to Transferable Adversarial Attacks
- Pattern Enhanced Multi-Turn Jailbreaking: Exploiting Structural Vulnerabilities in Large Language Models
- CREST-Search: Comprehensive Red-teaming for Evaluating Safety Threats in Large Language Models Powered by Web Search
- Fewer Weights, More Problems: A Practical Attack on LLM Pruning
- From Defender to Devil? Unintended Risk Interactions Induced by LLM Defenses
- Invisible to Humans, Triggered by Agents: Stealthy Jailbreak Attacks on Mobile Vision-Language Agents
- The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMs
- Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMs
- AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teaming
- VisualDAN: Exposing Vulnerabilities in VLMs with Visual-Driven DAN Commands
- PEAR: Planner-Executor Agent Robustness Benchmark
- On the Convergence of Moral Self-Correction in Large Language Models
- Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Experts
- PERSONA: Dynamic and Compositional Inference-Time Personality Control via Activation Vector Algebra
- Authenticated Workflows: A Systems Approach to Protecting Agentic AI
- Bypassing Prompt Guards in Production with Controlled-Release Prompting
- Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization
- LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics
- A Survey on Agentic Security: Applications, Threats and Defenses
- Evaluating LLM Safety Across Child Development Stages: A Simulated Agent Approach
- Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?
- RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contexts
- Learning to Interpret Weight Differences in Language Models
- Imperceptible Jailbreaking against Large Language Models
- Proactive defense against LLM Jailbreak
- SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests
- RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection
- Machine Learning for Detection and Analysis of Novel LLM Jailbreaks
- Read the Scene, Not the Script: Outcome-Aware Safety for LLMs
- AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agents
- Activation Steering with a Feedback Controller
- Quantitative Certification of Agentic Tool Selection
- From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMs
- Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models
- Backdoor-Powered Prompt Injection Attacks Nullify Defense Methods
- NonTextual Target Attack
- External Data Extraction Attacks against Retrieval-Augmented Large Language Models
- Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain
- Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs
- Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks
- Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks
- Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approach
- Microsaccade-Inspired Probing: Positional Encoding Perturbations Reveal LLM Misbehaviours
- Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors
- Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerability
- Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed
- Are Robust LLM Fingerprints Adversarially Robust?
- OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!
- STaR-Attack: A Spatio-Temporal and Narrative Reasoning Attack Framework for Unified Multimodal Understanding and Generation Models
- SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Models
- Better Privilege Separation for Agents by Restricting Data Types
- ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attack
- LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Models
- VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models
- MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources
- Fingerprinting LLMs via Prompt Injection
- OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment
- FuncPoison: Poisoning Function Library to Hijack Multi-agent Autonomous Driving Systems
- HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
- DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models
- Model Correlation Detection via Random Selection Probing
- Think Twice, Generate Once: Safeguarding by Progressive Self-Reflection
- RADAR: A Risk-Aware Dynamic Multi-Agent Framework for LLM Safety Evaluation via Role-Specialized Collaboration
- Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
- Dual-Space Smoothness for Robust and Balanced LLM Unlearning
- A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Models
- Preventing Robotic Jailbreaking via Multimodal Domain Adaptation
- Decision Potential Surface: A Theoretical and Practical Approximation of Large Language Model Decision Boundary
- Your RAG is Unfair: Exposing Fairness Vulnerabilities in Retrieval-Augmented Generation via Backdoor Attacks
- The Rogue Scalpel: Activation Steering Compromises LLM Safety
- Active Attacks: Red-teaming LLMs via Adaptive Environments
- You Can't Steal Nothing: Mitigating Prompt Leakages in LLMs via System Vectors
- PSRT: Accelerating LRM-based Guard Models via Prefilled Safe Reasoning Traces
- Backdoor Attribution: Elucidating and Controlling Backdoor in Language Models
- Where Did It Go Wrong? Attributing Undesirable LLM Behaviors via Representation Gradient Tracing
- Bidirectional Intention Inference Enhances LLMs' Defense Against Multi-Turn Jailbreak Attacks
- FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction
- GEP: A GCG-Based method for extracting personally identifiable information from chatbots built on small language models
- JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation
- SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models
- Responsible AI Technical Report
- LatentGuard: Controllable Latent Steering for Robust Refusal of Attacks and Reliable Response Generation
- FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Models
- bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs
- Compliance2LoRA: Personalizable On-Demand Safety Alignment on Arbitrary Policy Subsets via Hypernetwork-Generated LoRA Adapters
- Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation
- FAVA: Formal Authorization for Verified Agents with Evidence-Backed Permission Graphs
- Beyond the Bidirectional Promise: Re-evaluating the Robustness of Diffusion Language Models
- LLM2Vec-Gen: Generative Embeddings from Large Language Models
- Pressure Reveals Character: Behavioural Alignment Evaluation at Depth
- Speculative Safety-Aware Decoding
- Trigger Where It Hurts: Unveiling Hidden Backdoors through Sensitivity with Sensitron
- LLMZ+: Contextual Prompt Whitelist Principles for Agentic LLMs
- Steering Multimodal Large Language Models Decoding for Context-Aware Safety
- Attacking LLMs and AI Agents: Advertisement Embedding Attacks Against Large Language Models
- The Ranking Blind Spot: Decision Hijacking in LLM-based Text Ranking
- Anecdoctoring: Automated Red-Teaming Across Language and Place
- When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMs
- Jailbreaking LLMs via Semantically Relevant Nested Scenarios with Targeted Toxic Knowledge
- Evaluating the Safety and Skill Reasoning of Large Reasoning Models Under Compute Constraints
- D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models
- DecipherGuard: Understanding and Deciphering Jailbreak Prompts for a Safer Deployment of Intelligent Software Systems
- SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
- Psychometric Personality Shaping Modulates Capabilities and Safety in Language Models
- Semantic Representation Attack against Aligned Large Language Models
- LLM Jailbreak Detection for (Almost) Free!
- Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs
- A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
- From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
- Jailbreaking Large Language Models Through Content Concretization
- Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models
- Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks
- NeuroStrike: Neuron-Level Attacks on Aligned LLMs
- Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check
- POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization
- Collapse of Irrelevant Representations (CIR) Ensures Robust and Non-Disruptive LLM Unlearning
- ERIS: Evolutionary Real-world Interference Scheme for Jailbreaking Audio Large Models
- A Biosecurity Agent for Lifecycle LLM Biosecurity Alignment
- When Smiley Turns Hostile: Interpreting How Emojis Trigger LLMs' Toxicity
- Harmful Prompt Laundering: Jailbreaking LLMs with Abductive Styles and Symbolic Encoding
- LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
- Steering MoE LLMs via Expert (De)Activation
- SALMAN: Stability Analysis of Language Models Through the Maps Between Graph-based Manifolds
- X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templates
- ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation
- AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agents
- Getting In Contract with Large Language Models -- An Agency Theory Perspective On Large Language Model Alignment
- Towards Post-mortem Data Management Principles for Generative AI
- Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling
- Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm
- MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Security
- Probabilistic Modeling of Latent Agentic Substructures in Deep Neural Networks
- Mind Your Server: A Systematic Study of Parasitic Toolchain Attacks on the MCP Ecosystem
- Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?
- Uncovering the Vulnerability of Large Language Models in the Financial Domain via Risk Concealment
- Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint
- BinaryShield: Cross-Service Threat Intelligence in LLM Services using Privacy-Preserving Fingerprints
- Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models
- Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
- HAMSA: Hijacking Aligned Compact Models via Stealthy Automation
- NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models
- MEUV: Achieving Fine-Grained Capability Activation in Large Language Models via Mutually Exclusive Unlock Vectors
- False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize
- A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
- Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain
- PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs
- Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs
- Strata-Sword: A Hierarchical Safety Evaluation towards LLMs based on Reasoning Complexity of Jailbreak Instructions
- Web Fraud Attacks Against LLM-Driven Multi-Agent Systems
- CARE: Decoding Time Safety Alignment via Rollback and Introspection Intervention
- Unraveling LLM Jailbreaks Through Safety Knowledge Neurons
- ConceptGuard: Neuro-Symbolic Safety Guardrails via Sparse Interpretable Jailbreak Concepts
- The Resurgence of GCG Adversarial Attacks on Large Language Models
- Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models
- ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety
- Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs
- Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol
- Embodied AI: Emerging Risks and Opportunities for Policy Action
- SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Models
- IntentionReasoner: Facilitating Adaptive LLM Safeguards through Intent Reasoning and Selective Query Refinement
- Evaluating Language Model Reasoning about Confidential Information
- SoK: Large Language Model Copyright Auditing via Fingerprinting
- On Surjectivity of Neural Networks: Can you elicit any behavior from your model?
- UniC-RAG: Universal Knowledge Corruption Attacks to Retrieval-Augmented Generation
- Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
- Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience
- SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attacks
- Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection
- Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation
- Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent
- CIA+TA Risk Assessment for AI Reasoning Vulnerabilities
- Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation
- CCFC: Core & Core-Full-Core Dual-Track Defense for LLM Jailbreak Protection
- Involuntary Jailbreak: On Self-Prompting Attacks
- RAJ-PGA: Reasoning-Activated Jailbreak and Principle-Guided Alignment Framework for Large Reasoning Models
- Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Position
- Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
- Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions
- Mitigating Jailbreaks with Intent-Aware LLMs
- Invitation Is All You Need! Promptware Attacks Against LLM-Powered Assistants in Production Are Practical and Dangerous
- When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs
- ORFuzz: Fuzzing the "Other Side" of LLM Safety -- Testing Over-Refusal
- Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends
- Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
- Jailbreaking Commercial Black-Box LLMs with Explicitly Harmful Prompts
- Slow Tuning and Low-Entropy Masking for Safe Chain-of-Thought Distillation
- NeuronTune: Fine-Grained Neuron Modulation for Balanced Safety-Utility Alignment in LLMs
- Special-Character Adversarial Attacks on Open-Source Language Model
- From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training
- Never compromise with vulnerabilities: a comprehensive survey on AI governance
- Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity
- Jinx: Unlimited LLMs for Probing Alignment Failures
- VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models
- A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detection
- Gradient Surgery for Safe LLM Fine-Tuning
- Context Misleads LLMs: The Role of Context Filtering in Maintaining Safe Alignment of LLMs
- Many-Turn Jailbreaking
- Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
- ScamAgents: How AI Agents Can Simulate Human-Level Scam Calls
- Quantifying Conversation Drift in MCP via Latent Polytope
- LLM Unlearning Without an Expert Curated Dataset
- Keep It Real: Challenges in Attacking Compression-Based Adversarial Purification
- JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering
- Automatic LLM Red Teaming
- ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Moments
- AttnTrace: Attention-based Context Traceback for Long-Context LLMs
- When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs
- Large Reasoning Models Are Autonomous Jailbreak Agents
- Simple Methods Defend RAG Systems Well Against Real-World Attacks
- Hidden in the Noise: Unveiling Backdoors in Audio LLMs Alignment through Latent Acoustic Pattern Triggers
- TRACEALIGN -- Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs
- Defend LLMs Through Self-Consciousness
- Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models
- PUZZLED: Jailbreaking LLMs through Word-Based Puzzles
- Activation-Guided Local Editing for Jailbreaking Attacks
- Lessons from complex systems science for AI governance
- Universally Unfiltered and Unseen:Input-Agnostic Multimodal Jailbreaks against Text-to-Image Model Safeguards
- Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems
- Strategic Deflection: Defending LLMs from Logit Manipulation
- UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases
- PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreaking
- Enhancing Jailbreak Attacks on LLMs via Persona Prompts
- Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition
- The Blessing and Curse of Dimensionality in Safety Alignment
- PromptArmor: Simple yet Effective Prompt Injection Defenses
- A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction
- MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?
- PrompTrend: Continuous Community-Driven Vulnerability Discovery and Assessment for Large Language Models
- Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment
- Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection
- Understanding the Supply Chain and Risks of Large Language Model Applications
- From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Models
- AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning
- Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree
- AgentTrace: A Structured Logging Framework for Agent System Observability
- GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Prompt
- Shaping capabilities with token-level data filtering
- An Uncertainty-Driven Adaptive Self-Alignment Framework for Large Language Models
- Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques
- The Geometry of Harmfulness in LLMs through Subconcept Probing
- Agent Identity Evals: Measuring Agentic Identity
- TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
- When LLMs Copy to Think: Uncovering Copy-Guided Attacks in Reasoning LLMs
- Analysis of Threat-Based Manipulation in Large Language Models: A Dual Perspective on Vulnerabilities and Performance Enhancement Opportunities
- Innocence in the Crossfire: Roles of Skip Connections in Jailbreaking Visual Language Models
- DEMONSTRATE: Zero-shot Language to Robotic Control via Multi-task Demonstration Learning
- Watch, Listen, Understand, Mislead: Tri-modal Adversarial Attacks on Short Videos for Content Appropriateness Evaluation
- LLMs Encode Harmfulness and Refusal Separately
- LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
- Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility
- AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basin
- SUA: Stealthy Multimodal Large Language Model Unlearning Attack
- The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMs
- Scaling laws for activation steering with Llama 2 models and refusal mechanisms
- PRM-Free Security Alignment of Large Models via Red Teaming and Adversarial Training
- ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoning
- Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks
- SoK: Machine Unlearning for Large Language Models
- SafeCoT: Improving VLM Safety with Minimal Reasoning
- SEALGuard: Safeguarding the Multilingual Conversations in Southeast Asian Languages for LLM Software Systems
- Lightweight Safety Guardrails via Synthetic Data and RL-guided Adversarial Training
- Defending Against Prompt Injection With a Few DefensiveTokens
- May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks
- Mitigating Watermark Forgery in Generative Models via Randomized Key Selection
- Attention-Aware GNN-based Input Defense against Multi-Turn LLM Jailbreak
- A Mathematical Theory of Discursive Networks
- On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks
- Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates
- CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representations
- How Not to Detect Prompt Injections with an LLM
- Search-based Selection of Metamorphic Relations for Optimized Robustness Testing of Large Language Models
- The bitter lesson of misuse detection
- Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigms
- The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models
- Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion
- Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message
- Attacker's Noise Can Manipulate Your Audio-based LLM in the Real World
- Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models
- GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models
- Attention Slipping: A Mechanistic Understanding of Jailbreak Attacks and Defenses in LLMs
- Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking
- Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuning
- A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models
- Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attacks
- Control at Stake: Evaluating the Security Landscape of LLM-Driven Email Agents
- Where Do Reasoning Models Refuse?
- MGC: A Compiler Framework Exploiting Compositional Blindness in Aligned LLMs for Malware Generation
- GPT, But Backwards: Exactly Inverting Language Model Outputs
- When Safety Becomes a Vulnerability: Exploiting LLM Alignment Homogeneity for Transferable Blocking in RAG
- ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks
- Reasoning as an Adaptive Defense for Safety
- Linearly Decoding Refused Knowledge in Aligned Language Models
- A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
- QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA
- STACK: Adversarial Attacks on LLM Safeguard Pipelines
- Logit-Gap Steering: A Forward-Pass Diagnostic for Alignment Robustness
- A Representation Engineering Perspective on the Effectiveness of Multi-Turn Jailbreaks
- TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs
- Perspective Dial: Measuring Perspective of Text and Guiding LLM Outputs
- MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMs
- Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses
- RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards
- TAI3: Testing Agent Integrity in Interpreting User Intent
- From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment
- Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation
- When Style Breaks Safety: Defending LLMs Against Superficial Style Alignment
- The Singapore Consensus on Global AI Safety Research Priorities
- Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures
- RedCoder: Automated Multi-Turn Red Teaming for Code LLMs
- Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models
- Is Long-to-Short a Free Lunch? Investigating Inconsistency and Reasoning Efficiency in LRMs
- Model Guidance via Robust Feature Attribution
- SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models
- NSFW-Classifier Guided Prompt Sanitization for Safe Text-to-Image Generation
- Command-V: Pasting LLM Behaviors via Activation Profiles
- GRAF: Multi-turn Jailbreaking via Global Refinement and Active Fabrication
- TwinBreak: Jailbreaking LLM Security Alignments based on Twin Prompts
- SAFEx: Analyzing Vulnerabilities of MoE-Based LLMs via Stable Safety-critical Expert Identification
- MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuning
- From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers
- Cross-Modal Obfuscation for Jailbreak Attacks on Large Vision-Language Models
- Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models
- Probing the Robustness of Large Language Models Safety to Latent Perturbations
- LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning
- InverseScope: Scalable Activation Inversion for Interpreting Large Language Models
- From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?
- Sysformer: Safeguarding Frozen Large Language Models with Adaptive System Prompts
- From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystem
- LLM Jailbreak Oracle
- OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents
- AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions
- Doppelganger Method: Breaking Role Consistency in LLM Agent via Prompt-based Transferable Adversarial Attack
- Reward Model Interpretability via Optimal and Pessimal Tokens
- RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?
- FORTRESS: Frontier Risk Evaluation for National Security and Public Safety
- What Makes a Good Natural Language Prompt?
- Empirical Evidence for Alignment Faking in a Small LLM and Prompt-Based Mitigation Techniques
- RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models
- Excessive Reasoning Attack on Reasoning LLMs
- DualEdit: Mitigating Safety Fallback in LLM Backdoor Editing via Affirmation-Refusal Regulation
- Transferring Linear Features Across Language Models With Model Stitching
- Mind the Web: The Security of Web Use Agents
- ExtendAttack: Attacking Servers of LRMs via Extending Reasoning
- Weakest Link in the Chain: Security Vulnerabilities in Advanced Reasoning Models
- Jailbreak Transferability Emerges from Shared Representations
- ContextBench: Modifying Contexts for Targeted Latent Activation
- Universal Jailbreak Suffixes Are Strong Attention Hijackers
- Enhancing the Safety of Medical Vision-Language Models by Synthetic Demonstrations
- The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Models
- Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025
- AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint
- Exploring the Secondary Risks of Large Language Models
- QGuard:Question-based Zero-shot Guard for Multi-modal LLM Safety
- Robust LLM Unlearning with MUDMAN: Meta-Unlearning with Disruption Masking And Normalization
- Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models
- Feedback Friction: LLMs Struggle to Fully Incorporate External Feedback
- Tokenized Bandit for LLM Decoding and Alignment
- Black-Box Access is Insufficient for Rigorous AI Audits
- From Emergence to Control: Probing and Modulating Self-Reflection in Language Models
- SDialog: A Python Toolkit for End-to-End Agent Building, User Simulation, Dialog Generation, and Evaluation
- SoK: Evaluating Jailbreak Guardrails for Large Language Models
- Prompt Attacks Reveal Superficial Knowledge Removal in Unlearning Methods
- Textual Bayes: Quantifying Uncertainty in LLM-Based Systems
- The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
- Benchmarking Misuse Mitigation Against Covert Adversaries
- What Really is a Member? Discrediting Membership Inference via Poisoning
- Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systems
- SECNEURON: Reliable and Flexible Abuse Control in Local LLMs via Hybrid Neuron Encryption
- HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
- Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction
- OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models
- A Proposal for Evaluating the Operational Risk for ChatBots based on Large Language Models
- Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG
- Misalignment or misuse? The AGI alignment tradeoff
- TracLLM: A Generic Framework for Attributing Long Context LLMs
- ATAG: AI-Agent Application Threat Assessment with Attack Graphs
- BitBypass: A New Direction in Jailbreaking Aligned Large Language Models with Bitstream Camouflage
- Modelship Attribution: Tracing Multi-Stage Manipulations Across Generative Models
- Robustness in Both Domains: CLIP Needs a Robust Text Encoder
- Adversarial Attacks on Robotic Vision Language Action Models
- Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
- IF-GUIDE: Influence Function-Guided Detoxification of LLMs
- ReGA: Representation-Guided Abstraction for Model-based Safeguarding of LLMs
- Comprehensive Vulnerability Analysis is Necessary for Trustworthy LLM-MAS
- Adversarial Attacks in Multimodal Systems: A Practitioner's Survey
- Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models
- CoP: Agentic Red-teaming for Large Language Models using Composition of Principles
- SafeGenes: Evaluating the Adversarial Robustness of Genomic Foundation Models
- AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents
- Spectral Insights into Data-Oblivious Critical Layers in Large Language Models
- Con Instruction: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalities
- Existing Large Language Model Unlearning Evaluations Are Inconclusive
- SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuning
- The Security Threat of Compressed Projectors in Large Vision-Language Models
- SafeTy Reasoning Elicitation Alignment for Multi-Turn Dialogues
- Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models
- Learning Safety Constraints for Large Language Models
- Adversarial Preference Learning for Robust LLM Alignment
- From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Models
- TRAPDOC: Deceiving LLM Users by Injecting Imperceptible Phantom Tokens into Documents
- A Red Teaming Roadmap Towards System-Level Safety
- LLM Agents Should Employ Security Principles
- SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents
- MEF: A Capability-Aware Multi-Encryption Framework for Evaluating Vulnerabilities in Black-Box Large Language Models
- Does Machine Unlearning Truly Remove Knowledge?
- Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusion
- A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluation Methods
- OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalities
- EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions
- SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
- Understanding Refusal in Language Models with Sparse Autoencoders
- Operationalizing CaMeL: Strengthening LLM Defenses for Enterprise Deployment
- Jailbreak Distillation: Renewable Safety Benchmarking
- Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack
- Mitigating Overthinking in Large Reasoning Models via Manifold Steering
- Risks of AI-driven product development and strategies for their mitigation
- From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization
- System Prompt Extraction Attacks and Defenses in Large Language Models
- The Multilingual Divide and Its Impact on Global AI Safety
- SAGE-Eval: Evaluating LLMs for Systematic Generalizations of Safety Facts
- Breaking the Ceiling: Exploring the Potential of Jailbreak Attacks through Expanding Strategy Space
- SoSBench: Benchmarking Safety Alignment on Six Scientific Domains
- Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis
- Semantic-Preserving Adversarial Attacks on LLMs: An Adaptive Greedy Binary Search Approach
- Attention! Your Vision Language Model Could Be Maliciously Manipulated
- PAM: Training Policy-Aligned Moderation Filters at Scale
- Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMs
- JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models
- Benign-to-Toxic Jailbreaking: Inducing Harmful Responses from Harmless Prompts
- CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Models
- Lifelong Safety Alignment for Language Models
- An Embarrassingly Simple Defense Against LLM Abliteration Attacks
- Dynamic Optimization and Safety Indicator Injection for Jailbreaking Text-to-Image Models with Multimodal Safety Filters
- Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representations
- Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression
- Security Concerns for Large Language Models: A Survey
- Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulation
- Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer
- The Silent Saboteur: Imperceptible Adversarial Attacks against Black-Box Retrieval-Augmented Generation Systems
- PD3F: A Pluggable and Dynamic DoS-Defense Framework Against Resource Consumption Attacks Targeting Large Language Models
- Audio Jailbreak Attacks: Exposing Vulnerabilities in SpeechGPT in a White-Box Framework
- EVADE: Multimodal Benchmark for Evasive Content Detection in E-Commerce Applications
- Does Chain-of-Thought Reasoning Really Reduce Harmfulness from Jailbreaking?
- Emergent Standing Wave Dynamics and Attractor Basins in Transformer Latent Spaces via Prompt Driven ConstraintsAnathema to Corporate Control by Ingrid Johnson
- One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs
- JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models
- Towards Evaluating Proactive Risk Awareness of Multimodal Language Models
- Speechless: Speech Instruction Training Without Speech for Low Resource Languages
- Misaligning Reasoning with Answers -- A Framework for Assessing LLM CoT Robustness
- Discovering Forbidden Topics in Language Models
- Chain-of-Lure: A Universal Jailbreak Attack Framework using Unconstrained Synthetic Narratives
- A Critical Evaluation of Defenses against Prompt Injection Attacks
- Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
- Towards medical AI misalignment: a preliminary study
- Refusal Direction is Universal Across Safety-Aligned Languages
- LaCache: Robust Semantic Caching for LLM Serving
- Shape it Up! Restoring LLM Safety during Finetuning
- MixAT: Combining Continuous and Discrete Adversarial Training for LLMs
- CAIN: Hijacking LLM-Humans Conversations via Malicious System Prompts
- Harry Potter is Still Here! Probing Knowledge Leakage in Targeted Unlearned Large Language Models via Automated Adversarial Prompting
- CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning
- MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming
- Robustifying Vision-Language Models via Dynamic Token Reweighting
- Guiding Giants: Lightweight Controllers for Weighted Activation Steering in LLMs
- Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning
- CoTSRF: Utilize Chain of Thought as Stealthy and Robust Fingerprint of Large Language Models
- Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards
- Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models
- Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
- SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning
- In-Context Watermarks for Large Language Models
- OpenEthics: A Comprehensive Ethical Evaluation of Open-Source Generative Large Language Models
- Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey
- Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defenses
- EduZone: A Framework for Evaluating LLM Safety for K-12 Students and Teachers
- How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study
- A Linear Approach to Data Poisoning
- Advancing LLM Safe Alignment with Safety Representation Ranking
- Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
- Improving LLM First-Token Predictions in Multiple-Choice Question Answering via Output Prefilling
- Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders
- SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment
- Context Reasoner: Incentivizing Reasoning Capability for Contextualized Privacy and Safety Compliance via Reinforcement Learning
- EVA: Red-Teaming GUI Agents via Evolving Indirect Prompt Injection
- Phonetic Perturbations Reveal Tokenizer-Rooted Safety Gaps in LLMs
- Adversarially Pretrained Transformers may be Universally Robust In-Context Learners
- Causes and Consequences of Representational Similarity in Machine Learning Models
- Robust and Efficient AI-Based Attack Recovery in Autonomous Drones
- AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Models
- From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs
- SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks
- Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study
- PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
- Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations
- Safety Alignment Can Be Not Superficial With Explicit Safety Signals
- PromptPrism: A Linguistically-Inspired Taxonomy for Prompts
- Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks
- Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Dataset
- RAFP: Identifying LLM Lineages via Rare-Region Fingerprints
- Web Intellectual Property at Risk: Preventing Unauthorized Real-Time Retrieval by Large Language Models
- Revealing the Deceptiveness of Knowledge Editing: A Mechanistic Analysis of Superficial Editing
- Bullying the Machine: How Personas Increase LLM Vulnerability
- A Survey of Attacks on Large Language Models
- Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integration
- BadNAVer: Exploring Jailbreak Attacks On Vision-and-Language Navigation
- Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression
- Uncovering Competing Poisoning Attacks in Retrieval-Augmented Generation
- SPIRIT: Patching Speech Language Models against Jailbreak Attacks
- IP Leakage Attacks Targeting LLM-Based Multi-Agent Systems
- SafeVid: Toward Safety Aligned Video Large Multimodal Models
- Multilingual Collaborative Defense for Large Language Models
- Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets
- Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs
- Why Not Act on What You Know? Unleashing Safety Potential of LLMs via Self-Aware Guard Enhancement
- EcoSafeRAG: Efficient Security through Context Analysis in Retrieval-Augmented Generation
- Generative artificial intelligence (AI) in higher education: a comprehensive review of challenges, opportunities, and implications
- AutoRAN: Automated Hijacking of Safety Reasoning in Large Reasoning Models
- A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?
- LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMs
- Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction
- CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs
- Signal in the Noise: Polysemantic Interference Transfers and Predicts Cross-Model Influence
- Adversarial Attack on Large Language Models using Exponentiated Gradient Descent
- PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization
- Adversarial Suffix Filtering: a Defense Pipeline for LLMs
- A Two-Step, Multidimensional Account of Deception in Language Models
- Layered Unlearning for Adversarial Relearning
- Can LLMs Reliably Self-Report Adversarial Prefills, and How?
- One Trigger Token Is Enough: A Defense Strategy for Balancing Safety and Usability in Large Language Models
- Concept-Level Explainability for Auditing & Steering LLM Responses
- FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning
- POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Models
- System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection
- OTora: A Unified Red Teaming Framework for Reasoning-Level Denial-of-Service in LLM Agents
- Minimal, Local, Causal Explanations for Jailbreak Success in Large Language Models
- A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook
- LiteLMGuard: Seamless and Lightweight On-Device Prompt Filtering for Safeguarding Small Language Models against Quantization-induced Risks and Vulnerabilities
- Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
- Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs
- Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization System
- Security and Privacy Challenges of Large Language Models: A Survey
- Attack and defense techniques in large language models: A survey and new perspectives
- Embracing Anisotropy: Turning Massive Activations into Interpretable Control Knobs for Large Language Models
- πCreds: Privately Inferred Credentials
- TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering
- Graph-based Agent Memory: Taxonomy, Techniques, and Applications
- Mechanistic Interpretability of LLM Jailbreaks via Internal Attribution Graphs
- Fast Multi-dimensional Refusal Subspaces via RFM-AGOP
- Towards Unsupervised Adversarial Document Detection in Retrieval Augmented Generation Systems
- When Behavioral Safety Evaluation Fails: A Representation-Level Perspective
- The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey
- Triggering Hallucinations in LLMs: A Quantitative Study of Prompt-Induced Hallucination in Large Language Models
- Semantic Invariance in Agentic AI
- Colluding LoRA: A Compositional Vulnerability in LLM Safety Alignment
- A Definition of Good Explanations and the Challenges Explaining LLM Outputs
- Assessing Automated Prompt Injection Attacks in Agentic Environments
- Multi-Stream Perturbation Attack: Breaking Safety Alignment of Thinking LLMs Through Concurrent Task Interference
- Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms
- Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
- Hiding in Plain Text: Detecting Concealed Jailbreaks via Activation Disentanglement
- Prompt Injection as Role Confusion
- Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis
- Positive Alignment: Artificial Intelligence for Human Flourishing
- Intent Laundering: AI Safety Datasets Are Not What They Seem
- SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
- A New Framework for Cybersecurity Refusals in AI Agents
- Trust The Typical
- XBreaking: Understanding how LLMs security alignment can be broken
- Hoist with His Own Petard: Inducing Guardrails to Facilitate Denial-of-Service Attacks on Retrieval-Augmented Generation of LLMs
- CachePrune: Neural-Based Attribution Defense Against Indirect Prompt Injection Attacks
- ADR: An Agentic Detection System for Enterprise Agentic AI Security
- Certified Robustness under Heterogeneous Perturbations via Hybrid Randomized Smoothing
- AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems
- Estimating Tail Risks in Language Model Output Distributions
- Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction
- NeuRel-Attack: Neuron Relearning for Safety Disalignment in Large Language Models
- When Memory Becomes a Vulnerability: Towards Multi-turn Jailbreak Attacks against Text-to-Image Generation Systems
- Prompt Injection Attack to Tool Selection in LLM Agents
- Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models
- When Local Monitors Miss Compositional Harm: Diagnosing Distributed Backdoors in Multi-Agent Systems
- Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
- Ethics Testing: Proactive Identification of Generative AI System Harms
- Future of AI Models: A Computational perspective on Model collapse
- Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testing
- Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types
- Synthetic Data for any Differentiable Target
- Sequential Data Poisoning in LLM Post-Training
- How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition
- Buy versus Build an LLM: A Decision Framework for Governments
- HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defense
- A Cryptographic Perspective on Mitigation vs. Detection in Machine Learning
- JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift
- Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Models
- Temporal UI State Inconsistency in Desktop GUI Agents: Formalizing and Defending Against TOCTOU Attacks on Computer-Use Agents
- LLM Safety From Within: Detecting Harmful Content with Internal Representations
- Graph of Attacks: Improved Black-Box and Interpretable Jailbreaks for LLMs
- Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
- RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models
- Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections
- MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attacks
- AISafetyBenchExplorer: A Metric-Aware Catalogue of AI Safety Benchmarks Reveals Fragmented Measurement and Weak Benchmark Governance
- TEMPLATEFUZZ: Fine-Grained Chat Template Fuzzing for Jailbreaking and Red Teaming LLMs
- Large Language Lobotomy: Jailbreaking Mixture-of-Experts via Expert Silencing
- Conflicts Make Large Reasoning Models Vulnerable to Attacks
- PIArena: A Platform for Prompt Injection Evaluation
- Learning to Inject: Automated Prompt Injection via Reinforcement Learning
- Ethical Risks in Deploying Large Language Models: An Evaluation of Medical Ethics Jailbreaking
- Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasure
- ICO: Enhancing Semantic-Shift Jailbreaks via Iterative Context Optimization
- AI Security Leaderboard: Methodology, Results and Minimal Standard
- Overlooked Safety Vulnerability in LLMs: Malicious Intelligent Optimization Algorithm Request and its Jailbreak
- DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models
- Looking in the Mirror: Introspecting Side-Effect Misalignments Induced by Fine-Tuning
- "Allow" to Achieve, Over-Privileged Inadvertently: The Unintended Cost of Task-Completion-Driven Pop-up Decisions in Mobile GUI Agents
- A Blind Spot in Alignment: Quantifying Biosecurity Risks in Large Language Models
- Behavioral Skill Reconstruction: Reconstructing Hidden Functionality from LLM Agent Skills
- Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models
- Eliciting Intrinsic Hallucinations in LLMs via Semantically Equivalent Adversarial Attacks
- A Multimodal Automatic Redteaming Evaluation based on Atomic Jailbreak Strategy Decoupling and Combination
- Governing Execution Risk in Agentic AI Systems: A Trajectory-Guided Framework for Red Teaming
- How Should AI Safety Benchmarks Benchmark Safety?
- Safety Pretraining: Toward the Next Generation of Safe AI
- A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Models
- Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Control
- Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debate
- WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks
- Inducing Vulnerable Code Generation in LLM Coding Assistants
- Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Models
- Safety Co-Option and Compromised National Security: The Self-Fulfilling Prophecy of Weakened AI Risk Thresholds
- RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Search
- DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimization
- Manipulating Multimodal Agents via Cross-Modal Prompt Injection
- Thought Manipulation: External Thought Can Be Efficient for Large Reasoning Models
- DETAM: Defending LLMs Against Jailbreak Attacks via Targeted Attention Modification
- Q-FAKER: Query-free Hard Black-box Attack via Controlled Generation
- AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
- ELAB: Extensive LLM Alignment Benchmark in Persian Language
- Antidistillation Sampling
- Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretability
- CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits
- From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMs
- MMAligner: Safeguarding Multimodal Large Language Models through Representation Calibration
- Robust Context-Aware Detection of Malicious Instructions in Text
- Mood Matters: How Syntactic Sensitivity Undermines Safety Alignment
- RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs
- Where Agent Frameworks Fall Short: Examining Functional Challenges and Usability Concerns
- Token-Level Constraint Boundary Search for Jailbreaking Text-to-Image Models
- REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective
- StruPhantom: Evolutionary Injection Attacks on Black-Box Tabular Agents Powered by Large Language Models
- The Jailbreak Tax: How Useful are Your Jailbreak Outputs?
- Ctrl-Z: Controlling AI Agents via Resampling
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?
- AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defender
- The Structural Safety Generalization Problem
- Feature-Aware Malicious Output Detection and Mitigation
- AttentionDefense: Leveraging System Prompt Attention for Explainable Defense Against Novel Jailbreaks
- Geneshift: Impact of different scenario shift on Jailbreaking LLM
- PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimization
- A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
- Bypassing Safety Guardrails in LLMs Using Humor
- NLP Security and Ethics, in the Wild
- Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separators
Discussions
- here's my favorite paper on how to do this. if anyone figures out if it's a crime or not lmk arxiv.org/abs/2307.150... [bsky, 9 points, 1 comments]
- Universal and Transferable Adversarial Attacks on LLM [hn, 3 points, 0 comments]
- Currently the answer is yes, attacks transfer pretty well across models! See e.g. arxiv.org/pdf/2307.15043 (old but still applicable) But I think stopping attacks from transferring is more tenable tha [bsky, 2 points, 0 comments]
- Universal and Transferable Adversarial Attacks on Aligned Language Models [hn, 1 points, 0 comments]
- Researchers have "jailbroken" commercial AIs, including OpenAI and Bard, by developing code to aid in contriving strings of characters to bypass input filtering. They have published both their techniq [bsky, 0 points, 0 comments]
Related