From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Models

2025/05/30 by Haibo Jin, Peiyan Zhang, Jin, Haibo +7 · 2 citations
Social Sciences · Decision Sciences · #Crime, Illicit Activities, and Governance #Big Data Technologies and Applications

paper · pdf · doi:10.48550/arxiv.2505.24232

Abstract

Large foundation models (LFMs) are susceptible to two distinct vulnerabilities: hallucinations and jailbreak attacks. While typically studied in isolation, we observe that defenses targeting one often affect the other, hinting at a deeper connection. We propose a unified theoretical framework that models jailbreaks as token-level optimization and hallucinations as attention-level optimization. Within this framework, we establish two key propositions: (1) Similar Loss Convergence - the loss functions for both vulnerabilities converge similarly when optimizing for target-specific outputs; and (2) Gradient Consistency in Attention Redistribution - both exhibit consistent gradient behavior driven by shared attention dynamics. We validate these propositions empirically on LLaVA-1.5 and MiniGPT-4, showing consistent optimization trends and aligned gradients. Leveraging this connection, we demonstrate that mitigation techniques for hallucinations can reduce jailbreak success rates, and vice versa. Our findings reveal a shared failure mode in LFMs and suggest that robustness strategies should jointly address both vulnerabilities.

Citations

• Jailbreak Large Vision-Language Models Through Multi-Modal Linkage
• Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps
• JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models
• InternalInspector I²: Robust Confidence Estimation in LLMs through Internal States
• AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models
• Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters
• Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach
• JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks
• AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting
• Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models
• DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers
• Query-Based Adversarial Prompt Generation
• Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States
• INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
• GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
• HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
• Jailbreaking Black Box Large Language Models in Twenty Queries
• Improved Baselines with Visual Instruction Tuning
• LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples
• Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models
• On the Adversarial Robustness of Multi-Modal Foundation Models
• GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
• Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
• Jailbroken: How Does LLM Safety Training Fail?
• Are aligned neural networks adversarially aligned?
• On Evaluating Adversarial Robustness of Large Vision-Language Models
• The Internal State of an LLM Knows When It's Lying
• Visual Instruction Tuning
• Multi-step Jailbreaking Privacy Attacks on ChatGPT
• SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles
• What Does BERT Look At? An Analysis of BERT's Attention
• Are Sixteen Heads Really Better than One?
• CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion
• Visual Adversarial Examples Jailbreak Aligned Large Language Models
• AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models

Cited by

• The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMs
• GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testing

Related

• Breaking the Ceiling: Exploring the Potential of Jailbreak Attacks through Expanding Strategy Space
• Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalization
• Attention Slipping: A Mechanistic Understanding of Jailbreak Attacks and Defenses in LLMs
• Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility
• What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks