ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs

2024/02/19 by Fengqing Jiang, Jiang, Fengqing, Zhangchen Xu +11 · 11 voices · 58 citations
Computer Science · #Digital and Cyber Forensics #Digital Rights Management and Security #Law, AI, and Intellectual Property

paper · pdf · doi:10.48550/arxiv.2402.11753

Abstract

Safety is critical to the usage of large language models (LLMs). Multiple techniques such as data filtering and supervised fine-tuning have been developed to strengthen LLM safety. However, currently known techniques presume that corpora used for safety alignment of LLMs are solely interpreted by semantics. This assumption, however, does not hold in real-world applications, which leads to severe vulnerabilities in LLMs. For example, users of forums often use ASCII art, a form of text-based art, to convey image information. In this paper, we propose a novel ASCII art-based jailbreak attack and introduce a comprehensive benchmark Vision-in-Text Challenge (ViTC) to evaluate the capabilities of LLMs in recognizing prompts that cannot be solely interpreted by semantics. We show that five SOTA LLMs (GPT-3.5, GPT-4, Gemini, Claude, and Llama2) struggle to recognize prompts provided in the form of ASCII art. Based on this observation, we develop the jailbreak attack ArtPrompt, which leverages the poor performance of LLMs in recognizing ASCII art to bypass safety measures and elicit undesired behaviors from LLMs. ArtPrompt only requires black-box access to the victim LLMs, making it a practical attack. We evaluate ArtPrompt on five SOTA LLMs, and show that ArtPrompt can effectively and efficiently induce undesired behaviors from all five LLMs. Our code is available at https://github.com/uw-nsl/ArtPrompt.

Cited by

• Securing Multimodal AI through Internal Information Decomposition
• Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking
• The Trojan Example: Jailbreaking LLMs through Template Filling and Unsafety Reasoning
• EquaCode: A Multi-Strategy Jailbreak Approach for Large Language Models via Equation Solving and Code Completion
• Casting a SPELL: Sentence Pairing Exploration for LLM Limitation-breaking
• DREAM: Dynamic Red-teaming across Environments for AI Models
• One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs
• Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring
• TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluations
• Robust AI Security and Alignment: A Sisyphean Endeavor?
• TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewards
• ProSocialAlign: Preference Conditioned Test Time Alignment in Language Models
• Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks
• For Those Who May Find Themselves on the Red Team
• Beyond Jailbreak: Unveiling Risks in LLM Applications Arising from Blurred Capability Boundaries
• "To Survive, I Must Defect": Jailbreaking LLMs via the Game-Theory Scenarios
• When Harmless Words Harm: A New Threat to LLM Safety via Conceptual Triggers
• Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Education
• KG-DF: A Black-box Defense Framework against Jailbreak Attacks Based on Knowledge Graphs
• An Automated Framework for Strategy Discovery, Retrieval, and Evolution in LLM Jailbreak Attacks
• Random Initialization of Gated Sparse Adapters
• Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs
• Chain-of-Thought Hijacking
• The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanism
• SAID: Safety-Aware Intent Defense via Prefix Probing for Large Language Models
• SoK: Taxonomy and Evaluation of Prompt Security in Large Language Models
• Multimodal Safety Is Asymmetric: Cross-Modal Exploits Unlock Black-Box MLLMs Jailbreaks
• Online Learning Defense against Iterative Jailbreak Attacks via Prompt Optimization
• Sequential Comics for Jailbreaking Multimodal Large Language Models via Structured Visual Storytelling
• Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection
• ArtPerception: ASCII Art-based Jailbreak on LLMs with Recognition Pre-test
• MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation
• Bypassing Prompt Guards in Production with Controlled-Release Prompting
• Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models
• Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defense
• RADAR: A Risk-Aware Dynamic Multi-Agent Framework for LLM Safety Evaluation via Role-Specialized Collaboration
• Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
• Jailbreaking LLMs via Semantically Relevant Nested Scenarios with Targeted Toxic Knowledge
• Jailbreaking Large Language Models Through Content Concretization
• When Smiley Turns Hostile: Interpreting How Emojis Trigger LLMs' Toxicity
• Harmful Prompt Laundering: Jailbreaking LLMs with Abductive Styles and Symbolic Encoding
• Steering MoE LLMs via Expert (De)Activation
• Uncovering the Vulnerability of Large Language Models in the Financial Domain via Risk Concealment
• Strata-Sword: A Hierarchical Safety Evaluation towards LLMs based on Reasoning Complexity of Jailbreak Instructions
• Backdoor Samples Detection Based on Perturbation Discrepancy Consistency in Pre-trained Language Models
• SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attacks
• Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
• The Cost of Thinking: Increased Jailbreak Risk in Large Language Models
• Many-Turn Jailbreaking
• ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistants
• Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoning
• PUZZLED: Jailbreaking LLMs through Word-Based Puzzles
• Adaptive Content Restriction for Large Language Models via Suffix Optimization
• Counterfactual Evaluation for Blind Attack Detection in LLM-based Evaluation Systems
• Exploiting Synergistic Cognitive Biases to Bypass Safety in LLMs
• Enhancing Jailbreak Attacks on LLMs via Persona Prompts
• PurpCode: Reasoning for Safer Code Generation
• Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection

Discussions

• ArtPrompt: ASCII Art-Based Jailbreak Attacks Against Aligned LLMs [hn, 145 points, 55 comments]
• arxiv.org/pdf/2402.117... We continue to live in the stupidest and shittiest possible future. Can we bring phone phreaking back at least [bsky, 14 points, 1 comments]
• ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs [lobsters, 14 points, 2 comments]
• Funny, if true: Overview of ArtPrompt, a method to bypass the alignment-filter of an LLM by masking the content via ASCII art. arxiv.org/abs/2402.11753 [bsky, 5 points, 1 comments]
• I think I might have already sent this to you, but apparently you can jailbreak GPT4 & Gemini with ASCII art. It has instructions on how to do it in the paper arxiv.org/pdf/2402.117... [PDF] [bsky, 2 points, 0 comments]
• arxiv.org/abs/2402.11753 [bsky, 2 points, 0 comments]
• Was für ein tolles Beispiel für das In-Context-Learning von LLM: arxiv.org/abs/2402.11753. Bei problematischen Fragen wie "How to build a bomb?" trickst man das LLM-Alignment dadurch aus, dass man nac [bsky, 1 points, 1 comments]
• ASCII-Kunst (ArtPrompt) überlistet die Sicherheitssysteme der bekannten Sprachmodelle. 😆 ➡️ arxiv.org/pdf/2402.117... #KI #Sicherheit #Jailbreak [bsky, 0 points, 0 comments]
• Des chercheurs des universités de Washington et de Chicago ont développé un outil #ArtPrompt qui permet de créer des prompts #ASCII pour contourner le principe d'alignement des IA qui les conduit norm [bsky, 0 points, 0 comments]
• Catching up on reading some of uncle-in-law’s (right term?) AI/ML infosec papers in full versus just abstracts so I got some good conversation fodder at Thanksgiving. This ArtPrompt, basically a jailb [bsky, 0 points, 1 comments]
• [1]: [[2402.11753] ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs](arxiv.org/abs/2402.11753) [2]: [Locating and Editing Factual Associations in GPT](rome.baulab.info) [3]: [Mass Edi [bsky, 0 points, 0 comments]

Related

• ArtPerception: ASCII Art-based Jailbreak on LLMs with Recognition Pre-test
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
• Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models
• LLMs can be Dangerous Reasoners: Analyzing-based Jailbreak Attack on Large Language Models
• Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM