REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective

2025/02/24 by Simon Geisler, Geisler, Simon, Tom Wollschläger +9 · 1 voice · 3 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.LG

paper · pdf · doi:10.48550/arxiv.2502.17254

Abstract

To circumvent the alignment of large language models (LLMs), current optimization-based adversarial attacks usually craft adversarial prompts by maximizing the likelihood of a so-called affirmative response. An affirmative response is a manually designed start of a harmful answer to an inappropriate request. While it is often easy to craft prompts that yield a substantial likelihood for the affirmative response, the attacked model frequently does not complete the response in a harmful manner. Moreover, the affirmative objective is usually not adapted to model-specific preferences and essentially ignores the fact that LLMs output a distribution over responses. If low attack success under such an objective is taken as a measure of robustness, the true robustness might be grossly overestimated. To alleviate these flaws, we propose an adaptive and semantic optimization problem over the population of responses. We derive a generally applicable objective via the REINFORCE policy-gradient formalism and demonstrate its efficacy with the state-of-the-art jailbreak algorithms Greedy Coordinate Gradient (GCG) and Projected Gradient Descent (PGD). For example, our objective doubles the attack success rate (ASR) on Llama3 and increases the ASR from 2% to 50% with circuit breaker defense.

Cited by

• Injecting Falsehoods: Adversarial Man-in-the-Middle Attacks Undermining Factual Recall in LLMs
• AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
• Diffusion LLMs are Natural Adversaries for any LLM

Discussions

• Want to dive deeper and better understand LLM robustness? Read our paper for details on our REINFORCE objective, its implementation, and experimental evaluation. Let us know what you think! 💬👇 arxiv [bsky, 4 points, 1 comments]

Related

• Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent
• The Resurgence of GCG Adversarial Attacks on Large Language Models
• Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Models
• Baseline Defenses for Adversarial Attacks Against Aligned Language Models
• Jailbreaker in Jail: Moving Target Defense for Large Language Models