Universal Adversarial Suffixes for Language Models Using Reinforcement Learning with Calibrated Reward

2025/12/09 by Soor, Sampriti, Ghosh, Suklav, Sur, Arijit
Computer Science · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · doi:10.48550/arxiv.2512.08131

openalex publication_date 2025/12/09 · openalex created_date 2025/12/11 · openalex updated_date 2026/07/28

Abstract

Language models are vulnerable to short adversarial suffixes that can reliably alter predictions. Previous works usually find such suffixes with gradient search or rule-based methods, but these are brittle and often tied to a single task or model. In this paper, a reinforcement learning framework is used where the suffix is treated as a policy and trained with Proximal Policy Optimization against a frozen model as a reward oracle. Rewards are shaped using calibrated cross-entropy, removing label bias and aggregating across surface forms to improve transferability. The proposed method is evaluated on five diverse NLP benchmark datasets, covering sentiment, natural language inference, paraphrase, and commonsense reasoning, using three distinct language models: Qwen2-1.5B Instruct, TinyLlama-1.1B Chat, and Phi-1.5. Results show that RL-trained suffixes consistently degrade accuracy and transfer more effectively across tasks and models than previous adversarial triggers of similar genres.

Citations

• Universal and Transferable Adversarial Attacks on Aligned Language Models
• Jailbroken: How Does LLM Safety Training Fail?
• Training language models to follow instructions with human feedback
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Surface Form Competition: Why the Highest Probability Answer Isn't\n Always Right
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Exploiting Cloze Questions for Few Shot Text Classification and Natural\n Language Inference
• Fine-Tuning Language Models from Human Preferences
• Universal Adversarial Triggers for Attacking and Analyzing NLP
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples
• Proximal Policy Optimization Algorithms
• Sequence Level Training with Recurrent Neural Networks
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Cited by

• C-LEAD: Contrastive Learning for Enhanced Adversarial Defense

Related

• Universal Adversarial Suffixes Using Calibrated Gumbel-Softmax Relaxation
• Fine-Tuning Language Models from Human Preferences
• Finding a Needle in the Adversarial Haystack: A Targeted Paraphrasing Approach For Uncovering Edge Cases with Minimal Distribution Distortion
• Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
• Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning