Should LLM Safety Be More Than Refusing Harmful Instructions?

2025/06/03 by Maskey, Utsav, Dras, Mark, Naseem, Usman · 3 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.02442

Abstract

This paper presents a systematic evaluation of Large Language Models' (LLMs) behavior on long-tail distributed (encrypted) texts and their safety implications. We introduce a two-dimensional framework for assessing LLM safety: (1) instruction refusal-the ability to reject harmful obfuscated instructions, and (2) generation safety-the suppression of generating harmful responses. Through comprehensive experiments, we demonstrate that models that possess capabilities to decrypt ciphers may be susceptible to mismatched-generalization attacks: their safety mechanisms fail on at least one safety dimension, leading to unsafe responses or over-refusal. Based on these findings, we evaluate a number of pre-LLM and post-LLM safeguards and discuss their strengths and limitations. This work contributes to understanding the safety of LLM in long-tail text scenarios and provides directions for developing robust safety mechanisms.

Citations

• Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilities
• CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenges
• SaRO: Enhancing LLM Safety through Reasoning-based Alignment
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Endless Jailbreaks with Bijection Learning
• Jailbreaking Large Language Models with Symbolic Mathematics
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods
• Efficient Adversarial Training in LLMs with Continuous Attacks
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
• ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs
• When "Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Ciphers
• SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding
• Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey
• Rethinking Machine Unlearning for Large Language Models
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
• DeepInception: Hypnotize Large Language Model to Be Jailbreaker
• Mistral 7B
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
• Baseline Defenses for Adversarial Attacks Against Aligned Language Models
• Detecting Language Model Attacks with Perplexity
• LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked
• GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
• Jailbroken: How Does LLM Safety Training Fail?
• TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• SALSA: Attacking Lattice Cryptography with Transformers
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Finetuned Language Models Are Zero-Shot Learners
• Language Models are Few-Shot Learners
• BPE-Dropout: Simple and Effective Subword Regularization
• BERTScore: Evaluating Text Generation with BERT

Cited by

• DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models
• InvThink: Premortem Reasoning for Safer Language Models
• Too Helpful, Too Harmless, Too Honest or Just Right?
• XGUARD: A Graded Benchmark for Evaluating Safety Failures of Large Language Models on Extremist Content
• Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilities

Related

• Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilities
• Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level
• LLM-Safety Evaluations Lack Robustness
• No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks
• Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks