2025/01/30 by Haoyu Liang, Youran Sun, Liang, Haoyu +8 · 1 voice · 2 citations
Computer Science · Social Sciences · #Artificial Intelligence in Law #Digital Rights Management and Security #Digital and Cyber Forensics #cs.AI #cs.CL #cs.LG #cs.NE
paper · pdf · doi:10.48550/arxiv.2501.18280
arxiv published 2025/01/30 · arxiv updated 2025/05/17
The security issue of large language models (LLMs) has gained wide attention recently, with various defense mechanisms developed to prevent harmful output, among which safeguards based on text embedding models serve as a fundamental defense. Through testing, we discover that the output distribution of text embedding models is severely biased with a large mean. Inspired by this observation, we propose novel, efficient methods to search for **universal magic words** that attack text embedding models. Universal magic words as suffixes can shift the embedding of any text towards the bias direction, thus manipulating the similarity of any text pair and misleading safeguards. Attackers can jailbreak the safeguards by appending magic words to user prompts and requiring LLMs to end answers with magic words. Experiments show that magic word attacks significantly degrade safeguard performance on JailbreakBench, cause real-world chatbots to produce harmful outputs in full-pipeline attacks, and generalize across input/output texts, models, and languages. To eradicate this security risk, we also propose defense methods against such attacks, which can correct the bias of text embeddings and improve downstream performance in a train-free manner.