Adversarial Semantic Collisions

2020/11/09 by Congzheng Song, Alexander M. Rush, Song, Congzheng +3 · 4 citations
Computer Science · #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #cs.CL #cs.CR

paper · pdf · doi:10.48550/arxiv.2011.04743

arxiv created 2020/11/09 · arxiv updated 2020/11/11

Abstract

We study semantic collisions: texts that are semantically unrelated but judged as similar by NLP models. We develop gradient-based approaches for generating semantic collisions and demonstrate that state-of-the-art models for many tasks which rely on analyzing the meaning and similarity of texts-- including paraphrase identification, document retrieval, response suggestion, and extractive summarization-- are vulnerable to semantic collisions. For example, given a target query, inserting a crafted collision into an irrelevant document can shift its retrieval rank from 1000 to top 3. We show how to generate semantic collisions that evade perplexity-based filtering and discuss other potential mitigations. Our code is available at https://github.com/csong27/collision-bert.

Cited by

• RobustMask: Certified Robustness against Adversarial Neural Ranking Attack via Randomized Masking
• Statistically Undetectable Backdoors in Deep Neural Networks
• Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systems
• Unsupervised Corpus Poisoning Attacks in Continuous Space for Dense Retrieval

Related

• Semantic Representation Attack against Aligned Large Language Models
• On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex
• Semantic Stealth: Adversarial Text Attacks on NLP Using Several Methods
• Saliency Attention and Semantic Similarity-Driven Adversarial Perturbation
• BERT is Robust! A Case Against Synonym-Based Adversarial Examples in Text Classification