TRAPDOC: Deceiving LLM Users by Injecting Imperceptible Phantom Tokens into Documents

2025/05/30 by Sicheol Sung, Jin, Hyundong, Sung, Sicheol +6 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computers and Society (cs.CY) #Digital Media Forensic Detection #Digital and Cyber Forensics #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2506.00089

openalex publication_date 2025/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The reasoning, writing, text-editing, and retrieval capabilities of proprietary large language models (LLMs) have advanced rapidly, providing users with an ever-expanding set of functionalities. However, this growing utility has also led to a serious societal concern: the over-reliance on LLMs. In particular, users increasingly delegate tasks such as homework, assignments, or the processing of sensitive documents to LLMs without meaningful engagement. This form of over-reliance and misuse is emerging as a significant social issue. In order to mitigate these issues, we propose a method injecting imperceptible phantom tokens into documents, which causes LLMs to generate outputs that appear plausible to users but are in fact incorrect. Based on this technique, we introduce TRAPDOC, a framework designed to deceive over-reliant LLM users. Through empirical evaluation, we demonstrate the effectiveness of our framework on proprietary LLMs, comparing its impact against several baselines. TRAPDOC serves as a strong foundation for promoting more responsible and thoughtful engagement with language models. Our code is available at https://github.com/jindong22/TrapDoc.

Citations

• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• An LLM can Fool Itself: A Prompt-Based Adversarial Attack
• Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• Red Teaming Language Models with Language Models
• Program Synthesis with Large Language Models
• A Dataset of Information-Seeking Questions and Answers Anchored in\n Research Papers
• FAWA: Fast Adversarial Watermark Attack on Optical Character Recognition (OCR) Systems
• Extracting Training Data from Large Language Models
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• On Faithfulness and Factuality in Abstractive Summarization
• BERT-ATTACK: Adversarial Attack Against BERT Using BERT
• Universal Adversarial Triggers for Attacking and Analyzing NLP
• DuTongChuan: Context-aware Translation Model for Simultaneous Interpreting
• Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
• BERTScore: Evaluating Text Generation with BERT
• Generating Natural Language Adversarial Examples
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks

Cited by

• RAG-Pull: Imperceptible Attacks on RAG Systems for Code Generation
• Model Misalignment and Language Change: Traces of AI-Associated Language in Unscripted Spoken English

Related

• How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts
• Can LLMs be Fooled? Investigating Vulnerabilities in LLMs
• Under the Surface: Tracking the Artifactuality of LLM-Generated Data
• A Lightweight Method to Disrupt Memorized Sequences in LLM
• Misinforming LLMs: vulnerabilities, challenges and opportunities