Watermark Stealing in Large Language Models

2024/02/29 by Nikola Jovanović, Robin Staab, Jovanović, Nikola +3 · 20 citations
Computer Science · #Advanced Steganography and Watermarking Techniques #Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2402.19361

openalex publication_date 2024/02/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

LLM watermarking has attracted attention as a promising way to detect AI-generated content, with some works suggesting that current schemes may already be fit for deployment. In this work we dispute this claim, identifying watermark stealing (WS) as a fundamental vulnerability of these schemes. We show that querying the API of the watermarked LLM to approximately reverse-engineer a watermark enables practical spoofing attacks, as hypothesized in prior work, but also greatly boosts scrubbing attacks, which was previously unnoticed. We are the first to propose an automated WS algorithm and use it in the first comprehensive study of spoofing and scrubbing in realistic settings. We show that for under 50 an attacker can both spoof and scrub state-of-the-art schemes previously considered safe, with average success rate of over 80%. Our findings challenge common beliefs about LLM watermarking, stressing the need for more robust schemes. We make all our code and additional examples available at https://watermark-stealing.org.

Cited by

• MarkTune: Improving the Quality-Detectability Trade-off in Open-Weight LLM Watermarking
• Watermarking Discrete Diffusion Language Models
• Advancing Machine-Generated Text Detection from an Easy to Hard Supervision Perspective
• AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Models
• A Reinforcement Learning Framework for Robust and Secure LLM Watermarking
• On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?
• Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption
• Verifiable Fine-Tuning for LLMs: Zero-Knowledge Training Proofs Bound to Data Provenance and Policy
• SimKey: A Semantically Aware Key Module for Watermarking Language Models
• SynthID-Image: Image watermarking at internet scale
• On the Empirical Power of Goodness-of-Fit Tests in Watermark Detection
• DMark: Order-Agnostic Watermarking for Diffusion Large Language Models
• Are Robust LLM Fingerprints Adversarially Robust?
• Watermarking Diffusion Language Models
• LLM Watermark Evasion via Bias Inversion
• RLCracker: Evaluating the Worst-Case Vulnerability of LLM Watermarks with Adaptive RL Attacks
• MetaSeal: Defending Against Image Attribution Forgery Through Content-Dependent Cryptographic Watermarks
• Character-Level Perturbations Disrupt LLM Watermarks
• Improving Detection of Watermarked Language Models
• Majority Bit-Aware Watermarking For Large Language Models

Related

• Breaking Distortion-free Watermarks in Large Language Models
• Discovering Spoofing Attempts on Language Model Watermarks
• No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices
• Turning Your Strength into Watermark: Watermarking Large Language Model via Knowledge Injection
• Large Language Model Watermark Stealing With Mixed Integer Programming