Antidistillation Fingerprinting

2026/02/03 by Yixuan Even Xu, John Kirchenbauer, Yash Savani +5 · 1 voice
Computer Science · #Advanced Graph Neural Networks #Adversarial Robustness in Machine Learning #Emulation #Fingerprint (computing) #Heuristic #Machine Learning and Data Classification #Mixture model #Pareto principle #Profiling (computer programming) #Quality (philosophy) #Sample (material) #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2602.03812

openalex publication_date 2026/02/03 · arxiv published 2026/02/03 · openalex created_date 2026/02/06 · arxiv updated 2026/05/15 · openalex updated_date 2026/07/28

Abstract

Model distillation enables efficient emulation of frontier large language models (LLMs), creating a need for robust mechanisms to detect when a third-party student model has trained on a teacher model's outputs. However, existing fingerprinting techniques that could be used to detect such distillation rely on heuristic perturbations that impose a steep trade-off between generation quality and fingerprinting strength, often requiring significant degradation of utility to ensure the fingerprint is effectively internalized by the student. We introduce antidistillation fingerprinting (ADFP), a principled approach that aligns the fingerprinting objective with the student's learning dynamics. Building upon the gradient-based framework of antidistillation sampling, ADFP utilizes a proxy model to identify and sample tokens that directly maximize the expected detectability of the fingerprint in the student after fine-tuning, rather than relying on the incidental absorption of the un-targeted biases of a more naive watermark. Experiments on GSM8K, OASST1, and MBPP demonstrate that ADFP achieves a significant Pareto improvement over state-of-the-art baselines, yielding stronger detection confidence with minimal impact on utility across mathematical reasoning, dialogue, and code generation, even when the student model's architecture is unknown.

Citations

• Exploring the limits of strong membership inference attacks on large language models
• Extracting memorized pieces of (copyrighted) books from open-weight language models
• Antidistillation Sampling
• Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
• Detecting Benchmark Contamination Through Watermarking
• Scalable watermarking for identifying large language model outputs
• The Llama 3 Herd of Models
• Qwen2 Technical Report
• Watermarking Makes Language Models Radioactive
• A Survey on Knowledge Distillation of Large Language Models
• Do Membership Inference Attacks Work on Large Language Models?
• On the Learnability of Watermarks for Language Models
• Mark My Words: Analyzing and Evaluating Language Model Watermarks
• Publicly-Detectable Watermarking for Language Models
• Robust Distortion-free Watermarks for Language Models
• Three Bricks to Consolidate Watermarks for Large Language Models
• Students Parrot Their Teachers: Membership Inference on Model Distillation
• A Watermark for Large Language Models
• Quantifying Memorization Across Neural Language Models
• Training Verifiers to Solve Math Word Problems
• Program Synthesis with Large Language Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Dataset Inference: Ownership Resolution in Machine Learning
• Extracting Training Data from Large Language Models
• Knowledge Distillation: A Survey
• Distilling the Knowledge in a Neural Network

Discussions

• We don’t need to trade off AI quality for fingerprinting anymore. By aligning watermarks with a model’s own learning process, antidistillation fingerprinting lets us trace student models without the c [bsky, 0 points, 0 comments]

Related

• ImF: Implicit Fingerprint for Large Language Models
• SeedPrints: Fingerprints Can Even Tell Which Seed Your Large Language Model Was Trained From
• Scalable Fingerprinting of Large Language Models
• A Fingerprint for Large Language Models
• On the Learnability of Watermarks for Language Models