DE-COP: Detecting Copyrighted Content in Language Models Training Data

2024/02/15 by André V. Duarte, Xuandong Zhao, Duarte, André V. +5 · 14 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2 #Machine Learning (cs.LG) #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2402.09910

openalex publication_date 2024/02/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

How can we detect if copyrighted content was used in the training process of a language model, considering that the training data is typically undisclosed? We are motivated by the premise that a language model is likely to identify verbatim excerpts from its training text. We propose DE-COP, a method to determine whether a piece of copyrighted content was included in training. DE-COP's core approach is to probe an LLM with multiple-choice questions, whose options include both verbatim text and their paraphrases. We construct BookTection, a benchmark with excerpts from 165 books published prior and subsequent to a model's training cutoff, along with their paraphrases. Our experiments show that DE-COP surpasses the prior best method by 9.6% in detection performance (AUC) on models with logits available. Moreover, DE-COP also achieves an average accuracy of 72% for detecting suspect books on fully black-box models where prior methods give approximately 4% accuracy. The code and datasets are available at https://github.com/LeiLiLab/DE-COP.

Cited by

• Copyright Detection in Large Language Models: An Ethical Approach to Generative AI Development
• Large Language Models as Search Engines: Societal Challenges
• Forgetting-MarI: LLM Unlearning via Marginal Information Regularization
• EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMs
• RECAP: Reproducing Copyrighted Data from LLMs Training with an Agentic Pipeline
• FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fiction
• Provable Training Data Identification for Large Language Models
• Membership Inference Attacks on Tokenizers of Large Language Models
• Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique
• Leave No TRACE: Black-box Detection of Copyrighted Dataset Usage in Large Language Models via Watermarking
• LogitTrace: Detecting Benchmark Contamination via Layerwise Logit Trajectories
• Causal Understanding by LLMs: The Role of Uncertainty
• Beyond Data Privacy: New Privacy Risks for Large Language Models
• The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage

Related

• Digger: Detecting Copyright Content Mis-usage in Large Language Model Training
• Copyright Traps for Large Language Models
• Measuring Copyright Risks of Large Language Model via Partial Information Probing
• Detecting Pretraining Data from Large Language Models
• Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?