Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data

2024/09/29 by Jie Zhang, Debeshee Das, Zhang, Jie +5 · 1 voice · 22 citations
Computer Science · Decision Sciences · #Cryptography and Security (cs.CR) #Data Quality and Management #FOS: Computer and information sciences #Machine Learning (cs.LG) #Scientific Computing and Data Management #cs.CR #cs.LG

paper · pdf · doi:10.48550/arxiv.2409.19798

openalex publication_date 2024/09/29 · arxiv published 2024/09/29 · arxiv updated 2025/03/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider the problem of a training data proof, where a data creator or owner wants to demonstrate to a third party that some machine learning model was trained on their data. Training data proofs play a key role in recent lawsuits against foundation models trained on web-scale data. Many prior works suggest to instantiate training data proofs using membership inference attacks. We argue that this approach is fundamentally unsound: to provide convincing evidence, the data creator needs to demonstrate that their attack has a low false positive rate, i.e., that the attack's output is unlikely under the null hypothesis that the model was not trained on the target data. Yet, sampling from this null hypothesis is impossible, as we do not know the exact contents of the training set, nor can we (efficiently) retrain a large foundation model. We conclude by offering two paths forward, by showing that data extraction attacks and membership inference on special canary data can be used to create sound training data proofs.

Cited by

• Descend or Rewind? Stochastic Gradient Descent Unlearning
• Observational Auditing of Label Privacy
• Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard
• Provable Training Data Identification for Large Language Models
• Membership Inference Attacks on Tokenizers of Large Language Models
• Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique
• Train Once, Answer All: Many Pretraining Experiments for the Cost of One
• Beyond Data Privacy: New Privacy Risks for Large Language Models
• Privacy Risks in Time Series Forecasting: User- and Record-Level Membership Inference
• The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage
• The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models
• The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation
• Unlocking Post-hoc Dataset Inference with Synthetic Data
• Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning
• SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
• What Really is a Member? Discrediting Membership Inference via Poisoning
• Membership Inference Attacks on Sequence Models
• Practical Bayes-Optimal Membership Inference Attacks
• Auditing Data Provenance in LLM Fine-tuning via Intrinsic Distributional Fingerprints
• From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs
• Efficient Machine Unlearning by Model Splitting and Core Sample Selection
• STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasings

Discussions

• ⚖️ Just because it looks like your data… Jie Zhang argues that membership inference attacks fall short of proving actual training inclusion. Paper: arxiv.org/abs/2409.19798 #SaTML25 [bsky, 3 points, 0 comments]

Related

• Can Membership Inferencing be Refuted?
• Blind Baselines Beat Membership Inference Attacks for Foundation Models
• Membership Inference Attacks via Adversarial Examples
• Membership Inference Attacks From First Principles
• Enhanced Membership Inference Attacks against Machine Learning Models