Membership Inference Attacks on Sequence Models

2025/06/05 by Lorenzo Rossi, Michael Aerni, Rossi, Lorenzo +5 · 1 citation
Computer Science · Social Sciences · #Advanced Malware Detection Techniques #Adversarial Robustness in Machine Learning #Cryptography and Security (cs.CR) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2506.05126

openalex publication_date 2025/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Sequence models, such as Large Language Models (LLMs) and autoregressive image generators, have a tendency to memorize and inadvertently leak sensitive information. While this tendency has critical legal implications, existing tools are insufficient to audit the resulting risks. We hypothesize that those tools' shortcomings are due to mismatched assumptions. Thus, we argue that effectively measuring privacy leakage in sequence models requires leveraging the correlations inherent in sequential generation. To illustrate this, we adapt a state-of-the-art membership inference attack to explicitly model within-sequence correlations, thereby demonstrating how a strong existing attack can be naturally extended to suit the structure of sequence models. Through a case study, we show that our adaptations consistently improve the effectiveness of memorization audits without introducing additional computational costs. Our work hence serves as an important stepping stone toward reliable memorization audits for large sequence models.

Citations

• Measuring Non-Adversarial Reproduction of Training Data in Large Language Models
• Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data
• Extracting Training Data from Document-Based VQA Models
• Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Do Membership Inference Attacks Work on Large Language Models?
• Memorization in Self-Supervised Learning Improves Downstream Generalization
• Scalable Pre-training of Large Autoregressive Image Models
• Low-Cost High-Power Membership Inference Attacks
• Scalable Extraction of Training Data from (Production) Language Models
• Detecting Pretraining Data from Large Language Models
• Large Language Models
• Membership Inference Attacks against Language Models via Neighbourhood Comparison
• PaLM 2 Technical Report
• Privacy Auditing with One (1) Training Run
• A Note on Normalized Emergence Timing (in Pythia Language Model Evaluations)
• GPT-4 Technical Report
• Extracting Training Data from Diffusion Models
• Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining
• Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models
• The Privacy Onion Effect: Memorization is Relative
• Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets
• Quantifying Memorization Across Neural Language Models
• What Does it Mean for a Language Model to Preserve Privacy?
• Membership Inference Attacks From First Principles
• Enhanced Membership Inference Attacks against Machine Learning Models
• On the Importance of Difficulty Calibration in Membership Inference Attacks
• Extracting Training Data from Large Language Models
• Revisiting Membership Inference Under Realistic Assumptions
• Does Learning Require Memorization? A Short Tale about a Long Tail
• White-box vs Black-box: Bayes Optimal Strategies for Membership Inference
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Attention Is All You Need
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• Membership Inference Attacks against Machine Learning Models
• Long Short-Term Memory

Cited by

• Privacy Risks in Time Series Forecasting: User- and Record-Level Membership Inference

Related

• Membership Inference Attacks on Large-Scale Models: A Survey
• Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
• SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)
• (Token-Level) InfoRMIA: Stronger Membership Inference and Memorization Assessment for LLMs
• A new membership inference attack that spots memorization in generative and predictive models: Loss-Based with Reference Model algorithm (LBRM)