Extracting Training Data from Large Language Models

2020/12/14 by Nicholas Carlini, Florian Tramèr, Florian Tramer +25 · 8 voices · 493 citations
Computer Science · #Adversarial Robustness in Machine Learning #Privacy-Preserving Technologies in Data #Topic Modeling #cs.CL #cs.CR #cs.LG

paper · pdf · doi:10.48550/arxiv.2012.07805

openalex publication_date 2020/12/14 · arxiv created 2021/06/15 · arxiv updated 2021/06/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.

Citations

• Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.
• Language Models as Knowledge Bases?
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Scaling Laws for Neural Language Models
• Language Models are Few-Shot Learners

Cited by

• Threat Modeling for AI: The Case for an Asset-Centric Approach
• DECAF: De-Clustering for Adaptive Representational Unlearning
• Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization
• Multi-Agent Privacy Game in Federated Learning: A Unified Mean-Field View
• PANOPTICON: A PII-Based Assemblage of Naturalistic Output Tokens for Investigating Privacy Leakage Within LLM Context Window
• Reading Without a Reader: Large Language Models Collapse Reading and Writing into a Single Entangled Code
• Assessing the Effectiveness of Membership Inference on Generative Music
• Beyond Context: Large Language Models' Failure to Grasp Users' Intent
• The Erasure Illusion: Stress-Testing the Generalization of LLM Forgetting Evaluation
• Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
• A Comparative Study of Light-weight Language Models for PII Masking and their Deployment for Real Conversational Texts
• SecureCode: A Production-Grade Multi-Turn Dataset for Training Security-Aware Code Generation Models
• PermuteV: A Performant Side-channel-Resistant RISC-V Core Securing Edge AI Inference
• Perturb Your Data: Paraphrase-Guided Training Data Watermarking
• Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation
• How Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code?
• ContextLeak: Auditing Leakage in Private In-Context Learning Methods
• PerProb: Indirectly Evaluating Memorization in Large Language Models
• IntentMiner: Intent Inversion Attack via Tool Call Analysis in the Model Context Protocol
• CTIGuardian: A Few-Shot Framework for Mitigating Privacy Leakage in Fine-Tuned LLMs
• On the Effectiveness of Membership Inference in Targeted Data Extraction from Large Language Models
• Forgetful but Faithful: A Cognitive Memory Architecture and Benchmark for Privacy-Aware Generative Agents
• Towards Privacy-Preserving Code Generation: Differentially Private Code Language Models
• Unconsciously Forget: Mitigating Memorization; Without Knowing What is being Memorized
• Membership and Dataset Inference Attacks on Large Audio Generative Models
• Reference Recommendation based Membership Inference Attack against Hybrid-based Recommender Systems
• Black-Box Behavioral Distillation Breaks Safety Alignment in Medical LLMs
• When Tables Leak: Attacking String Memorization in LLM-Based Tabular Data Generation
• Exposing and Defending Membership Leakage in Vulnerability Prediction Models
• A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties
• AgentCrypt: Advancing Privacy and (Secure) Computation in AI Agent Collaboration
• Understanding Privacy Risks in Code Models Through Training Dynamics: A Causal Approach
• Privacy Practices of Browser Agents
• LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples
• Recover-to-Forget: Gradient Reconstruction from LoRA for Efficient LLM Unlearning
• Protecting Bystander Privacy via Selective Hearing in Audio LLMs
• PrivCode: When Code Generation Meets Differential Privacy
• Executable Governance for AI: Translating Policies into Rules Using LLMs
• Efficient Public Verification of Private ML via Regularization
• Context-Aware Hierarchical Learning: A Two-Step Paradigm towards Safer LLMs
• SRPG: Semantically Reconstructed Privacy Guard for Zero-Trust Privacy in Educational Multi-Agent Systems
• Grokked Models are Better Unlearners
• Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs
• Towards Contextual Sensitive Data Detection
• Ensemble Privacy Defense for Knowledge-Intensive LLMs against Membership Inference Attacks
• What Is Preference Optimization Doing, How and Why?
• Privacy Preserving Diffusion Models for Mixed-Type Tabular Data Generation
• Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning
• Decomposed Trust: Exploring Privacy, Adversarial Robustness, Fairness, and Ethics of Low-Rank LLMs
• Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?
• Memories Retrieved from Many Paths: A Multi-Prefix Framework for Robust Detection of Training Data Leakage in Large Language Models
• Quantifying the Privacy Implications of High-Fidelity Synthetic Network Traffic
• Exploiting the Experts: Unauthorized Compression in MoE-LLMs
• Geometric-disentangelment Unlearning
• Membership Inference Attacks Beyond Overfitting
• How to Train Private Clinical Language Models: A Comparative Study of Privacy-Preserving Pipelines for ICD-9 Coding
• As If We've Met Before: LLMs Exhibit Certainty in Recognizing Seen Files
• Effective Code Membership Inference for Code Completion Models via Adversarial Prompts
• GRPO Privacy Is at Risk: A Membership Inference Attack Against Reinforcement Learning With Verifiable Rewards
• Observational Auditing of Label Privacy
• Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probing
• AI Bill of Materials and Beyond: Systematizing Security Assurance through the AI Risk Scanning (AIRS) Framework
• Forgetting-MarI: LLM Unlearning via Marginal Information Regularization
• InData: Towards Secure Multi-Step, Tool-Based Data Analysis
• Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models
• Hierarchical Memorization in Large Language Models: Evidence from Citation Generation
• Beyond Superficial Forgetting: Thorough Unlearning through Knowledge Density Estimation and Block Re-insertion
• SALT: Steering Activations towards Leakage-free Thinking in Chain of Thought
• Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety
• Biologically-Informed Hybrid Membership Inference Attacks on Generative Genomic Models
• Uncovering Pretraining Code in LLMs: A Syntax-Aware Attribution Approach
• ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversations
• REMIND: Input Loss Landscapes Reveal Residual Memorization in Post-Unlearning LLMs
• Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations
• Contamination Detection for VLMs using Multi-Modal Semantic Perturbation
• Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning
• Secure PAC Learning: Sample-Budget Laws and Quantum Data-Path Admissibility
• Remembering Unequally: Global and Disciplinary Bias in LLM-Generated Co-Authorship Networks
• EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMs
• Detecting Data Contamination in LLMs via In-Context Learning
• Questionnaire meets LLM: A Benchmark and Empirical Study of Structural Skills for Understanding Questions and Responses
• RECAP: Reproducing Copyrighted Data from LLMs Training with an Agentic Pipeline
• Impossible to hide secret ...: Uncovering Security and Privacy Issues in LLM-native IDEs
• RA-Det: Towards Universal Detection of AI-Generated Images via Robustness Asymmetry
• Unintended Memorization of Sensitive Information in Fine-Tuned Language Models
• A Survey on Unlearning in Large Language Models
• The seven roles of generative AI: Potential & pitfalls in combatting misinformation
• On the Impossibility of Retrain Equivalence in Machine Unlearning
• PrivacyGuard: A Modular Framework for Privacy Auditing in Machine Learning
• Retracing the Past: LLMs Emit Training Data When They Get Lost
• Accelerating Materials Design via LLM-Guided Evolutionary Search
• Leverage Unlearning to Sanitize LLMs
• Adversarially-Aware Architecture Design for Robust Medical AI Systems
• Black Box Absorption: LLMs Undermining Innovative Ideas
• CircuitGuard: Mitigating LLM Memorization in RTL Code Generation Against IP Leakage
• Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?
• Machine Text Detectors are Membership Inference Attacks
• Extracting alignment data in open models
• One Size Fits All? A Modular Adaptive Sanitization Kit (MASK) for Customizable Privacy-Preserving Phone Scam Detection
• ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control
• Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models
• Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting
• Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures
• MAGPIE: A benchmark for Multi-AGent contextual PrIvacy Evaluation
• Backdoor Unlearning by Linear Task Decomposition
• VaultGemma: A Differentially Private Gemma Model
• ShuffleV: A Microarchitectural Defense Strategy against Electromagnetic Side-Channel Attacks in Microprocessors
• Local Differential Privacy for Federated Learning with Fixed Memory Usage and Per-Client Privacy
• A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness
• An Investigation of Memorization Risk in Healthcare Foundation Models
• CoSPED: Consistent Soft Prompt Targeted Data Extraction and Defense
• Secret-Protected Evolution for Differentially Private Synthetic Text Generation
• Quantifying Information Disclosure During Gradient Descent Using Gradient Uniqueness
• SoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Training
• Early Detection and Reduction of Memorisation for Domain Adaptation and Instruction Tuning
• ArtPerception: ASCII Art-based Jailbreak on LLMs with Recognition Pre-test
• CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs
• RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation
• Provable Training Data Identification for Large Language Models
• Getting Your Indices in a Row: Full-Text Search for LLM Training Data for Real World
• From Defender to Devil? Unintended Risk Interactions Induced by LLM Defenses
• Position: Privacy Is Not Just Memorization!
• The Model's Language Matters: A Comparative Privacy Analysis of LLMs
• Cocoon: A System Architecture for Differentially Private Training with Correlated Noises
• LLMs Show Surface-Form Brittleness Under Paraphrase Stress Tests
• Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models
• Memorization in Language Models through the Lens of Intrinsic Dimension
• Empirical Comparison of Membership Inference Attacks in Deep Transfer Learning
• Membership Inference Attacks on Tokenizers of Large Language Models
• (Token-Level) InfoRMIA: Stronger Membership Inference and Memorization Assessment for LLMs
• Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique
• InvThink: Premortem Reasoning for Safer Language Models
• Detecting Distillation Data from Reasoning Models
• VortexPIA: Indirect Prompt Injection Attack against LLMs for Efficient Extraction of User Privacy
• AgentHub: A Registry for Discoverable, Verifiable, and Reproducible AI Agents
• External Data Extraction Attacks against Retrieval-Augmented Large Language Models
• Leave No TRACE: Black-box Detection of Copyrighted Dataset Usage in Large Language Models via Watermarking
• Re-examining Low Rank adaptation for private LLM fine-tuning
• "We are not Future-ready": Understanding AI Privacy Risks and Existing Mitigation Strategies from the Perspective of AI Developers in Europe
• Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbations
• On The Fragility of Benchmark Contamination Detection in Reasoning Models
• Attention over Scene Graphs: Indoor Scene Representations Toward CSAI Classification
• Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusions
• Controlled Generation for Private Synthetic Text
• When Hallucination Costs Millions: Benchmarking AI Agents in High-Stakes Adversarial Financial Markets
• BloomAPR: A Bloom's Taxonomy-based Framework for Assessing the Capabilities of LLM-Powered APR Solutions
• Comparing Open-Source and Commercial LLMs for Domain-Specific Analysis and Reporting: Software Engineering Challenges and Design Trade-offs
• Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmark
• Adaptive Token-Weighted Differential Privacy for LLMs: Not All Tokens Require Equal Protection
• A Law of Data Reconstruction for Random Features (and Beyond)
• No Prior, No Leakage: Revisiting Reconstruction Attacks in Trained Neural Networks
• LogitTrace: Detecting Benchmark Contamination via Layerwise Logit Trajectories
• Can Federated Learning Safeguard Private Data in LLM Training? Vulnerabilities, Attacks, and Defense Evaluation
• CURE: Centroid-guided Unsupervised Representation Erasure for Facial Recognition Systems
• The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation From Recognition to Reasoning
• From moral panic to pragmatic governance: reframing AI’s societal impacts in employment, education, and ethics
• Ghost in the cache: How data decay shapes the unseen landscape of AI memory
• ISACL: Internal State Analyzer for Copyrighted Training Data Leakage
• Memory in Large Language Models: Mechanisms, Evaluation and Evolution
• Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs
• RL-Finetuned LLMs for Privacy-Preserving Synthetic Rewriting
• Can You Trust Your Copilot? A Privacy Scorecard for AI Coding Assistants
• SLM-Based Agentic AI with P-C-G: Optimized for Korean Tool Use
• Sparse-Autoencoder-Guided Internal Representation Unlearning for Large Language Models
• Concept Unlearning in Large Language Models via Self-Constructed Knowledge Triplets
• Enterprise AI Must Enforce Participant-Aware Access Control
• SynBench: A Benchmark for Differentially Private Text Generation
• Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning
• Risk Assessment and Security Analysis of Large Language Models
• Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning
• Beyond Data Privacy: New Privacy Risks for Large Language Models
• Membership Inference Attack against Large Language Model-based Recommendation Systems: A New Distillation-based Paradigm
• The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration
• Exposing Privacy Risks in Graph Retrieval-Augmented Generation
• Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inference
• Amulet: a Python Library for Assessing Interactions Among ML Defenses and Risks
• From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teaming
• A Biosecurity Agent for Lifecycle LLM Biosecurity Alignment
• LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
• Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora
• Generative Data Refinement: Just Ask for Better Data
• Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implications
• Imitative Membership Inference Attack
• "Abuse Risks are Often Inherent to Product Features": Exploring AI Vendors' Bug Bounty and Responsible Disclosure Policies
• Preventing Another Tessa: Modular Safety Middleware For Health-Adjacent AI Assistants
• Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysis
• Privacy Risks in Time Series Forecasting: User- and Record-Level Membership Inference
• MEUV: Achieving Fine-Grained Capability Activation in Large Language Models via Mutually Exclusive Unlock Vectors
• EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint
• Gaming and Cooperation in Federated Learning: What Can Happen and How to Monitor It
• The Need for Verification in AI-Driven Scientific Discovery
• DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks
• Localizing and Mitigating Memorization in Image Autoregressive Models
• Comparative Analysis of Large Language Models for the Machine-Assisted Resolution of User Intentions
• Going over Fine Web with a Fine-Tooth Comb: Technical Report of Indexing Fine Web for Problematic Content Search and Retrieval
• Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data
• Evaluating Differentially Private Generation of Domain-Specific Text
• Assessing the Geolocation Capabilities, Limitations and Societal Risks of Generative Vision-Language Models
• Secure Multi-LLM Agentic AI and Agentification for Edge General Intelligence by Zero-Trust: A Survey
• Data Cartography for Detecting Memorization Hotspots and Guiding Data Interventions in Generative Models
• Network-Level Prompt and Trait Leakage in Local Research Agents
• Memorization in Graph Neural Networks
• Membership Inference Attacks on LLM-based Recommender Systems
• A Study of Privacy-preserving Language Modeling Approaches
• Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent
• Incident Analysis for AI Agents
• Promoting Self-Regulation Progress and Knowledge Construction in Blended Learning via ChatGPT-Based Learning Aid
• KLUE: Korean Language Understanding Evaluation
• RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns
• Unlearning at Scale: Implementing the Right to be Forgotten in Large Language Models
• Demystifying Foreground-Background Memorization in Diffusion Models
• Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends
• Searching for Privacy Risks in LLM Agents via Simulation
• Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
• The PacifAIst Benchmark:Would an Artificial Intelligence Choose to Sacrifice Itself for Human Safety?
• The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage
• Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference
• Security Analysis of ChatGPT: Threats and Privacy Risks
• Securing Educational LLMs: A Generalised Taxonomy of Attacks on LLMs and DREAD Risk Assessment
• Several Issues Regarding Data Governance in AGI
• Towards Aligning Personalized Conversational Recommendation Agents with Users' Privacy Preferences
• Assessing and Mitigating Data Memorization Risks in Fine-Tuned Large Language Models
• Dynamic Benchmark Construction for Evaluating Large Language Models on Real-World Codes
• HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathways
• LLM Unlearning using Gradient Ratio-Based Influence Estimation and Noise Injection
• PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction
• DP-DocLDM: Differentially Private Document Image Generation using Latent Diffusion Models
• Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks
• Current State in Privacy-Preserving Text Preprocessing for Domain-Agnostic NLP
• Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generation
• Guess or Recall? Training CNNs to Classify and Localize Memorization in LLMs
• A Survey on Data Security in Large Language Models
• ConfGuard: A Simple and Effective Backdoor Detection for Large Language Models
• Towards Evaluation for Real-World LLM Unlearning
• Win-k: Improved Membership Inference Attacks on Small Language Models
• LeakyCLIP: Extracting Training Data from CLIP
• How Quantization Impacts Privacy Risk on LLMs for Code?
• Evaluating the Dynamics of Membership Privacy in Deep Learning
• Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems
• Rote Learning Considered Useful: Generalizing over Memorized Data in LLMs
• Cascading and Proxy Membership Inference Attacks
• Memorization in Fine-Tuned Large Language Models
• Uncovering Gradient Inversion Risks in Practical Language Model Training
• Bridging AI Innovation and Healthcare Needs: Lessons Learned from Incorporating Modern NLP at The BC Cancer Registry
• PromptArmor: Simple yet Effective Prompt Injection Defenses
• A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction
• Privacy-Preserving Synthetic Review Generation with Diverse Writing Styles Using LLMs
• AnalogFed: Privacy-Preserving Discovery of Analog Circuits at Scale with Federated Generative AI
• LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Models
• Privacy Risks of LLM-Empowered Recommender Systems: An Inversion Attack Perspective
• Rethinking Memorization Measures and their Implications in Large Language Models
• Reading the Finetuning Prior: Verbatim Content Recovery via Contrastive Decoding Diffing
• VectorSmuggle: Steganographic Exfiltration in Embedding Stores and a Cryptographic Provenance Defense
• Characterizing Linear Alignment Across Language Models
• Tab-MIA: A Benchmark Dataset for Membership Inference Attacks on Tabular Data in LLMs
• Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generation
• Differentially Private Conformal Prediction via Quantile Binary Search
• Adversarial machine learning :
• What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests
• On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalance
• PRM-Free Security Alignment of Large Models via Red Teaming and Adversarial Training
• SoK: Machine Unlearning for Large Language Models
• On the Relevance of Logic for Artificial Intelligence, and the Promise of Neurosymbolic Learning
• VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models
• Reconstructing Template-Memorized Images from Natural Prompts
• A Unified Framework for Adversary-Aware Differential Privacy Bounds
• GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testing
• FlexOlmo: Open Language Models for Flexible Data Use
• TextCloak: Thwarting Unauthorized LLM Exploitation via RL-Driven Unlearnable Text
• The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation
• Data Compressibility Quantifies LLM Memorization
• The Hidden Threat in Plain Text: Attacking RAG Data Loaders
• Adversarial Machine Learning Attacks on Financial Reporting via Maximum Violated Multi-Objective Attack
• Efficient Unlearning with Privacy Guarantees
• Model Inversion Attacks on Llama 3: Extracting PII from Large Language Models
• UniAud: A Unified Auditing Framework for High Auditing Power and Utility with One Training Run
• Blackbox Dataset Inference for LLM
• PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage
• Rethinking Data Protection in the (Generative) Artificial Intelligence Era
• Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Training
• Low-Perplexity LLM-Generated Sequences and Where To Find Them
• Pitfalls of Evaluating Language Models with Open Benchmarks
• A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
• Agentic Surgical AI: Surgeon Style Fingerprinting and Privacy Risk Quantification via Discrete Diffusion in a Vision-Language-Action Framework
• Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Models
• Securing AI Systems: A Guide to Known Attacks and Impacts
• SoK: Data Reconstruction Attacks Against Machine Learning Models: Definition, Metrics, and Benchmark
• The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements
• Revisiting the Past: Data Unlearning with Model State History
• A Survey on Model Extraction Attacks and Defenses for Large Language Models
• Counterfactual Influence as a Distributional Quantity
• JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation
• MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation
• PrivacyXray: Detecting Privacy Breaches in LLMs through Semantic Consistency and Probability Certainty
• Automated Detection of Pre-training Text in Black-box LLMs
• Classifying Hate: Legal and Ethical Evaluations of ML-Assisted Hate Crime Classification and Estimation in Sweden
• Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models
• LastingBench: Defend Benchmarks Against Knowledge Leakage
• AI Safety vs. AI Security: Demystifying the Distinction and Boundaries
• Federated In-Context Learning: Iterative Refinement for Improved Answer Quality
• Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs
• Black-Box Privacy Attacks on Shared Representations in Multitask Learning
• From Teacher to Student: Tracking Memorization Through Model Distillation
• Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediaries
• Approximating Language Model Training Data from Weights
• Unlocking Post-hoc Dataset Inference with Synthetic Data
• Memory-Efficient Differentially Private Training with Gradient Random Projection
• Understanding Verbatim Memorization in LLMs Through Circuit Discovery
• Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning
• LexiMark: Robust Watermarking via Lexical Substitutions to Enhance Membership Verification of an LLM's Textual Training Data
• Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble
• RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimality
• A Practical Guide for Evaluating LLMs and LLM-Reliant Systems
• SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
• Differentially Private Bilevel Optimization: Efficient Algorithms with Near-Optimal Rates
• Beyond Frequency: The Role of Redundancy in Large Language Model Memorization
• OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
• Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Models
• Privacy Reasoning in Ambiguous Contexts
• Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information
• Black-Box Access is Insufficient for Rigorous AI Audits
• GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models
• Can We Infer Confidential Properties of Training Data from LLMs?
• SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
• UCD: Unlearning in LLMs via Contrastive Decoding
• Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completeness
• System-Aware Unlearning Algorithms: Use Lesser, Forget Faster
• Hey, That's My Data! Label-Only Dataset Inference in Large Language Models
• FedShield-LLM: A Secure and Scalable Federated Fine-Tuned Large Language Model
• What Really is a Member? Discrediting Membership Inference via Poisoning
• Benchmarking Multimodal AutoML for Tabular Data with Text Fields
• Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models
• Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models
• Membership Inference Attacks on Sequence Models
• Quantifying Cross-Modality Memorization in Vision-Language Models
• UNO: Unlearning via Orthogonalization in Generative models
• Survey on the Evaluation of Generative Models in Music
• Differentially Private Learning Needs Better Features (or Much More Data)
• FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisition
• Learning to Diagnose Privately: DP-Powered LLMs for Radiology Report Classification
• OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models
• Hyperparameter Tuning with Renyi Differential Privacy
• Defending Medical Image Diagnostics against Privacy Attacks using Generative Methods
• DMRL: Data- and Model-aware Reward Learning for Data Extraction
• Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning
• Dynamic Social Networks in Dairy Cows
• IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Data
• Privacy Leaks by Adversaries: Adversarial Iterations for Membership Inference Attack
• Trade-offs in Data Memorization via Strong Data Processing Inequalities
• SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Design
• Adversarial Attacks in Multimodal Systems: A Practitioner's Survey
• Automatic Calibration for Membership Inference Attack on Large Language Models
• Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning
• Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution
• A new membership inference attack that spots memorization in generative and predictive models: Loss-Based with Reference Model algorithm (LBRM)
• Practical Bayes-Optimal Membership Inference Attacks
• Hush! Protecting Secrets During Model Training: An Indistinguishability Approach
• TRAPDOC: Deceiving LLM Users by Injecting Imperceptible Phantom Tokens into Documents
• Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM
• The End Of Universal Lifelong Identifiers: Identity Systems For The AI Era
• Bayesian Perspective on Memorization and Reconstruction
• Vid-SME: Membership Inference Attacks against Large Video Understanding Models
• Permissioned LLMs: Enforcing Access Control in Large Language Models
• OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literature
• A Closer Look on Memorization in Tabular Diffusion Model: A Data-Centric Perspective
• CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building
• Differential Privacy for Text Analytics via Natural Text Sanitization
• Automated Privacy Information Annotation in Large Language Model Interactions
• Privacy Risks and Preservation Methods in Explainable Artificial Intelligence: A Scoping Review
• SHE-LoRA: Selective Homomorphic Encryption for Federated Tuning with Heterogeneous LoRA
• The Impact of a Chatbot's Ephemerality-Framing on Self-Disclosure Perceptions
• Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis
• Spurious Privacy Leakage in Neural Networks
• Querying Kernel Methods Suffices for Reconstructing their Training Data
• Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects
• Smoothie: Smoothing Diffusion on Token Embeddings for Text Generation
• SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks
• CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting
• Auditing Data Provenance in LLM Fine-tuning via Intrinsic Distributional Fingerprints
• Discovering Forbidden Topics in Language Models
• Cracking Aegis: An Adversarial LLM-based Game for Raising Awareness of Vulnerabilities in Privacy Protection
• BitHydra: Towards Bit-flip Inference Cost Attack against Large Language Models
• LLM Fingerprinting via Semantically Conditioned Watermarks
• Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models
• Does Localization Inform Unlearning? A Rigorous Examination of Local Parameter Attribution for Knowledge Unlearning in Language Models
• Covariance-Aware Private Mean Estimation Without Private Covariance\n Estimation
• Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities
• DUSK: Do Not Unlearn Shared Knowledge
• Covert Attacks on Machine Learning Training in Passively Secure MPC
• Secrets Everywhere: Auditing Memorization in Mobility Prediction Models
• Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!
• R-TOFU: Unlearning in Large Reasoning Models
• Context Reasoner: Incentivizing Reasoning Capability for Contextualized Privacy and Safety Compliance via Reinforcement Learning
• Adversarially Pretrained Transformers may be Universally Robust In-Context Learners
• Fragments to Facts: Partial-Information Fragment Inference from LLMs
• Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generation
• MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning
• Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
• GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection
• Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integration
• Automated Profile Inference with Language Model Agents
• How should the advancement of large language models affect the practice of science?
• PANORAMA: A synthetic PII-laced dataset for studying sensitive data memorization in LLMs
• On Membership Inference Attacks in Knowledge Distillation
• Artificial Intelligence and Modeling & Simulation: An Overview
• Adversarial Attack on Large Language Models using Exponentiated Gradient Descent
• PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization
• Toward a Public and Secure Generative AI: A Comparative Analysis of Open and Closed LLMs
• AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMs
• Similarity-Aware Machine Unlearning
• Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models
• Improved Algorithms for Differentially Private Language Model Alignment
• Security of Internet of Agents: Attacks and Countermeasures
• Minimizing Risk Through Minimizing Model-Data Interaction: A Protocol For Relying on Proxy Tasks When Designing Child Sexual Abuse Imagery Detection Models
• Dependency-Aware Privacy for Multi-turn Agents
• Efficient Full-Stack Private Federated Deep Learning with Post-Quantum Security
• MAPLE: Metadata Augmented Private Language Evolution
• A Survey on Privacy Risks and Protection in Large Language Models
• Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs
• Attack and defense techniques in large language models: A survey and new perspectives
• LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasures
• Training Data Leakage Analysis in Language Models
• Antidistillation Fingerprinting
• On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance
• Private Alternating Least Squares: Practical Private Matrix Completion with Tighter Rates
• Hybrid privacy-aware semantic search: SVD-truncated document geometry and CKKS-encrypted query reranking under a restricted threat model
• Internal Data Repetition Destroys Language Models
• What LLMs Think When You Don't Tell Them What to Think About?
• STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations
• OptiLeak: Efficient Prompt Reconstruction via Reinforcement Learning in Multi-tenant LLM Services
• Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
• Intent Laundering: AI Safety Datasets Are Not What They Seem
• Causal methods for LLM development and evaluation
• We Should Separate Memorization from Copyright
• Trust The Typical
• The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggers
• XBreaking: Understanding how LLMs security alignment can be broken
• AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems
• Multimodal Large Language Models for Medicine: A Comprehensive Survey
• Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts
• An LLM-Powered Semantic Alignment Framework for Journal Recommendation
• LLM Features Can Hurt GNNs: Concatenation Interference on Homophilous Graph Benchmarks
• Deployment-Time Memorization in Foundation-Model Agents
• LLMSurgeon: Diagnosing Data Mixture of Large Language Models
• Reproducibility is the New Copyleft: Defining AGI-oriented Reproducible Builds
• Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation
• The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques
• Memorization Dynamics in Knowledge Distillation for Language Models
• A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage
• A Catalog of Data Smells for Coding Tasks
• The Automation Advantage in AI Red Teaming
• Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generation
• Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Models
• Revisiting Data Auditing in Large Vision-Language Models
• Enhancing Privacy-Utility Trade-offs to Mitigate Memorization in Diffusion Models
• Per-parameter Task Arithmetic for Unlearning in Large Language Models
• Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance
• Guardrails for trust, safety, and ethical development and deployment of Large Language Models (LLM)
• Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning
• Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks
• Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation
• Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores
• HalluLens: LLM Hallucination Benchmark
• Assessing the Potential of Generative Agents in Crowdsourced Fact-Checking
• Large Language Models and Social Media Information Integrity: Opportunities, Challenges, and Research Directions
• Private Direct Preference Optimization for LLM Alignment
• When Do PEFT Adaptations Leak Structure? Measuring Black-Box Structural Bounds in Public-Base Model Services
• Behavioral Skill Reconstruction: Reconstructing Hidden Functionality from LLM Agent Skills
• Rethinking Reflection in Pre-Training
• SoK: How Frontier AI Reshapes System-Level Security Risk Dynamics in Critical Infrastructure
• How Should AI Safety Benchmarks Benchmark Safety?
• Frontier AI's Impact on the Cybersecurity Landscape
• Certified Mitigation of Worst-Case LLM Copyright Infringement
• How Private is Your Attention? Bridging Privacy with In-Context Learning
• ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
• DP2Unlearning: An Efficient and Guaranteed Unlearning Framework for LLMs
• STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasings
• SHA256 at SemEval-2025 Task 4: Selective Amnesia -- Constrained Unlearning for Large Language Models via Knowledge Isolation
• GROM: Gradient-Free Rapid One-Shot Machine Unlearning
• Disparate Privacy Vulnerability: Targeted Attribute Inference Attacks and Defenses
• Where Privacy Risk Lives in English-Source Multilingual RAG: A Stage-Decomposed Audit Across Five Query Languages
• Agentic Nesting: A New Methodology for Existing Enterprise Application Integration and Services
• Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuning
• The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections
• ControlNET: A Firewall for RAG-based LLM System
• Measuring LLM Novelty As The Frontier Of Original And High-Quality Output
• Preserving Privacy Without Compromising Accuracy: Machine Unlearning for Handwritten Text Recognition
• Large Language Models Could Be Rote Learners
• Toward Holistic Evaluation of Recommender Systems Powered by Generative Models
• Sharpness-Aware Parameter Selection for Machine Unlearning
• Measuring Déjà vu Memorization Efficiently
• Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs
• Prεεmpt: Sanitizing Sensitive Prompts for LLMs

Discussions

• ところで、AIの学習データをモデルから引き出す方法は、こんなふうにいつでも読める文献にあります。 こちらの論文は4年前のもので、効率は33%です。引き出せた情報の中に個人情報の件数はこの表のとおりです。 arxiv.org/abs/2012.07805 [bsky, 4 points, 0 comments]
• Extracting Training Data from Large Language Models [hn, 2 points, 0 comments]
• Here's one example against GPT2 (trying to look for more recent ones, I see a lot of papers with math but not (yet) the kind that's useful to determine what fraction of samples are likely to be recove [bsky, 1 points, 1 comments]
• Extracting Training Data from Large Language Models [hn, 1 points, 0 comments]
• Extracting Training Data from Large Language Models [hn, 1 points, 0 comments]
• Not entirely, no. This is not the first paper to show that transformer based LLMs memorize verbatim data and that it is possible to extract it: arxiv.org/pdf/2012.07805 [bsky, 1 points, 0 comments]
• Training Data Extraction Attack arxiv.org/pdf/2012.078... [bsky, 0 points, 0 comments]
• and yet, "Our results show that state-of-the-art LMs do memorize their training data in practice" arxiv.org/pdf/2012.07805 [bsky, 0 points, 1 comments]

Related

• Towards More Realistic Extraction Attacks: An Adversarial Perspective
• Pandora's White-Box: Precise Training Data Detection and Extraction in Large Language Models
• Scalable Extraction of Training Data from (Production) Language Models
• Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Models
• On the Effectiveness of Membership Inference in Targeted Data Extraction from Large Language Models