Extracting Training Data from Diffusion Models

2023/01/30 by Nicholas Carlini, Jamie Hayes, Carlini, Nicholas +15 · 9 voices · 101 citations
Computer Science · #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2301.13188

Abstract

Image diffusion models such as DALL-E 2, Imagen, and Stable Diffusion have attracted significant attention due to their ability to generate high-quality synthetic images. In this work, we show that diffusion models memorize individual images from their training data and emit them at generation time. With a generate-and-filter pipeline, we extract over a thousand training examples from state-of-the-art models, ranging from photographs of individual people to trademarked company logos. We also train hundreds of diffusion models in various settings to analyze how different modeling and data decisions affect privacy. Overall, our results show that diffusion models are much less private than prior generative models such as GANs, and that mitigating these vulnerabilities may require new advances in privacy-preserving training.

Cited by

• DCS: A Unified Conditional Sensitivity Framework for Cross-Modal Copyright Infringement Detection
• Agentic Evaluation of Copyright Law Compliance
• Membership Inference Attacks for Unseen Classes
• Dominant vs. Dominated: Concept-Level Generative Collapse in Diffusion Models
• Signed Rectified Flow: Negativity-Controlled Generation
• RRAM-DP: Device-Calibrated Differential Privacy for In-Memory Edge Learning
• Points as Tori: Fast Pointwise Signed Distance for Point Clouds
• Semi-Supervised Conditional Diffusion via Label Augmentation
• Privacy-Aware Synthetic Video Benchmarking and Relational Evaluation for Worker-Under-Suspended-Load Detection
• On the Interpolation Effect of Score Smoothing in Diffusion Models
• A Common Pool of Privacy Problems: Legal and Technical Lessons from a Large-Scale Web-Scraped Machine Learning Dataset
• How much do language models memorize?
• On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticity
• Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
• Memorization in 3D Shape Generation: An Empirical Study
• Taxing Artificial Intelligence
• A Reinforcement Learning Approach to Synthetic Data Generation
• Generalization of Diffusion Models Arises with a Balanced Representation Space
• AlignDP: Hybrid Differential Privacy with Rarity-Aware Protection for LLMs
• Copyright Infringement Risk Reduction via Chain-of-Thought and Task Instruction Prompting
• Quantifying Return on Security Controls in LLM Systems
• LCMem: A Universal Model for Robust Image Memorization Detection
• Beyond Memorization: Selective Learning for Copyright-Safe Diffusion Model Training
• CAPTAIN: Semantic Feature Injection for Memorization Mitigation in Text-to-Image Diffusion Models
• Unconsciously Forget: Mitigating Memorization; Without Knowing What is being Memorized
• Membership and Dataset Inference Attacks on Large Audio Generative Models
• PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learning
• Prediction with Expert Advice under Local Differential Privacy
• SUGAR: A Sweeter Spot for Generative Unlearning of Many Identities
• Self-Supervised AI-Generated Image Detection: A Camera Metadata Perspective
• Privacy Preserving Diffusion Models for Mixed-Type Tabular Data Generation
• Latent Diffusion Inversion Requires Understanding the Latent Space
• Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?
• Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks
• SPQR: A Standardized Benchmark for Modern Safety Alignment Methods in Text-to-Image Diffusion Models
• The Persistence of Cultural Memory: Investigating Multimodal Iconicity in Diffusion Models
• A PDE Perspective on Generative Diffusion Models
• A Parallel Region-Adaptive Differential Privacy Framework for Image Pixelization
• Dialogues Towards Sociologies of Generative AI
• BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation
• Cyclic Denoising Reveals Ultrastable Memories in Diffusion Models
• RA-Det: Towards Universal Detection of AI-Generated Images via Robustness Asymmetry
• Learning to Attack: Uncovering Privacy Risks in Sequential Data Releases
• On the Anisotropy of Score-Based Generative Models
• Black Box Absorption: LLMs Undermining Innovative Ideas
• Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?
• Nonparametric Data Attribution for Diffusion Models
• Local Differential Privacy for Federated Learning with Fixed Memory Usage and Per-Client Privacy
• DiffEM: Learning from Corrupted Data with Diffusion Models via Expectation Maximization
• A Black-Box Debiasing Framework for Conditional Sampling
• Sensitivity, Specificity, and Consistency: A Tripartite Evaluation of Privacy Filters for Synthetic Data Generation
• Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptive
• Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models
• Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique
• Knowledge Distillation Detection for Open-weights Models
• External Data Extraction Attacks against Retrieval-Augmented Large Language Models
• Selective Underfitting in Diffusion Models
• How Diffusion Models Memorize
• Score-based Membership Inference on Diffusion Models
• A phase transition in diffusion models reveals the hierarchical nature of data
• Copyright Infringement Detection in Text-to-Image Diffusion Models via Differential Privacy
• A Law of Data Reconstruction for Random Features (and Beyond)
• Review of Hallucination Understanding in Large Language and Vision Models
• A Unified Framework for Diffusion Model Unlearning with f-Divergence
• RAG Security and Privacy: Formalizing the Threat Model and Attack Surface
• Responsible Diffusion: A Comprehensive Survey on Safety, Ethics, and Trust in Diffusion Models
• Latent Iterative Refinement Flow: A Geometric-Constrained Approach for Few-Shot Generation
• Adversarial machine learning :
• On the Edge of Memorization in Diffusion Models
• A Novel Metric for Detecting Memorization in Generative Models for Brain MRI Synthesis
• A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective
• Mitigating data replication in text-to-audio generative diffusion models through anti-memorization guidance
• Enterprise AI Must Enforce Participant-Aware Access Control
• MIA-EPT: Membership Inference Attack via Error Prediction for Tabular Data
• ReTrack: Data Unlearning in Diffusion Models through Redirecting the Denoising Trajectory
• Beyond Data Privacy: New Privacy Risks for Large Language Models
• Fundamental Limits of Membership Inference Attacks on Machine Learning Models
• Robust Concept Erasure in Diffusion Models: A Theoretical Perspective on Security and Robustness
• AI-Generated Content in Cross-Domain Applications: Research Trends, Challenges and Propositions
• Prompt Pirates Need a Map: Stealing Seeds helps Stealing Prompts
• ArtifactGen: Benchmarking WGAN-GP vs Diffusion for Label-Aware EEG Artifact Synthesis
• Towards Post-mortem Data Management Principles for Generative AI
• Imitative Membership Inference Attack
• On the MIA Vulnerability Gap Between Private GANs and Diffusion Models
• On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Models
• AMCR: A Framework for Assessing and Mitigating Copyright Risks in Generative Models
• Localizing and Mitigating Memorization in Image Autoregressive Models
• The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang
• Data Cartography for Detecting Memorization Hotspots and Guiding Data Interventions in Generative Models
• Towards 6G Intelligence: The Role of Generative AI in Future Wireless Networks
• Mitigating Data Exfiltration Attacks through Layer-Wise Learning Rate Decay Fine-Tuning
• Demystifying Foreground-Background Memorization in Diffusion Models
• SoK: Data Minimization in Machine Learning
• Towards Effective Prompt Stealing Attack against Text-to-Image Diffusion Models
• Data Exfiltration by Compression Attack: Definition and Evaluation on Medical Image Data
• FedMP: Tackling Medical Feature Heterogeneity in Federated Learning from a Manifold Perspective
• Zero-Residual Concept Erasure via Progressive Alignment in Text-to-Image Model
• TRACEALIGN -- Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs
• A Survey on Data Security in Large Language Models
• LeakyCLIP: Extracting Training Data from CLIP
• LoReUn: Data Itself Implicitly Provides Cues to Improve Machine Unlearning

Discussions

• Extracting training data from diffusion models [hn, 163 points, 309 comments]
• no. arxiv.org/abs/2301.13188 [bsky, 7 points, 2 comments]
• this research paper from 2023 should be similarly interesting. i would go as far as to say the only thing modern “ai” companies have brought to the table to forward the tech is their disregard for cop [bsky, 6 points, 0 comments]
• He do not address the fact that in the paper they address the copyright point and support it independent of the opinion a posterior participants do not change it. and he ignores the teorical corpus ov [bsky, 5 points, 1 comments]
• Extracting Training Data from Diffusion Models [hn, 3 points, 0 comments]
• 潜在拡散モデルで同じ画像を学習時に100回使うと記憶して酷似画像を生成するという論文。ピクセルベースの拡散モデルでは記憶する割合がさらに増えると書かれています。 arxiv.org/abs/2301.13188 私がSD1.5を使用した経験では、データセットに多数画像のあるマリオやプーチンは、名前を書くだけで何度も生成できました。 記憶して再現できるAIを悪用させないために、再現させない技術と法規 [bsky, 2 points, 0 comments]
• Stable Diffusion Memorizes Training Points [bsky, 0 points, 0 comments]
• Esto es falso, las IAGs no imitan ningún proceso mental. Son algoritmos que funcionan a base de scaping data. arxiv.org/abs/2301.13188 [bsky, 0 points, 1 comments]
• I don't want to go too far into this because I don't have time right now, but there's plenty of instances of modern generative models replicating artists work. But for now, here's a paper on a couple [bsky, 0 points, 1 comments]

Related

• Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation
• Differentially Private Diffusion Models Generate Useful Synthetic Images
• Model Will Tell: Training Membership Inference for Diffusion Models
• On the Importance of Conditioning for Privacy-Preserving Data Augmentation
• Leveraging Model Guidance to Extract Training Data from Personalized Diffusion Models