MACE: Mass Concept Erasure in Diffusion Models
2024/03/10 by Shilin Lu, Lu, Shilin, Zilan Wang +7 · 72 citations
Chemistry · Engineering · #Analytical Chemistry and Chromatography #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Nuclear reactor physics and engineering
paper · pdf · doi:10.48550/arxiv.2403.06135
openalex publication_date 2024/03/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
The rapid expansion of large-scale text-to-image diffusion models has raised growing concerns regarding their potential misuse in creating harmful or misleading content. In this paper, we introduce MACE, a finetuning framework for the task of mass concept erasure. This task aims to prevent models from generating images that embody unwanted concepts when prompted. Existing concept erasure methods are typically restricted to handling fewer than five concepts simultaneously and struggle to find a balance between erasing concept synonyms (generality) and maintaining unrelated concepts (specificity). In contrast, MACE differs by successfully scaling the erasure scope up to 100 concepts and by achieving an effective balance between generality and specificity. This is achieved by leveraging closed-form cross-attention refinement along with LoRA finetuning, collectively eliminating the information of undesirable concepts. Furthermore, MACE integrates multiple LoRAs without mutual interference. We conduct extensive evaluations of MACE against prior methods across four different tasks: object erasure, celebrity erasure, explicit content erasure, and artistic style erasure. Our results reveal that MACE surpasses prior methods in all evaluated tasks. Code is available at https://github.com/Shilin-LU/MACE.
Cited by
- PurifyGen: A Risk-Discrimination and Semantic-Purification Model for Safe Text-to-Image Generation
- M-ErasureBench: A Comprehensive Multimodal Evaluation Benchmark for Concept Erasure in Diffusion Models
- LU-500: A Logo Benchmark for Concept Unlearning
- To Erase, or Not to Erase: Robust Training-Free Concept Erasure with Preservation aware Adaptive Ranked Subspace Expansion
- ReVision : A Post-Hoc, Vision-Based Technique for Replacing Unacceptable Concepts in Image Generation Pipeline
- EMMA: Concept Erasure Benchmark with Comprehensive Semantic Metrics and Diverse Categories
- Bi-Erasing: A Bidirectional Framework for Concept Removal in Diffusion Models
- From Unlearning to UNBRANDING: A Benchmark for Trademark-Safe Text-to-Image Generation
- Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimality
- Unconsciously Forget: Mitigating Memorization; Without Knowing What is being Memorized
- SUGAR: A Sweeter Spot for Generative Unlearning of Many Identities
- Metaphor-based Jailbreak Attacks on Text-to-Image Models
- HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval
- Distill, Forget, Repeat: A Framework for Continual Unlearning in Text-to-Image Diffusion Models
- BideDPO: Conditional Image Generation with Simultaneous Text and Condition Alignment
- SPQR: A Standardized Benchmark for Modern Safety Alignment Methods in Text-to-Image Diffusion Models
- Now You See It, Now You Don't - Instant Concept Erasure for Safe Text-to-Image and Video Generation
- Nested Unfolding Network for Real-World Concealed Object Segmentation
- GrOCE:Graph-Guided Online Concept Erasure for Text-to-Image Diffusion Models
- CompTrack: Information Bottleneck-Guided Low-Rank Dynamic Token Compression for Point Cloud Tracking
- Value-Aligned Prompt Moderation via Zero-Shot Agentic Rewriting for Safe Image Generation
- Diffusion-Based Image Editing: An Unforeseen Adversary to Robust Invisible Watermarks
- EraseFlow: Learning Concept Erasure Policies via GFlowNet-Driven Alignment
- Rethinking Robust Adversarial Concept Erasure in Diffusion Models
- Beyond Fixed Anchors: Precisely Erasing Concepts with Sibling Exclusive Counterparts
- AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts
- Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models
- T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model
- UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Dataset
- One-step Diffusion Models with Bregman Density Ratio Matching
- Enhancing Time Series Forecasting through Selective Representation Spaces: A Patch Perspective
- STAR: Boosting Time Series Foundation Models for Anomaly Detection through State-aware Adapter
- DragFlow: Unleashing DiT Priors with Region Based Supervision for Drag Editing
- DeRainMamba: A Frequency-Aware State Space Model with Detail Enhancement for Image Deraining
- Diffusion-Based Image Editing for Breaking Robust Watermarks
- Erased, But Not Forgotten: Erased Rectified Flow Transformers Still Remain Unsafe Under Concept Attack
- Revoking Amnesia: RL-based Trajectory Optimization to Resurrect Erased Concepts in Diffusion Models
- ASTGI: Adaptive Spatio-Temporal Graph Interactions for Irregular Multivariate Time Series Forecasting
- Aurora: Towards Universal Generative Multimodal Time Series Forecasting
- Unlocking the Power of Mixture-of-Experts for Task-Aware Time Series Analytics
- Memory Self-Regeneration: Uncovering Hidden Knowledge in Unlearned Models
- DyME: Dynamic Multi-Concept Erasure in Diffusion Models with Bi-Level Orthogonal LoRA Adaptation
- Does FLUX Already Know How to Perform Physically Plausible Image Composition?
- A Unified Framework for Diffusion Model Unlearning with f-Divergence
- A Single Neuron Works: Precise Concept Erasure in Text-to-Image Diffusion Models
- Robust RGB-T Tracking via Learnable Visual Fourier Prompt Fine-tuning and Modality Fusion Prompt Generation
- SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoders
- SSCM: A Spatial-Semantic Consistent Model for Multi-Contrast MRI Super-Resolution
- VCE: Safe Autoregressive Image Generation via Visual Contrast Exploitation
- Robust Concept Erasure in Diffusion Models: A Theoretical Perspective on Security and Robustness
- Beyond Frame-wise Tracking: A Trajectory-based Paradigm for Efficient Point Cloud Tracking
- SuMa: A Subspace Mapping Approach for Robust and Effective Concept Erasure in Text-to-Image Diffusion Models
- VideoEraser: Concept Erasure in Text-to-Video Diffusion Models
- PractiLight: Practical Light Control Using Foundational Diffusion Models
- Safe-Control: A Safety Patch for Mitigating Unsafe Content in Text-to-Image Generation Models
- Side Effects of Erasing Concepts from Diffusion Models
- VisionLaw: Inferring Interpretable Intrinsic Dynamics from Visual Observations via Bilevel Optimization
- Noise Matters: Optimizing Matching Noise for Diffusion Classifiers
- UnGuide: Learning to Forget with LoRA-Guided Diffusion Models
- Zero-Residual Concept Erasure via Progressive Alignment in Text-to-Image Model
- Personalized Safety Alignment for Text-to-Image Diffusion Models
- CylinderPlane: Nested Cylinder Representation for 3D-aware Image Generation
- ZIUM: Zero-Shot Intent-Aware Adversarial Attack on Unlearned Models
- Decoupled Spatio-Temporal Consistency Learning for Self-Supervised Tracking
- CineVision: An Interactive Pre-visualization Storyboard System for Director-Cinematographer Collaboration
- Towards Universal Modal Tracking with Online Dense Temporal Token Learning
- A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction
- FADE: Adversarial Concept Erasure in Flow Models
- Minimalist Concept Erasure in Generative Models
- Show and Polish: Reference-Guided Identity Preservation in Face Video Restoration
- PLA: Prompt Learning Attack against Text-to-Image Generative Models
- Butter: Frequency Consistency and Hierarchical Fusion for Autonomous Driving Object Detection
Related