Remember What You Want to Forget: Algorithms for Machine Unlearning
2021/03/04 by Ayush Sekhari, Sekhari, Ayush, Jayadev Acharya +5 · 77 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2103.03279
arxiv created 2021/07/22 · arxiv updated 2021/07/23
Abstract
We study the problem of unlearning datapoints from a learnt model. The learner first receives a dataset S drawn i.i.d. from an unknown distribution, and outputs a model \widehatw that performs well on unseen samples from the same distribution. However, at some point in the future, any training datapoint z ∈ S can request to be unlearned, thus prompting the learner to modify its output model while still ensuring the same accuracy guarantees. We initiate a rigorous study of generalization in machine unlearning, where the goal is to perform well on previously unseen datapoints. Our focus is on both computational and storage complexity. For the setting of convex losses, we provide an unlearning algorithm that can unlearn up to O(n/d1/4) samples, where d is the problem dimension. In comparison, in general, differentially private learning (which implies unlearning) only guarantees deletion of O(n/d1/2) samples. This demonstrates a novel separation between differential privacy and machine unlearning.
Cited by
- Face Identity Unlearning for Retrieval via Embedding Dispersion
- On the Accuracy of Newton Step and Influence Function Data Attributions
- Robust MLLM Unlearning via Visual Knowledge Distillation
- Fully Decentralized Certified Unlearning
- Adaptive-lambda Subtracted Importance Sampled Scores in Machine Unlearning for DDPMs and VAEs
- ModHiFi: Identifying High Fidelity predictive components for Model Modification
- POUR: A Provably Optimal Method for Unlearning Representations via Neural Collapse
- Descend or Rewind? Stochastic Gradient Descent Unlearning
- Selective Forgetting in Option Calibration: An Operator-Theoretic Gauss-Newton Framework
- Beyond Uniform Deletion: A Data Value-Weighted Framework for Certified Machine Unlearning
- FiCABU: A Fisher-Based, Context-Adaptive Machine Unlearning Processor for Edge AI
- On the Impossibility of Retrain Equivalence in Machine Unlearning
- Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
- LEGO: A Lightweight and Efficient Multiple-Attribute Unlearning Framework for Recommender Systems
- LLM Unlearning with LLM Beliefs
- Gaussian Certified Unlearning in High Dimensions: A Hypothesis Testing Approach
- Approximate Domain Unlearning for Vision-Language Models
- Unlearning in Diffusion models under Data Constraints: A Variational Inference Approach
- SMS: Self-supervised Model Seeding for Verification of Machine Unlearning
- CURE: Centroid-guided Unsupervised Representation Erasure for Facial Recognition Systems
- Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning
- ToFU: Transforming How Federated Learning Systems Forget User Data
- The Measure of Deception: An Analysis of Data Forging in Machine Unlearning
- Curriculum Approximate Unlearning for Session-based Recommendation
- Efficient Knowledge Graph Unlearning with Zeroth-order Information
- Demystifying Foreground-Background Memorization in Diffusion Models
- Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
- Mo' Memory, Mo' Problems: Stream-Native Machine Unlearning
- The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage
- Invisible Watermarks, Visible Gains: Steering Machine Unlearning with Bi-Level Watermarking Design
- Conformal Unlearning: A New Paradigm for Unlearning in Conformal Predictors
- IMU: Influence-guided Machine Unlearning
- LetheViT: Selective Machine Unlearning for Vision Transformers via Attention-Guided Contrastive Learning
- Towards Evaluation for Real-World LLM Unlearning
- Efficient Machine Unlearning via Influence Approximation
- LoReUn: Data Itself Implicitly Provides Cues to Improve Machine Unlearning
- Machine Unlearning for Streaming Forgetting
- Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence
- What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests
- DICE: Data Influence Cascade in Decentralized Learning
- Efficient Unlearning with Privacy Guarantees
- Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs
- Rescaled Influence Functions: Accurate Data Attribution in High Dimension
- On the Necessity of Output Distribution Reweighting for Effective Class Unlearning
- BLUR: A Bi-Level Optimization Approach for LLM Unlearning
- Large Language Model Unlearning for Source Code
- Towards Reliable Forgetting: A Survey on Machine Unlearning Verification
- Train Once, Forget Precisely: Anchored Optimization for Efficient Post-Hoc Unlearning
- Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective
- Certified Unlearning for Neural Networks
- UCD: Unlearning in LLMs via Contrastive Decoding
- Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models
- System-Aware Unlearning Algorithms: Use Lesser, Forget Faster
- Distillation Robustifies Unlearning
- A Certified Unlearning Approach without Access to Source Data
- OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models
- Targeted Forgetting of Image Subgroups in CLIP Models
- Unlearning's Blind Spots: Over-Unlearning and Prototypical Relearning Attack
- Existing Large Language Model Unlearning Evaluations Are Inconclusive
- Model Immunization from a Condition Number Perspective
- Machine Unlearning under Overparameterization
- Leveraging Per-Instance Privacy for Machine Unlearning
- Redirection for Erasing Memory (REM): Towards a universal unlearning method for corrupted data
- A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning
- Unlearning Algorithmic Biases over Graphs
- SEPS: A Separability Measure for Robust Unlearning in LLMs
- MUBox: A Critical Evaluation Framework of Deep Machine Unlearning
- Online Learning and Unlearning
- Mirror Mirror on the Wall, Have I Forgotten it All? A New Framework for Evaluating Machine Unlearning
- Certified Data Removal Under High-dimensional Settings
- Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
- Protecting the Undeleted in Machine Unlearning
- Exact Unlearning in Reinforcement Learning
- A Model Merging Approach for Continual MLLM Unlearning
- DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers
- Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware Optimization
- A Neuro-inspired Interpretation of Unlearning in Large Language Models through Sample-level Unlearning Difficulty
Related