Counterfactual Memorization in Neural Language Models

2021/12/24 by Chiyuan Zhang, Zhang, Chiyuan, Daphne Ippolito +9 · 12 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2112.12938

openalex publication_date 2021/12/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Modern neural language models that are widely used in various NLP tasks risk memorizing sensitive information from their training data. Understanding this memorization is important in real world applications and also from a learning-theoretical perspective. An open question in previous studies of language model memorization is how to filter out "common" memorization. In fact, most memorization criteria strongly correlate with the number of occurrences in the training set, capturing memorized familiar phrases, public knowledge, templated texts, or other repeated data. We formulate a notion of counterfactual memorization which characterizes how a model's predictions change if a particular document is omitted during training. We identify and study counterfactually-memorized training examples in standard text datasets. We estimate the influence of each memorized training example on the validation set and on generated texts, showing how this can provide direct evidence of the source of memorization at test time.

Cited by

• CTIGuardian: A Few-Shot Framework for Mitigating Privacy Leakage in Fine-Tuned LLMs
• The Persistence of Cultural Memory: Investigating Multimodal Iconicity in Diffusion Models
• Differentially Private In-Context Learning with Nearest Neighbor Search
• Memorizing Long-tail Data Can Help Generalization Through Composition
• Leverage Unlearning to Sanitize LLMs
• Extracting alignment data in open models
• The Model's Language Matters: A Comparative Privacy Analysis of LLMs
• (Token-Level) InfoRMIA: Stronger Membership Inference and Memorization Assessment for LLMs
• Train Once, Answer All: Many Pretraining Experiments for the Cost of One
• A Law of Data Reconstruction for Random Features (and Beyond)
• Beyond Data Privacy: New Privacy Risks for Large Language Models
• Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
• Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks

Related

• Quantifying Memorization Across Neural Language Models
• An Empirical Study of Memorization in NLP
• Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy
• Detecting Memorization in Large Language Models
• Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models