An Empirical Study of Memorization in NLP

2022/03/23 by Zheng, Xiaosen, Jiang, Jing · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2203.12171

Abstract

A recent study by Feldman (2020) proposed a long-tail theory to explain the memorization behavior of deep learning models. However, memorization has not been empirically verified in the context of NLP, a gap addressed by this work. In this paper, we use three different NLP tasks to check if the long-tail theory holds. Our experiments demonstrate that top-ranked memorized training instances are likely atypical, and removing the top-memorized training instances leads to a more serious drop in test accuracy compared with removing training instances randomly. Furthermore, we develop an attribution method to better understand why a training instance is memorized. We empirically show that our memorization attribution method is faithful, and share our interesting finding that the top-memorized parts of a training instance tend to be features negatively correlated with the class label.

Cited by

• What do Reward Models Memorize?
• Retrieval-augmented Prompt Learning for Pre-trained Foundation Models

Related

• Quantifying Memorization Across Neural Language Models
• What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
• Exploring Memorization in Fine-tuned Language Models
• Guess or Recall? Training CNNs to Classify and Localize Memorization in LLMs
• Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models