From Teacher to Student: Tracking Memorization Through Model Distillation

2025/06/19 by Singh, Simardeep · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.16170

openalex publication_date 2025/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security. While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches.

Citations

• Improving Domain Adaptation through Extended-Text Reading Comprehension
• Analyzing Leakage of Personally Identifiable Information in Language Models
• Are Large Pre-Trained Language Models Leaking Your Personal Information?
• Extracting Training Data from Large Language Models
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• Sequence-Level Knowledge Distillation
• Distilling the Knowledge in a Neural Network

Cited by

• The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation

Related

• Exploring Memorization in Fine-tuned Language Models
• Membership and Memorization in LLM Knowledge Distillation
• The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation
• Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation
• Skewed Memorization in Large Language Models: Quantification and Decomposition