2025/06/03 by Xiangyu Zhou, Zhou, Xiangyu, Yao Qiang +9 · 3 voices · 2 citations
Computer Science · #Big Data and Digital Economy #Face (sociological concept) #Forgetting #Identifier #Information loss #Language model #Memorization #Multimodal Machine Learning Applications #Preference #Process (computing) #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2506.03142
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/06/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Large Language Models (LLMs), pre-trained on massive text corpora, exhibit remarkable human-level language understanding, reasoning, and decision-making abilities. However, they tend to memorize unwanted information, such as private or copyrighted content, raising significant privacy and legal concerns. Unlearning has emerged as a promising solution, but existing methods face a significant challenge of over-forgetting. This issue arises because they indiscriminately suppress the generation of all the tokens in forget samples, leading to a substantial loss of model utility. To overcome this challenge, we introduce the Targeted Information Forgetting (TIF) framework, which consists of (1) a flexible targeted information identifier designed to differentiate between unwanted words (UW) and general words (GW) in the forget samples, and (2) a novel Targeted Preference Optimization approach that leverages Logit Preference Loss to unlearn unwanted information associated with UW and Preservation Loss to retain general information in GW, effectively improving the unlearning process while mitigating utility degradation. Extensive experiments on the TOFU and MUSE benchmarks demonstrate that the proposed TIF framework enhances unlearning effectiveness while preserving model utility and achieving state-of-the-art results.