Lacuna Inc. at SemEval-2025 Task 4: LoRA-Enhanced Influence-Based Unlearning for LLMs

2025/06/04 by Aleksey Kudelya, Kudelya, Aleksey, Alexander Shirnin +1
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.04044

openalex publication_date 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper describes LIBU (LoRA enhanced influence-based unlearning), an algorithm to solve the task of unlearning - removing specific knowledge from a large language model without retraining from scratch and compromising its overall utility (SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models). The algorithm combines classical influence functions to remove the influence of the data from the model and second-order optimization to stabilize the overall utility. Our experiments show that this lightweight approach is well applicable for unlearning LLMs in different kinds of task.

Citations

• LUME: LLM Unlearning with Multitask Evaluations
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models
• OLMo: Accelerating the Science of Language Models
• Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening
• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
• Measuring Massive Multitask Language Understanding
• Machine Unlearning
• Understanding Black-box Predictions via Influence Functions
• Adam: A Method for Stochastic Optimization

Related

• Unified Parameter-Efficient Unlearning for LLMs
• LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples
• UNLEARN Efficient Removal of Knowledge in Large Language Models
• LLM Unlearning using Gradient Ratio-Based Influence Estimation and Noise Injection
• SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning