Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks

2025/04/01 by Haque, Naimul · 4 citations
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.01241

openalex publication_date 2025/04/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have significantly advanced Natural Language Processing (NLP), particularly in Natural Language Understanding (NLU) tasks. As we progress toward an agentic world where LLM-based agents autonomously handle specialized tasks, it becomes crucial for these models to adapt to new tasks without forgetting previously learned information - a challenge known as catastrophic forgetting. This study evaluates the continual fine-tuning of various open-source LLMs with different parameter sizes (specifically models under 10 billion parameters) on key NLU tasks from the GLUE benchmark, including SST-2, MRPC, CoLA, and MNLI. By employing prompt engineering and task-specific adjustments, we assess and compare the models' abilities to retain prior knowledge while learning new tasks. Our results indicate that models such as Phi-3.5-mini exhibit minimal forgetting while maintaining strong learning capabilities, making them well-suited for continual learning environments. Additionally, models like Orca-2-7b and Qwen2.5-7B demonstrate impressive learning abilities and overall performance after fine-tuning. This work contributes to understanding catastrophic forgetting in LLMs and highlights prompting engineering to optimize model performance for continual learning scenarios.

Cited by

• How Much is Too Much? Exploring LoRA Rank Trade-offs for Retaining Knowledge and Domain Robustness
• MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models
• Mapping Post-Training Forgetting in Language Models at Scale
• Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs

Related

• Scaling Laws for Forgetting When Fine-Tuning Large Language Models
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
• Examining Forgetting in Continual Pre-training of Aligned Large Language Models
• Can LLMs Learn New Concepts Incrementally without Forgetting?
• Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models