Routing-Based Continual Learning for Multimodal Large Language Models

2025/11/03 by Mohta, Jay, Lee, Gwang, Ak, Kenan Emir +3
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · doi:10.48550/arxiv.2511.01831

openalex publication_date 2025/11/03 · openalex created_date 2025/11/06 · openalex updated_date 2026/07/28

Abstract

Multimodal Large Language Models (MLLMs) struggle with continual learning, often suffering from catastrophic forgetting when adapting to sequential tasks. We introduce a routing-based architecture that integrates new capabilities while robustly preserving foundational knowledge. While Multi-Task Learning (MTL) offers a theoretical performance upper bound, it incurs a linearly scaling computational overhead as the number of tasks increases. In contrast, our method maintains fixed data and compute requirements regardless of the task sequence length. Across models ranging from 2B to 8B parameters, we demonstrate that our routing approach performs on par with MTL while retaining the training efficiency of sequential fine-tuning. Beyond merely mitigating forgetting, we observe that token-level routing facilitates cross-modal transfer, leveraging knowledge from one modality to bolster performance in another. Ablation studies confirm the approach's scalability: routing remains robust even with large expert pools and effectively capitalizes on task relatedness. Finally, we show that our method scales favorably, with larger models exhibiting minimal degradation compared to fully specialized fine-tuning.

Citations

• SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
• ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
• Unique Security and Privacy Threats of Large Language Models: A Comprehensive Survey
• Towards Modular LLMs by Building and Reusing a Library of LoRAs
• Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning
• Beyond Text: Frozen Large Language Models in Visual Signal Comprehension
• Learning to Route Among Specialized Experts for Zero-Shot Generalization
• Orthogonal Subspace Learning for Language Model Continual Learning
• Improved Baselines with Visual Instruction Tuning
• Understanding Catastrophic Forgetting in Language Models via Implicit Inference
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• Progressive Prompts: Continual Learning for Language Models
• Language Models are Multilingual Chain-of-Thought Reasoners
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• Learning to Prompt for Continual Learning
• Merging Models with Fisher-Weighted Averaging
• Scaling Vision with Sparse Mixture of Experts
• Learning Transferable Visual Models From Natural Language Supervision
• DocVQA: A Dataset for VQA on Document Images
• Language Models are Few-Shot Learners
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Experience Replay for Continual Learning
• XNLI: Evaluating Cross-lingual Sentence Representations
• Measuring Catastrophic Forgetting in Neural Networks
• An Overview of Multi-Task Learning in Deep Neural Networks
• Gated Multimodal Units for Information Fusion
• iCaRL: Incremental Classifier and Representation Learning
• A large annotated corpus for learning natural language inference
• CIDEr: Consensus-based Image Description Evaluation
• Microsoft COCO: Common Objects in Context

Related

• LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models
• CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks
• Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning
• Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models
• Continual Learning in Vision-Language Models via Aligned Model Merging