Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models

2025/12/02 by Xiwen Wei, Wei, Xiwen, Mustafa Munir +2 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Forgetting #Generative Adversarial Networks and Image Synthesis #Generative grammar #Generative model #Machine Learning (cs.LG) #Modalities #Modality (human–computer interaction) #Multimodal Machine Learning Applications #Scalability

paper · pdf · doi:10.48550/arxiv.2512.03125

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/12/02 · openalex created_date 2025/12/05 · openalex updated_date 2026/08/05

Abstract

Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings. Codes will be publicly available: https://github.com/Christina200/MoDE-official.git

Citations

• Emerging Properties in Unified Multimodal Pretraining
• ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement
• UGen: Unified Autoregressive Multimodal Model with Progressive Vocabulary Learning
• LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models
• HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model
• OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models
• CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering
• Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
• MetaMorph: Multimodal Understanding and Generation via Instruction Tuning
• Modality-Inconsistent Continual Learning of Multimodal Large Language Models
• ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance
• TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation
• Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads
• Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
• Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection
• Recent Advances of Multimodal Continual Learning: A Comprehensive Survey
• Emu3: Next-Token Prediction is All You Need
• VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation
• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
• Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
• ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation
• Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
• LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models
• Wings: Learning Multimodal LLMs without Text-only Forgetting
• Chameleon: Mixed-Modal Early-Fusion Foundation Models
• MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts
• Mixture of LoRA Experts
• Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters
• CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
• MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models
• Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models
• MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning
• World Model on Million-Length Video And Language With Blockwise RingAttention
• Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer
• Generative Multimodal Models are In-Context Learners
• Continual Instruction Tuning for Large Multimodal Models
• Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
• GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
• Investigating the Catastrophic Forgetting in Multimodal Large Language Models
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization
• Visual Instruction Tuning with Polite Flamingo
• Lifelong Language Pretraining with Distribution-Specialized Experts
• Visual Instruction Tuning
• Sigmoid Loss for Language Image Pre-Training
• Scaling Vision-Language Models with Sparse Mixture of Experts
• A Comprehensive Survey of Continual Learning: Theory, Method and Application
• Multi-Concept Customization of Text-to-Image Diffusion
• CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
• DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning
• High-Resolution Image Synthesis with Latent Diffusion Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Scaling Vision with Sparse Mixture of Experts
• Learning Transferable Visual Models From Natural Language Supervision
• Taming Transformers for High-Resolution Image Synthesis
• Towards VQA Models That Can Read
• VizWiz Grand Challenge: Answering Visual Questions from Blind People
• Neural Discrete Representation Learning
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Generation and Comprehension of Unambiguous Object Descriptions
• ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt

Cited by

• From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms

Related

• Beyond Unimodal Learning: The Importance of Integrating Multiple Modalities for Lifelong Learning
• Modality-Inconsistent Continual Learning of Multimodal Large Language Models
• Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation
• Continual Learning for Multiple Modalities
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting