Orthogonal Subspace Learning for Language Model Continual Learning

2023/10/22 by Xinghuan Wang, Tianze Chen, Wang, Xiao +15 · 91 citations
Computer Science · Mathematics · #Adaptation (eye) #Algorithm #Artificial intelligence #Artificial neural network #Cognitive psychology #Computation and Language (cs.CL) #Computer science #Domain Adaptation and Few-Shot Learning #Extreme learning machine #FOS: Computer and information sciences #Forgetting #Generalization #Linear subspace #Machine Learning (cs.LG) #Machine learning #Mathematics #Multimodal Machine Learning Applications #Rank (graph theory) #State (computer science) #Subspace topology

paper · pdf · doi:10.48550/arxiv.2310.14152

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/10/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.

Cited by

• Merge before Forget: A Single LoRA Continual Learning via Continual Merging
• Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
• Mitigating Forgetting in Low Rank Adaptation
• Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces
• SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning
• An Efficient LLM-based Evolutional Recommendation with Locate-Forget-Update Paradigm
• Parameter Importance-Driven Continual Learning for Foundation Models
• Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learning
• Teaching Prompts to Coordinate: Hierarchical Layer-Grouped Prompt Tuning for Continual Learning
• Routing-Based Continual Learning for Multimodal Large Language Models
• Random Initialization of Gated Sparse Adapters
• Progressive Multimodal Alignment for Continual Instruction Tuning
• Continual Low-Rank Adapters for LLM-based Generative Recommender Systems
• PLAN: Proactive Low-Rank Allocation for Continual Learning
• RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging
• KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls
• Personalized Federated Fine-Tuning of Vision Foundation Models for Healthcare
• ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
• Logits Replay + MoClip: Stabilized, Low-Cost Post-Training with Minimal Forgetting
• Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Models
• How to Teach Large Multimodal Models New Skills
• FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
• Deep Generative Continual Learning using Functional LoRA: FunLoRA
• Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettings
• Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection
• SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
• Null-Space Filtering for Data-Free Continual Model Merging: Preserving Transparency, Promoting Fidelity
• Policy Compatible Skill Incremental Learning via Lazy Learning Interface
• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning
• DevFD: Developmental Face Forgery Detection by Learning Shared and Orthogonal LoRA Subspaces
• AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learning
• K-DeCore: Facilitating Knowledge Transfer in Continual Structured Knowledge Reasoning via Knowledge Decoupling
• LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization
• LLM Agents at the Roundtable: A Multi-Perspective and Dialectical Reasoning Framework for Essay Scoring
• Reveal and Release: Iterative LLM Unlearning with Self-generated Data
• Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning
• Self-Evolving LLMs via Continual Instruction Tuning
• Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models
• SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment
• Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection
• Efficient Modular Learning through Naive LoRA Summation: Leveraging Orthogonality in High-Dimensional Models
• MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
• GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
• CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptation
• RegCL: Continual Adaptation of Segment Anything Model via Model Merging
• LLaVA-c: Continual Improved Visual Instruction Tuning
• Remember Past, Anticipate Future: Learning Continual Multimodal Misinformation Detectors
• Continual Multiple Instance Learning with Enhanced Localization for Histopathological Whole Slide Image Analysis
• Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
• Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts
• Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models
• MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation
• Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
• Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
• Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learning
• MLLM-CL: Continual Learning for Multimodal Large Language Models
• Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning
• Flashbacks to Harmonize Stability and Plasticity in Continual Learning
• CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning
• Continual Learning in Vision-Language Models via Aligned Model Merging
• When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations
• Adaptive Budget Allocation for Orthogonal-Subspace Adapter Tuning in LLMs Continual Learning
• LoKI: Low-damage Knowledge Implanting of Large Language Models
• One Rank at a Time: Cascading Error Dynamics in Sequential Learning
• SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting
• Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning
• SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
• ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation
• HD-PiSSA: High-Rank Distributed Orthogonal Adaptation
• T2I-ConBench: Text-to-Image Benchmark for Continual Post-training
• Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
• MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging
• Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation
• Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation
• Task-Core Memory Management and Consolidation for Long-term Continual Learning
• Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback
• In-Context Collapse in Vision-Language Models and How to Mitigate it?
• MedCL-Bench: Benchmarking stability-efficiency trade-offs and scaling in biomedical continual learning
• Reverse Distillation: Consistently Scaling Protein Language Model Representations
• Panini: Continual Learning in Token Space via Structured Memory
• Shared LoRA Subspaces for almost Strict Continual Learning
• Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces
• TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning
• ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning
• PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
• FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning
• Omega-S: A Functional Resilience Index for LLM Fine-Tuning
• LoRA-Based Continual Learning with Constraints on Critical Parameter Changes
• CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
• LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation

Related

• Merge before Forget: A Single LoRA Continual Learning via Continual Merging
• Is Parameter Collision Hindering Continual Learning in LLMs?
• Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning
• Adaptive Budget Allocation for Orthogonal-Subspace Adapter Tuning in LLMs Continual Learning
• Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models