ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation

2025/05/24 by Liang, Jian, Huang, Wenke, Guo, Xianda +3 · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2505.18640

Abstract

Low-Rank Adaptation (LoRA) is widely adopted for downstream fine-tuning of foundation models due to its efficiency and zero additional inference cost. Many real-world applications require foundation models to specialize in several specific tasks simultaneously, motivating the need for efficient multi-task downstream adaptation. To address this need, existing studies have primarily explored two directions: Model Merging with LoRA, which shows advantages in training-free scenarios but still lags behind multi-task training in overall performance; and MoE-based LoRA approaches, which improve multi-task learning performance but introduce routers that hinder the mergeability of LoRA parameters and incur considerable inference overhead, thereby limiting real-world deployment practicality. To this end, we propose ThanoRA, a Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation framework that enables effective, efficient and unified multi-task downstream adaptation without introducing additional structure. ThanoRA performs multi-task learning by tailoring subspace allocation at initialization and enforcing diversity preservation throughout training: it allocates varying dimensions to construct task-specific low-rank subspaces driven by inter-task heterogeneity, enabling fine-grained knowledge injection, while diversity-preserving regularization mitigates task interference and subspace collapse, thereby fully exploiting the low-rank capacity. Extensive experiments across multimodal and text-only benchmarks under varying multi-task mixtures demonstrate that ThanoRA consistently outperforms strong baselines, surpassing even separate task-specific fine-tuning, while introducing no additional structures or inference overhead. Our code will be publicly available at: https://github.com/LiangJian24/ThanoRA.

Citations

• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning
• Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging
• Qwen3 Technical Report
• LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models
• Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
• RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness
• Qwen2.5-VL Technical Report
• A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations
• Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
• Cross-modal Information Flow in Multimodal Large Language Models
• MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning
• Model merging with SVD to tie the Knots
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• LLaVA-OneVision: Easy Visual Task Transfer
• The Llama 3 Herd of Models
• Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization
• CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
• MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts
• Mixture of LoRA Experts
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
• MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning
• Multimodal Instruction Tuning with Conditional Mixture of LoRA
• Fair Resource Allocation in Multi-Task Learning
• LoRA+: Efficient Low Rank Adaptation of Large Models
• DoRA: Weight-Decomposed Low-Rank Adaptation
• Merging Multi-Task Models via Weight-Ensembling Mixture of Experts
• LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs
• LLaMA Pro: Progressive LLaMA with Block Expansion
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
• Orthogonal Subspace Learning for Language Model Continual Learning
• When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications
• VeRA: Vector-based Random Matrix Adaptation
• Improved Baselines with Visual Instruction Tuning
• LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• LLaMA: Open and Efficient Foundation Language Models
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Task Adaptive Parameter Sharing for Multi-Task Learning
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Multi-Task Learning with Deep Neural Networks: A Survey
• DocVQA: A Dataset for VQA on Document Images
• Learning to Branch for Multi-Task Learning
• Parameter-Efficient Transfer Learning for NLP
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• A Survey on Multi-Task Learning
• A Survey on Multi-Task Learning
• Trace Norm Regularised Deep Multi-Task Learning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Cited by

• Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning

Related

• MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning
• Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
• MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning
• LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
• Batched Low-Rank Adaptation of Foundation Models