Subspace-Boosted Model Merging

2025/06/19 by Ronald Skorobogat, Karsten Roth, Skorobogat, Ronald +3 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2506.16506

openalex publication_date 2025/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Model merging enables the combination of multiple specialized expert models into a single model capable of performing multiple tasks. However, the benefits of merging an increasing amount of specialized experts generally lead to diminishing returns and reduced overall performance gains. In this work, we empirically and theoretically analyze this limitation, proving that for Task Arithmetic-based methods, as more experts are merged, the common information dominates the task-specific information, leading to inevitable rank collapse. To mitigate this issue, we introduce Subspace Boosting, which operates on the singular value decomposed task vector space and maintains task vector ranks. Subspace Boosting raises merging efficacy for up to 20 experts by large margins of more than 10% when evaluated on both vision and language benchmarks. Moreover, we propose employing Higher-Order Generalized Singular Value Decomposition to quantify task similarity, offering a new interpretable perspective on model merging. Code and models are available at https://github.com/ronskoro/Subspace-Boosting.

Citations

• AdaRank: Adaptive Rank Pruning for Enhanced Model Merging
• Transformers without Normalization
• No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
• Revisiting Weight Averaging for Model Merging
• How to Merge Your Multimodal Models Over Time?
• Multi-Task Model Merging via Adaptive Weight Disentanglement
• Task Singular Vectors: Reducing Task Interference in Model Merging
• Collective Model Intelligence Requires Compatible Specialization
• Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
• Model merging with SVD to tie the Knots
• LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging
• The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse
• What Matters for Model Merging at Scale?
• DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation
• Realistic Evaluation of Model Merging for Compositional Generalization
• A Practitioner's Guide to Continual Multimodal Pretraining
• Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic
• Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
• A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
• MagMax: Leveraging Model Merging for Seamless Continual Learning
• Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models
• PLeaS -- Merging Models with Permutations and Least Squares
• WARP: On the Benefits of Weight Averaged Rewarded Policies
• DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling
• Localizing Task Information for Improved Model Merging and Compression
• Model Stock: All we need is just a few fine-tuned models
• Arcee's MergeKit: A Toolkit for Merging Large Language Models
• DiffRed: Dimensionality Reduction guided by stable rank
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
• DoRA: Weight-Decomposed Low-Rank Adaptation
• Representation Surgery for Multi-Task Model Merging
• Time is Encoded in the Weights of Finetuned Language Models
• Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks
• Merging by Matching Models in Task Parameter Subspaces
• Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion
• AdaMerging: Adaptive Model Merging for Multi-Task Learning
• Fine-tuning can cripple your foundation model; preserving features may be the solution
• Early Weight Averaging meets High Learning Rates for LLM Pre-training
• TIES-Merging: Resolving Interference When Merging Models
• An Empirical Study of Multimodal Model Merging
• Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization
• Dataless Knowledge Fusion by Merging Weights of Language Models
• Editing Models with Task Arithmetic
• Linear Interpolation In Parameter Space is Good Enough for Fine-Tuned Language Models
• Disentanglement of Correlated Factors via Hausdorff Factorized Support
• Git Re-Basin: Merging Models modulo Permutation Symmetries
• Diverse Weight Averaging for Out-of-Distribution Generalization
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks
• Robust fine-tuning of zero-shot models
• On the Opportunities and Risks of Foundation Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Learning Transferable Visual Models From Natural Language Supervision
• A Higher-Order Generalized Singular Value Decomposition for Rank Deficient Matrices
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• DiVA: Diverse Visual Feature Aggregation for Deep Metric Learning
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
• Revisiting Training Strategies and Generalization Performance in Deep Metric Learning
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Stable Rank Normalization for Improved Generalization in Neural Networks and GANs
• Averaging Weights Leads to Wider Optima and Better Generalization
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Deep Learning of Representations: Looking Forward
• Stable Principal Component Pursuit
• A Generalized Solution of the Orthogonal Procrustes Problem
• FREE-Merging: Fourier Transform for Efficient Model Merging
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• A Systematic Study of In-the-Wild Model Merging for Large Language Models
• Steering Language Models with Weight Arithmetic

Related

• No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
• Purifying Task Vectors in Knowledge-Aware Subspace for Model Merging
• Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
• Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration
• Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging