Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities

2024/08/14 by Enneng Yang, Shen Li, Li Shen +12 · 1 voice · 54 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Semantic Web and Ontologies #cs.AI #cs.CL #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2408.07666

openalex publication_date 2024/08/14 · arxiv published 2024/08/14 · openalex created_date 2024/10/19 · arxiv updated 2025/12/31 · openalex updated_date 2026/07/28

Abstract

Model merging is an efficient empowerment technique in the machine learning community that does not require the collection of raw training data and does not require expensive computation. As model merging becomes increasingly prevalent across various fields, it is crucial to understand the available model merging techniques comprehensively. However, there is a significant gap in the literature regarding a systematic and thorough review of these techniques. This survey provides a comprehensive overview of model merging methods and theories, their applications in various domains and settings, and future research directions. Specifically, we first propose a new taxonomic approach that exhaustively discusses existing model merging methods. Secondly, we discuss the application of model merging techniques in large language models, multimodal large language models, and more than ten machine learning subfields, including continual learning, multi-task learning, few-shot learning, etc. Finally, we highlight the remaining challenges of model merging and discuss future research directions. A comprehensive list of papers about model merging is available at https://github.com/EnnengYang/Awesome-Model-Merging-Methods-Theories-Applications.

Cited by

• Rethinking Expert Training for Model Merging with Prompt Learning
• Model Merging via Multi-Teacher Knowledge Distillation
• MAGIC: Achieving Superior Model Merging via Magnitude Calibration
• GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training
• BAMBO: Construct Ability and Efficiency LLM Pareto Set via Bayesian Adaptive Multi-objective Block-wise Optimization
• Exploring Test-time Scaling via Prediction Merging on Large-Scale Recommendation
• Basis-Oriented Low-rank Transfer for Few-Shot and Test-Time Adaptation
• An Empirical Survey of Model Merging Algorithms for Social Bias Mitigation
• Weight Space Representation Learning via Neural Field Adaptation
• Group-Aware Partial Model Merging for Children's Automatic Speech Recognition
• A Systematic Study of In-the-Wild Model Merging for Large Language Models
• Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transport
• MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agent
• Learning to Debug: LLM-Organized Knowledge Trees for Solving RTL Assertion Failures
• Evaluating Embedding Generalization: How LLMs, LoRA, and SLERP Shape Representational Geometry
• Retrofit: Continual Learning with Bounded Forgetting for Security Applications
• Defending Unauthorized Model Merging via Dual-Stage Weight Protection
• Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective
• Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captions
• Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance
• Beyond Neural Incompatibility: Easing Cross-Scale Knowledge Transfer in Large Language Models through Latent Semantic Alignment
• World Simulation with Video Foundation Models for Physical AI
• PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs
• Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
• Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?
• Revisiting Model Interpolation for Efficient Reasoning
• Towards Efficient Multimodal Unified Reasoning Model via Model Merging
• Deploying Tiny LVLM Judges for Real-World Evaluation of Chart Models: Lessons Learned and Best Practices
• Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Models
• Timber: Training-free Instruct Model Refining with Base via Effective Rank
• Toward a Holistic Approach to Continual Model Merging
• Faster, Smaller, and Smarter: Task-Aware Expert Merging for Online MoE Inference
• Accurate and Efficient Low-Rank Model Merging in Core Space
• Variational Task Vector Composition
• Continually Adding New Languages to Multilingual Language Models
• Merge-of-Thought Distillation
• Delta Activations: A Representation for Finetuned Large Language Models
• Surrogate Benchmarks for Model Merging Optimization
• Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing
• Efficient Multi-Source Knowledge Transfer by Model Merging
• Consiglieres in the Shadow: Understanding the Use of Uncensored Large Language Models in Cybercrimes
• CoDiEmb: A Collaborative yet Distinct Framework for Unified Representation Learning in Information Retrieval and Semantic Textual Similarity
• Hierarchical Adaptive networks with Task vectors for Test-Time Adaptation
• DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging
• Forgetting of task-specific knowledge in model merging-based continual learning
• Uncertainty-driven Embedding Convolution
• MeMo: Memory as a Model
• FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data
• Temporal Information Retrieval via Time-Specifier Model Merging
• Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planning
• DOTResize: Reducing LLM Width via Discrete Optimal Transport-based Neuron Merging
• DC-TTA: Divide-and-Conquer Framework for Test-Time Adaptation of Interactive Segmentation
• CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning
• Subspace-Boosted Model Merging

Discussions

• Model Merging in LLMs, MLLMs, and Beyond [hn, 2 points, 0 comments]

Related

• A Systematic Study of In-the-Wild Model Merging for Large Language Models
• MergeBench: A Benchmark for Merging Domain-Specialized LLMs
• Exploring Model Kinship for Merging Large Language Models
• Training-free LLM Merging for Multi-task Learning
• Activation-Informed Merging of Large Language Models