Harmonizing and Merging Source Models for CLIP-based Domain Generalization

2025/06/11 by Yuhe Ding, Jian Liang, Ding, Yuhe +9 · 1 citation
Computer Science · #Advanced Graph Neural Networks #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.09446

openalex publication_date 2025/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

CLIP-based domain generalization aims to improve model generalization to unseen domains by leveraging the powerful zero-shot classification capabilities of CLIP and multiple source datasets. Existing methods typically train a single model across multiple source domains to capture domain-shared information. However, this paradigm inherently suffers from two types of conflicts: 1) sample conflicts, arising from noisy samples and extreme domain shifts among sources; and 2) optimization conflicts, stemming from competition and trade-offs during multi-source training. Both hinder the generalization and lead to suboptimal solutions. Recent studies have shown that model merging can effectively mitigate the competition of multi-objective optimization and improve generalization performance. Inspired by these findings, we propose Harmonizing and Merging (HAM), a novel source model merging framework for CLIP-based domain generalization. During the training process of the source models, HAM enriches the source samples without conflicting samples, and harmonizes the update directions of all models. Then, a redundancy-aware historical model merging method is introduced to effectively integrate knowledge across all source models. HAM comprehensively consolidates source domain information while enabling mutual enhancement among source models, ultimately yielding a final model with optimal generalization capabilities. Extensive experiments on five widely used benchmark datasets demonstrate the effectiveness of our approach, achieving state-of-the-art performance.

Citations

• Balanced Direction from Multifarious Choices: Arithmetic Meta-Learning for Domain Generalization
• Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes
• LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization
• Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
• Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization
• Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion
• Diffusion Soup: Model Merging for Text-to-Image Diffusion Models
• Soft Prompt Generation for Domain Generalization
• Evolutionary optimization of model merging recipes
• Domain Generalization with Small Data
• Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification
• AdaMerging: Adaptive Model Merging for Multi-Task Learning
• Jointly Training Large Autoregressive Multimodal Models
• Understanding Hessian Alignment for Domain Generalization
• TIES-Merging: Resolving Interference When Merging Models
• CLIPood: Generalizing CLIP to Out-of-Distributions
• Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization
• Editing Models with Task Arithmetic
• Context-Aware Robust Fine-Tuning
• Domain Generalization by Mutual-Information Regularization with Pre-trained Models
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• Exploiting Domain-Specific Features to Enhance Domain Generalization
• Robust fine-tuning of zero-shot models
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Invariant Information Bottleneck for Domain Generalization
• AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation
• Semi-Supervised Domain Generalization with Stochastic StyleMatch
• Gradient Matching for Domain Generalization
• SelfReg: Self-supervised Contrastive Regularization for Domain Generalization
• SWAD: Domain Generalization by Seeking Flat Minima
• Sparsity in Deep Learning: Pruning and growth for efficient inference\n and training in neural networks
• Learning to Generalize Unseen Domains via Memory-based Multi-Source Meta-Learning for Person Re-Identification
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Respecting Domain Relations: Hypothesis Invariance for Domain Generalization
• Self-Supervised Learning Across Domains
• Shape-aware Meta-learning for Generalizing Prostate MRI Segmentation to Unseen Domains
• In Search of Lost Domain Generalization
• Efficient Domain Generalization via Common-Specific Low-Rank Decomposition
• Improving Sample Efficiency in Model-Free Reinforcement Learning from\n Images
• Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data
• Invariant Risk Minimization
• Episodic Training for Domain Generalization
• Moment Matching for Multi-Source Domain Adaptation
• Recognition in Terra Incognita
• Averaging Weights Leads to Wider Optima and Better Generalization
• mixup: Beyond Empirical Risk Minimization
• Learning to Generalize: Meta-Learning for Domain Generalization
• Deeper, Broader and Artier Domain Generalization
• Unified Deep Supervised Domain Adaptation and Generalization
• Deep Hashing Network for Unsupervised Domain Adaptation
• Deep CORAL: Correlation Alignment for Deep Domain Adaptation
• Deep Residual Learning for Image Recognition
• Domain Generalization via Invariant Feature Representation

Cited by

• Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging

Related

• Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations
• Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization
• In Search of Lost Domain Generalization
• Training-Free Model Merging for Multi-target Domain Adaptation
• Learning to Diversify for Single Domain Generalization