CoGraM: Context-sensitive granular optimization method with rollback for robust model fusion

2025/12/03 by Julius Lenz, Lenz, Julius
Computer Science · #Advanced Graph Neural Networks #Advanced Neural Network Applications #F.2.0 #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #I.2.6 #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2512.03610

openalex publication_date 2025/12/03 · openalex created_date 2025/12/05 · openalex updated_date 2026/07/28

Abstract

Merging neural networks without retraining is central to federated and distributed learning. Common methods such as weight averaging or Fisher merging often lose accuracy and are unstable across seeds. CoGraM (Contextual Granular Merging) is a multi-stage, context-sensitive, loss-based, and iterative optimization method across layers, neurons, and weight levels that aligns decisions with loss differences and thresholds and prevents harmful updates through rollback. CoGraM is an optimization method that addresses the weaknesses of methods such as Fisher and can significantly improve the merged network.

Citations

• Generating Synthetic Oracle Datasets to Analyze Noise Impact: A Study on Building Function Classification Using Tweets
• Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent
• Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyond
• On the Impact of Data Heterogeneity in Federated Learning Environments with Application to Healthcare Networks
• Understanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thought
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• Merging Models with Fisher-Weighted Averaging
• On the Hyperparameters in Stochastic Gradient Descent with Momentum
• Heterogeneous Federated Learning
• Perverse equivalence in SL(2,q)
• Advances and Open Problems in Federated Learning
• Unsupervised Deep Embedding for Clustering Analysis
• Adam: A Method for Stochastic Optimization
• On the difficulty of training Recurrent Neural Networks
• Selection of relevant features and examples in machine learning
• Least squares quantization in PCM

Related

• Rethinking Layer-wise Model Merging through Chain of Merges
• Soft Merging: A Flexible and Robust Soft Model Merging Approach for Enhanced Neural Network Performance
• MeGA: Merging Multiple Independently Trained Neural Networks Based on Genetic Algorithm
• MIN-Merging: Merge the Important Neurons for Model Merging
• Model Fusion via Optimal Transport