Robustness and Regularization in Hierarchical Re-Basin

2025/10/10 by Franke, Benedikt, Heinrich, Florian, Lange, Markus +1 · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.09174

Abstract

This paper takes a closer look at Git Re-Basin, an interesting new approach to merge trained models. We propose a hierarchical model merging scheme that significantly outperforms the standard MergeMany algorithm. With our new algorithm, we find that Re-Basin induces adversarial and perturbation robustness into the merged models, with the effect becoming stronger the more models participate in the hierarchical merging scheme. However, in our experiments Re-Basin induces a much bigger performance drop than reported by the original authors.

Citations

• Exploring Mode Connectivity for Pre-trained Language Models
• Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks
• Git Re-Basin: Merging Models modulo Permutation Symmetries
• The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks
• Analytical bounds on the local Lipschitz constants of ReLU networks
• Linear Mode Connectivity in Multitask and Continual Learning
• Federated Learning with Matched Averaging
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Adversarial Attacks on Neural Network Policies
• DeepFool: a simple and accurate method to fool deep neural networks

Cited by

• Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifiers

Related

• Git Re-Basin: Merging Models modulo Permutation Symmetries
• If You Can't Use Them, Recycle Them: Optimizing Merging at Scale Mitigates Performance Tradeoffs
• Disrupting Model Merging: A Parameter-Level Defense Without Sacrificing Accuracy
• Why Do More Experts Fail? A Theoretical Analysis of Model Merging
• Vanishing Feature: Diagnosing Model Merging and Beyond