The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

2021/10/12 by Rahim Entezari, Entezari, Rahim, Hanie Sedghi +5 · 46 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques #cs.LG

paper · pdf · doi:10.48550/arxiv.2110.06296

openalex publication_date 2021/10/12 · arxiv created 2022/07/05 · arxiv updated 2022/07/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training, and ensemble methods.

Cited by

• Fast Cross-Scenario Adaptation of CSI Models via Channel Conditional Parameter Generation
• A Systematic Study of In-the-Wild Model Merging for Large Language Models
• Model Agreement via Anchoring
• Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Models
• Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifiers
• Keys in the Weights: Transformer Authentication Using Model-Bound Latent Representations
• Weight Weaving: Parameter Pooling for Data-Free Model Merging
• One Token Embedding Is Enough to Deadlock Your Large Reasoning Model
• Robustness and Regularization in Hierarchical Re-Basin
• Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivity
• Riddled basin geometry sets fundamental limits to predictability and reproducibility in deep learning
• How does the optimizer implicitly bias the model merging loss landscape?
• Topological Invariance and Breakdown in Learning
• Null-Space Filtering for Data-Free Continual Model Merging: Preserving Transparency, Promoting Fidelity
• nDNA -- the Semantic Helix of Artificial Cognition
• On Task Vectors and Gradients
• Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing
• Characterizing Linear Alignment Across Language Models
• Sharper Convergence Rates for Nonconvex Optimisation via Reduction Mappings
• Weighted Ensemble Models Are Strong Continual Learners
• Generalized Linear Mode Connectivity for Transformers
• Circumventing Backdoor Space via Weight Symmetry
• SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging
• Subspace-Boosted Model Merging
• The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
• Symmetry in Neural Network Parameter Spaces
• A correlation-permutation approach for speech-music encoders model merging
• Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
• A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models
• Smooth Model Compression without Fine-Tuning
• Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking
• Understanding Mode Connectivity via Parameter Space Symmetry
• Update Your Transformer to the Latest Release: Re-Basin of Task Vectors
• Connecting Independently Trained Modes via Layer-Wise Connectivity
• Robust fine-tuning of speech recognition models via model merging: application to disordered speech
• Leveraging Per-Instance Privacy for Machine Unlearning
• PEER pressure: Model-to-Model Regularization for Single Source Domain Generalization
• Distilling a speech and music encoder with task arithmetic
• SAFE-Merge: Data-Free Continual Model Merging with General Knowledge Preservation
• MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging
• Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
• Evidence of an Emergent "Self" in Continual Robot Learning
• Transformers Learn the Mestre-Nagao Heuristic
• Dynamic Fisher-weighted Model Merging via Bayesian Optimization
• MASS: MoErging through Adaptive Subspace Selection
• Convergent Evolution in Algorithmic Space
• Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations

Related

• Analyzing the Role of Permutation Invariance in Linear Mode Connectivity
• Simultaneous linear connectivity of neural networks modulo permutation
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• Random initialisations performing above chance and how to find them
• On Symmetry and Initialization for Neural Networks