2023/02/09 by Almog Gueta, Gueta, Almog, Elad Venezian +9 · 3 voices · 6 citations
Computer Science · #Artificial intelligence #Computer science #Explainable Artificial Intelligence (XAI) #Language model #Leverage (statistics) #Machine learning #Natural Language Processing Techniques #Space (punctuation) #Task (project management) #Theoretical computer science #Topic Modeling #Traverse
paper · pdf · doi:10.48550/arxiv.2302.04863
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/02/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Research on neural networks has focused on understanding a single model trained on a single dataset. However, relatively little is known about the relationships between different models, particularly those trained or tested on different datasets. We address this by studying how the weight space and the underlying loss landscape of different models are interconnected. Specifically, we demonstrate that finetuned models that were optimized for high performance, reside in well-defined regions in weight space, and vice versa -- that any model that resides anywhere in those regions also exhibits high performance. Notably, we show that language models that have been finetuned on the same dataset form a tight cluster in the weight space, while models finetuned on different datasets from the same underlying task form a looser cluster. Moreover, traversing around the region between the models leads to new models that perform comparably or even better than models obtained via finetuning, even on tasks that the original models were not finetuned on. Our findings provide insight into the relationships between models, demonstrating that a model positioned between two similar models can acquire the knowledge of both. We leverage this and design a method for selecting a better model for efficient finetuning. Specifically, we show that starting from the center of the region is as effective, if not more, than using the pretrained model in 11 out of 12 datasets, resulting in an average accuracy improvement of 3.06.