2025/10/07 by Prabhant Singh, Singh, Prabhant, Joaquin Vanschoren +2 · 1 citation
Business, Management and Accounting · Computer Science · #Artificial Intelligence (cs.AI) #Business Process Modeling and Analysis #FOS: Computer and information sciences #Machine Learning (cs.LG) #Semantic Web and Ontologies
paper · doi:10.48550/arxiv.2510.06448
openalex publication_date 2025/10/07 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28
Transferability estimation metrics are used to find a high-performing pre-trained model for a given target task without fine-tuning models and without access to the source dataset. Despite the growing interest in developing such metrics, the benchmarks used to measure their progress have gone largely unexamined. In this work, we empirically show the shortcomings of widely used benchmark setups to evaluate transferability estimation metrics. We argue that the benchmarks on which these metrics are evaluated are fundamentally flawed. We empirically demonstrate that their unrealistic model spaces and static performance hierarchies artificially inflate the perceived performance of existing metrics, to the point where simple, dataset-agnostic heuristics can outperform sophisticated methods. Our analysis reveals a critical disconnect between current evaluation protocols and the complexities of real-world model selection. To address this, we provide concrete recommendations for constructing more robust and realistic benchmarks to guide future research in a more meaningful direction.