2024/10/31 by Rachel Longjohn, Longjohn, Rachel, Markelle Kelly +5 · 3 citations
Computer Science · #Digital Libraries (cs.DL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Semantic Web and Ontologies
paper · pdf · doi:10.48550/arxiv.2410.24100
openalex publication_date 2024/10/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In machine learning research, it is common to evaluate algorithms via their performance on standard benchmark datasets. While a growing body of work establishes guidelines for -- and levies criticisms at -- data and benchmarking practices in machine learning, comparatively less attention has been paid to the data repositories where these datasets are stored, documented, and shared. In this paper, we analyze the landscape of these benchmark data repositories and the role they can play in improving benchmarking. This role includes addressing issues with both datasets themselves (e.g., representational harms, construct validity) and the manner in which evaluation is carried out using such datasets (e.g., overemphasis on a few datasets and metrics, lack of reproducibility). To this end, we identify and discuss a set of considerations surrounding the design and use of benchmark data repositories, with a focus on improving benchmarking practices in machine learning.