Reproducibility Requires Consolidated Artifacts

2023/05/21 by Iordanis Fostiropoulos, Fostiropoulos, Iordanis, Bowman Brown +3
Computer Science · Decision Sciences · Materials Science · #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Materials Science #Scientific Computing and Data Management #Software Engineering (cs.SE)

paper · pdf · doi:10.48550/arxiv.2305.12571

openalex publication_date 2023/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Machine learning is facing a 'reproducibility crisis' where a significant number of works report failures when attempting to reproduce previously published results. We evaluate the sources of reproducibility failures using a meta-analysis of 142 replication studies from ReScience C and 204 code repositories. We find that missing experiment details such as hyperparameters are potential causes of unreproducibility. We experimentally show the bias of different hyperparameter selection strategies and conclude that consolidated artifacts with a unified framework can help support reproducibility.

Related

• Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers
• Reproducibility in machine‐learning‐based research: Overview, barriers, and drivers
• Laying foundations to quantify the "Effort of Reproducibility"
• Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
• Reproducibility in Machine Learning-Driven Research