Guaranteeing Reproducibility in Deep Learning Competitions

2020/05/12 by Brandon Houghton, Stephanie Milani, Houghton, Brandon +13
Computer Science · Decision Sciences · #Adversarial Robustness in Machine Learning #Auction Theory and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2005.06041

openalex publication_date 2020/05/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

To encourage the development of methods with reproducible and robust training behavior, we propose a challenge paradigm where competitors are evaluated directly on the performance of their learning procedures rather than pre-trained agents. Since competition organizers re-train proposed methods in a controlled setting they can guarantee reproducibility, and -- by retraining submissions using a held-out test set -- help ensure generalization past the environments on which they were trained.

Citations

• Generative Adversarial Imitation Learning
• Run, skeleton, run: skeletal model in a physics-based simulation
• MineRL: A Large-Scale Dataset of Minecraft Demonstrations

Related

• Towards robust and domain agnostic reinforcement learning competitions
• Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
• Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning
• On "Deep Learning" Misconduct
• DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation