Climbing a shaky ladder: Better adaptive risk estimation

2017/06/08 by Moritz Hardt, Hardt, Moritz · 3 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1706.02733

openalex publication_date 2017/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We revisit the leaderboard problem introduced by Blum and Hardt (2015) in an effort to reduce overfitting in machine learning benchmarks. We show that a randomized version of their Ladder algorithm achieves leaderboard error O(1/n^0.4) compared with the previous best rate of O(1/n^1/3). Short of proving that our algorithm is optimal, we point out a major obstacle toward further progress. Specifically, any improvement to our upper bound would lead to asymptotic improvements in the general adaptive estimation setting as have remained elusive in recent years. This connection also directly leads to lower bounds for specific classes of algorithms. In particular, we exhibit a new attack on the leaderboard algorithm that both theoretically and empirically distinguishes between our algorithm and previous leaderboard algorithms.

Cited by

• LLM Personas as a Substitute for Field Experiments in Method Benchmarking
• Utility is in the Eye of the User: A Critique of NLP Leaderboards
• Overtuning in Hyperparameter Optimization

Related

• The Ladder: A Reliable Leaderboard for Machine Learning Competitions
• Toward a Better Understanding of Leaderboard
• Reducing overfitting in challenge-based competitions
• A Second-order Bound with Excess Losses
• Efficient learning with robust gradient descent