Questionable practices in machine learning
2024/07/17 by Gavin Leech, Leech, Gavin, Juan J. Vazquez +7 · 10 voices · 1 citation
#cs.LG #cs.CL #cs.CY
paper · pdf · doi:10.48550/arxiv.2407.12220
Abstract
Evaluating modern ML models is hard. The strong incentive for researchers and companies to report a state-of-the-art result on some metric often leads to questionable research practices (QRPs): bad practices which fall short of outright research fraud. We describe 44 such practices which can undermine reported results, giving examples where possible. Our list emphasises the evaluation of large language models (LLMs) on public benchmarks. We also discuss "irreproducible research practices", i.e. decisions that make it difficult or impossible for other researchers to reproduce, build on or audit previous research.
Cited by
Discussions
- Questionable practices in machine learning arxiv.org/pdf/2407.12220 #machinelearning #artificialintelligence #datascience [bsky, 53 points, 1 comments]
- "Questionable practices in machine learning" arxiv.org/pdf/2407.12220 [bsky, 48 points, 4 comments]
- Best practices... no wait... Good enough practices... err... how about Questionable practices in machine learning https://arxiv.org/abs/2407.12220 [bsky, 32 points, 2 comments]
- Questionable Practices in Machine Learning [hn, 6 points, 1 comments]
- This is a bit of a tangent, but still related and interesting perspective on the topic (and the authors seem to have read Ben there) arxiv.org/abs/2407.12220 [bsky, 4 points, 0 comments]
- Questionable practices in machine learning arxiv.org/abs/2407.12220 [bsky, 3 points, 0 comments]
- AI research isn’t always as reliable as it seems. Researchers sometimes use questionable practices (QRPs) to make their models look better. From training on test data to cherry-picking results, these [bsky, 0 points, 5 comments]
- Questionable practices in machine learning https://arxiv.org/abs/2407.12220 (preprint, Leech et al, 2024) [bsky, 0 points, 0 comments]
- Why is it hard to trust LLM benchmarks? This paper lists lots of good reasons why arxiv.org/pdf/2407.12220 #llm #benchmarks Credit: Leech et al. [bsky, 0 points, 0 comments]
- Link here : arxiv.org/abs/2407.12220 [bsky, 0 points, 0 comments]
Related