Flaws of ImageNet, Computer Vision's Favourite Dataset

2024/11/26 by Nikita Kisel, Illia Volkov, Kisel, Nikita +7 · 7 citations
Computer Science · Medicine · #AI in cancer detection #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2412.00076

openalex publication_date 2024/11/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Since its release, ImageNet-1k dataset has become a gold standard for evaluating model performance. It has served as the foundation for numerous other datasets and training tasks in computer vision. As models have improved in accuracy, issues related to label correctness have become increasingly apparent. In this blog post, we analyze the issues in the ImageNet-1k dataset, including incorrect labels, overlapping or ambiguous class definitions, training-evaluation domain shifts, and image duplicates. The solutions for some problems are straightforward. For others, we hope to start a broader conversation about refining this influential dataset to better serve future research.

Cited by

• Multimodal Large Language Models as Image Classifiers
• Robust Context-Aware Object Recognition
• Image Recognition with Vision and Language Embeddings of VLMs
• I Am Big, You Are Little; I Am Right, You Are Wrong
• Meta-Learning Transformers to Improve In-Context Generalization
• Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning
• DRAGON: A Large-Scale Dataset of Realistic Images Generated by Diffusion Models

Related

• Are we done with ImageNet?
• Automated Classification of Model Errors on ImageNet
• When does dough become a bagel? Analyzing the remaining mistakes on ImageNet
• ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations
• Does progress on ImageNet transfer to real-world datasets?