Do CIFAR-10 Classifiers Generalize to CIFAR-10?

2018/06/01 by Benjamin Recht, Rebecca Roelofs, Recht, Benjamin +5 · 2 voices · 22 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Medical Image Segmentation Techniques #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1806.00451

openalex publication_date 2018/06/01 · arxiv published 2018/06/01 · arxiv updated 2018/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Machine learning is currently dominated by largely experimental work focused on improvements in a few key tasks. However, the impressive accuracy numbers of the best performing models are questionable because the same test sets have been used to select these models for multiple years now. To understand the danger of overfitting, we measure the accuracy of CIFAR-10 classifiers by creating a new test set of truly unseen images. Although we ensure that the new test set is as close to the original data distribution as possible, we find a large drop in accuracy (4% to 10%) for a broad range of deep learning models. Yet more recent models with higher original accuracy show a smaller drop and better overall performance, indicating that this drop is likely not due to overfitting based on adaptivity. Instead, we view our results as evidence that current accuracy numbers are brittle and susceptible to even minute natural variations in the data distribution.

Citations

• Improving neural networks by preventing co-adaptation of feature detectors

Cited by

• Federated Learning with Feedback Alignment
• On the Bayes Inconsistency of Disagreement Discrepancy Surrogates
• Benchmarking Corruption Robustness of LVLMs: A Discriminative Benchmark and Robustness Alignment Metric
• ODP-Bench: Benchmarking Out-of-Distribution Performance Prediction
• How does the optimizer implicitly bias the model merging loss landscape?
• Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Ranking
• Probabilistic Runtime Verification, Evaluation and Risk Assessment of Visual Deep Learning Systems
• SSFN -- Self Size-estimating Feed-forward Network with Low Complexity, Limited Need for Human Intervention, and Consistent Behaviour across Trials
• Dual-View Alignment Learning with Hierarchical-Prompt for Class-Imbalance Multi-Label Classification
• A Modern Look at Simplicity Bias in Image Classification Tasks
• Natural Perturbed Training for General Robustness of Neural Network\n Classifiers
• Adapt in the Wild: Test-Time Entropy Minimization with Sharpness and Feature Regularization
• Domain Adaptation Techniques for Natural and Medical Image Classification
• ALSA: Anchors in Logit Space for Out-of-Distribution Accuracy Estimation
• Optimal multiclass overfitting by sequence reconstruction from Hamming\n queries
• NeuroView: Explainable Deep Network Decision Making
• Rip van Winkle's Razor: A Simple Estimate of Overfit to Test Data
• Decentralized Relaxed Smooth Optimization with Gradient Descent Methods
• Meta Transition Adaptation for Robust Deep Learning with Noisy Labels
• Adversarial Examples Are Not Bugs, They Are Features
• Consensus-Driven Active Model Selection
• Wiggling Weights to Improve the Robustness of Classifiers

Discussions

• Do CIFAR-10 Classifiers Generalize to CIFAR-10? [pdf] [hn, 3 points, 0 comments]
• Do CIFAR-10 Classifiers Generalize to CIFAR-10? [hn, 1 points, 0 comments]

Related

• Do We Train on Test Data? Purging CIFAR of Near-Duplicates
• Do ImageNet Classifiers Generalize to ImageNet?
• CIFAR-10: KNN-based Ensemble of Classifiers
• CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysis
• Improving image classifiers for small datasets by learning rate adaptations