Identifying Mislabeled Data using the Area Under the Margin Ranking

2020/01/28 by Geoff Pleiss, Tianyi Zhang, Pleiss, Geoff +5 · 26 citations
Computer Science · #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2001.10528

openalex publication_date 2020/01/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Not all data in a typical training set help with generalization; some samples can be overly ambiguous or outrightly mislabeled. This paper introduces a new method to identify such samples and mitigate their impact when training neural networks. At the heart of our algorithm is the Area Under the Margin (AUM) statistic, which exploits differences in the training dynamics of clean and mislabeled samples. A simple procedure - adding an extra class populated with purposefully mislabeled threshold samples - learns a AUM upper bound that isolates mislabeled data. This approach consistently improves upon prior work on synthetic and real-world datasets. On the WebVision50 classification task our method removes 17% of training data, yielding a 1.6% (absolute) improvement in test error. On CIFAR100 removing 13% of the data leads to a 1.2% drop in error.

Citations

• WebVision Database: Visual Learning and Understanding from Web Data
• How does Disagreement Help Generalization against Label Corruption?
• Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Learning Multiple Layers of Features from Tiny Images

Cited by

• RS-Prune: Training-Free Data Pruning at High Ratios for Efficient Remote Sensing Diffusion Foundation Models
• Semi-Supervised Learning for Large Language Models Safety and Content Moderation
• Investigating Data Pruning for Pretraining Biological Foundation Models at Scale
• Adaptive Dataset Quantization: A New Direction for Dataset Pruning
• Anomaly Detection with Adaptive and Aggressive Rejection for Contaminated Training Data
• Pre-train to Gain: Robust Learning Without Clean Labels
• Diffusion Reconstruction-based Data Likelihood Estimation for Core-Set Selection
• UNSEEN: Enhancing Dataset Pruning from a Generalization Perspective
• Hard Samples, Bad Labels: Robust Loss Functions That Know When to Back Off
• SNAP: Low-Latency Test-Time Adaptation with Sparse Updates
• Mitigating Negative Flips via Margin Preserving Training
• Calibration improves detection of mislabeled examples
• Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling
• Benchmarking noisy label detection methods
• Predictive Modeling and Explainable AI for Veterinary Safety Profiles, Residue Assessment, and Health Outcomes Using Real-World Data and Physicochemical Properties
• SubZeroCore: A Submodular Approach with Zero Training for Coreset Selection
• HyperCore: Coreset Selection under Noise via Hypersphere Models
• LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training
• Data-Efficient Stream-Based Active Distillation for Scalable Edge Model Deployment
• Learning to Detect Label Errors by Making Them: A Method for Segmentation and Object Detection Datasets
• Pseudo-D: Informing Multi-View Uncertainty Estimation with Calibrated Neural Training Dynamics
• Hadamard-Riemannian Optimization for Margin-Variance Ensemble
• Coresets from Trajectories: Selecting Data via Correlation of Loss Differences
• Guiding Noisy Label Conditional Diffusion Models with Score-based Discriminator Correction
• Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection
• Unlocking Robust Semantic Segmentation Performance via Label-only Elastic Deformations against Implicit Label Noise
• Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap

Related

• Assessing the Quality of the Datasets by Identifying Mislabeled Samples
• Enhancing Sample Selection Against Label Noise by Cutting Mislabeled Easy Examples
• Learning from Training Dynamics: Identifying Mislabeled Data Beyond Manually Designed Features
• Towards a New Understanding of the Training of Neural Networks with Mislabeled Training Data
• Late Stopping: Avoiding Confidently Learning from Mislabeled Examples