Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

2019/06/18 by Kaidi Cao, Colin Wei, Cao, Kaidi +7 · 63 citations
Computer Science · #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Imbalanced Data Classification Techniques #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1906.07413

openalex publication_date 2019/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep learning algorithms can fare poorly when the training dataset suffers from heavy class-imbalance but the testing criterion requires good generalization on less frequent classes. We design two novel methods to improve performance in such scenarios. First, we propose a theoretically-principled label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound. This loss replaces the standard cross-entropy objective during training and can be applied with prior strategies for training with class-imbalance such as re-weighting or re-sampling. Second, we propose a simple, yet effective, training schedule that defers re-weighting until after the initial stage, allowing the model to learn an initial representation while avoiding some of the complications associated with re-weighting or re-sampling. We test our methods on several benchmark vision tasks including the real-world imbalanced dataset iNaturalist 2018. Our experiments show that either of these methods alone can already improve over existing techniques and their combination achieves even better performance gains.

Citations

• Gradient-based learning applied to document recognition
• ImageNet Large Scale Visual Recognition Challenge
• YFCC100M
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• A systematic study of the class imbalance problem in convolutional neural networks
• Stronger generalization bounds for deep nets via a compression approach
• Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Spectral-Aware Analytic Class-Incremental Learning for Long-Tailed Distributions
• Dual-Margin Embedding for Fine-Grained Long-Tailed Plant Taxonomy
• Interaction-via-Actions: Cattle Interaction Detection with Joint Learning of Action-Interaction Latent Space
• Balanced Learning for Domain Adaptive Semantic Segmentation
• Operator-Theoretic Framework for Gradient-Free Federated Learning
• The Multiclass Score-Oriented Loss (MultiSOL) on the Simplex
• Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition
• Space Alignment Matters: The Missing Piece for Inducing Neural Collapse in Long-Tailed Learning
• Bayesian-based Online Label Shift Estimation with Dynamic Dirichlet Priors
• Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation
• Exploiting Inter-Sample Information for Long-tailed Out-of-Distribution Detection
• Oya: Deep Learning for Accurate Global Precipitation Estimation
• Intrinsic Dimensionality as a Model-Free Measure of Class Imbalance
• GraphSB: Boosting Imbalanced Node Classification on Graphs through Structural Balance
• PANDA -- Patch And Distribution-Aware Augmentation for Long-Tailed Exemplar-Free Continual Learning
• Mitigating Negative Flips via Margin Preserving Training
• LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups
• Spurious Correlation-Aware Embedding Regularization for Worst-Group Robustness
• SPECTRA: Spectral Domain-Aware Graph Generation for Imbalanced Molecular Property Regression
• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
• Long-tailed Species Recognition in the NACTI Wildlife Dataset
• Concentration and excess risk bounds for imbalanced classification with synthetic oversampling
• Exploring "Many in Few" and "Few in Many" Properties in Long-Tailed, Highly-Imbalanced IC Defect Classification
• Beyond Frequency: Scoring-Driven Debiasing for Object Detection via Blueprint-Prompted Image Synthesis
• Large Language Models for Imbalanced Classification: Diversity makes the difference
• Long-Tailed Recognition via Information-Preservable Two-Stage Learning
• Dual-granularity Sinkhorn Distillation for Enhanced Learning from Long-tailed Noisy Data
• Long-tailed Recognition with Model Rebalancing
• Revisiting Mixout: An Overlooked Path to Robust Finetuning
• Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Ranking
• Ordinal Label-Distribution Learning with Constrained Asymmetric Priors for Imbalanced Retinal Grading
• Reweighted Flow Matching via Unbalanced OT for Label-free Long-tailed Generation
• Convolutional Neural Nets vs Vision Transformers: A SpaceNet Case Study with Balanced vs Imbalanced Regimes
• FracAug: Fractional Augmentation boost Graph-level Anomaly Detection under Limited Supervision
• What Makes Deep Learning Work for Traditional Chinese Medicine Tongue Diagnosis? A Comprehensive Ablation Study
• Dual-View Alignment Learning with Hierarchical-Prompt for Class-Imbalance Multi-Label Classification
• Intra-Cluster Mixup: An Effective Data Augmentation Technique for Complementary-Label Learning
• Efficient Long-Tail Learning in Latent Space by sampling Synthetic Data
• Long-Tailed Out-of-Distribution Detection with Refined Separate Class Learning
• Learning by Imagining: Debiased Feature Augmentation for Compositional Zero-Shot Learning
• Curvature Learning for Generalization of Hyperbolic Neural Networks
• Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning
• Optimizing Class Distributions for Bias-Aware Multi-Class Learning
• ACE and Diverse Generalization via Selective Disagreement
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments
• Jointly De-biasing Face Recognition and Demographic Attribute Estimation
• Self-Damaging Contrastive Learning
• Conditional-t³VAE: Equitable Latent Space Allocation for Fair Generation
• MSE Loss with Outlying Label for Imbalanced Classification
• Mixture of Balanced Information Bottlenecks for Long-Tailed Visual Recognition
• ART: Adaptive Resampling-based Training for Imbalanced Classification
• Learning with Multiclass AUC: Theory and Algorithms
• Divide, Weight, and Route: Difficulty-Aware Optimization with Dynamic Expert Fusion for Long-tailed Recognition
• Class-wise Flooding Regularization for Imbalanced Image Classification
• LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions
• Conformal Prediction Meets Long-tail Classification
• GazeLT: Visual attention-guided long-tailed disease classification in chest radiographs
• Deep Neural Network Calibration by Reducing Classifier Shift with Stochastic Masking
• SPA++: Generalized Graph Spectral Alignment for Versatile Domain Adaptation
• Investigate the Essence of Long-Tailed Recognition from a Unified Perspective
• An Enhanced Focal Loss Function to Mitigate Class Imbalance in Auto Insurance Fraud Detection with Explainable AI
• Generative Active Learning for Long-tail Trajectory Prediction via Controllable Diffusion Model

Related

• Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Data
• Class-Wise Difficulty-Balanced Loss for Solving Class-Imbalance
• Learning with Imbalanced Noisy Data by Preventing Bias in Sample Selection
• Phased Progressive Learning with Coupling-Regulation-Imbalance Loss for Imbalanced Data Classification
• Enlarged Large Margin Loss for Imbalanced Classification