Training Over-parameterized Models with Non-decomposable Objectives

2021/07/09 by Harikrishna Narasimhan, Aditya Krishna Menon, Narasimhan, Harikrishna +1
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2107.04641

openalex publication_date 2021/07/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Many modern machine learning applications come with complex and nuanced design goals such as minimizing the worst-case error, satisfying a given precision or recall target, or enforcing group-fairness constraints. Popular techniques for optimizing such non-decomposable objectives reduce the problem into a sequence of cost-sensitive learning tasks, each of which is then solved by re-weighting the training loss with example-specific costs. We point out that the standard approach of re-weighting the loss to incorporate label costs can produce unsatisfactory results when used to train over-parameterized models. As a remedy, we propose new cost-sensitive losses that extend the classical idea of logit adjustment to handle more general cost matrices. Our losses are calibrated, and can be further improved with distilled labels from a teacher model. Through experiments on benchmark image datasets, we showcase the effectiveness of our approach in training ResNet models with common robust and constrained optimization objectives.

Citations

• Distilling the Knowledge in a Neural Network
• ImageNet Large Scale Visual Recognition Challenge
• Convex Calibration Dimension for Multiclass Loss Matrices
• Equality of Opportunity in Supervised Learning
• Policy Distillation
• A systematic study of the class imbalance problem in convolutional neural networks
• Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment
• Training Well-Generalizing Classifiers for Fairness Metrics and Other Data-Dependent Constraints
• Self-Training With Noisy Student Improves ImageNet Classification
• Learning Multiple Layers of Features from Tiny Images
• Distribution Alignment: A Unified Framework for Long-tail Visual Recognition

Related

• Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining
• You Only Train Once
• Negotiated Representations to Prevent Overfitting in Machine Learning Applications
• Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws
• Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learning