Positive-Congruent Training: Towards Regression-Free Model Updates

2020/11/18 by Sijie Yan, Yuanjun Xiong, Yan, Sijie +13 · 4 citations
Computer Science · #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2011.09161

openalex publication_date 2020/11/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reducing inconsistencies in the behavior of different versions of an AI system can be as important in practice as reducing its overall error. In image classification, sample-wise inconsistencies appear as "negative flips": A new model incorrectly predicts the output for a test sample that was correctly classified by the old (reference) model. Positive-congruent (PC) training aims at reducing error rate while at the same time reducing negative flips, thus maximizing congruency with the reference model only on positive predictions, unlike model distillation. We propose a simple approach for PC training, Focal Distillation, which enforces congruence with the reference model by giving more weights to samples that were correctly classified. We also found that, if the reference model itself can be chosen as an ensemble of multiple deep neural networks, negative flips can be further reduced without affecting the new model's accuracy.

Citations

• Distilling the Knowledge in a Neural Network
• The Loss Surfaces of Multilayer Networks
• Stochastic Estimation of the Maximum of a Regression Function
• Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• ImageNet Large Scale Visual Recognition Challenge
• Learning without Forgetting
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

Cited by

• Mitigating Negative Flips via Margin Preserving Training
• DeepTrust: Multi-Step Classification through Dissimilar Adversarial Representations for Robust Android Malware Detection
• \boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning
• On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI

Related

• Mitigating Negative Flips via Margin Preserving Training
• ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training
• Bag of Tricks for Image Classification with Convolutional Neural Networks
• DiscrimLoss: A Universal Loss for Hard Samples and Incorrect Samples Discrimination
• More Is More -- Narrowing the Generalization Gap by Adding Classification Heads