Contrastive Representations for Label Noise Require Fine-Tuning

2021/08/20 by Pierre Nodet, Vincent Lemaire, Nodet, Pierre +5
Computer Science · #Advanced Multi-Objective Optimization Algorithms #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2108.09154

openalex publication_date 2021/08/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper we show that the combination of a Contrastive representation with a label noise-robust classification head requires fine-tuning the representation in order to achieve state-of-the-art performances. Since fine-tuned representations are shown to outperform frozen ones, one can conclude that noise-robust classification heads are indeed able to promote meaningful representations if provided with a suitable starting point. Experiments are conducted to draw a comprehensive picture of performances by featuring six methods and nine noise instances of three different kinds (none, symmetric, and asymmetric). In presence of noise the experiments show that fine tuning of Contrastive representation allows the six methods to achieve better results than end-to-end learning and represent a new reference compare to the recent state of art. Results are also remarkable stable versus the noise level.

Citations

• A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Understanding deep learning (still) requires rethinking generalization
• Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
• Improved Baselines with Momentum Contrastive Learning
• Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
• Momentum Contrast for Unsupervised Visual Representation Learning
• Rethinking Importance Weighting for Deep Learning under Distribution Shift

Related

• Channel-Wise Contrastive Learning for Learning with Noisy Labels
• Fine-Grained Classification with Noisy Labels
• An Inclusive Theoretical Framework of Robust Supervised Contrastive Loss against Label Noise
• A Survey of Label-noise Representation Learning: Past, Present and Future
• On Learning Contrastive Representations for Learning with Noisy Labels