2021/03/25 by Haozhi Zhang, Zhang, Haozhi, Xun Wang +5 · 1 citation
Computer Science · Engineering · #Advanced Image and Video Retrieval Techniques #Algorithm #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Embedding #Engineering #FOS: Computer and information sciences #Face recognition and analysis #Intuition #Key (lock) #Machine learning #Metric (unit) #Multimodal Machine Learning Applications #Simple (philosophy) #Theoretical computer science #Weighting #cs.CV
paper · pdf · doi:10.48550/arxiv.2103.14003
published in arXiv (Cornell University) (Cornell University) · Under review
arxiv created 2021/03/25 · openalex publication_date 2021/03/25 · arxiv updated 2021/03/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Pair-wise loss functions have been extensively studied and shown to continuously improve the performance of deep metric learning (DML). However, they are primarily designed with intuition based on simple toy examples, and experimentally identifying the truly effective design is difficult in complicated, real-world cases. In this paper, we provide a new methodology for systematically studying weighting strategies of various pair-wise loss functions, and rethink pair weighting with an embedding memory. We delve into the weighting mechanisms by decomposing the pair-wise functions, and study positive and negative weights separately using direct weight assignment. This allows us to study various weighting functions deeply and systematically via weight curves, and identify a number of meaningful, comprehensive and insightful facts, which come up with our key observation on memory-based DML: it is critical to mine hard negatives and discard easy negatives which are less informative and redundant, but weighting on positive pairs is not helpful. This results in an efficient but surprisingly simple rule to design the weighting scheme, making it significantly different from existing mini-batch based methods which design various sophisticated loss functions to weight pairs carefully. Finally, we conduct extensive experiments on three large-scale visual retrieval benchmarks, and demonstrate the superiority of memory-based DML over recent mini-batch based approaches, by using a simple contrastive loss with momentum-updated memory.