DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer

2017/07/05 by Yuntao Chen, Naiyan Wang, Chen, Yuntao +3 · 21 citations
Computer Science · Engineering · Mathematics · #Advanced Image and Video Retrieval Techniques #Artificial intelligence #Artificial neural network #Baseline (sea) #Cluster analysis #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Deep learning #Engineering #FOS: Computer and information sciences #Generalization #Human Pose and Action Recognition #Identification (biology) #Machine Learning (cs.LG) #Machine learning #Margin (machine learning) #Mathematics #Metric (unit) #Neural and Evolutionary Computing (cs.NE) #Rank (graph theory) #Sample (material) #Transfer of learning #Video Surveillance and Tracking Methods #cs.CV #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1707.01220

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2017/07/05 · arxiv created 2017/12/18 · arxiv updated 2017/12/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

We have witnessed rapid evolution of deep neural network architecture design in the past years. These latest progresses greatly facilitate the developments in various areas such as computer vision and natural language processing. However, along with the extraordinary performance, these state-of-the-art models also bring in expensive computational cost. Directly deploying these models into applications with real-time requirement is still infeasible. Recently, Hinton etal. have shown that the dark knowledge within a powerful teacher model can significantly help the training of a smaller and faster student network. These knowledge are vastly beneficial to improve the generalization ability of the student model. Inspired by their work, we introduce a new type of knowledge -- cross sample similarities for model compression and acceleration. This knowledge can be naturally derived from deep metric learning model. To transfer them, we bring the "learning to rank" technique into deep metric learning formulation. We test our proposed DarkRank method on various metric learning tasks including pedestrian re-identification, image retrieval and image clustering. The results are quite encouraging. Our method can improve over the baseline method by a large margin. Moreover, it is fully compatible with other existing methods. When combined, the performance can be further boosted.

Citations

• Distilling the Knowledge in a Neural Network
• ImageNet Large Scale Visual Recognition Challenge
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Binarized Neural Networks
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Large-Margin Softmax Loss for Convolutional Neural Networks
• In Defense of the Triplet Loss for Person Re-Identification
• A Good Practice Towards Top Performance of Face Recognition: Transferred Deep Feature Fusion
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Unsupervised Learning of Visual Representations using Videos

Cited by

• Knowledge Distillation Detection for Open-weights Models
• Expandable Residual Approximation for Knowledge Distillation
• Unified Knowledge Distillation Framework: Fine-Grained Alignment and Geometric Relationship Preservation for Deep Face Recognition
• Fine-grained Image Retrieval via Dual-Vision Adaptation
• Model compression using knowledge distillation with integrated gradients
• It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMs
• Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
• BackSlash: Rate Constrained Optimized Training of Large Language Models
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation

Related

• Revisiting Training Strategies and Generalization Performance in Deep Metric Learning
• Deep Metric Learning for Computer Vision: A Brief Overview
• DiVA: Diverse Visual Feature Aggregation for Deep Metric Learning
• Visual Explanation for Deep Metric Learning
• Deep Metric Learning by Online Soft Mining and Class-Aware Attention