Correlation Congruence for Knowledge Distillation

2019/04/03 by Baoyun Peng, Jin Xiao, Peng, Baoyun +13 · 15 citations
Computer Science · Medicine · #Advanced Neural Network Applications #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.1904.01802

openalex publication_date 2019/04/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Most teacher-student frameworks based on knowledge distillation (KD) depend on a strong congruent constraint on instance level. However, they usually ignore the correlation between multiple instances, which is also valuable for knowledge transfer. In this work, we propose a new framework named correlation congruence for knowledge distillation (CCKD), which transfers not only the instance-level information, but also the correlation between instances. Furthermore, a generalized kernel method based on Taylor series expansion is proposed to better capture the correlation between instances. Empirical experiments and ablation studies on image classification tasks (including CIFAR-100, ImageNet-1K) and metric learning tasks (including ReID and Face Recognition) show that the proposed CCKD substantially outperforms the original KD and achieves state-of-the-art accuracy compared with other SOTA KD-based methods. The CCKD can be easily deployed in the majority of the teacher-student framework such as KD and hint-based learning methods.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition
• Learning Multiple Layers of Features from Tiny Images

Cited by

• AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition
• Rhea: Role-aware Heuristic Episodic Attention for Conversational LLMs
• Heterogeneous Complementary Distillation
• Global Multiple Extraction Network for Low-Resolution Facial Expression Recognition
• Approaching Low-Cost Cardiac Intelligence with Semi-Supervised Knowledge Distillation
• UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representations
• Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity
• Learning Task-Agnostic Representations through Multi-Teacher Distillation
• Data-Augmented Quantization-Aware Knowledge Distillation
• Expandable Residual Approximation for Knowledge Distillation
• ATMS-KD: Adaptive Temperature and Mixed Sample Knowledge Distillation for a Lightweight Residual CNN in Agricultural Embedded Systems
• UNIFORM: Unifying Knowledge from Large-scale and Diverse Pre-trained Models
• Unified Knowledge Distillation Framework: Fine-Grained Alignment and Geometric Relationship Preservation for Deep Face Recognition
• NT-ML: Backdoor Defense via Non-target Label Training and Mutual Learning
• Cross-Architecture Distillation Made Simple with Redundancy Suppression

Related

• Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching
• An Embarrassingly Simple Approach for Knowledge Distillation
• Multi-level Knowledge Distillation via Knowledge Alignment and Correlation
• Knowledge Distillation from A Stronger Teacher
• Deep Collective Knowledge Distillation