Distilling Pixel-Wise Feature Similarities for Semantic Segmentation

2019/10/31 by Yuhu Shan, Shan, Yuhu
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #cs.CV

paper · pdf · doi:10.48550/arxiv.1910.14226

arxiv created 2019/10/31 · openalex publication_date 2019/10/31 · arxiv updated 2019/11/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Among the neural network compression techniques, knowledge distillation is an effective one which forces a simpler student network to mimic the output of a larger teacher network. However, most of such model distillation methods focus on the image-level classification task. Directly adapting these methods to the task of semantic segmentation only brings marginal improvements. In this paper, we propose a simple, yet effective knowledge representation referred to as pixel-wise feature similarities (PFS) to tackle the challenging distillation problem of semantic segmentation. The developed PFS encodes spatial structural information for each pixel location of the high-level convolutional features, which helps guide the distillation process in an easier way. Furthermore, a novel weighted pixel-level soft prediction imitation approach is proposed to enable the student network to selectively mimic the teacher network's output, according to their pixel-wise knowledge-gaps. Extensive experiments are conducted on the challenging datasets of Pascal VOC 2012, ADE20K and Pascal Context. Our approach brings significant performance improvements compared to several strong baselines and achieves new state-of-the-art results.

Citations

• FitNets: Hints for Thin Deep Nets
• Compressing Deep Convolutional Networks using Vector Quantization
• Tensorizing Neural Networks
• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?
• ImageNet classification with deep convolutional neural networks
• Net2Net: Accelerating Learning via Knowledge Transfer
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• An Analysis of Deep Neural Network Models for Practical Applications
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification
• Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks

Related

• AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentation
• Knowledge Adaptation for Efficient Semantic Segmentation
• Distilling Inter-Class Distance for Semantic Segmentation
• Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation
• Attention-guided Feature Distillation for Semantic Segmentation