Distilling Critical Paths in Convolutional Neural Networks

2018/10/28 by Fuxun Yu, Yu, Fuxun, Zhuwei Qin +3 · 2 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.1811.02643

openalex publication_date 2018/10/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Neural network compression and acceleration are widely demanded currently due to the resource constraints on most deployment targets. In this paper, through analyzing the filter activation, gradients, and visualizing the filters' functionality in convolutional neural networks, we show that the filters in higher layers learn extremely task-specific features, which are exclusive for only a small subset of the overall tasks, or even a single class. Based on such findings, we reveal the critical paths of information flow for different classes. And by their intrinsic property of exclusiveness, we propose a critical path distillation method, which can effectively customize the convolutional neural networks to small ones with much smaller model size and less computation.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Understanding Neural Networks Through Deep Visualization
• Pruning Filters for Efficient ConvNets
• On the importance of single directions for generalization
• Revisiting the Importance of Individual Units in CNNs via Ablation

Cited by

• Fed2: Feature-Aligned Federated Learning
• Anomaly Detection Based on Critical Paths for Deep Neural Networks

Related

• Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
• A scalable convolutional neural network for task-specified scenarios via knowledge distillation
• Compact CNN Structure Learning by Knowledge Distillation
• Knowledge Distillation: Enhancing Neural Network Compression with Integrated Gradients
• Information Bottleneck Theory on Convolutional Neural Networks