Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

2019/05/17 by Linfeng Zhang, Jiebo Song, Zhang, Linfeng +9 · 83 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Computer science #Convolutional neural network #Distillation #Enhanced Data Rates for GSM Evolution #Flexibility (engineering) #Generalization #Inference #Machine Learning and Data Classification #Machine learning #Mathematics #Scalability #Softmax function #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1905.08094

published in arXiv (Cornell University) (Cornell University) · 10pages

arxiv created 2019/05/17 · openalex publication_date 2019/05/17 · arxiv updated 2019/05/21 · openalex created_date 2019/05/29 · openalex updated_date 2026/08/04

Abstract

Convolutional neural networks have been widely deployed in various application scenarios. In order to extend the applications' boundaries to some accuracy-crucial domains, researchers have been investigating approaches to boost accuracy through either deeper or wider network structures, which brings with them the exponential increment of the computational and storage cost, delaying the responding time. In this paper, we propose a general training framework named self distillation, which notably enhances the performance (accuracy) of convolutional neural networks through shrinking the size of the network rather than aggrandizing it. Different from traditional knowledge distillation - a knowledge transformation methodology among networks, which forces student neural networks to approximate the softmax layer outputs of pre-trained teacher neural networks, the proposed self distillation framework distills knowledge within network itself. The networks are firstly divided into several sections. Then the knowledge in the deeper portion of the networks is squeezed into the shallow ones. Experiments further prove the generalization of the proposed self distillation framework: enhancement of accuracy at average level is 2.65%, varying from 0.61% in ResNeXt as minimum to 4.07% in VGG19 as maximum. In addition, it can also provide flexibility of depth-wise scalable inference on resource-limited edge devices.Our codes will be released on github soon.

Citations

• ImageNet classification with deep convolutional neural networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Cited by

• ReLKD: Inter-Class Relation Learning with Knowledge Distillation for Generalized Category Discovery
• Self-supervised prior learning improves structured illumination microscopy resolution
• Hear What Matters! Text-conditioned Selective Video-to-Audio Generation
• FRAMER: Frequency-Aligned Self-Distillation with Adaptive Modulation Leveraging Diffusion Priors for Real-World Image Super-Resolution
• DuoTeach: Dual Role Self-Teaching for Coarse-to-Fine Decision Coordination in Vision--Language Models
• Real-Time 3D Object Detection with Inference-Aligned Learning
• Mitigating Negative Flips via Margin Preserving Training
• Beyond ImageNet: Understanding Cross-Dataset Robustness of Lightweight Vision Models
• Extending Label Smoothing Regularization with Self-Knowledge Distillation
• Attention Residual Fusion Network with Contrast for Source-free Domain Adaptation
• GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs
• Data-Centric Lessons To Improve Speech-Language Pretraining
• LLM Unlearning with LLM Beliefs
• ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder
• Learning Task-Agnostic Representations through Multi-Teacher Distillation
• Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
• From Reasoning LLMs to BERT: A Two-Stage Distillation Framework for Search Relevance
• Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers
• Artificial Hippocampus Networks for Efficient Long-Context Modeling
• StaR-KVQA: Structured Reasoning Traces for Implicit-Knowledge Visual Question Answering
• Knowledge Distillation Detection for Open-weights Models
• Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training
• Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Crops
• Replication and Information Extraction in a Minimal Agent-Environment Model
• Enriching Knowledge Distillation with Intra-Class Contrastive Learning
• Dark3R: Learning Structure from Motion in the Dark
• Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception
• Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding
• GhostNetV3-Small: A Tailored Architecture and Comparative Study of Distillation Strategies for Tiny Images
• Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation
• Formal Algorithms for Model Efficiency
• Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving
• LPGNet: A Lightweight Network with Parallel Attention and Gated Fusion for Multimodal Emotion Recognition
• BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
• NT-ML: Backdoor Defense via Non-target Label Training and Mutual Learning
• CORE-ReID: Comprehensive Optimization and Refinement through Ensemble fusion in Domain Adaptation for person re-identification
• DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging
• ProGMLP: A Progressive Framework for GNN-to-MLP Knowledge Distillation with Efficient Trade-offs
• Unified Medical Image Segmentation with State Space Modeling Snake
• ATAS: Any-to-Any Self-Distillation for Enhanced Open-Vocabulary Dense Prediction
• SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models
• GeoDistill: Geometry-Guided Self-Distillation for Weakly Supervised Cross-View Localization
• The Power of Certainty: How Confident Models Lead to Better Segmentation
• HMID-Net: An Exploration of Masked Image Modeling and Knowledge Distillation in Hyperbolic Space
• PRIME: Large Language Model Personalization with Cognitive Dual-Memory and Personalized Thought Process
• Information-Bottleneck Driven Binary Neural Network for Change Detection
• Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation
• Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification
• Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap
• MonoVQD: Monocular 3D Object Detection with Variational Query Denoising and Self-Distillation
• Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
• Ground Reaction Force Estimation via Time-aware Knowledge Distillation
• How to Train Your Metamorphic Deep Neural Network
• Smooth Model Compression without Fine-Tuning
• Matryoshka Model Learning for Improved Elastic Student Models
• Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
• Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects
• Shifting AI Efficiency From Model-Centric to Data-Centric Compression
• Intra-class Patch Swap for Self-Distillation
• DCSNet: A Lightweight Knowledge Distillation-Based Model with Explainable AI for Lung Cancer Diagnosis from Histopathological Images
• Erased but Not Forgotten: How Backdoors Compromise Concept Erasure
• When are likely answers right? On Sequence Probability and Correctness in LLMs
• Weak-Driven Learning: How Weak Agents make Strong Agents Stronger
• Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
• iFAN: Inference-Aware Learning for Plain Mask Transformers
• Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data
• BioKD: Selective Physiology-to-Video Knowledge Distillation via Reliability Gate for Emotion Recognition
• Position: It's Time to Optimize LLMs for Self-Consistency
• Reinforced Multi-teacher Knowledge Distillation for Efficient General Image Forgery Detection and Localization

Related

• Training convolutional neural networks with cheap convolutions and online distillation
• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
• A New Training Framework for Deep Neural Network
• Knowledge Distillation: Enhancing Neural Network Compression with Integrated Gradients
• On Self-Distilling Graph Neural Network