2025/08/30 by Zhen Li, Li, Zhengda
Computer Science · Medicine · #Alpha (finance) #Anomaly Detection Techniques and Applications #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #Contextual image classification #Distillation #FOS: Computer and information sciences #Hyperparameter #Image (mathematics) #Image Retrieval and Classification Techniques #Pattern recognition (psychology) #Weighting
paper · pdf · doi:10.48550/arxiv.2509.05319
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/08/30 · openalex created_date 2025/10/11 · openalex updated_date 2026/08/05
Knowledge distillation (KD) is a widely used technique to transfer knowledge from a large teacher network to a smaller student model. Traditional KD uses a fixed balancing factor alpha as a hyperparameter to combine the hard-label cross-entropy loss with the soft-label distillation loss. However, a static alpha is suboptimal because the optimal trade-off between hard and soft supervision can vary during training. In this work, we propose an Adaptive Knowledge Distillation (AKD) framework. First we try to make alpha as learnable parameter that can be automatically learned and optimized during training. Then we introduce a formula to reflect the gap between the student and the teacher to compute alpha dynamically, guided by student-teacher discrepancies, and further introduce a Context-Aware Module (CAM) using MLP + Attention to adaptively reweight class-wise teacher outputs. Experiments on CIFAR-10 with ResNet-50 as teacher and ResNet-18 as student demonstrate that our approach achieves superior accuracy compared to fixed-weight KD baselines, and yields more stable convergence.