2025/09/15 by Florian Zager, Zager, Florian, Hamza A. A. Gardi +1
Computer Science · Engineering · #Adaptation (eye) #Architecture #Artificial Intelligence (cs.AI) #Artificial neural network #Baseline (sea) #Computer Vision and Pattern Recognition (cs.CV) #Distillation #FOS: Computer and information sciences #Focus (optics) #Generative Adversarial Networks and Image Synthesis #Image Processing Techniques and Applications #Inference #Machine Learning (cs.LG) #Software deployment #Visual Attention and Saliency Detection
paper · pdf · doi:10.48550/arxiv.2509.12380
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/09/15 · openalex created_date 2025/10/18 · openalex updated_date 2026/08/05
Deep neural networks have achieved remarkable success across a range of tasks, however their computational demands often make them unsuitable for deployment on resource-constrained edge devices. This paper explores strategies for compressing and adapting models to enable efficient inference in such environments. We focus on GhostNetV3, a state-of-the-art architecture for mobile applications, and propose GhostNetV3-Small, a modified variant designed to perform better on low-resolution inputs such as those in the CIFAR-10 dataset. In addition to architectural adaptation, we provide a comparative evaluation of knowledge distillation techniques, including traditional knowledge distillation, teacher assistants, and teacher ensembles. Experimental results show that GhostNetV3-Small significantly outperforms the original GhostNetV3 on CIFAR-10, achieving an accuracy of 93.94%. Contrary to expectations, all examined distillation strategies led to reduced accuracy compared to baseline training. These findings indicate that architectural adaptation can be more impactful than distillation in small-scale image classification tasks, highlighting the need for further research on effective model design and advanced distillation techniques for low-resolution domains.