2020/07/20 by Anthony Berthelier, Berthelier, Anthony, Yongzhe Yan +9 · 1 citation
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Neural and Evolutionary Computing (cs.NE)
paper · pdf · doi:10.48550/arxiv.2007.10022
openalex publication_date 2020/07/20 · openalex created_date 2022/02/25 · openalex updated_date 2026/07/28
While deep neural networks (DNNs) have proven to be efficient for numerous\ntasks, they come at a high memory and computation cost, thus making them\nimpractical on resource-limited devices. However, these networks are known to\ncontain a large number of parameters. Recent research has shown that their\nstructure can be more compact without compromising their performance. In this\npaper, we present a sparsity-inducing regularization term based on the ratio\nl1/l2 pseudo-norm defined on the filter coefficients. By defining this\npseudo-norm appropriately for the different filter kernels, and removing\nirrelevant filters, the number of kernels in each layer can be drastically\nreduced leading to very compact Deep Convolutional Neural Networks (DCNN)\nstructures. Unlike numerous existing methods, our approach does not require an\niterative retraining process and, using this regularization term, directly\nproduces a sparse model during the training process. Furthermore, our approach\nis also much easier and simpler to implement than existing methods.\nExperimental results on MNIST and CIFAR-10 show that our approach significantly\nreduces the number of filters of classical models such as LeNet and VGG while\nreaching the same or even better accuracy than the baseline models. Moreover,\nthe trade-off between the sparsity and the accuracy is compared to other loss\nregularization terms based on the l1 or l2 norm as well as the SSL, NISP and\nGAL methods and shows that our approach is outperforming them.\n