2018/11/04 by Mohammad Saeed Shafiee, Shafiee, Mohammad Saeed, Mohammad Javad Shafiee +3
Computer Science · Engineering · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Industrial Vision Systems and Defect Detection #Machine Learning (cs.LG) #Machine Learning and Data Classification #Neural and Evolutionary Computing (cs.NE)
paper · pdf · doi:10.48550/arxiv.1811.01476
openalex publication_date 2018/11/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
While deep neural networks extract rich features from the input data, the\ncurrent trade-off between depth and computational cost makes it difficult to\nadopt deep neural networks for many industrial applications, especially when\ncomputing power is limited. Here, we are inspired by the idea that, while\ndeeper embeddings are needed to discriminate difficult samples (i.e.,\nfine-grained discrimination), a large number of samples can be well\ndiscriminated via much shallower embeddings (i.e., coarse-grained\ndiscrimination). In this study, we introduce the simple yet effective concept\nof decision gates (d-gate), modules trained to decide whether a sample needs to\nbe projected into a deeper embedding or if an early prediction can be made at\nthe d-gate, thus enabling the computation of dynamic representations at\ndifferent depths. The proposed d-gate modules can be integrated with any deep\nneural network and reduces the average computational cost of the deep neural\nnetworks while maintaining modeling accuracy. The proposed d-gate framework is\nexamined via different network architectures and datasets, with experimental\nresults showing that leveraging the proposed d-gate modules led to a ~43%\nspeed-up and 44% FLOPs reduction on ResNet-101 and 55% speed-up and 39% FLOPs\nreduction on DenseNet-201 trained on the CIFAR10 dataset with only ~2% drop in\naccuracy. Furthermore, experiments where d-gate modules are integrated into\nResNet-101 trained on the ImageNet dataset demonstrate that it is possible to\nreduce the computational cost of the network by 1.5 GFLOPs without any drop in\nthe modeling accuracy.\n