Efficient Inferencing of Compressed Deep Neural Networks

2017/11/01 by Dharma Teja Vooturi, Saurabh Goyal, Vooturi, Dharma Teja +7 · 1 citation
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Distributed #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Parallel #and Cluster Computing (cs.DC) #cs.DC #cs.LG

paper · pdf · doi:10.48550/arxiv.1711.00244

arxiv created 2017/11/01 · openalex publication_date 2017/11/01 · arxiv updated 2017/11/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large number of weights in deep neural networks makes the models difficult to be deployed in low memory environments such as, mobile phones, IOT edge devices as well as "inferencing as a service" environments on cloud. Prior work has considered reduction in the size of the models, through compression techniques like pruning, quantization, Huffman encoding etc. However, efficient inferencing using the compressed models has received little attention, specially with the Huffman encoding in place. In this paper, we propose efficient parallel algorithms for inferencing of single image and batches, under various memory constraints. Our experimental results show that our approach of using variable batch size for inferencing achieves 15-25% performance improvement in the inference throughput for AlexNet, while maintaining memory and latency constraints.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Compressing Deep Convolutional Networks using Vector Quantization
• Gradient-based learning applied to document recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Trained Ternary Quantization
• Caffe: Convolutional Architecture for Fast Feature Embedding

Cited by

• PipeSim: Trace-driven Simulation of Large-Scale AI Operations Platforms

Related

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• SlimNets: An Exploration of Deep Model Compression and Acceleration
• Deep Model Compression Via Two-Stage Deep Reinforcement Learning
• To Compress, or Not to Compress: Characterizing Deep Learning Model Compression for Embedded Inference
• Data-Driven Compression of Convolutional Neural Networks