Efficient and Accurate Approximations of Nonlinear Convolutional Networks

2014/11/16 by Xiangyu Zhang, Jianhua Zou, Zhang, Xiangyu +8
Computer Science · Engineering · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning and Algorithms #Sparse and Compressive Sensing Techniques #cs.CV

paper · pdf · doi:10.48550/arxiv.1411.4229

arxiv created 2014/11/16 · openalex publication_date 2014/11/16 · arxiv updated 2014/11/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper aims to accelerate the test-time computation of deep convolutional neural networks (CNNs). Unlike existing methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We minimize the reconstruction error of the nonlinear responses, subject to a low-rank constraint which helps to reduce the complexity of filters. We develop an effective solution to this constrained nonlinear optimization problem. An algorithm is also presented for reducing the accumulated error when multiple layers are approximated. A whole-model speedup ratio of 4x is demonstrated on a large network trained for ImageNet, while the top-5 error rate is only increased by 0.9%. Our accelerated model has a comparably fast speed as the "AlexNet", but is 4.7% more accurate.

Related

• Accelerating Very Deep Convolutional Networks for Classification and Detection
• A Unified Approximation Framework for Compressing and Accelerating Deep Neural Networks
• Convolutional Neural Networks at Constrained Time Cost
• NL-CNN: A Resources-Constrained Deep Learning Model based on Nonlinear\n Convolution
• Fast Training of Convolutional Neural Networks via Kernel Rescaling