Convolutional neural networks with low-rank regularization

2015/11/19 by Cheng Tai, Tai, Cheng, Tong Xiao +7 · 8 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Face and Expression Recognition #Image and Signal Denoising Methods #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1511.06067

openalex publication_date 2015/11/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large CNNs have delivered impressive performance in various computer vision applications. But the storage and computation requirements make it problematic for deploying these models on mobile devices. Recently, tensor decompositions have been used for speeding up CNNs. In this paper, we further develop the tensor decomposition technique. We propose a new algorithm for computing the low-rank tensor decomposition for removing the redundancy in the convolution kernels. The algorithm finds the exact global optimizer of the decomposition and is more effective than iterative methods. Based on the decomposition, we further propose a new method for training low-rank constrained CNNs from scratch. Interestingly, while achieving a significant speedup, sometimes the low-rank constrained CNNs delivers significantly better performance than their non-constrained counterparts. On the CIFAR-10 dataset, the proposed low-rank NIN model achieves 91.31% accuracy (without data augmentation), which also improves upon state-of-the-art result. We evaluated the proposed method on CIFAR-10 and ILSVRC12 datasets for a variety of modern CNNs, including AlexNet, NIN, VGG and GoogleNet with success. For example, the forward time of VGG-16 is reduced by half while the performance is still comparable. Empirical success suggests that low-rank tensor decompositions can be a very useful tool for speeding up large CNNs.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• ImageNet Large Scale Visual Recognition Challenge
• Speeding-up Convolutional Neural Networks Using Fine-tuned\n CP-Decomposition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Fully Convolutional Networks for Semantic Segmentation

Cited by

• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples
• Quantization Range Estimation for Convolutional Neural Networks
• Fast and accurate neural reflectance transformation imaging through knowledge distillation
• FastCaps: A Design Methodology for Accelerating Capsule Network on Field Programmable Gate Arrays
• GASL: Guided Attention for Sparsity Learning in Deep Neural Networks
• Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning
• Convolutional neural networks compression with low rank and sparse tensor decompositions
• Matrix and tensor decompositions for training binary neural networks

Related

• Stable Low-rank Tensor Decomposition for Compression of Convolutional\n Neural Network
• Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization
• Speeding-up Convolutional Neural Networks Using Fine-tuned\n CP-Decomposition
• Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks
• CNN Acceleration by Low-rank Approximation with Quantized Factors