Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

2014/04/02 by Emily Denton, Wojciech Zaremba, Denton, Remi +7 · 118 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.1404.0736

openalex publication_date 2014/04/02 · openalex created_date 2016/06/24 · openalex updated_date 2026/07/28

Abstract

We present techniques for speeding up the test-time evaluation of large convolutional networks, designed for object recognition tasks. These models deliver impressive accuracy but each image evaluation requires millions of floating point operations, making their deployment on smartphones and Internet-scale clusters problematic. The computation is dominated by the convolution operations in the lower layers of the model. We exploit the linear structure present within the convolutional filters to derive approximations that significantly reduce the required computation. Using large state-of-the-art models, we demonstrate we demonstrate speedups of convolutional layers on both CPU and GPU by a factor of 2x, while keeping the accuracy within 1% of the original model.

Citations

• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• ImageNet classification with deep convolutional neural networks
• Building high-level features using large scale unsupervised learning

Cited by

• High-Dimensional Search, Low-Dimensional Solution: Decoupling Optimization from Representation
• Element-wise Modulation of Random Matrices for Efficient Neural Layers
• Learning Single-Image Super-Resolution in the JPEG Compressed Domain
• CompressNAS : A Fast and Efficient Technique for Model Compression using Decomposition
• An Efficient Gradient-Aware Error-Bounded Lossy Compressor for Federated Learning
• Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks
• Feedback Alignment Meets Low-Rank Manifolds: A Structured Recipe for Local Learning
• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples
• CAST: Compositional Analysis via Spectral Tracking for Understanding Transformer Layer Functions
• Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility
• Layer-wise dynamic rank for compressing large language models
• BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression
• COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
• Learning from Compressed CT: Feature Attention Style Transfer and Structured Factorized Projections for Resource-Efficient Medical Image Analysis
• Formal Algorithms for Model Efficiency
• ONG: One-Shot NMF-based Gradient Masking for Efficient Model Sparsification
• Investigating 1-Bit Quantization in Transformer-Based Top Tagging
• FGFP: A Fractional Gaussian Filter and Pruning for Deep Neural Networks Compression
• COLI: A Hierarchical Efficient Compressor for Large Images
• MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression
• On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization
• Model Compression using Progressive Channel Pruning
• Importance-Aware Activation Space Reconstruction
• Reaching Data Confidentiality and Model Accountability on the CalTrain
• Multilinear Map Layer: Prediction Regularization by Structural Constraint
• Principal Component Networks: Parameter Reduction Early in Training
• Pruning Convolutional Neural Networks for Image Instance Retrieval
• Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System
• Fully Learnable Group Convolution for Acceleration of Deep Neural Networks
• Dynamic Zoom-in Network for Fast Object Detection in Large Images
• Structured Convolutions for Efficient Neural Network Design
• Knowledge Projection for Deep Neural Networks
• Neural Machine Translation: A Review and Survey
• ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
• WSNet: Compact and Efficient Networks Through Weight Sampling
• Coordinating Filters for Faster Deep Neural Networks
• Multi-objective Evolutionary Approach for Efficient Kernel Size and Shape for CNN
• Structure-Aware Automatic Channel Pruning by Searching with Graph Embedding
• A Lite Distributed Semantic Communication System for Internet of Things
• A One-step Pruning-recovery Framework for Acceleration of Convolutional Neural Networks
• Partition Pruning: Parallelization-Aware Pruning for Deep Neural Networks
• Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization
• Toward Compact Deep Neural Networks via Energy-Aware Pruning
• No Multiplication? No Floating Point? No Problem! Training Networks for Efficient Inference
• Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation
• LFA applied to CNNs: Efficient Singular Value Decomposition of Convolutional Mappings by Local Fourier Analysis
• HOTCAKE: Higher Order Tucker Articulated Kernels for Deeper CNN Compression
• Image Super-Resolution Using Deep Convolutional Networks
• Rate Distortion For Model Compression: From Theory To Practice
• Localized Compression: Applying Convolutional Neural Networks to Compressed Images
• Deep Model Compression Via Two-Stage Deep Reinforcement Learning
• To Compress, or Not to Compress: Characterizing Deep Learning Model Compression for Embedded Inference
• Provable Filter Pruning for Efficient Neural Networks
• Cogradient Descent for Dependable Learning
• Binary embeddings with structured hashed projections
• Energy Considerations for Large Pretrained Neural Networks
• Knowledge Distillation: A Survey
• Vision Transformer Pruning
• Constructing Deep Neural Networks by Bayesian Network Structure Learning
• A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank
• Towards thinner convolutional neural networks through Gradually Global Pruning
• Learning Instance-wise Sparsity for Accelerating Deep Models
• Resource-Efficient Neural Networks for Embedded Systems
• Full-Stack Filters to Build Minimum Viable CNNs
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Identifying and Exploiting Structures for Reliable Deep Learning
• LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing
• Layer Reduction: Accelerating Conformer-Based Self-Supervised Model via Layer Consistency
• Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer
• Layer Pruning via Fusible Residual Convolutional Block for Deep Neural Networks
• Composite Binary Decomposition Networks
• LatentLLM: Attention-Aware Joint Tensor Compression
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• Searching for Low-Bit Weights in Quantized Neural Networks
• Faster Neural Network Training with Approximate Tensor Operations
• Learning Structured Sparsity in Deep Neural Networks
• Detecting Dead Weights and Units in Neural Networks
• Adaptive Low-Rank Regularization with Damping Sequences to Restrict Lazy Weights in Deep Networks
• Accelerating Deep Unsupervised Domain Adaptation with Transfer Channel Pruning
• Dynamic Group Convolution for Accelerating Convolutional Neural Networks
• A practical convolutional neural network as loop filter for intra frame
• Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• Quantized Neural Networks via -1, +1 Encoding Decomposition and Acceleration
• Learning Strict Identity Mappings in Deep Residual Networks
• Reducing the Model Order of Deep Neural Networks Using Information Theory
• Shapley Value as Principled Metric for Structured Network Pruning
• Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• Flattened Convolutional Neural Networks for Feedforward Acceleration
• Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression
• OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networks
• Training Binary Neural Networks through Learning with Noisy Supervision
• ApproxNet: Content and Contention-Aware Video Analytics System for Embedded Clients
• AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference
• Data-Independent Structured Pruning of Neural Networks via Coresets
• Few Sample Knowledge Distillation for Efficient Network Compression
• Deep learning in bioinformatics: introduction, application, and perspective in big data era
• Virtual Experience to Real World Application: Sidewalk Obstacle Avoidance Using Reinforcement Learning for Visually Impaired
• Knowledge Distillation By Sparse Representation Matching
• Collaborative Distillation for Ultra-Resolution Universal Style Transfer
• Data Efficient Stagewise Knowledge Distillation
• Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
• Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
• Convolutional Neural Networks using Logarithmic Data Representation
• Radius Adaptive Convolutional Neural Network
• ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multipliers
• GDRQ: Group-based Distribution Reshaping for Quantization
• CGaP: Continuous Growth and Pruning for Efficient Deep Learning
• An Efficient Batch Solver for the Singular Value Decomposition on GPUs
• ViP: Virtual Pooling for Accelerating CNN-based Image Classification and Object Detection
• Coding for Computation: Efficient Compression of Neural Networks for Reconfigurable Hardware
• BitNet: Bit-Regularized Deep Neural Networks
• TrojanDam: Detection-Free Backdoor Defense in Federated Learning through Proactive Model Robustification utilizing OOD Data
• REPrune: Filter Pruning via Representative Election
• The Effects of Grouped Structural Global Pruning of Vision Transformers on Domain Generalisation

Related

• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Fast Algorithms for Convolutional Neural Networks
• Accelerating Very Deep Convolutional Networks for Classification and Detection
• Rethinking the Inception Architecture for Computer Vision
• Efficient and Accurate Approximations of Nonlinear Convolutional Networks