2019/06/19 by Qing Yang, Yang, Qing, Wei Wen +5
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Algorithm #Artificial intelligence #Artificial neural network #Computation #Computer science #Constraint (computer-aided design) #Deep neural networks #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Inference #Joint probability distribution #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematics #Pruning #Regularization (linguistics) #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1906.07875
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2019/06/19 · arxiv created 2019/09/13 · arxiv updated 2019/09/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
With the rapid scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for improving deployment efficiency. This work aims to advance the compression beyond the weights to neuron activations. We propose the joint regularization technique which simultaneously regulates the distribution of weights and activations. By distinguishing and leveraging the significance difference among neuron responses and connections during learning, the jointly pruned network, namely JPnet, optimizes the sparsity of activations and weights for improving execution efficiency. The derived deep sparsification of JPnet reveals more optimization space for the existing DNN accelerators dedicated for sparse matrix operations. We thoroughly evaluate the effectiveness of joint regularization through various network models with different activation functions and on different datasets. With 0.4% degradation constraint on inference accuracy, a JPnet can save 72.3% ∼ 98.8% of computation cost compared to the original dense models, with up to 5.2× and 12.3× reductions in activation and weight numbers, respectively.