Soft Weight-Sharing for Neural Network Compression

2017/02/13 by Ullrich, Karen, Meeds, Edward, Welling, Max · 7 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1702.04008

Abstract

The success of deep learning in numerous application domains created the de- sire to run and train them on mobile devices. This however, conflicts with their computationally, memory and energy intense nature, leading to a growing interest in compression. Recent work by Han et al. (2015a) propose a pipeline that involves retraining, pruning and quantization of neural network weights, obtaining state-of-the-art compression rates. In this paper, we show that competitive compression rates can be achieved by using a version of soft weight-sharing (Nowlan & Hinton, 1992). Our method achieves both quantization and pruning in one simple (re-)training procedure. This point of view also exposes the relation between compression and the minimum description length (MDL) principle.

Cited by

• FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivative
• Theoretical Compression Bounds for Wide Multilayer Perceptrons
• CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformer
• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples
• SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
• Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformers
• Deep Lookup Network

Related

• Improved Bayesian Compression
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
• Balanced and Deterministic Weight-sharing Helps Network Performance
• A Novel Structure-Agnostic Multi-Objective Approach for Weight-Sharing Compression in Deep Neural Networks
• Entropy-Constrained Training of Deep Neural Networks