SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions

2025/10/10 by Ziyi Wang, Nan Jiang, Wang, Ziyi +5
Computer Science · #Advanced Data Compression Techniques #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2510.08999

openalex publication_date 2025/10/10 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Compressing large-scale neural networks is essential for deploying models on resource-constrained devices. Most existing methods adopt weight pruning or low-bit quantization individually, often resulting in suboptimal compression rates to preserve acceptable performance drops. We introduce a unified framework for simultaneous pruning and low-bit quantization via Bayesian variational learning (SQS), which achieves higher compression rates than prior baselines while maintaining comparable performance. The key idea is to employ a spike-and-slab prior to inducing sparsity and model quantized weights using Gaussian Mixture Models (GMMs) to enable low-bit precision. In theory, we provide the consistent result of our proposed variational approach to a sparse and quantized deep neural network. Extensive experiments on compressing ResNet, BERT-base, Llama3, and Qwen2.5 models show that our method achieves higher compression rates than a line of existing methods with comparable performance drops.

Citations

• Stochastic Weight Sharing for Bayesian Neural Networks
• Low-bit Model Quantization for Deep Neural Networks: A Survey
• Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?
• Optimizing Large Language Model Training Using FP4 Quantization
• Scaling Laws for Precision
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
• Unified Data-Free Compression: Pruning and Quantization without Fine-Tuning
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• QLoRA: Efficient Finetuning of Quantized LLMs
• Stable and low-precision training for large-scale vision-language models
• LLaMA: Open and Efficient Foundation Language Models
• Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
• PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance
• Dual Lottery Ticket Hypothesis
• Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networks
• A Survey of Quantization Methods for Efficient Neural Network Inference
• Consistent Sparse Deep Learning: Theory and Computation
• Efficient Variational Inference for Sparse Deep Learning with\n Theoretical Guarantee
• Differentiable Joint Pruning and Quantization for Hardware Efficiency
• Knowledge Distillation: A Survey
• APQ: Joint Search for Network Architecture, Pruning and Quantization Policy
• Up or Down? Adaptive Rounding for Post-Training Quantization
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• An Adaptive Empirical Bayesian Method for Sparse Deep Learning
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• Convergence Rates of Variational Inference in Sparse Deep Learning
• EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
• Relational Knowledge Distillation
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
• Scalable Methods for 8-bit Training of Neural Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• High-Accuracy Low-Precision Training
• Training and Inference with Integers in Deep Neural Networks
• Quantization and Training of Neural Networks for Efficient\n Integer-Arithmetic-Only Inference
• Learning Sparse Neural Networks through L₀ Regularization
• Progressive Neural Architecture Search
• Learning Discrete Weights Using the Local Reparameterization Trick
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Bayesian Compression for Deep Learning
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• Soft Weight-Sharing for Neural Network Compression
• Trained Ternary Quantization
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Pruning Filters for Efficient ConvNets
• Dynamic Network Surgery for Efficient DNNs
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural\n Networks
• Variational Inference: A Review for Statisticians
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Deep Learning with Limited Numerical Precision
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters
• Variational Bayesian Quantization
• Efficient Model Compression for Bayesian Neural Networks
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Compression with Bayesian Implicit Neural Representations