Compression Aware Certified Training
2025/06/13 by Xu, Changming, Singh, Gagandeep
#FOS: Computer and information sciences #Machine Learning (cs.LG)
paper · doi:10.48550/arxiv.2506.11992
Abstract
Deep neural networks deployed in safety-critical, resource-constrained environments must balance efficiency and robustness. Existing methods treat compression and certified robustness as separate goals, compromising either efficiency or safety. We propose CACTUS (Compression Aware Certified Training Using network Sets), a general framework for unifying these objectives during training. CACTUS models maintain high certified accuracy even when compressed. We apply CACTUS for both pruning and quantization and show that it effectively trains models which can be efficiently compressed while maintaining high accuracy and certifiable robustness. CACTUS achieves state-of-the-art accuracy and certified performance for both pruning and quantization on a variety of datasets and input specifications.
Citations
- Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness
- A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations
- A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
- Expressive Losses for Verified Robustness via Convex Combinations
- TAPS: Connecting Certified and Adversarial Training
- Exploring the Performance of Pruning Methods in Neural Networks: An Empirical Study of the Lottery Ticket Hypothesis
- SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
- Quantization-aware Interval Bound Propagation for Training Certifiably Robust Quantized Neural Networks
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Certified Training: Small Boxes are All You Need
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Can pruning improve certified robustness of neural networks?
- Compression-aware Training of Neural Networks using Frank-Wolfe
- Fast Certified Robust Training with Short Warmup
- Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers
- Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
- On Adaptive Attacks to Adversarial Example Defenses
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Towards Stable and Efficient Training of Verifiably Robust Neural Networks
- Learned Step Size Quantization
- HAQ: Hardware-Aware Automated Quantization with Mixed Precision
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Quantization and Training of Neural Networks for Efficient\n Integer-Arithmetic-Only Inference
- Towards Deep Learning Models Resistant to Adversarial Attacks
- End to End Learning for Self-Driving Cars
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural\n Networks
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Learning both Weights and Connections for Efficient Neural Networks
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]
- Machine learning for medical diagnosis: history, state of the art and perspective
Related