Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

2019/01/28 by Ritchie Zhao, Yuwei Hu, Zhao, Ritchie +7 · 10 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1901.09504

openalex publication_date 2019/01/28 · openalex created_date 2019/03/02 · openalex updated_date 2026/07/28

Abstract

Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS), which duplicates channels containing outliers, then halves the channel values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.

Citations

• Recurrent Neural Network Regularization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Resiliency of Deep Neural Networks under Quantization
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Pointer Sentinel Mixture Models
• Quantizing Convolutional Neural Networks for Low-Power High-Throughput Inference Engines
• Learning Multiple Layers of Features from Tiny Images

Cited by

• HQ-DM: Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models
• AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models
• Optimization of the quantization of dense neural networks from an exact QUBO formulation
• Explaining How Quantization Disparately Skews a Model
• Sensitivity-Aware Post-Training Quantization for Deep Neural Networks
• Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications
• Search-Optimized Quantization in Biomedical Ontology Alignment
• BiVM: Accurate Binarized Neural Network for Efficient Video Matting
• EfficientQuant: An Efficient Post-Training Quantization for CNN-Transformer Hybrid Models on Edge Devices
• FPTQuant: Function-Preserving Transforms for LLM Quantization

Related

• OverQ: Opportunistic Outlier Quantization for Neural Network Accelerators
• SplitQuant: Layer Splitting for Low-Bit Neural Network Quantization
• A White Paper on Neural Network Quantization
• Quantization for Rapid Deployment of Deep Neural Networks
• AUSN: Approximately Uniform Quantization by Adaptively Superimposing Non-uniform Distribution for Deep Neural Networks