Symmetry Regularization and Saturating Nonlinearity for Robust Quantization

2022/07/31 by Sein Park, Park, Sein, Yeongsang Jang +3 · 1 citation
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Image Enhancement Techniques #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2208.00338

openalex publication_date 2022/07/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Robust quantization improves the tolerance of networks for various implementations, allowing reliable output in different bit-widths or fragmented low-precision arithmetic. In this work, we perform extensive analyses to identify the sources of quantization error and present three insights to robustify a network against quantization: reduction of error propagation, range clamping for error minimization, and inherited robustness against quantization. Based on these insights, we propose two novel methods called symmetry regularization (SymReg) and saturating nonlinearity (SatNL). Applying the proposed methods during training can enhance the robustness of arbitrary neural networks against quantization on existing post-training quantization (PTQ) and quantization-aware training (QAT) algorithms and enables us to obtain a single weight flexible enough to maintain the output quality under various conditions. We conduct extensive studies on CIFAR and ImageNet datasets and validate the effectiveness of the proposed methods.

Cited by

• Assessing the Potential for Catastrophic Failure in Dynamic Post-Training Quantization

Related

• Gradient ℓ₁ Regularization for Quantization Robustness
• Robust Quantization: One Model to Rule Them All
• Oscillations Make Neural Networks Robust to Quantization
• Post-training Quantization for Neural Networks with Provable Guarantees
• QGen: On the Ability to Generalize in Quantization Aware Training