On Periodic Functions as Regularizers for Quantization of Neural Networks

2018/11/24 by Maxim Naumov, Naumov, Maxim, Utku Diril +9 · 1 citation
Computer Science · Physics and Astronomy · #68T05 #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #I.2.6 #I.5.0 #Image and Signal Denoising Methods #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1811.09862

openalex publication_date 2018/11/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep learning models have been successfully used in computer vision and many other fields. We propose an unorthodox algorithm for performing quantization of the model parameters. In contrast with popular quantization schemes based on thresholds, we use a novel technique based on periodic functions, such as continuous trigonometric sine or cosine as well as non-continuous hat functions. We apply these functions component-wise and add the sum over the model parameters as a regularizer to the model loss during training. The frequency and amplitude hyper-parameters of these functions can be adjusted during training. The regularization pushes the weights into discrete points that can be encoded as integers. We show that using this technique the resulting quantized models exhibit the same accuracy as the original ones on CIFAR-10 and ImageNet datasets.

Citations

• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Gradient-based learning applied to document recognition
• Backpropagation Applied to Handwritten Zip Code Recognition
• Ternary Weight Networks
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Deep Residual Learning for Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding

Cited by

• Training with Multi-Layer Embeddings for Model Reduction

Related

• WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization
• SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep\n Quantized Training
• BiPer: Binary Neural Networks using a Periodic Function
• Foothill: A Quasiconvex Regularization for Edge Computing of Deep Neural Networks
• QGen: On the Ability to Generalize in Quantization Aware Training