Pyramid Vector Quantization for Deep Learning

2017/04/10 by Vincenzo Liguori, Liguori, Vincenzo
Computer Science · Engineering · #FOS: Computer and information sciences #Image Processing Techniques and Applications #Machine Learning (cs.LG) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Parallel Computing and Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1704.02681

openalex publication_date 2017/04/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper explores the use of Pyramid Vector Quantization (PVQ) to reduce the computational cost for a variety of neural networks (NNs) while, at the same time, compressing the weights that describe them. This is based on the fact that the dot product between an N dimensional vector of real numbers and an N dimensional PVQ vector can be calculated with only additions and subtractions and one multiplication. This is advantageous since tensor products, commonly used in NNs, can be re-conduced to a dot product or a set of dot products. Finally, it is stressed that any NN architecture that is based on an operation that can be re-conduced to a dot product can benefit from the techniques described here.

Citations

• Gradient-based learning applied to document recognition
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Related

• Pyramid Vector Quantization and Bit Level Sparsity in Weights for Efficient Neural Networks Inference
• Vector Quantization for Machine Vision
• Neural Networks Weights Quantization: Target None-retraining Ternary (TNT)
• DBQ: A Differentiable Branch Quantizer for Lightweight Deep Neural Networks
• Pyramid Vector Quantization for LLMs