Differentiable Product Quantization for End-to-End Embedding Compression

2019/08/26 by Ting Chen, Chen, Ting, Lala Li +3 · 5 citations
Computer Science · Mathematics · #Advanced Data Compression Techniques #Algorithms and Data Compression #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.AI #cs.CL #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1908.09756

ICML'2020. Code at https://github.com/chentingpc/dpq_embedding_compression

openalex publication_date 2019/08/26 · arxiv created 2020/06/25 · arxiv updated 2020/06/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Embedding layers are commonly used to map discrete symbols into continuous embedding vectors that reflect their semantic meanings. Despite their effectiveness, the number of parameters in an embedding layer increases linearly with the number of symbols and poses a critical challenge on memory and storage constraints. In this work, we propose a generic and end-to-end learnable compression framework termed differentiable product quantization (DPQ). We present two instantiations of DPQ that leverage different approximation techniques to enable differentiability in end-to-end learning. Our method can readily serve as a drop-in alternative for any existing embedding layer. Empirically, DPQ offers significant compression ratios (14-238×) at negligible or no performance cost on 10 datasets across three different language tasks.

Citations

• Recurrent Neural Network Regularization
• Matrix Factorization Techniques for Recommender Systems
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Categorical Reparameterization with Gumbel-Softmax
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Neural Discrete Representation Learning

Cited by

• A Survey on Deep Text Hashing: Efficient Semantic Text Retrieval with Binary Representation
• Semantic Compression via Multimodal Representation Learning
• OneShot: Index-in-Ranking with Neural Scoring for Large-Scale Retrieval
• Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models
• Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey

Related

• Embedding Compression with Isotropic Iterative Quantization
• Differentiable Model Compression via Pseudo Quantization Noise
• Stacked Quantizers for Compositional Vector Compression
• CARVQ: Corrective Adaptor with Group Residual Vector Quantization for LLM Embedding Compression
• QuantEase: Optimization-based Quantization for Language Models