Tensorized Embedding Layers for Efficient Model Compression

2019/01/30 by Oleksii Hrinchuk, Valentin Khrulkov, Hrinchuk, Oleksii +6 · 3 citations
Computer Science · Mathematics · #Advanced Data Compression Techniques #Algorithms and Data Compression #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Tensor decomposition and applications

paper · pdf · doi:10.48550/arxiv.1901.10787

openalex publication_date 2019/01/30 · openalex created_date 2019/03/11 · openalex updated_date 2026/07/28

Abstract

The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Tensorizing Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Analysis of Individual Differences in Multidimensional Scaling Via an N-way Generalization of “Eckart-Young” Decomposition
• Long Short-Term Memory
• Tensor Ring Decomposition
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Pointer Sentinel Mixture Models
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Tensor-Train Recurrent Neural Networks for Video Classification
• Learning K-way D-dimensional Discrete Code For Compact Embedding Representations
• Wide Compression: Tensor Ring Nets
• Adaptive Input Representations for Neural Language Modeling
• A Tensorized Transformer for Language Modeling
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Cited by

• Adversarial Learning-Based Radio Map Reconstruction for Fingerprinting Localization
• On the Downstream Performance of Compressed Word Embeddings
• Weight Squeezing: Reparameterization for Knowledge Transfer and Model\n Compression

Related

• TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition
• Partial Tensorized Transformers for Natural Language Processing
• TQCompressor: improving tensor decomposition methods in neural networks via permutations
• Tensorizing Neural Networks
• Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models