TernaryBERT: Distillation-aware Ultra-low Bit BERT

2020/09/27 by Wei Zhang, Zhang, Wei, Lu Hou +10 · 5 citations
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Sound (cs.SD) #Topic Modeling #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2009.12812

openalex publication_date 2020/09/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks.However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices. In this work, we propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Specifically, we use both approximation-based and loss-aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT. Moreover, to reduce the accuracy degradation caused by the lower capacity of low bits, we leverage the knowledge distillation technique in the training process. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods, and even achieves comparable performance as the full-precision model while being 14.9x smaller.

Citations

• Ternary Weight Networks
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• LLM-Supported Manufacturing Mapping Generation
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

Cited by

• FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference
• TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge
• PT²-LLM: Post-Training Ternarization for Large Language Models
• Otters: An Energy-Efficient SpikingTransformer via Optical Time-to-First-Spike Encoding
• TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge

Related

• BinaryBERT: Pushing the Limit of BERT Quantization
• TinyBERT: Distilling BERT for Natural Language Understanding
• MKQ-BERT: Quantized BERT with 4-bits Weights and Activations
• Q8BERT: Quantized 8Bit BERT
• TernaryLLM: Ternarized Large Language Model