BinaryBERT: Pushing the Limit of BERT Quantization

2020/12/31 by Haoli Bai, Bai, Haoli, Wei Zhang +14 · 9 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2012.15701

openalex publication_date 2020/12/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24x smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks.

Citations

• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Ternary Weight Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• DARTS: Differentiable Architecture Search
• ProxQuant: Quantized Neural Networks via Proximal Operators
• LLM-Supported Manufacturing Mapping Generation
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Exploiting Generative AI to Scale up Intelligent Tutoring Systems

Cited by

• SingleQuant: Efficient Quantization of Large Language Models in a Single Pass
• T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganization
• FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference
• Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
• NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
• EGGS-PTP: An Expander-Graph Guided Structured Post-training Pruning Method for Large Language Models
• Task complexity shapes internal representations and robustness in neural networks
• Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
• A Lower Bound for the Number of Linear Regions of Ternary ReLU Regression Neural Networks

Related

• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• Q8BERT: Quantized 8Bit BERT
• MKQ-BERT: Quantized BERT with 4-bits Weights and Activations
• NeoBERT: A Next-Generation BERT
• FP8-BERT: Post-Training Quantization for Transformer