Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models

2025/08/09 by Zhijun Tu, Tu, Zhijun, Chen, Hanting +11
Computer Science · #Advanced Neural Network Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2508.06974

openalex publication_date 2025/08/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

1-bit LLM quantization offers significant advantages in reducing storage and computational costs. However, existing methods typically train 1-bit LLMs from scratch, failing to fully leverage pre-trained models. This results in high training costs and notable accuracy degradation. We identify that the large gap between full precision and 1-bit representations makes naive adaptation difficult. In this paper, we introduce a consistent progressive training for both forward and backward, smoothly converting the full-precision weights into the binarized ones. Additionally, we incorporate binary-aware initialization and dual-scaling compensation to reduce the difficulty of progressive training and improve the performance. Experimental results on LLMs of various sizes demonstrate that our method outperforms existing approaches. Our results show that high-performance 1-bit LLMs can be achieved using pre-trained models, eliminating the need for expensive training from scratch.

Related

• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
• Rethinking Output Alignment For 1-bit Post-Training Quantization of Large Language Models
• OneBit: Towards Extremely Low-bit Large Language Models
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models