BitNet b1.58 2B4T Technical Report

2025/04/16 by Shuming Ma, Ma, Shuming, Hongyu Wang +14 · 10 voices · 13 citations
Computer Science · #Advanced Neural Network Applications #Big Data and Digital Economy #Natural Language Processing Techniques #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2504.12285

openalex publication_date 2025/04/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce BitNet b1.58 2B4T, the first open-source, native 1-bit Large Language Model (LLM) at the 2-billion parameter scale. Trained on a corpus of 4 trillion tokens, the model has been rigorously evaluated across benchmarks covering language understanding, mathematical reasoning, coding proficiency, and conversational ability. Our results demonstrate that BitNet b1.58 2B4T achieves performance on par with leading open-weight, full-precision LLMs of similar size, while offering significant advantages in computational efficiency, including substantially reduced memory footprint, energy consumption, and decoding latency. To facilitate further research and adoption, the model weights are released via Hugging Face along with open-source inference implementations for both GPU and CPU architectures.

Citations

• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• BitNet: Scaling 1-bit Transformers for Large Language Models
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
• Attention Is All You Need

Cited by

• BitNet Text Embeddings
• Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices
• Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in \± 1, ± i\
• Tekum: Balanced Ternary Tapered Precision Real Arithmetic
• BitSkip: An Empirical Analysis of Quantization and Early Exit Composition
• Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions
• BitNet Distillation
• DP-Adam-AC: Privacy-preserving Fine-Tuning of Localizable Language Models Using Adam Optimization with Adaptive Clipping
• Tequila: Trapping-free Ternary Quantization for Large Language Models
• PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
• TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
• Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs
• iFairy: the First 2-bit Complex LLM with All Parameters in \±1, ± i\
• 1.58-bit large language model [wikipedia]

Discussions

• BitNet b1.58 2B4T Technical Report [hn, 111 points, 30 comments]
• MSR introduces BitNet b1.58 2B4T, a 1-bit LLM with 2 billion parameters, optimizing performance and efficiency for AI applications at the pareto frontier arxiv.org/abs/2504.12285 [bsky, 2 points, 0 comments]
• https://bsky.app/profile/hackernews.com.web.brid.gy/post/3lmyw4wfhmfj2 [bsky, 0 points, 0 comments]
• BitNet b1.58 2B4T Technical Report [bsky, 0 points, 0 comments]
• BitNet b1.58 2B4T Technical Report https://arxiv.org/abs/2504.12285 (https://news.ycombinator.com/item?id=43714004) [bsky, 0 points, 0 comments]
• BitNet b1.58 2B4T Technical Report https://arxiv.org/abs/2504.12285 [bsky, 0 points, 0 comments]
• I remember Bitnet from the 80s. Seems it's making a comeback. arxiv.org/abs/2504.12285 en.wikipedia.org/wiki/BITNET [bsky, 0 points, 0 comments]
• https://arxiv.org/abs/2504.12285 BitNet b1.58 2B4Tに関する技術レポートです。 詳細はリンク先のWebコンテンツをご覧ください。 AIに関する技術的な情報がまとめられています。 [bsky, 0 points, 0 comments]
• BitNet b1.58 2B4T Technical Report https://arxiv.org/abs/2504.12285 [bsky, 0 points, 0 comments]
• BitNet b1.58 2B4T Technical Report https://arxiv.org/abs/2504.12285 (https://news.ycombinator.com/item?id=43714004) [bsky, 0 points, 0 comments]

Related

• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• BitNet a4.8: 4-bit Activations for 1-bit LLMs
• 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
• GEB-1.3B: Open Lightweight Large Language Model
• BitNet: Scaling 1-bit Transformers for Large Language Models