BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs

2025/04/25 by Hongyu Wang, Shuming Ma, Wang, Hongyu +3 · 3 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2504.18415

openalex publication_date 2025/04/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which complicate quantization to low bit-widths. We introduce BitNet v2, a novel framework enabling native 4-bit activation quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward network activations, we propose H-BitLinear, a module applying an online Hadamard transformation prior to activation quantization. This transformation smooths sharp activation distributions into more Gaussian-like forms, suitable for low-bit representation. Experiments show BitNet v2 trained from scratch with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2 achieves minimal performance degradation when trained with native 4-bit activations, significantly reducing memory footprint and computational cost for batched inference.

Cited by

• Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in \± 1, ± i\
• BitSkip: An Empirical Analysis of Quantization and Early Exit Composition
• TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
• iFairy: the First 2-bit Complex LLM with All Parameters in \±1, ± i\

Related

• BitNet a4.8: 4-bit Activations for 1-bit LLMs
• Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMs
• Achieving binary weight and activation for LLMs using Post-Training Quantization
• BitNet b1.58 2B4T Technical Report
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits