Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in \± 1, ± i\

2025/12/02 by Wang, Feiyu, Tan, Xinyu, Yihao Zhang +8
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Big Data and Digital Economy #FOS: Computer and information sciences #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2512.02901

openalex publication_date 2025/12/02 · openalex created_date 2025/12/04 · openalex updated_date 2026/08/03

Abstract

Large language models (LLMs) have revolutionized artificial intelligence, yet their massive memory and computational demands necessitate aggressive quantization, increasingly pushing representations toward the theoretical limit of a single bit. While complex-valued LLMs, such as iFairy, offer a superior chance for low-bit representation compared to real-valued counterparts, they require training from scratch, preventing the utilization of the vast ecosystem of pre-trained real-valued foundation models. Here we present Fairy2i, a universal framework that transforms pre-trained real-valued layers into an equivalent widely-linear complex form, enabling extremely low-bit quantization while reusing existing checkpoints. By proving a lossless mathematical equivalence between real and widely-linear maps, we convert standard Transformers into the complex domain and employ a phase-aware quantization scheme with a highly efficient codebook of fourth roots of unity. Furthermore, we introduce a recursive residual quantization mechanism that iteratively minimizes quantization error, allowing inference to proceed via efficient multiplication-free accumulation. We demonstrate that Fairy2i restores the performance of LLaMA-2 7B at an effective 2-bit precision to levels nearly comparable with full-precision baselines, significantly outperforming state-of-the-art real-valued binary and ternary quantization methods. This work bridges the gap between the representational efficiency of complex-valued arithmetic and the practical utility of pre-trained models, paving a new way for efficient inference on commodity hardware. We open-source the Fairy2i model and code at https://huggingface.co/PKU-DS-LAB/Fairy2i-W2 and https://github.com/PKULab1806/Fairy2i-W2.

Citations

• iFairy: the First 2-bit Complex LLM with All Parameters in \±1, ± i\
• MiniCPM4: Ultra-Efficient LLMs on End Devices
• Qwen3 Technical Report
• BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs
• BitNet b1.58 2B4T Technical Report
• Binary Neural Networks for Large Language Model: A Survey
• Bitnet.cpp: Efficient Edge Inference for Ternary LLMs
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• RedPajama: an Open Dataset for Training Large Language Models
• The Llama 3 Herd of Models
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
• EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
• Low-Rank Quantization-Aware Training for LLMs
• SpinQuant: LLM quantization with learned rotations
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• Extreme Compression of Large Language Models via Additive Quantization
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
• Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
• Efficient Large Language Models: A Survey
• BitNet: Scaling 1-bit Transformers for Large Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
• A Survey on Model Compression for Large Language Models
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• OPT: Open Pre-trained Transformer Language Models
• A Survey of Complex-Valued Neural Networks
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Complex Transformer: A Framework for Modeling Complex-Valued Sequence
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Deep Complex Networks
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Related

• iFairy: the First 2-bit Complex LLM with All Parameters in \±1, ± i\
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
• I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
• SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
• LLM-QAT: Data-Free Quantization Aware Training for Large Language Models