BitSkip: An Empirical Analysis of Quantization and Early Exit Composition

2025/10/27 by Bhuvaneswaran, Ramshankar, Liu, Handan
#68T05 #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.6 #I.2.7

paper · doi:10.48550/arxiv.2510.23766

Abstract

The pursuit of efficient Large Language Models (LLMs) has led to increasingly complex techniques like extreme quantization and dynamic routing. While individual benefits of these methods are well-documented, their compositional effects remain poorly understood. This paper introduces BitSkip, a hybrid architectural framework for systematically exploring these interactions. Counter-intuitively, our findings reveal that a simple 8-bit quantized model without Hadamard transform (BitSkip-V1) not only outperforms its more complex 4-bit and Hadamard-enhanced counterparts but also competes the full-precision baseline in quality (perplexity of 1.13 vs 1.19) . The introduction of Hadamard transforms, even at 8-bit precision, catastrophically degraded performance by over 37,000%, tracing fundamental training instability. Our BitSkip-V1 recipe demonstrates superior early-exit characteristics, with layer 18 providing optimal 32.5% speed gain for minimal 4% quality loss.

Citations

• BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs
• BitNet b1.58 2B4T Technical Report
• DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding
• The Llama 3 Herd of Models
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• Efficient Large Language Models: A Survey
• BitNet: Scaling 1-bit Transformers for Large Language Models
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
• Fast Inference from Transformers via Speculative Decoding
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Confident Adaptive Language Modeling
• Primer: Searching for Efficient Transformers for Language Modeling
• A White Paper on Neural Network Quantization
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
• Depth-Adaptive Transformer
• Quantization and Training of Neural Networks for Efficient\n Integer-Arithmetic-Only Inference
• BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks
• Pointer Sentinel Mixture Models
• Adaptive Computation Time for Recurrent Neural Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Related

• Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques
• SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
• Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization
• AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models
• BAQ: Efficient Bit Allocation Quantization for Large Language Models