When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon

2026/05/07 by Mohamed Amine Bergach · 1 voice
Computer Science · #cs.AI #cs.PF

paper · pdf · doi:10.48550/arxiv.2605.05699

Abstract

KV-cache quantization is framed as a quality--latency trade-off. We show it is inverted on Apple Silicon's unified memory: a single fused Metal kernel (sign-randomized FFT + per-channel λ + per-group abs-max + int4 nibble pack), exposed as a HuggingFace Cache subclass, runs faster than fp16 across 256--4096-token prefixes on Gemma-3 1B (-3 to -8% ms/tok) and at short context on Qwen2.5-1.5B (-0.7 to -2.6% through 1K), with 3× persistent memory compression and quality preserved (\dPPL = 0.000 Qwen short-prompt; +3.6 hook \dPPL Gemma). The kernel's ∼ 25 ns/vec overhead is below the bandwidth savings from 3× compression. The fused kernel also closes Qwen's 4-bit per-token catastrophe (\dPPL = +7975 → +638.6, 12.5× reduction) at 182 GFLOPS / D=128. Supporting findings: \SRFT and \SRHT are statistically indistinguishable for KV quality (we pick \SRFT for mixed-radix and matrix-multiply alignment); a learned-rotation ablation surfaces a regularization role for the fixed random SRFT base (learning R+λ without SRFT lowers calibration MSE 84.9% vs 50.3% but yields worse PPL); Householder rotations at k=d/2 reflectors are effectively lossless at d=256.

Citations

• DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
• ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transforms
• Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache
• FPTQuant: Function-Preserving Transforms for LLM Quantization
• Formal Verification of TurboQuant: Machine-Checked Proofs and Gap Closures
• KurTail : Kurtosis-based LLM Quantization
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• The Llama 3 Herd of Models
• Qwen2 Technical Report
• SpinQuant: LLM quantization with learned rotations
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• KIVI : Plug-and-play 2bit KV Cache Quantization with Streaming Asymmetric Quantization
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions
• The Fast Johnson–Lindenstrauss Transform and Approximate Nearest Neighbors
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• A fast randomized algorithm for the approximation of matrices

Discussions

• Mohamed Amine Bergach: When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon https://arxiv.org/abs/2605.05699 https://arxiv.org/pdf/2605.05699 https://arxiv.org/html/2605.0569 [bsky, 0 points, 0 comments]

Related

• A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs
• SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
• Smarter and Cheaper at Once: Byte-Exact KV-Cache Grafting Turns a Frozen Small Model into a Verified-Knowledge Flywheel