70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)

2025/04/15 by Tianyi Zhang, Zhang, Tianyi, Mohsen Hariri +11 · 17 voices · 5 citations
Computer Science · #Parallel Computing and Optimization Techniques #Big Data and Digital Economy #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2504.11651

Abstract

Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs), have grown rapidly in size, creating significant challenges for efficient deployment on resource-constrained hardware. In this paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model. DFloat11 is motivated by the low entropy in the BFloat16 weight representation of LLMs, which reveals significant inefficiency in the existing storage format. By applying entropy coding, DFloat11 assigns dynamic-length encodings to weights based on frequency, achieving near information-optimal compression without any loss of precision. To facilitate efficient inference with dynamic-length encodings, we develop a custom GPU kernel for fast online decompression. Our design incorporates the following: (i) compact, hierarchical lookup tables (LUTs) that fit within GPU SRAM for efficient decoding, (ii) a two-phase GPU kernel for coordinating thread read/write positions using lightweight auxiliary variables, and (iii) transformer-block-level decompression to minimize latency. Experiments on Llama 3.3, Qwen 3, Mistral 3, FLUX.1, and others validate our hypothesis that DFloat11 achieves around 30% model size reduction while preserving bit-for-bit identical outputs. Compared to a potential alternative of offloading parts of an uncompressed model to the CPU to meet memory constraints, DFloat11 achieves 2.3--46.2x higher throughput in token generation. With a fixed GPU memory budget, DFloat11 enables 5.7--14.9x longer generation lengths than uncompressed models. Notably, our method enables lossless inference of Llama 3.1 405B, an 810GB model, on a single node equipped with 8x80GB GPUs.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• QLoRA: Efficient Finetuning of Quantized LLMs
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• LoRA: Low-Rank Adaptation of Large Language Models
• Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding

Cited by

• Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
• PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
• Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
• To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentration
• Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications

Discussions

• Lossless LLM compression for efficient GPU inference via dynamic-length float [hn, 411 points, 117 comments]
• >In this paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression framework that reduces LLM size by 30% while preserving outputs that are bit-for-bit identical to the original mode [bsky, 2 points, 1 comments]
• https://arxiv.org/abs/2504.11651 この論文は、大規模言語モデル（LLM）の効率的なGPU推論のための新しい圧縮手法を提案しています。 Dynamic-Length Float（DyLoFloat）という手法を用いて、LLMのサイズを70%削減しつつ、精度を維持します。 これにより、メモリ使用量を削減し、推論速度を向上させることが可能になります。 [bsky, 1 points, 0 comments]
• Lossless LLM compression for efficient GPU inference via dynamic-length float https://arxiv.org/abs/2504.11651 arxiv.org [bsky, 1 points, 0 comments]
• Really cool idea and implementation, model compression via huffman coding, fast enough to be used live ! Side note is that there are probably model arch questions stemming from the initial redundancy [bsky, 1 points, 2 comments]
• 70% Size, 100% Accuracy: Lossless LLM Compression via Dynamic-Length Float https://arxiv.org/abs/2504.11651 (https://news.ycombinator.com/item?id=43796935) [bsky, 0 points, 0 comments]
• "Lossless LLM compression for efficient GPU inference via dynamic-length float" Lossless LLM compression lets huge AI models run on fewer GPUs. This is great for smaller labs and companies, making pow [bsky, 0 points, 0 comments]
• 70% Size, 100% Accuracy: Lossless LLM Compression via Dynamic-Length Float [bsky, 0 points, 0 comments]
• 70% Size, 100% Accuracy: Lossless LLM Compression via Dynamic-Length Float #HackerNews https://arxiv.org/abs/2504.11651 [bsky, 0 points, 0 comments]
• 70% Size, 100% Accuracy: Lossless LLM Compression via Dynamic-Length Float https://arxiv.org/abs/2504.11651 https://news.ycombinator.com/item?id=43796935 [bsky, 0 points, 0 comments]
• Lossless LLM compression for efficient GPU inference via dynamic-length float https://arxiv.org/abs/2504.11651 [bsky, 0 points, 0 comments]
• Lossless LLM compression for efficient GPU inference via dynamic-length float https://arxiv.org/abs/2504.11651 [comments] [193 points] [bsky, 0 points, 0 comments]
• Lossless LLM compression for efficient GPU inference via dynamic-length float https://arxiv.org/abs/2504.11651 (http://news.ycombinator.com/item?id=43796935) [bsky, 0 points, 0 comments]
• Lossless LLM compression for efficient GPU inference via dynamic-length float https://arxiv.org/abs/2504.11651 (http://news.ycombinator.com/item?id=43796935) [bsky, 0 points, 0 comments]
• 70% Size, 100% Accuracy: Lossless LLM Compression via Dynamic-Length Float view on hacker news [bsky, 0 points, 0 comments]
• 70% Size, 100% Accuracy: Lossless LLM Compression via Dynamic-Length Float https://arxiv.org/abs/2504.11651 (https://news.ycombinator.com/item?id=43796935) [bsky, 0 points, 0 comments]
• Lossless LLM compression for efficient GPU inference via dynamic-length float https://arxiv.org/abs/2504.11651 (https://news.ycombinator.com/item?id=43796935) [bsky, 0 points, 0 comments]

Related

• Endor: Hardware-Friendly Sparse Format for Offloaded LLM Inference
• FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs
• Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
• Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching