Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference

2026/02/04 by Xinyu Wang, Jieyu Li, Yanan Sun +1
#cs.AR

paper · pdf

Abstract

Large Language Models (LLMs) incur substantial memory and computation costs. Prior works reduce FP-INT arithmetic overhead by converting linear-layer activations to block floating point (BFP), but retain FP activations in attention layers due to accuracy concerns. We propose Harmonia, an algorithm-hardware co-design framework that enables BFP representation and computation across both linear and attention layers. Harmonia first explores BFP configurations to balance model accuracy and activation compression. It then combines asymmetric bit allocation with hybrid offline-online outlier smoothing to compress the KV cache from FP16 to 4-bit-mantissa BFP with less than 1% average accuracy loss on LongBench. To efficiently support all-layer BFP computation, Harmonia incorporates a reconfigurable PE unit for mixed data formats and precisions, a real-time FP16-to-BFP converter, and a flexible tiling-aware dataflow that reduces external memory traffic. Evaluations across eight widely used LLMs show that Harmonia achieves 3.84x higher area efficiency, 2.03x better energy efficiency, and 3.08x speedup on average, with maximum improvements of 5.05x, 3.90x, and 4.62x, respectively.

Citations

• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
• BitNet: Scaling 1-bit Transformers for Large Language Models
• Mistral 7B
• Efficient Streaming Language Models with Attention Sinks
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• OPT: Open Pre-trained Transformer Language Models
• Training Compute-Optimal Large Language Models
• Language Models are Few-Shot Learners
• Attention Is All You Need

Related

• Pushing the Limits of BFP on Narrow Precision LLM Inference
• F-BFQ: Flexible Block Floating-Point Quantization Accelerator for LLMs
• AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Design
• NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs
• FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference