2026/01/01 by Chao Fang, Jun Yin, Man Shi +1
Computer Science · #Acceleration #Cache #Decoding methods #Hardware acceleration #Integrated circuit #Integrated circuit design #Network Packet Processing and Optimization #Parallel Computing and Optimization Techniques #Security and Verification in Computing #cs.AI #cs.AR #cs.LG
paper · pdf · doi:10.1109/tcsi.2026.3716602
published in IEEE Transactions on Circuits and Systems I Regular Papers, 1-14 (Institute of Electrical and Electronics Engineers) · To appear in the IEEE Transactions on Circuits and Systems I: Regular Papers (TCAS-I)
openalex publication_date 2026/01/01 · arxiv created 2026/07/24 · openalex created_date 2026/08/01 · arxiv updated 2026/08/04 · openalex updated_date 2026/08/06
With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck. To tackle this challenge, we propose HiKV, a novel algorithm-hardware co-design that exploits KV cache redundancy through hierarchical importance awareness. Algorithmically, HiKV compresses the KV cache at two granularities: Stage I evicts unimportant tokens within a fixed budget, and Stage II further loads only the significant elements of each retained token, reaching compression ratios unattainable at a single granularity. Architecturally, we develop a dedicated accelerator centered on a reconfigurable importance sorter that switches between the distinct sorting datapaths each stage requires, unifying the two-stage acceleration in one circuit with minimal overhead. Evaluated on representative LLMs, HiKV achieves up to 7.95x speedup and 90% energy reduction in the attention computation over the vanilla KV cache baseline within negligible 1% accuracy loss. Under iso-accuracy constraints, HiKV outperforms state-of-the-art importance-based methods by achieving an additional 1.82~4.87x reduction in external memory accesses. These benefits are enabled by specialized hardware components that add only 8% to the system area.