HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placement
2025/10/23 by Ge, Danying, Gao, Jianhua, Yang, Yixue +1
#Artificial Intelligence (cs.AI) #C.4 #E.4 #FOS: Computer and information sciences #I.2 #Machine Learning (cs.LG)
paper · doi:10.48550/arxiv.2510.20878
Abstract
Retrieval-Augmented Generation (RAG) improves model output accuracy by leveraging external knowledge bases, serving as an effective solution to address hallucination issues and knowledge-update delays in Large Language Models (LLMs). However, the introduction of external knowledge bases presents RAG with challenges in long-context processing, significantly increasing memory consumption and inference latency. Existing research accelerates inference by precomputing Key and Value (KV) of the knowledge base and loading them on-demand during inference. Based on the access frequency of different KV chunks within the external knowledge base, this paper proposes a hotness-aware RAG (HA-RAG) inference optimization system. First, leveraging the numerical distribution of KV chunks, we introduce a hotness-aware mixed-precision compressing and loading method to reduce disk I/O and memory access overhead. Second, we design a hotness-aware data placement strategy that prioritizes storing frequently accessed KV chunks in high-speed memory to improve data access efficiency. Experimental results demonstrate that, compared with TurboRAG, the proposed HA-RAG achieves an average speedup of 2.10x and maximum speedup of 10.49x in Time-To-First-Token (TTFT) with negligible accuracy loss.
Citations
- Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps
- Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models
- RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU
- Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs
- Efficient Distributed Retrieval-Augmented Generation for Enhancing Language Model Performance
- HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse
- KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
- Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
- TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval
- KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
- HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Tasks
- Does RAG Really Perform Bad For Long-Context Processing?
- APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding
- Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation
- Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads
- Improving Factuality with Explicit Working Memory
- Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
- Precision-Aware Iterative Algorithms Based on Group-Shared Exponents of Floating-Point Numbers
- EPIC: Efficient Position-Independent Caching for Serving Large Language Models
- TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text
- LightRAG: Simple and Fast Retrieval-Augmented Generation
- Fast State Restoration in LLM Serving with HCache
- Compute Or Load KV Cache? Why Not Both?
- SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation
- Block-Attention for Efficient Prefilling
- CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
- Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection
- A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
- FlashBack:Efficient Retrieval-Augmented Language Modeling for Long Context Inference
- XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
- RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
- KMMLU: Measuring Massive Multitask Language Understanding in Korean
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
- Prompt Cache: Modular Attention Reuse for Low-Latency Inference
- CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
- Do Large Language Models Know about Facts?
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
- Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
- Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
- Hallucinations in Large Multilingual Translation Models
- Natural Language Reasoning, A Survey
- A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
- Benchmarking Large Language Models for News Summarization
- Leveraging Large Language Models for Multiple Choice Question Answering
- FP8 Formats for Deep Learning
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Survey of Hallucination in Natural Language Generation
- BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
- Unsupervised Dense Information Retrieval with Contrastive Learning
- The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey
- Conv-Linformer: Boosting Linformer's Performance with Convolution in Small-Scale Settings
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
- IC-Cache: Efficient Large Language Model Serving via In-context Caching
- METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation
Related