2023/11/08 by Haim Barad, Barad, Haim, Ekaterina Aidova +3
Computer Science · #Artificial Intelligence (cs.AI) #Caching and Content Delivery #FOS: Computer and information sciences #Machine Learning (cs.LG) #Performance (cs.PF) #Recommender Systems and Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2311.04951
openalex publication_date 2023/11/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Inference optimizations are critical for improving user experience and reducing infrastructure costs and power consumption. In this article, we illustrate a form of dynamic execution known as speculative sampling to reduce the overall latency of text generation and compare it with standard autoregressive sampling. This can be used together with model-based optimizations (e.g. quantization) to provide an optimized solution. Both sampling methods make use of KV caching. A Jupyter notebook and some sample executions are provided.