LLM Inference Serving: Survey of Recent Advances and Opportunities

2024/07/17 by Baolin Li, Yankai Jiang, Li, Baolin +5 · 9 citations
Decision Sciences · #Artificial Intelligence (cs.AI) #Data Quality and Management #Distributed #FOS: Computer and information sciences #Parallel #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2407.12391

openalex publication_date 2024/07/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This survey offers a comprehensive overview of recent advancements in Large Language Model (LLM) serving systems, focusing on research since the year 2023. We specifically examine system-level enhancements that improve performance and efficiency without altering the core LLM decoding mechanisms. By selecting and reviewing high-quality papers from prestigious ML and system venues, we highlight key innovations and practical considerations for deploying and scaling LLMs in real-world production environments. This survey serves as a valuable resource for LLM practitioners seeking to stay abreast of the latest developments in this rapidly evolving field.

Cited by

• MatKV: Trading Compute for Flash Storage in LLM Inference
• Joint Partitioning and Placement of Foundation Models for Real-Time Edge AI
• Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles
• Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks
• Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving
• Reasoning Language Model Inference Serving Unveiled: An Empirical Study
• Large Reasoning Models Learn Better Alignment from Flawed Thinking
• Prompt-Aware Scheduling for Low-Latency LLM Serving
• Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs

Related

• LLM Inference Unveiled: Survey and Roofline Model Insights
• Challenges and Applications of Large Language Models
• Reasoning Language Model Inference Serving Unveiled: An Empirical Study
• Understanding LLMs: A Comprehensive Overview from Training to Inference
• Can abstract concepts from LLM improve SLM performance?