Towards Resource-Efficient Serverless LLM Inference with SLINFER

2025/07/01 by Chao‐Nan Xu, Xu, Chuhao, Zijun Li +7 · 1 citation
Computer Science · #Advanced Data Storage Technologies #Cloud Computing and Resource Management #Distributed #FOS: Computer and information sciences #Parallel #Security and Verification in Computing #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2507.00507

openalex publication_date 2025/07/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

The rise of LLMs has driven demand for private serverless deployments, characterized by moderate-sized models and infrequent requests. While existing serverless solutions follow exclusive GPU allocation, we take a step back to explore modern platforms and find that: Emerging CPU architectures with built-in accelerators are capable of serving LLMs but remain underutilized, and both CPUs and GPUs can accommodate multiple LLMs simultaneously. We propose SLINFER, a resource-efficient serverless inference scheme tailored for small- to mid-sized LLMs that enables elastic and on-demand sharing across heterogeneous hardware. SLINFER tackles three fundamental challenges: (1) precise, fine-grained compute resource allocation at token-level to handle fluctuating computational demands; (2) a coordinated and forward-looking memory scaling mechanism to detect out-of-memory hazards and reduce operational overhead; and (3) a dual approach that consolidates fragmented instances through proactive preemption and reactive bin-packing. Experimental results on 4 32-core CPUs and 4 A100 GPUs show that SLINFER improves serving capacity by 47% - 62% through sharing, while further leveraging CPUs boosts this to 86% - 154%.

Citations

• Dilu: Enabling GPU Resourcing-on-Demand for Serverless DL Serving via Introspective Elasticity
• DeepServe: Serverless Large Language Model Serving at Scale
• Pie: Pooling CPU Memory for LLM Inference
• NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
• Gemma 2: Improving Open Language Models at a Practical Size
• Llumnix: Dynamic Scheduling for Large Language Model Serving
• vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
• LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
• MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving
• FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines
• Gemma: Open Models Based on Gemini Research and Technology
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
• ServerlessLLM: Low-Latency Serverless Inference for Large Language Models
• CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
• Splitwise: Efficient generative LLM inference using phase splitting
• SpotServe: Serving Generative Large Language Models on Preemptible Instances
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• In-Storage Domain-Specific Acceleration for Serverless Computing
• Evaluating Large Language Models Trained on Code
• Llama: A Heterogeneous & Serverless Framework for Auto-Tuning Video Analytics Pipelines
• TurboTransformers: An Efficient GPU Serving System For Transformer Models
• Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads
• Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
• Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
• PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
• Clipper: A Low-Latency Online Prediction Serving System
• HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
• AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

Cited by

• TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity

Related

• ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs
• PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling
• HAS-GPU: Efficient Hybrid Auto-scaling with Fine-grained GPU Allocation for SLO-aware Serverless Inferences
• Scaling Up Throughput-oriented LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management
• LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference