Predictable LLM Serving on GPU Clusters

2025/08/27 by Erfan Darzi, Darzi, Erfan, Shreeanant Bharadwaj +3
Engineering · #Vehicle License Plate Recognition #Power Systems and Technologies #Power Line Inspection Robots

paper · pdf · doi:10.48550/arxiv.2508.20274

Abstract

Latency-sensitive inference on shared A100 clusters often suffers noisy-neighbor interference on the PCIe fabric, inflating tail latency and SLO violations. We present a fabric-agnostic, VM-deployable host-level controller that combines dynamic Multi-Instance GPU (MIG) reconfiguration, PCIe-aware placement, and lightweight guardrails (MPS quotas, cgroup I/O). It samples per-tenant tails and system signals, uses topology hints to avoid PCIe hot spots, and gates actions with dwell/cool-down to avoid thrash. On a single host and a 2-node (16-GPU) cluster, SLO miss-rate is reduced by \(≈\)32% (\(≈\)1.5) and p99 latency improves \(≈\)15% with \(≤\)5% throughput cost versus static MIG and naive placement; ablations show MIG and placement contribute comparably. We also evaluate LLM serving with vLLM on OLMo 2 7B Instruct: TTFT p99 improves \(≈\)10--15% at \(≤\)5% cost without changing the controller.

Citations

• Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfiguration
• Themis: Fair and Efficient GPU Cluster Scheduling
• Clipper: A Low-Latency Online Prediction Serving System

Related

• PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling
• GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving
• WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
• Cronus: Efficient LLM inference on Heterogeneous GPU Clusters via Partially Disaggregated Prefill
• DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing