Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads

2024/01/20 by Cunchen Hu, Hu, Cunchen, Heyang Huang +21 · 31 citations
Computer Science · Engineering · #Cloud Computing and Resource Management #Distributed #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Parallel #Topic Modeling #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2401.11181

openalex publication_date 2024/01/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computationsaturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.

Cited by

• RollArt: Disaggregated Multi-Task Agentic RL Training at Scale
• PTStore (Prefix Tensor Store): Distributed Prefix Caching and Replication for High Throughput Inference Serving
• Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing
• Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inference
• ProServe: Unified Multi-Priority Request Scheduling for LLM Serving
• RollMux: Phase-Level Multiplexing for Disaggregated RL Post-Training
• GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inference
• LaMoSys3.5D: Enabling 3.5D-IC-Based Large Language Model Inference Serving Systems via Hardware/Software Co-Design
• SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
• DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing
• Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving
• MoEntwine: Unleashing the Potential of Wafer-scale Chips for Large-scale Expert Parallel Inference
• Reasoning Language Model Inference Serving Unveiled: An Empirical Study
• xLLM Technical Report
• FairBatching: Fairness-Aware Batch Formation for LLM Inference
• Adaptive Rescheduling in Prefill-Decode Disaggregated LLM Inference
• BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure
• FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters
• SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV
• Prompt-Aware Scheduling for Low-Latency LLM Serving
• SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
• Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs
• ExpertWeave: Efficiently Serving Expert-Specialized Fine-Tuned Adapters at Scale
• Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism
• VoltanaLLM: Feedback-Driven Frequency Control and State-Space Routing for Energy-Efficient LLM Serving
• PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference
• HyperFlexis: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling
• Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference
• P/D-Device: Disaggregated Large Language Model between Cloud and Devices
• Block: Balancing Load in LLM Serving with Context, Knowledge and Predictive Scheduling
• xDeepServe: Model-as-a-Service on Huawei CloudMatrix384

Related

• PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference
• From Attention to Disaggregation: Tracing the Evolution of LLM Inference
• Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving
• InferCept: Efficient Intercept Support for Augmented Large Language Model Inference
• PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications