Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow

2024/06/03 by Yixuan Mei, Mei, Yixuan, Yonghao Zhuang +9 · 30 citations
Computer Science · #Computation and Language (cs.CL) #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #Topic Modeling #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2406.01566

openalex publication_date 2024/06/03 · openalex created_date 2024/06/07 · openalex updated_date 2026/07/28

Abstract

This paper introduces Helix, a distributed system for high-throughput, low-latency large language model (LLM) serving in heterogeneous GPU clusters. The key idea behind Helix is to formulate inference computation of LLMs over heterogeneous GPUs and network connections as a max-flow problem on directed, weighted graphs, whose nodes represent GPU instances and edges capture both GPU and network heterogeneity through their capacities. Helix then uses a mixed integer linear programming (MILP) algorithm to discover highly optimized strategies to serve LLMs on heterogeneous GPUs. This approach allows Helix to jointly optimize model placement and request scheduling, two highly entangled tasks in heterogeneous LLM serving. Our evaluation on several heterogeneous clusters ranging from 24 to 42 GPU nodes shows that Helix improves serving throughput by up to 3.3x and reduces prompting and decoding latency by up to 66% and 24%, respectively, compared to existing approaches. Helix is available at https://github.com/Thesys-lab/Helix-ASPLOS25.

Cited by

• HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization
• UCCL-EP: Portable Expert-Parallel Communication
• Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Serving
• Janus: Disaggregating Attention and Experts for Scalable MoE Inference
• HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments
• GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inference
• AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs
• ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform
• GOGH: Correlation-Guided Orchestration of GPUs in Heterogeneous Clusters
• TetriServe: Efficiently Serving Mixed DiT Workloads
• Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Framework
• Parallax: Efficient LLM Inference Service over Decentralized Environment
• Agentic Services Computing
• SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
• ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
• Hardwired-Neurons Language Processing Units as General-Purpose Cognitive Substrates
• Simulating LLM training workloads for heterogeneous compute and network infrastructure
• VLH: Vision-Language-Haptics Foundation Model
• Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving
• MoLink: Distributed and Efficient Serving Framework for Large Models
• TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Network
• RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference
• L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models
• HEXGEN-FLOW: Optimizing LLM Inference Request Scheduling for Agentic Text-to-SQL
• Patchwork: A Unified Framework for RAG Serving
• Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation
• NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
• Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI
• Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters
• SLOs-Serve: Optimized Serving of Multi-SLO LLMs

Related

• MoLink: Distributed and Efficient Serving Framework for Large Models
• HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment
• NanoFlow: Towards Optimal Large Language Model Serving Throughput
• Parallax: Efficient LLM Inference Service over Decentralized Environment
• Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment