HexGen: Generative Inference of Large Language Model over Heterogeneous Environment

2023/11/20 by Youhe Jiang, Jiang, Youhe, Ran Yan +8 · 12 citations
Computer Science · Decision Sciences · Engineering · #Distributed #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #Modular Robots and Swarm Intelligence #Parallel #Simulation Techniques and Applications #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2311.11514

openalex publication_date 2023/11/20 · openalex created_date 2023/11/23 · openalex updated_date 2026/07/28

Abstract

Serving generative inference of the large language model is a crucial component of contemporary AI applications. This paper focuses on deploying such services in a heterogeneous and cross-datacenter setting to mitigate the substantial inference costs typically associated with a single centralized datacenter. Towards this end, we propose HexGen, a flexible distributed inference engine that uniquely supports the asymmetric partition of generative inference computations over both tensor model parallelism and pipeline parallelism and allows for effective deployment across diverse GPUs interconnected by a fully heterogeneous network. We further propose a sophisticated scheduling algorithm grounded in constrained optimization that can adaptively assign asymmetric inference computation across the GPUs to fulfill inference requests while maintaining acceptable latency levels. We conduct an extensive evaluation to verify the efficiency of HexGen by serving the state-of-the-art Llama-2 (70B) model. The results suggest that HexGen can choose to achieve up to 2.3 times lower latency deadlines or tolerate up to 4 times more request rates compared with the homogeneous baseline given the same budget.

Cited by

• HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments
• AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs
• BeLLMan: Controlling LLM Congestion
• Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Framework
• Parallax: Efficient LLM Inference Service over Decentralized Environment
• Cronus: Efficient LLM inference on Heterogeneous GPU Clusters via Partially Disaggregated Prefill
• ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
• Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism
• Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference
• LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind
• FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel
• MoLink: Distributed and Efficient Serving Framework for Large Models

Related

• HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices
• Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Models
• FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access