HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices

2024/03/02 by Xuanlei Zhao, Zhao, Xuanlei, Bin Jia +9 · 3 citations
Computer Science · #Artificial intelligence #Computer network #Computer science #Distributed computing #Inference #Parallel computing #Programming language #Resource (disambiguation) #Theoretical computer science #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2403.01164

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/03/02 · openalex created_date 2024/03/06 · openalex updated_date 2026/07/28

Abstract

In recent times, the emergence of Large Language Models (LLMs) has resulted in increasingly larger model size, posing challenges for inference on low-resource devices. Prior approaches have explored offloading to facilitate low-memory inference but often suffer from efficiency due to I/O bottlenecks. To achieve low-latency LLMs inference on resource-constrained devices, we introduce HeteGen, a novel approach that presents a principled framework for heterogeneous parallel computing using CPUs and GPUs. Based on this framework, HeteGen further employs heterogeneous parallel computing and asynchronous overlap for LLMs to mitigate I/O bottlenecks. Our experiments demonstrate a substantial improvement in inference speed, surpassing state-of-the-art methods by over 317% at most.

Cited by

• STADI: Fine-Grained Step-Patch Diffusion Parallelism for Heterogeneous GPUs
• Parallel CPU-GPU Execution for LLM Inference on Constrained GPUs
• WindVE: Collaborative CPU-NPU Vector Embedding

Related

• GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
• HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment
• HPIM: Heterogeneous Processing-In-Memory-based Accelerator for Large Language Models Inference
• Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
• Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems