2024/05/17 by T. Huang, Pengfei Chen, Huang, Tao +13 · 1 citation
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Blockchain Technology Applications and Security #Cloud Computing and Resource Management #Distributed #FOS: Computer and information sciences #Parallel #Smart Grid Security and Resilience #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2407.09486
openalex publication_date 2024/05/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Since the increasing popularity of large language model (LLM) backend systems, it is common and necessary to deploy stable serverless serving of LLM on multi-GPU clusters with autoscaling. However, there exist challenges because the diversity and co-location of applications in multi-GPU clusters will lead to low service quality and GPU utilization. To address them, we build ENOVA, a deployment, monitoring and autoscaling service towards serverless LLM serving. ENOVA deconstructs the execution process of LLM service comprehensively, based on which ENOVA designs a configuration recommendation module for automatic deployment on any GPU clusters and a performance detection module for autoscaling. On top of them, ENOVA implements a deployment execution engine for multi-GPU cluster scheduling. The experiment results show that ENOVA significantly outperforms other state-of-the-art methods and is suitable for wide deployment in large online systems.