GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving

2026/02/28 by Alessio Ricci Toniolo, Rome Thorstenson, Abinaya Dinesh
Computer Science · #cs.NI #cs.DC

paper · pdf

12 pages, 4 figures. Code: https://github.com/Arcadia-Research-Team/GORGO

arxiv created 2026/07/29 · arxiv updated 2026/07/31

Abstract

Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improve p95 TTFT by 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baseline load-balancing policies such as simple session affinity and prefix-cache. The code and ART-Chat-2.5M dataset can be found at https://github.com/Arcadia-Research-Team/GORGO.

Citations

• BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure
• KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
• SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference
• KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
• Llumnix: Dynamic Scheduling for Large Language Model Serving
• CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
• Preble: Efficient Distributed Prompt Scheduling for LLM Serving
• WildChat: 1M ChatGPT Interaction Logs in the Wild
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• SGLang: Efficient Execution of Structured Language Model Programs
• Splitwise: Efficient generative LLM inference using phase splitting
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Chord: a scalable peer-to-peer lookup protocol for internet applications

Related

• DeServe: Towards Affordable Offline LLM Inference via Decentralization
• ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
• HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
• JITServe: SLO-aware LLM Serving with Imprecise Request Information
• Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing