Equinox: Holistic Fair Scheduling in Serving Large Language Models

2025/08/19 by Wei, Zhixiang, Yen, James, Chen, Jingyi +11 · 1 citation
#Artificial Intelligence (cs.AI) #Distributed #FOS: Computer and information sciences #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.2508.16646

Abstract

We address the limitations of current LLM serving with a dual-counter framework separating user and operator perspectives. The User Fairness Counter measures quality of service via weighted tokens and latency; the Resource Fairness Counter measures operational efficiency through throughput and GPU utilization. Since these metrics are only available post-execution, creating a scheduling paradox, we introduce a deterministic Mixture of Prediction Experts (MoPE) framework to predict user-perceived latency, output tokens, throughput, and GPU utilization. These predictions enable calculation of a unified Holistic Fairness score that balances both counters through tunable parameters for proactive fairness-aware scheduling. We implement this in Equinox, an open-source system with other optimizations like adaptive batching, and stall-free scheduling. Evaluations on production traces (ShareGPT, LMSYS) and synthetic workloads demonstrate Equinox achieves up to 1.3× higher throughput, 60% lower time-to-first-token latency, and 13% higher fairness versus VTC while maintaining 94% GPU utilization, proving fairness under bounded discrepancy across heterogeneous platforms.

Citations

• Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference
• Fair-MoE: Fairness-Oriented Mixture of Experts in Vision-Language Models
• Multi-Bin Batching for Increasing LLM Inference Throughput
• DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
• Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction
• Parrot: Efficient Serving of LLM-based Applications with Semantic Variable
• KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation
• LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
• Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• SGLang: Efficient Execution of Structured Language Model Programs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
• Hierarchical Roofline Performance Analysis for Deep Learning Applications
• Language Models are Few-Shot Learners
• Attention Is All You Need
• Sequence to Sequence Learning with Neural Networks
• Are Emergent Abilities of Large Language Models a Mirage?
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• AdaFair-MARL: Enforcing Adaptive Fairness Constraints in Multi-Agent Reinforcement Learning

Related

• Fairness in Serving Large Language Models
• Justitia: Fair and Efficient Scheduling for LLM Applications
• FairBatching: Fairness-Aware Batch Formation for LLM Inference
• Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs
• Locality-aware Fair Scheduling in LLM Serving