Determinism-Preserving GPU Spatial Sharing with Vitamin-E

2026/03/31 by Zhenyuan Yang, Wenxin Zheng, Mingyu Li +1
Computer Science · #cs.DC #cs.OS

paper · pdf

arxiv created 2026/08/05 · arxiv updated 2026/08/06

Abstract

GPU sharing faces a determinism--utilization tradeoff: fixed bindings can strand capacity as demand fluctuates, while resource-driven kernel reshaping improves utilization by altering a launch's parallel structure, potentially changing output bits. We rethink modern GPU scheduling and observe that it decouples logical structure from physical width: one unmodified launch spans a family of widths through changes in block placement and wave count. From this observation, we derive the parallel-structure invariant: for fixed-structure deterministic workloads, keeping each launch immutable makes its output bits independent of physical width. Guided by this invariant, Vitamin-E late-binds immutable launches to pooled physical contexts, preserving bitwise equality across allocations, whereas resource-driven reshaping can alter the selected token under temperature-zero greedy decoding. Across all workload--baseline comparisons, Vitamin-E achieves up to 3.50× the aggregate normalized LLM training throughput, 62.5% lower inference p99 latency, and 1.43× the background-training throughput. With the same mechanism, TPOT-First reduces TPOT SLO violations by up to 46.1% over Throughput-Oriented on three serving workloads, demonstrating mechanism effectiveness and policy flexibility.

Citations

• Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch
• RepDL: Bit-level Reproducible Deep Learning Training and Inference
• ECLIP: Energy-efficient and Practical Co-Location of ML Inference on Spatially Partitioned GPUs
• LithOS: An Operating System for Efficient Machine Learning on GPUs
• Testing GPU Numerics: Finding Numerical Differences Between NVIDIA and AMD GPUs
• ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments
• NanoFlow: Towards Optimal Large Language Model Serving Throughput
• Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications
• DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
• The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
• Guardian: Safe GPU Sharing in Multi-Tenant Environments
• SGLang: Efficient Execution of Structured Language Model Programs
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• PiPAD: Pipelined and Parallel Dynamic GNN Training on GPUs
• Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
• Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
• TicTac: Accelerating Distributed Deep Learning with Communication Scheduling
• Deep Residual Learning for Image Recognition
• Kernelet: High-Throughput GPU Kernel Executions with Dynamic Slicing and Scheduling
• What every computer scientist should know about floating-point arithmetic
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding