gpu_ext: Extensible OS Policies for GPUs via eBPF

2025/12/14 by Yusheng Zheng, Tong Yu, Zheng, Yusheng +19 · 1 voice · 1 citation
Computer Science · #Advanced Data Storage Technologies #FOS: Computer and information sciences #Operating Systems (cs.OS) #Parallel Computing and Optimization Techniques #Security and Verification in Computing #cs.OS

paper · pdf · doi:10.48550/arxiv.2512.12615

openalex publication_date 2025/12/14 · arxiv published 2025/12/14 · openalex created_date 2025/12/17 · arxiv updated 2025/12/20 · openalex updated_date 2026/07/28

Abstract

Performance in modern GPU-centric systems increasingly depends on resource management policies, including memory placement, scheduling, and observability. However, uniform policies typically yield suboptimal performance across diverse workloads. Existing approaches present a tradeoff: user-space runtimes provide programmability and flexibility but lack cross-tenant visibility and fine-grained control of hardware resources; meanwhile, modifications to the OS kernel introduce significant complexity and safety risks. To address this, we argue that the GPU driver and device layer should provide an extensible OS interface for policy enforcement. While the emerging eBPF technology shows potential, directly applying existing host-side eBPF is insufficient because they lack visibility and control into critical device-side events, and directly embedding policy code into GPU kernels could compromise safety and efficiency. We propose gpu_ext, an eBPF-based runtime that treats the GPU driver and device as a programmable OS subsystem. gpu_ext extends GPU drivers by exposing safe programmable hooks and introduces a device-side eBPF runtime capable of executing verified policy logic within GPU kernels, enabling coherent and transparent policies. Evaluation across realistic workloads including inference, training, and vector search demonstrates that gpu_ext improves throughput by up to 4.8x and reduces tail latency by up to 2x, incurring low overhead, without modifying or restarting applications

Citations

• Pie: A Programmable Serving System for Emerging LLM Applications
• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
• LithOS: An Operating System for Efficient Machine Learning on GPUs
• GCAPS: GPU Context-Aware Preemptive Priority-based Scheduling for Real-Time Tasks
• Llumnix: Dynamic Scheduling for Large Language Model Serving
• Vidur: A Large-Scale Simulation Framework For LLM Inference
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• ServerlessLLM: Low-Latency Serverless Inference for Large Language Models
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Splitwise: Efficient generative LLM inference using phase splitting
• Survey of Vector Database Management Systems
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• Programmable System Call Security with eBPF
• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
• FastMoE: A Fast Mixture-of-Expert Training System
• A flow-based IDS using Machine Learning in eBPF
• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
• Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads
• UVMBench: A Comprehensive Benchmark Suite for Researching Unified Virtual Memory in GPUs
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads

Cited by

• KernelScript: Cross-Boundary Typed DSL for eBPF Applications

Discussions

• Yusheng Zheng, Tong Yu, Yiwei Yang, Minghui Jiang, Xiangyu Gao, Jianchang Su, Yanpeng Hu, Wenan Mao, Wei Zhang, Dan Williams, Andi Quinn: gpu_ext: Extensible OS Policies for GPUs via eBPF https://arxi [bsky, 0 points, 0 comments]

Related

• Guardian: Safe GPU Sharing in Multi-Tenant Environments
• The Fabric Is the Cluster Driver: Cross-Layer eBPF Policies for GPU-CXL Fabrics
• GPUReplay: A 50-KB GPU Stack for Client ML
• Glider: A GPU Library Driver for Improved System Security
• CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads