MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems

2026/04/14 by Yifei Wang, Hancheng Ye, Yechen Xu +8 · 1 voice · 1 citation
Computer Science · #cs.DC #cs.LG #cs.MA #cs.OS

paper · pdf · doi:10.48550/arxiv.2604.26963

arxiv published 2026/04/14 · arxiv updated 2026/06/15

Abstract

Large language models (LLMs) are increasingly deployed as the execution core of autonomous agents rather than as standalone text generators. Agentic workloads induce a temporal shift from single-turn inference to multi-turn LLM-tool loops, and a spatial shift from chat-scale, GPU-only execution to repository-scale, GPU-CPU co-located execution. Consequently, coordinating heterogeneous resource demands of agentic execution has emerged as a critical system challenge. We design and implement MARS, an efficient and adaptive co-scheduling system that globally coordinates heterogeneous agentic workloads under coupled GPU-CPU resource pressure. By establishing holistic visibility across GPU inference and CPU tool execution via a unified information stream, an external control plane in MARS decouples admission from execution to prevent heterogeneous resource oversubscription. An internal agent-centric scheduler further minimizes the end-to-end critical path by prioritizing latency-sensitive continuations and adaptively retaining KV cache state only when warm resumption yields a latency benefit. Our evaluations show that MARS reduces end-to-end latency by up to 5.94x while maintaining nearly maximal system throughput. We further integrate MARS as the serving backend for the OpenHands coding agent framework, demonstrating its real-world effectiveness by accelerating end-to-end task completion time by up to 1.87x. Our source code is publicly available at https://github.com/Afterglow231/MARS_preview .

Citations

• AgentOS: From Application Silos to a Natural Language-Driven Data Ecosystem
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
• Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
• gpt-oss-120b & gpt-oss-20b Model Card
• Autellix: An Efficient Serving Engine for LLM Agents as General Programs
• OpenHands: An Open Platform for AI Software Developers as Generalist Agents
• Teola: Towards End-to-End Optimization of LLM-based Applications
• Llumnix: Dynamic Scheduling for Large Language Model Serving
• Parrot: Efficient Serving of LLM-based Applications with Semantic Variable
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
• SGLang: Efficient Execution of Structured Language Model Programs
• Prompt Cache: Modular Attention Reuse for Low-Latency Inference
• MemGPT: Towards LLMs as Operating Systems
• CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures

Cited by

• Agentic Nesting: A New Methodology for Existing Enterprise Application Integration and Services

Discussions

• Wang, Ye, Xu, Guo, Wei, Wang, Li, Chen, Li, Zhuo, Chen: MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems https://arxiv.org/abs/2604.26963 https://arxiv.org/pdf/2604.26963 http [bsky, 0 points, 0 comments]

Related

• Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC
• Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines
• Towards Agentic OS: An LLM Agent Framework for Linux Schedulers
• Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
• Astraea: A State-Aware Scheduling Engine for LLM-Powered Agents