Aragog: Just-in-Time Model Routing for Scalable Serving of Agentic Workflows

2025/11/26 by Dai, Yinwei, Chen, Zhuofu, Iyer, Anand +1
#Distributed #FOS: Computer and information sciences #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.2511.20975

Abstract

Agentic workflows have emerged as a powerful paradigm for solving complex, multi-stage tasks, but serving them at scale is computationally expensive given the many LLM inferences that each request must pass through. Configuration selection, or the cost-aware assignment of workflow agents to specific LLMs, can reduce these costs, but existing approaches bind configuration decisions before request execution, making them ill-suited for the heterogeneous and lengthy execution of workflows. Specifically, system loads can fluctuate rapidly and substantially during a request's lifetime, causing fixed configurations to quickly become suboptimal. We present Aragog, a system that progressively adapts a request's configuration throughout its execution to match runtime dynamics. To make this practical despite the massive space of workflow configurations, Aragog decouples the problem into two core elements -- a one-time routing step that identifies all accuracy-preserving configurations, and a cheap per-stage scheduler that selects among them using up-to-date system observations -- and introduces novel strategies to accelerate each. Across diverse workflows and model families, Aragog increases maximum serving throughput by 50.0--217.0% and reduces median latency by 32.5--78.9% at peak request rates, while maintaining accuracy comparable to the most expensive configurations.

Citations

• Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloud
• Efficient Serving of LLM Applications with Probabilistic Demand Modeling
• HEXGEN-FLOW: Optimizing LLM Inference Request Scheduling for Agentic Text-to-SQL
• JITServe: SLO-aware LLM Serving with Imprecise Request Information
• MasRouter: Learning to Route LLMs for Multi-Agent Systems
• FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
• NanoFlow: Towards Optimal Large Language Model Serving Throughput
• Llumnix: Dynamic Scheduling for Large Language Model Serving
• Parrot: Efficient Serving of LLM-based Applications with Semantic Variable
• Mirage: A Multi-Level Superoptimizer for Tensor Programs
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
• Affective Coherence Monitoring for Transformer-Based Language Models
• Fast Inference from Transformers via Speculative Decoding
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Evaluating Large Language Models Trained on Code
• Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• SQuAD: 100,000+ Questions for Machine Comprehension of Text

Cited by

• Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms

Related

• Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms
• Explainable Model Routing for Agentic Workflows
• Flow: Modularized Agentic Workflow Automation
• AFlow: Automating Agentic Workflow Generation
• Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloud