Compass: SLO-aware Query Planner for Compound AI Serving at Scale

2025/04/23 by Banruo Liu, Liu, Banruo, Wei‐Yu Lin +6 · 2 citations
Computer Science · #Cloud computing #Coding theory and cryptography #Databases (cs.DB) #FOS: Computer and information sciences #Goodput #Machine Learning (cs.LG) #Pipeline (software) #Plan (archaeology) #Planner #Query optimization #Rendering (computer graphics) #Service (business) #Software deployment

paper · pdf · doi:10.48550/arxiv.2504.16397

openalex publication_date 2025/04/23 · openalex created_date 2025/10/11 · openalex updated_date 2026/08/05

Abstract

The rise of compound AI serving that integrates multiple operators in a pipeline enables end-user applications such as generative AI-powered meeting companions, autonomous driving, and immersive gaming. These workloads span diverse deployment spaces, from cloud-only queries to edge-assisted ones across infrastructure tiers, often including both within an application. Achieving high service goodput -- i.e., meeting service level objectives (SLOs) for pipeline latency, accuracy, and costs -- requires joint planning of operators' placement, configuration, and resource allocation. However, diverse SLOs, varying runtime environments (e.g., heterogeneous device speeds), and a large volume of queries competing for shared infrastructure explode the planning space, making real-time serving and cost-efficient deployment intractable with existing advances. This paper presents Compass, the first SLO-aware query planner that optimizes large-scale compound AI workloads across diverse deployment spaces. Compass decomposes the many-query, multi-SLO planning problem into tractable subproblems while preserving global decision quality, exploiting plan similarities within and across queries to slash the search steps. It further improves per-step efficiency with a plan profiler that performs selective profiling to achieve high-fidelity performance estimates at a fraction of the profiling cost. At runtime, Compass performs query-plan bipartite matching to maximize SLO goodput under resource contentions. Real-world evaluations show that Compass improves service goodput by 2.4--5.1x, reduces deployment costs by 3.8--4.5x, and accelerates planning by 4.2--10.5x, achieving service responsiveness within seconds and near-optimal decision quality.

Citations

• Recall: Empowering Multimodal Embedding for Edge Devices
• NanoFlow: Towards Optimal Large Language Model Serving Throughput
• The Llama 3 Herd of Models
• CascadeServe: Unlocking Model Cascades for Inference Serving
• Vidur: A Large-Scale Simulation Framework For LLM Inference
• Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
• vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training
• VidChapters-7M: Video Chapters at Scale
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• More is Different: Prototyping and Analyzing a New Form of Edge Server with Massive Mobile SoCs
• Robust Speech Recognition via Large-Scale Weak Supervision
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• Walle: An End-to-End, General-Purpose, and Large-Scale Production System for Device-Cloud Collaborative Machine Learning
• Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructures
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions
• Ekya: Continuous Learning of Video Analytics Models on Edge Compute\n Servers
• Multimodal Pretraining for Dense Video Captioning
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Voices Obscured in Complex Environmental Settings (VOICES) corpus
• An Efficient Bandit Algorithm for Realtime Multivariate Optimization
• Dense-Captioning Events in Videos
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Deep Residual Learning for Image Recognition
• You Only Look Once: Unified, Real-Time Object Detection
• Scalable Bayesian Optimization Using Deep Neural Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• Worst-Case Performance Bounds for Simple One-Dimensional Packing Algorithms
• HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Cited by

• Sherlock: Reliable and Efficient Agentic Workflow Execution
• Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale

Related

• ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput
• Batch Query Processing and Optimization for Agentic Workflows
• Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving
• Vortex: Hosting ML Inference and Knowledge Retrieval Services With Tight Latency and Throughput Requirements
• Beyond ReAct: A Planner-Centric Framework for Complex Tool-Augmented LLM Reasoning