Clipper: A Low-Latency Online Prediction Serving System

2016/12/09 by Daniel Crankshaw, Crankshaw, Daniel, Xin Wang +9 · 81 citations
Computer Science · #Artificial intelligence #Caching and Content Delivery #Clipper (electronics) #Cloud Computing and Resource Management #Computer architecture #Computer science #Data Stream Mining Techniques #Distributed computing #Latency (audio) #Machine learning #Operating system #Real-time computing #Robustness (evolution) #Software deployment #cs.DC #cs.LG

paper · pdf · doi:10.48550/arxiv.1612.03079

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2016/12/09 · arxiv created 2017/02/28 · arxiv updated 2017/03/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Machine learning is being deployed in a growing number of applications which demand real-time, accurate, and robust predictions under heavy query load. However, most machine learning frameworks and systems only address model training and not deployment. In this paper, we introduce Clipper, a general-purpose low-latency prediction serving system. Interposing between end-user applications and a wide range of machine learning frameworks, Clipper introduces a modular architecture to simplify model deployment across frameworks and applications. Furthermore, by introducing caching, batching, and adaptive model selection techniques, Clipper reduces prediction latency and improves prediction throughput, accuracy, and robustness without modifying the underlying machine learning frameworks. We evaluate Clipper on four common machine learning benchmark datasets and demonstrate its ability to meet the latency, accuracy, and throughput demands of online serving applications. Finally, we compare Clipper to the TensorFlow Serving system and demonstrate that we are able to achieve comparable throughput and latency while enabling model composition and online learning to improve accuracy and render more robust predictions.

Citations

• The Nonstochastic Multiarmed Bandit Problem

Cited by

• Efficient Multi-Model Orchestration for Self-Hosted Large Language Models
• ML Inference Scheduling with Predictable Latency
• An Online Fragmentation-Aware Scheduler for Managing GPU-Sharing Workloads on Multi-Instance GPUs
• IslandRun: Privacy-Aware Multi-Objective Orchestration for Distributed AI Inference
• InferF: Declarative Factorization of AI/ML Inferences over Joins
• Scalable Data Attribution via Forward-Only Test-Time Inference
• Argus: Quality-Aware High-Throughput Text-to-Image Inference Serving System
• FREESH: Fair, Resource- and Energy-Efficient Scheduling for LLM Serving on Heterogeneous GPUs
• ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform
• xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning
• Next-Generation Event-Driven Architectures: Performance, Scalability, and Intelligent Orchestration Across Messaging Frameworks
• HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units
• Multi-Layer Perceptron-Based Relay Node Selection for Next-Generation Intelligent Delay-Tolerant Networks
• MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaS
• STADI: Fine-Grained Step-Patch Diffusion Parallelism for Heterogeneous GPUs
• HADIS: Hybrid Adaptive Diffusion Model Serving for Efficient Text-to-Image Generation
• Illuminating Patterns of Divergence: DataDios SmartDiff for Large-Scale Data Difference Analysis
• Predictable LLM Serving on GPU Clusters
• Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts
• PPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via Pool-Based Pipeline Parallelism
• DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
• Towards Resource-Efficient Serverless LLM Inference with SLINFER
• Geography for AI sustainability and sustainability for GeoAI
• A Survey of LLM Inference Systems
• MEL: Multi-level Ensemble Learning for Resource-Constrained Environments
• Orthogonal Soft Pruning for Efficient Class Unlearning
• Filter-Centric Vector Indexing: Geometric Transformation for Efficient Filtered Vector Search
• ViFusion: In-Network Tensor Fusion for Scalable Video Feature Indexing
• TIP-Search: Time-Predictable Inference Scheduling for Market Prediction under Uncertain Load
• InstGenIE: Generative Image Editing Made Efficient with Mask-aware Caching and Scheduling
• HorizonServe: Coordinating Request Scheduling with GPU Sharing for Omni-Model Serving
• Software Engineering for AI-Based Systems: A Survey
• HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routing
• Action Chunk Scheduling for Batched Robot Policy Serving
• ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor
• Net2Vec: Deep Learning for the Network
• SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models
• SneakPeek: Data-Aware Model Selection and Scheduling for Inference Serving on the Edge
• Patchwork: A Unified Framework for RAG Serving
• Compass: Optimizing Compound AI Workflows for Dynamic Adaptation
• GDEV-AI: A Generalized Evaluation of Deep Learning Inference Scaling and Architectural Saturation
• Valve: Production Online-Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
• HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models
• FailLite: Failure-Resilient Model Serving for Resource-Constrained Edge Environments
• Guillotine: Hypervisors for Isolating Malicious AIs
• LithOS: An Operating System for Efficient Machine Learning on GPUs
• HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving

Related

• PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
• Optimizing Prediction Serving on Low-Latency Serverless Dataflow
• Parity Models: A General Framework for Coding-Based Resilience in ML Inference
• InferLine: ML Prediction Pipeline Provisioning and Management for Tight Latency Objectives
• CowClip: Reducing CTR Prediction Model Training Time from 12 hours to 10 minutes on 1 GPU