TensorFlow-Serving: Flexible, High-Performance ML Serving

2017/12/17 by Christopher Olston, Olston, Christopher, Noah Fiedel +15 · 33 citations
Computer Science · #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine Learning and Data Classification #Parallel #Parallel Computing and Optimization Techniques #and Cluster Computing (cs.DC) #cs.DC #cs.LG

paper · pdf · doi:10.48550/arxiv.1712.06139

Presented at NIPS 2017 Workshop on ML Systems (http://learningsys.org/nips17/acceptedpapers.html)

openalex publication_date 2017/12/17 · arxiv created 2017/12/27 · arxiv updated 2017/12/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We describe TensorFlow-Serving, a system to serve machine learning models inside Google which is also available in the cloud and via open-source. It is extremely flexible in terms of the types of ML platforms it supports, and ways to integrate with systems that convey new models and updated versions from training to serving. At the same time, the core code paths around model lookup and inference have been carefully optimized to avoid performance pitfalls observed in naive implementations. Google uses it in many production deployments, including a multi-tenant model hosting service called TFS².

Cited by

• ML Inference Scheduling with Predictable Latency
• Cloudy with high chance of DBMS: A 10-year prediction for Enterprise-Grade ML
• CardioLearn: A Cloud Deep Learning Service for Cardiac Disease Detection from Electrocardiogram
• Hoplite: Efficient and Fault-Tolerant Collective Communication for Task-Based Distributed Systems
• HADIS: Hybrid Adaptive Diffusion Model Serving for Efficient Text-to-Image Generation
• ModelHub.AI: Dissemination Platform for Deep Learning Models
• ModiPick: SLA-aware Accuracy Optimization For Mobile Deep Inference
• SequenceLayers: Sequence Processing and Streaming Neural Networks Made Easy
• No DNN Left Behind: Improving Inference in the Cloud with Multi-Tenancy
• DistrAttention: An Efficient and Flexible Self-Attention Mechanism on Modern GPUs
• Towards ML Engineering: A Brief History Of TensorFlow Extended (TFX)
• Kamae: Bridging Spark and Keras for Seamless ML Preprocessing
• Characterizing the Deep Neural Networks Inference Performance of Mobile Applications
• InferBench: Understanding Deep Learning Inference Serving with an Automatic Benchmarking System
• A Survey of LLM Inference Systems
• Golden Grain: Building a Secure and Decentralized Model Marketplace for MLaaS
• MEL: Multi-level Ensemble Learning for Resource-Constrained Environments
• Filter-Centric Vector Indexing: Geometric Transformation for Efficient Filtered Vector Search
• Tady: A Neural Disassembler without Structural Constraint Violations
• DeepRT: A Soft Real Time Scheduler for Computer Vision Applications on the Edge
• PACSET (Packed Serialized Trees): Reducing Inference Latency for Tree Ensemble Deployment
• FlexServe: Deployment of PyTorch Models as Flexible REST Endpoints
• Frustrated with Replicating Claims of a Shared Model? A Solution
• IDK Cascades: Fast Deep Learning by Learning not to Overthink
• ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor
• AIBench: An Industry Standard Internet Service AI Benchmark Suite
• ModelCI-e: Enabling Continual Learning in Deep Learning Serving Systems
• Towards Personalized and Semantic Retrieval: An End-to-End Solution for E-commerce Search via Embedding Learning
• Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning
• Representing and querying data tensors in RDF and SPARQL
• AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite
• HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models
• Large-Scale Intelligent Microservices
• LithOS: An Operating System for Efficient Machine Learning on GPUs
• Efficient LLM Serving on Hybrid Real-time and Best-effort Requests

Related

• TensorFlow: A system for large-scale machine learning
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Should I use TensorFlow
• Scaling TensorFlow to 300 million predictions per second
• Kafka-ML: connecting the data stream with ML/AI frameworks