2023/03/21 by Daniel S. Johnson, Igor L. Markov, Johnson, Daniel S +1
Computer Science · Neuroscience · #Artificial intelligence #Brain Tumor Detection and Classification #Cloud Computing and Resource Management #Code (set theory) #Computer science #Distributed computing #FOS: Computer and information sciences #Inference #Latency (audio) #Machine Learning (cs.LG) #Operating system #Overhead (engineering) #Programming language #Real-time computing #Software System Performance and Reliability #Telecommunications
paper · pdf · doi:10.48550/arxiv.2303.11580
openalex publication_date 2023/03/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Many ML applications and products train on medium amounts of input data but get bottlenecked in real-time inference. When implementing ML systems, conventional wisdom favors segregating ML code into services queried by product code via Remote Procedure Call (RPC) APIs. This approach clarifies the overall software architecture and simplifies product code by abstracting away ML internals. However, the separation adds network latency and entails additional CPU overhead. Hence, we simplify inference algorithms and embed them into the product code to reduce network communication. For public datasets and a high-performance real-time platform that deals with tabular data, we show that over half of the inputs are often amenable to such optimization, while the remainder can be handled by the original model. By applying our optimization with AutoML to both training and inference, we reduce inference latency by 1.3x, CPU resources by 30%, and network communication between application front-end and ML back-end by about 50% for a commercial end-to-end ML platform that serves millions of real-time decisions per second.