2020/06/22 by Thomas Rausch, Rausch, Thomas, Waldemar Hummer +3 · 1 citation
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Data Stream Mining Techniques #Data Visualization and Analytics #Distributed #FOS: Computer and information sciences #FOS: Electrical engineering #H.4 #I.2.m #I.6 #Machine Learning (cs.LG) #Parallel #Scientific Computing and Data Management #Systems and Control (eess.SY) #and Cluster Computing (cs.DC) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2006.12587
openalex publication_date 2020/06/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Operationalizing AI has become a major endeavor in both research and industry. Automated, operationalized pipelines that manage the AI application lifecycle will form a significant part of tomorrow's infrastructure workloads. To optimize operations of production-grade AI workflow platforms we can leverage existing scheduling approaches, yet it is challenging to fine-tune operational strategies that achieve application-specific cost-benefit tradeoffs while catering to the specific domain characteristics of machine learning (ML) models, such as accuracy, robustness, or fairness. We present a trace-driven simulation-based experimentation and analytics environment that allows researchers and engineers to devise and evaluate such operational strategies for large-scale AI workflow systems. Analytics data from a production-grade AI platform developed at IBM are used to build a comprehensive simulation model. Our simulation model describes the interaction between pipelines and system infrastructure, and how pipeline tasks affect different ML model metrics. We implement the model in a standalone, stochastic, discrete event simulator, and provide a toolkit for running experiments. Synthetic traces are made available for ad-hoc exploration as well as statistical analysis of experiments to test and examine pipeline scheduling, cluster resource allocation, and similar operational mechanisms.