2017/10/09 by Eduardo Rodrigues, Rodrigues, Eduardo, Ricardo Morla +2
Business, Management and Accounting · Computer Science · #Advanced Database Systems and Queries #Big Data and Business Intelligence #Cloud Computing and Resource Management #Data Stream Mining Techniques #Distributed #FOS: Computer and information sciences #Machine Learning and Data Classification #Parallel #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.1710.03040
openalex publication_date 2017/10/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Data science and machine learning algorithms running on big data infrastructure are increasingly important in activities ranging from business intelligence and analytics to cybersecurity, smart city management, and many fields of science and engineering. As these algorithms are further integrated into daily operations, understanding how long they take to run on a big data infrastructure is paramount to controlling costs and delivery times. In this paper we discuss the issues involved in understanding the run time of iterative machine learning algorithms and provide a case study of such an algorithm - including a statistical characterization and model of the run time of an implementation of K-Means for the Spark big data engine using the Edward probabilistic programming language.