2021/08/05 by Genoveva Vargas‐Solar, Vargas-Solar, Genoveva, Ali Akoglu +3
Computer Science · #Distributed and Parallel Computing Systems #Cloud Computing and Resource Management #Advanced Data Storage Technologies
paper · pdf · doi:10.48550/arxiv.2108.02558
This paper targets the execution of data science (DS) pipelines supported by\ndata processing, transmission and sharing across several resources executing\ngreedy processes. Current data science pipelines environments provide various\ninfrastructure services with computing resources such as general-purpose\nprocessors (GPP), Graphics Processing Units (GPUs), Field Programmable Gate\nArrays (FPGAs) and Tensor Processing Unit (TPU) coupled with platform and\nsoftware services to design, run and maintain DS pipelines. These one-fits-all\nsolutions impose the complete externalization of data pipeline tasks. However,\nsome tasks can be executed in the edge, and the backend can provide just in\ntime resources to ensure ad-hoc and elastic execution environments.\n This paper introduces an innovative composable "Just in Time Architecture"\nfor configuring DCs for Data Science Pipelines (JITA-4DS) and associated\nresource management techniques. JITA-4DS is a cross-layer management system\nthat is aware of both the application characteristics and the underlying\ninfrastructures to break the barriers between applications,\nmiddleware/operating system, and hardware layers. Vertical integration of these\nlayers is needed for building a customizable Virtual Data Center (VDC) to meet\nthe dynamically changing data science pipelines' requirements such as\nperformance, availability, and energy consumption. Accordingly, the paper shows\nan experimental simulation devoted to run data science workloads and determine\nthe best strategies for scheduling the allocation of resources implemented by\nJITA-4DS.\n