2022/09/06 by Aaron Saxton, Saxton, Aaron, Stephen Squaire +1
Computer Science · #Advanced Data Storage Technologies #Cloud Computing and Resource Management #Distributed #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #Parallel #Performance (cs.PF) #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2209.15390
openalex publication_date 2022/09/06 · openalex created_date 2022/10/04 · openalex updated_date 2026/07/28
Data stores are the foundation on which data science, in all its variations, is built upon. They provide a queryable interface to structured and unstructured data. Data science often starts by leveraging these query features to perform initial data preparation. However, most data stores are designed to run continuously to service disparate user requests with little or no downtime. Many HPC architectures process user requests by job queue scheduler and maintain a shard filesystem to store a jobs persistent data. We deploy a MongoDB sharded cluster with a run script that is designed to run a data science workload concurrently. As our test piece, we run data ingest and data queries to measure the performance with different configurations on the Blue Waters supper computer.