2020/06/22 by Jashwant Raj Gunasekaran, Michael M. Cui, Michael Cui +11
Computer Science · Engineering · #Cloud Computing and Resource Management #Cloud computing #Computer network #Computer science #Distributed #Distributed and Parallel Computing Systems #Distributed computing #Engineering #FOS: Computer and information sciences #Hypervisor #IoT and Edge/Fog Computing #Operating system #Parallel #Provisioning #Scheduling (production processes) #Temporal isolation among virtual machines #Throughput #Virtual machine #Virtualization #and Cluster Computing (cs.DC) #cs.DC
paper · pdf · doi:10.48550/arxiv.2006.12560
arxiv created 2020/06/22 · openalex publication_date 2020/06/22 · arxiv updated 2020/06/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Traditionally, HPC workloads have been deployed in bare-metal clusters; but\nthe advances in virtualization have led the pathway for these workloads to be\ndeployed in virtualized clusters. However, HPC cluster administrators/providers\nstill face challenges in terms of resource elasticity and virtual machine (VM)\nprovisioning at large-scale, due to the lack of coordination between a\ntraditional HPC scheduler and the VM hypervisor (resource management layer).\nThis lack of interaction leads to low cluster utilization and job completion\nthroughput. Furthermore, the VM provisioning delays directly impact the overall\nperformance of jobs in the cluster. Hence, there is a need for effectively\nprovisioning virtualized HPC clusters, which can best-utilize the physical\nhardware with minimal provisioning overheads.\n Towards this, we propose Multiverse, a VM provisioning framework, which can\ndynamically spawn VMs for incoming jobs in a virtualized HPC cluster, by\nintegrating the HPC scheduler along with VM resource manager. We have\nimplemented this framework on the Slurm scheduler along with the vSphere VM\nresource manager. In order to reduce the VM provisioning overheads, we use\ninstant cloning which shares both the disk and memory with the parent VM, when\ncompared to full VM cloning which has to boot-up a new VM from scratch.\nMeasurements with real-world HPC workloads demonstrate that, instant cloning is\n2.5x faster than full cloning in terms of VM provisioning time. Further, it\nimproves resource utilization by up to 40%, and cluster throughput by up to\n1.5x, when compared to full clone for bursty job arrival scenarios.\n