2015/07/04 by Derek Groen, Simon Portegies Zwart, Groen, Derek +1
Computer Science · Physics and Astronomy · #Advanced Data Storage Technologies #Cloud Computing and Resource Management #Computers and Society (cs.CY) #Distributed #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #FOS: Physical sciences #Instrumentation and Methods for Astrophysics (astro-ph.IM) #Parallel #Parallel Computing and Optimization Techniques #and Cluster Computing (cs.DC) #astro-ph.IM #cs.CY #cs.DC
paper · pdf · doi:10.48550/arxiv.1507.01138
Accepted for publication in IEEE conference on ERRORs
arxiv created 2015/07/04 · openalex publication_date 2015/07/04 · arxiv updated 2015/07/07 · openalex created_date 2022/10/01 · openalex updated_date 2026/07/28
We describe the political and technical complications encountered during the astronomical CosmoGrid project. CosmoGrid is a numerical study on the formation of large scale structure in the universe. The simulations are challenging due to the enormous dynamic range in spatial and temporal coordinates, as well as the enormous computer resources required. In CosmoGrid we dealt with the computational requirements by connecting up to four supercomputers via an optical network and make them operate as a single machine. This was challenging, if only for the fact that the supercomputers of our choice are separated by half the planet, as three of them are located scattered across Europe and fourth one is in Tokyo. The co-scheduling of multiple computers and the 'gridification' of the code enabled us to achieve an efficiency of up to 93% for this distributed intercontinental supercomputer. In this work, we find that high-performance computing on a grid can be done much more effectively if the sites involved are willing to be flexible about their user policies, and that having facilities to provide such flexibility could be key to strengthening the position of the HPC community in an increasingly Cloud-dominated computing landscape. Given that smaller computer clusters owned by research groups or university departments usually have flexible user policies, we argue that it could be easier to instead realize distributed supercomputing by combining tens, hundreds or even thousands of these resources.