2017/11/10 by Dingwen Tao, Sheng Di, Tao, Dingwen +5 · 1 citation
Computer Science · #Advanced Data Storage Technologies #Algorithms and Data Compression #Distributed #FOS: Computer and information sciences #Parallel #Parallel Computing and Optimization Techniques #and Cluster Computing (cs.DC) #cs.DC
paper · pdf · doi:10.48550/arxiv.1711.03888
Accepted by IEEE BigData 2017
arxiv created 2017/11/10 · openalex publication_date 2017/11/10 · arxiv updated 2017/11/13 · openalex created_date 2017/11/17 · openalex updated_date 2026/08/01
In situ lossy compression allowing user-controlled data loss can significantly reduce the I/O burden. For large-scale N-body simulations where only one snapshot can be compressed at a time, the lossy compression ratio is very limited because of the fairly low spatial coherence of the particle data. In this work, we assess the state-of-the-art single-snapshot lossy compression techniques of two common N-body simulation models: cosmology and molecular dynamics. We design a series of novel optimization techniques based on the two representative real-world N-body simulation codes. For molecular dynamics simulation, we propose three compression modes (i.e., best speed, best tradeoff, best compression mode) that can refine the tradeoff between the compression rate (a.k.a., speed/throughput) and ratio. For cosmology simulation, we identify that our improved SZ is the best lossy compressor with respect to both compression ratio and rate. Its compression ratio is higher than the second-best compressor by 11% with comparable compression rate. Experiments with up to 1024 cores on the Blues supercomputer at Argonne show that our proposed lossy compression method can reduce I/O time by 80% compared with writing data directly to a parallel file system and outperforms the second-best solution by 60%. Moreover, our proposed lossy compression methods have the best rate-distortion with reasonable compression errors on the tested N-body simulation data compared with state-of-the-art compressors.