2021/11/29 by David S. Smith, Smith, David S.
Computer Science · Decision Sciences · #Advanced Data Storage Technologies #Computational Physics and Python Applications #Databases (cs.DB) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mathematical Software (cs.MS) #Scientific Computing and Data Management #cs.DB #cs.LG #cs.MS
paper · pdf · doi:10.48550/arxiv.2112.01273
8 pages, 3 figures
openalex publication_date 2021/11/29 · arxiv created 2021/11/30 · arxiv updated 2022/01/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Raw data sizes are growing and proliferating in scientific research, driven by the success of data-hungry computational methods, such as machine learning. The preponderance of proprietary and shoehorned data formats make computations slower and make it harder to reproduce research and to port methods to new platforms. Here we present the RawArray format: a simple, fast, and extensible format for archival storage of multidimensional numeric arrays on disk. The RawArray file format is a simple concatenation of a header array and a data array. The header comprises seven or more 64-bit unsigned integers. The array data can be anything. Arbitrary user metadata can be appended to an RawArray file if desired, for example to store measurement details, color palettes, or geolocation data. We present benchmarks showing a factor of 2--3× speedup over HDF5 for a range of array sizes and a speedup of up to 20× in reading the common deep learning datasets MNIST and CIFAR10.