Baler -- Machine Learning Based Compression of Scientific Data

2023/05/03 by Fritjof Bengtsson, Bengtsson, Fritjof, C. Doglioni +15 · 1 citation
Computer Science · #Computational Physics and Python Applications #Algorithms and Data Compression #Advanced Data Storage Technologies

paper · pdf · doi:10.48550/arxiv.2305.02283

Abstract

Storing and sharing increasingly large datasets is a challenge across scientific research and industry. In this paper, we document the development and applications of Baler - a Machine Learning based data compression tool for use across scientific disciplines and industry. Here, we present Baler's performance for the compression of High Energy Physics (HEP) data, as well as its application to Computational Fluid Dynamics (CFD) toy data as a proof-of-principle. We also present suggestions for cross-disciplinary guidelines to enable feasibility studies for machine learning based compression for scientific data.

Cited by

• Overview of the ESCAPE Dark Matter Test Science Project for Astronomers

Related

• Scalable Hybrid Learning Techniques for Scientific Data Compression
• NeurLZ: An Online Neural Learning-Based Method to Enhance Scientific Lossy Compression
• Attention Based Machine Learning Methods for Data Reduction with Guaranteed Error Bounds
• EVOLVE: Efficient Learned Volume Compression with Variable-Rate Encoding on a Cross-Domain Database
• Machine Learning Techniques for Data Reduction of CFD Applications