2024/06/26 by François Mazen, Mazen, François, Louis Gombert +5 · 1 citation
Engineering · #Distributed #FOS: Computer and information sciences #Fault Detection and Control Systems #Parallel #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2406.18112
openalex publication_date 2024/06/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In this short paper, we present an innovative approach to limit the required bandwidth when transferring data during in transit analysis. This approach is called hybrid because it combines existing in situ and in transit solutions. It leverages the stable ABI of Catalyst version 2 and the Catalyst-ADIOS2 implementation to seamlessly switch from in situ, in transit and hybrid analysis without modifying the numerical simulation code. The typical use case is to perform data reduction in situ then generate a visualization in transit on the reduced data. This approach makes the numerical simulation workflows very flexible depending on the size of the data, the available computing resources or the analysis type. Our experiment with this hybrid approach, reducing data before sending it, demonstrated large cost reductions for some visualization pipelines compared to in situ and in transit solutions. The implementation is available under an open source permissive license to be usable broadly in any scientific community.