LowFive: In Situ Data Transport for High-Performance Workflows - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

LowFive: In Situ Data Transport for High-Performance Workflows


We describe LowFive, a new data transport layer based on the HDF5 data model, for in situ workflows. Executables using LowFive can communicate in situ (using in-memory data and MPI message passing), reading and writing traditional HDF5 files to physical storage, and combining the two modes. Minimal and often no source-code modification is needed for programs that already use HDF5. LowFive maintains deep copies or shallow references of datasets, configurable by the user. More than one task can produce (write) data, and more than one task can consume (read) data, accommodating fan-in and fan-out in the workflow task graph. LowFive supports data redistribution from n producer processes to m consumer processes. We demonstrate the above features in a series of experiments featuring both synthetic benchmarks as well as a representative use case from a scientific workflow, and we also compare with other data transport solutions in the literature.
Fichier principal
Vignette du fichier
lowfive-ipdps23.pdf (1.3 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04119925 , version 1 (07-06-2023)




  • HAL Id : hal-04119925 , version 1


Tom Peterka, Dmitriy Morozov, Arnur Nigmetov, Orcun Yildiz, Bogdan Nicolae, et al.. LowFive: In Situ Data Transport for High-Performance Workflows. IPDPS'23: The 37th IEEE International Parallel and Distributed Processing Symposium, ACM; IEEE, May 2023, St.Petersburg, United States. ⟨hal-04119925⟩
6 View
26 Download


Gmail Facebook Twitter LinkedIn More