Dask-Extended External Tasks for HPC/ML In Transit Workflows - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Dask-Extended External Tasks for HPC/ML In Transit Workflows

Amal Gueroudji
Julien Bigot
Bruno Raffin
Robert Ross


In situ workflows are inescapable to fully leverage exascale architectures. They can be complex to build, however, because simulation and data analytics come from two different software ecosystems with their own paradigms and programming models. This work extends the deisa bridging model between MPI+X simulations and distributed task-based analytics; it introduces the concept of external tasks to support the description of analytics graphs spanning multiple timesteps ahead of time while improving scalability. This new approach leads to a straightforward support for contracts between the simulation and analytics graph to limit the data transferred to that actually analyzed in a given execution. We implement this approach using Dask and MPI and evaluate it using an end-toend in-transit workflow that uses an unsupervised ML model for dimensionality reduction. We compare our work with plain Dask postprocessing and with the previous version of deisa. Our work performs better, up to ×7 for the simulation and ×3 for the analytics compared with deisa, and is ×18 less costly compared with plain Dask-all of these with similar development efforts.
Fichier principal
Vignette du fichier
deisa.pdf (1.09 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04409157 , version 1 (22-01-2024)




Amal Gueroudji, Julien Bigot, Bruno Raffin, Robert Ross. Dask-Extended External Tasks for HPC/ML In Transit Workflows. SC-W 2023 - Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, Nov 2023, Denver, United States. pp.831-838, ⟨10.1145/3624062.3624151⟩. ⟨hal-04409157⟩
59 View
44 Download



Gmail Mastodon Facebook X LinkedIn More