Melissa: coordinating large-scale ensemble runs for deep learning and sensitivity analyses - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Article Dans Une Revue Journal of Open Source Software Année : 2023

Melissa: coordinating large-scale ensemble runs for deep learning and sensitivity analyses

Résumé

Large-scale ensemble runs typically consist of executing thousands of physical simulation instances according to a range of different input parameters. These ensemble runs enable sensitivity analyses, deep surrogate trainings, reinforcement learning, and data assimilation, but they rely on volumes of data that are too large to store. For example, a recent data assimilation ensemble study generated 1.3 PB of data [@yashiro2020]. These enormous volumes of data hinder scientific analyses in two ways: first, the I/O speeds are the slowest component in supercomputers; the incongruence between slow read/write speeds compared to the rapid generation of data leads to a degradation and plateau of performance. Second, the file systems on supercomputers are not designed to allocate such large volumes of data to singular studies. To avoid this I/O limitation, scientists reduce their study size by running low resolution simulations or down-sampling output data in space and time. However, the I/O problem only becomes more pronounced as the speed and size of supercomputers continues to advance faster than I/O speeds of storage disks.
Fichier principal
Vignette du fichier
10.21105.joss.05291.pdf (754.18 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04145897 , version 1 (29-06-2023)

Licence

Paternité

Identifiants

Citer

Marc Schouler, Robert Alexander Caulk, Lucas Meyer, Théophile Terraz, Christoph Conrads, et al.. Melissa: coordinating large-scale ensemble runs for deep learning and sensitivity analyses. Journal of Open Source Software, 2023, 8 (86), pp.5291. ⟨10.21105/joss.05291⟩. ⟨hal-04145897⟩
36 Consultations
34 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More