Dynamic Load/Propagate/Store for Data Assimilation with Particle Filters on Supercomputers - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles Journal of computational science Year : 2024

Dynamic Load/Propagate/Store for Data Assimilation with Particle Filters on Supercomputers

Abstract

Several ensemble-based Data Assimilation (DA) methods rely on a prop- agate/update cycle, where a potentially compute intensive simulation code propagates multiple states for several consecutive time steps, that are then analyzed to update the states to be propagated for the next cycle. In this pa- per we focus on DA methods where the update can be computed by gathering only lightweight data obtained independently from each of the propagated states. This encompasses particle filters where one weight is computed from each state, but also methods like Approximate Bayesian Computation (ABC) or Markov Chain Monte Carlo (MCMC). Such methods can be very compute intensive and running efficiently at scale on supercomputers is challenging. This paper proposes a framework based on an elastic and fault-tolerant runner/server architecture minimizing data movements while enabling dynamic load balancing. Our approach relies on runners that load, propagate and store particles from an asynchronously managed distributed particle cache permitting particles to move from one runner to another in the back- ground while particle propagation proceeds. The framework is validated with a bootstrap particle filter with the WRF simulation code. We handle up to 2,555 particles on 20,442 compute cores. Compared to a file-based implementation, our solution spends up to 2.84 less resources (cores×seconds) per particle.
Fichier principal
Vignette du fichier
melissa-P2P-hal.pdf (4.47 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03927612 , version 1 (06-01-2023)
hal-03927612 , version 2 (14-02-2024)

Licence

Identifiers

Cite

Sebastian Friedemann, Kai Keller, Yen-Sen Lu, Bruno Raffin, Leonardo Bautista Gomez. Dynamic Load/Propagate/Store for Data Assimilation with Particle Filters on Supercomputers. Journal of computational science, 2024, pp.102229. ⟨10.1016/j.jocs.2024.102229⟩. ⟨hal-03927612v2⟩
111 View
112 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More