GPU-Enabled Asynchronous Multi-level Checkpoint Caching and Prefetching - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

GPU-Enabled Asynchronous Multi-level Checkpoint Caching and Prefetching

M Mustafa Rafique
Thierry Tonellot
  • Function : Author
  • PersonId : 1110240
Hussain J Alsalem
  • Function : Author
  • PersonId : 1259979
Franck Cappello
  • Function : Author
  • PersonId : 1102088
Bogdan Nicolae


Checkpointing is an I/O intensive operation increasingly used by High-Performance Computing (HPC) applications to revisit pre- vious intermediate datasets at scale. Unlike the case of resilience, where only the last checkpoint is needed for application restart and rarely accessed to recover from failures, in this scenario, it is important to optimize frequent reads and writes of an entire history of checkpoints. State-of-the-art checkpointing approaches often rely on asynchronous multi-level techniques to hide I/O overheads by writing to fast local tiers (e.g. an SSD) and asynchronously flush- ing to slower, potentially remote tiers (e.g. a parallel file system) in the background, while the application keeps running. However, such approaches have two limitations. First, despite the fact that HPC infrastructures routinely rely on accelerators (e.g. GPUs), and therefore a majority of the checkpoints involve GPU memory, ef- ficient asynchronous data movement between the GPU memory and host memory is lagging behind. Second, revisiting previous data often involves predictable access patterns, which are not exploited to accelerate read operations. In this paper, we address these limitations by proposing a scalable and asynchronous multi-level checkpointing approach optimized for both reading and writing of an arbitrarily long history of checkpoints. Our approach exploits GPU memory as a first-class citizen in the multi-level storage hierarchy to enable informed caching and prefetching of checkpoints by leveraging foreknowledge about the access order passed by the application as hints. Our evaluation using a variety of scenarios under I/O concurrency shows up to 74x faster checkpoint and restore throughput as compared to the state-of-art runtime and optimized unified virtual memory (UVM) based prefetching strategies and at least 2x shorter I/O wait time for the application across various workloads and configurations.
Fichier principal
Vignette du fichier
HPDC_23__VELOC_GPU.pdf (1.24 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04119928 , version 1 (07-06-2023)




Avinash Maurya, M Mustafa Rafique, Thierry Tonellot, Hussain J Alsalem, Franck Cappello, et al.. GPU-Enabled Asynchronous Multi-level Checkpoint Caching and Prefetching. HPDC'23: 32nd International Symposium on High-Performance Parallel and Distributed Computing, ACM; IEEE, Jun 2023, Orlando, United States. ⟨10.1145/3588195.3592987⟩. ⟨hal-04119928⟩
234 View
275 Download



Gmail Mastodon Facebook X LinkedIn More