Towards Efficient Cache Allocation for High-Frequency Checkpointing - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year :

Towards Efficient Cache Allocation for High-Frequency Checkpointing

Bogdan Nicolae
M Mustafa Rafique
  • Function : Author
  • PersonId : 1076142
Thierry Tonellot
  • Function : Author
  • PersonId : 1110240
Franck Cappello
  • Function : Author
  • PersonId : 1102088


While many HPC applications are known to have long runtimes, this is not always because of single large runs: in many cases, this is due to ensembles composed of many short runs (runtime in the order of minutes). When each such run needs to checkpoint frequently (e.g. adjoint computations using a checkpoint interval in the order of milliseconds), it is important to minimize both checkpointing overheads at each iteration, as well as initialization overheads. With the rising popularity of GPUs, minimizing both overheads simultaneously is challenging: while it is possible to take advantage of efficient asynchronous data transfers between GPU and host memory, this comes at the cost of high initialization overhead needed to allocate and pin host memory. In this paper, we contribute with an efficient technique to address this challenge. The key idea is to use an adaptive approach that delays the pinning of the host memory buffer holding the checkpoints until all memory pages are touched, which greatly reduces the overhead of registering the host memory with the CUDA driver. To this end, we use a combination of asynchronous touching of memory pages and direct writes of checkpoints to untouched and touched memory pages in order to minimize end-to-end checkpointing overheads based on performance modeling. Our evaluations show a significant improvement over a variety of alternative static allocation strategies and state-of-art approaches.
Fichier principal
Vignette du fichier
HiPC_22_VELOC_GPU.pdf (822.39 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03799226 , version 1 (05-10-2022)


  • HAL Id : hal-03799226 , version 1


Avinash Maurya, Bogdan Nicolae, M Mustafa Rafique, Amr Elsayed, Thierry Tonellot, et al.. Towards Efficient Cache Allocation for High-Frequency Checkpointing. HiPC'22: 29th IEEE International Conference on High Performance Computing, Data, and Analytics, Dec 2022, Bangalore, India. ⟨hal-03799226⟩
119 View
117 Download


Gmail Facebook Twitter LinkedIn More