Towards Efficient I/O Pipelines using Accumulated Compression - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Towards Efficient I/O Pipelines using Accumulated Compression

M Mustafa Rafique
Franck Cappello
  • Function : Author
  • PersonId : 1102088
Bogdan Nicolae


High-Performance Computing (HPC) workloads generate large volumes of data at high-frequency during their execution, which needs to be captured concurrently at scale. These workloads exploit accelerators such as GPU for faster performance. However, the limited onboard high-bandwidth memory (HBM) on the GPU, and slow device-to-host memory PCIe interconnects lead to I/O overheads during application execution, thereby exacerbating their overall runtime. To overcome the aforementioned limitations, techniques such as compression and asynchronous transfers have been used by data management runtimes. However, compressing small blocks of data leads to a significant runtime penalty on the application. In this paper, we design and develop strategies to optimize the tradeoff between compressing checkpoints instantly and enqueuing transfers immediately versus accumulating snapshots and delaying compression to achieve faster compression throughput. Our evaluations on synthetic and real-life workloads for different systems and workload configurations demonstrate 1.3× to 8.3× speedup compared to the existing checkpoint approaches.
Fichier principal
Vignette du fichier
HiPC_23__Accumulated_checkpoint_compression.pdf (841.9 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04343670 , version 1 (14-12-2023)




  • HAL Id : hal-04343670 , version 1


Avinash Maurya, M Mustafa Rafique, Franck Cappello, Bogdan Nicolae. Towards Efficient I/O Pipelines using Accumulated Compression. HIPC’23: 30th IEEE International Conference on High Performance Computing, Data, and Analytics, Dec 2023, Goa, India. ⟨hal-04343670⟩
25 View
41 Download


Gmail Facebook X LinkedIn More