Concealing Compression-accelerated I/O for HPC Applications through In Situ Task Scheduling - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2024

Concealing Compression-accelerated I/O for HPC Applications through In Situ Task Scheduling

Sian Jin
  • Fonction : Auteur
  • PersonId : 1289574
Sheng Di
  • Fonction : Auteur
  • PersonId : 1289575
Daoce Wang
  • Fonction : Auteur
  • PersonId : 1289576
Dingwen Tao
  • Fonction : Auteur
  • PersonId : 1289577
Franck Cappello
  • Fonction : Auteur
  • PersonId : 1102088

Résumé

Lossy compression and asynchronous I/O are two of the most effective solutions for reducing storage overhead and enhancing I/O performance in large-scale high-performance computing (HPC) applications. However, current approaches have limitations that prevent them from fully leveraging lossy compression, and they may also result in task collisions, which restrict the overall performance of HPC applications. To address these issues, we propose an optimization approach for the task scheduling problem that encompasses computation, compression, and I/O. Our algorithm adaptively selects the optimal compression and I/O queue to minimize the performance degradation of the computation. We also introduce an intra-node I/O workload balancing mechanism that evenly distributes the workload across different processes. Additionally, we design a framework that incorporates fine-grained compression, a compressed data buffer, and a shared Huffman tree to fully benefit from our proposed task scheduling. Experimental results with up to 16 nodes and 64 GPUs from ORNL Summit, as well as real-world HPC applications, demonstrate that our solution reduces I/O overhead by up to 3.8×× and 2.6×× compared to non-compression and asynchronous I/O solutions, respectively.
Fichier principal
Vignette du fichier
main.pdf (1.15 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04225758 , version 1 (30-09-2024)

Licence

Identifiants

  • HAL Id : hal-04225758 , version 1

Citer

Sian Jin, Sheng Di, Frédéric Vivien, Daoce Wang, Yves Robert, et al.. Concealing Compression-accelerated I/O for HPC Applications through In Situ Task Scheduling. EuroSys 2024, Apr 2024, Athens, Greece. ⟨hal-04225758⟩
97 Consultations
38 Téléchargements

Partager

More