Enabling Agile Analysis of I/O Performance Data with PyDarshan - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Enabling Agile Analysis of I/O Performance Data with PyDarshan

Jakob Luettgau
Shane Snyder
Tyler Reddy
Nikolaus Awtrey
Kevin Harms
Rui Wang
Rob Latham
Philip Carns

Résumé

Modern scientific applications utilize numerous software and hardware layers to efficiently access data. This approach poses a challenge for I/O optimization because of the need to instrument and correlate information across those layers. The Darshan characterization tool seeks to address this challenge by providing efficient, transparent, and compact runtime instrumentation of many common I/O interfaces. It also includes command-line tools to generate actionable insights and summary reports. However, the extreme diversity of today’s scientific applications means that not all applications are well served by one-size-fits-all analysis tools. In this work we present PyDarshan, a Python-based library that enables agile analysis of I/O performance data. PyDarshan caters to both novice and advanced users by offering ready-to-use HTML reports as well as a rich collection of APIs to facilitate custom analyses. We present the design of PyDarshan and demonstrate its effectiveness in four diverse real-world analysis use cases.
Fichier principal
Vignette du fichier
pydarshan_paper-5.pdf (2.66 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04390702 , version 1 (12-01-2024)

Licence

Paternité

Identifiants

Citer

Jakob Luettgau, Shane Snyder, Tyler Reddy, Nikolaus Awtrey, Kevin Harms, et al.. Enabling Agile Analysis of I/O Performance Data with PyDarshan. SC-W 2023 - Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, Nov 2023, Denver CO, United States. pp.1380-1391, ⟨10.1145/3624062.3624207⟩. ⟨hal-04390702⟩
40 Consultations
6 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More