Enabling Agile Analysis of I/O Performance Data with PyDarshan - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Enabling Agile Analysis of I/O Performance Data with PyDarshan

Jakob Luettgau
Shane Snyder
Tyler Reddy
Nikolaus Awtrey
Kevin Harms
Rui Wang
Rob Latham
Philip Carns

Abstract

Modern scientific applications utilize numerous software and hardware layers to efficiently access data. This approach poses a challenge for I/O optimization because of the need to instrument and correlate information across those layers. The Darshan characterization tool seeks to address this challenge by providing efficient, transparent, and compact runtime instrumentation of many common I/O interfaces. It also includes command-line tools to generate actionable insights and summary reports. However, the extreme diversity of today’s scientific applications means that not all applications are well served by one-size-fits-all analysis tools. In this work we present PyDarshan, a Python-based library that enables agile analysis of I/O performance data. PyDarshan caters to both novice and advanced users by offering ready-to-use HTML reports as well as a rich collection of APIs to facilitate custom analyses. We present the design of PyDarshan and demonstrate its effectiveness in four diverse real-world analysis use cases.
Fichier principal
Vignette du fichier
pydarshan_paper-5.pdf (2.66 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04390702 , version 1 (12-01-2024)

Licence

Attribution

Identifiers

Cite

Jakob Luettgau, Shane Snyder, Tyler Reddy, Nikolaus Awtrey, Kevin Harms, et al.. Enabling Agile Analysis of I/O Performance Data with PyDarshan. SC-W 2023 - Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, Nov 2023, Denver CO, United States. pp.1380-1391, ⟨10.1145/3624062.3624207⟩. ⟨hal-04390702⟩
40 View
6 Download

Altmetric

Share

Gmail Facebook X LinkedIn More