Mission Possible: Unify HPC and Big Data Stacks Towards Application-Defined Blobs at the Storage Layer - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles Future Generation Computer Systems Year : 2020

Mission Possible: Unify HPC and Big Data Stacks Towards Application-Defined Blobs at the Storage Layer

Abstract

HPC and Big Data stacks are completely separated today. The storage layer offers opportunities for convergence, as the challenges associated with HPC and Big Data storage are similar: trading versatility for performance. This motivates a global move towards dropping file-based, POSIX-IO compliance systems. However, on HPC platforms this is made difficult by the centralized storage architecture using file-based storage. In this paper we advocate that the growing trend of equipping HPC compute nodes with local storage redistributes the cards by enabling object storage to be deployed alongside the application on the compute nodes. Such integration of application and storage not only allows fine-grained configuration of the storage system, but also improves application portability across platforms. In addition, the single-user nature of such application-specific storage obviates the need for resource-consuming storage features like permissions or file hierarchies offered by traditional file systems. In this article we propose and evaluate Blobs (Binary Large Objects) as an alternative to distributed file systems. We factually demonstrate that it offers drop-in compatibility with a variety of existing applications while improving storage throughput by up to 28%.
Fichier principal
Vignette du fichier
HPC_BD_Convergence__Elsevier_-2.pdf (167.41 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-01892682 , version 1 (10-10-2018)

Identifiers

Cite

Pierre Matri, Yevhen Alforov, Alvaro Brandon, María Pérez, Alexandru Costan, et al.. Mission Possible: Unify HPC and Big Data Stacks Towards Application-Defined Blobs at the Storage Layer. Future Generation Computer Systems, 2020, 109, pp.668-677. ⟨10.1016/j.future.2018.07.035⟩. ⟨hal-01892682⟩
486 View
270 Download

Altmetric

Share

Gmail Facebook X LinkedIn More