Understanding the Performance of Erasure Codes in Hadoop Distributed File System - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2022

Understanding the Performance of Erasure Codes in Hadoop Distributed File System

Résumé

Replication has been successfully employed and practiced to ensure high data availability in large-scale distributed storage systems. However, with the relentless growth of gen- erated and collected data, replication has become expensive not only in terms of storage cost but also in terms of network cost and hardware cost. Traditionally, erasure coding (EC) is employed as a cost-efficient alternative to replication when high access latency to the data can be tolerated. However, with the continuous reduction in its CPU overhead, EC is performed on the critical path of data access. For instance, EC has been integrated into the last major release of Hadoop Distributed File System (HDFS) which is the primary storage backend for data analytic frameworks (e.g., Hadoop, Spark, etc.). In this work, we measure and compare the performance of data accesses in HDFS under both replication and EC. Our analysis indicates that EC is a feasible solution for data- intensive applications and it can outperform replication in many scenarios. Furthermore, we demonstrate that it is the block placement algorithm in HDFS that mostly impacts the performance under EC.

Domaines

Informatique
Fichier principal
Vignette du fichier
cheops-2022-CR.pdf (749.91 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03890398 , version 1 (26-12-2022)

Licence

Identifiants

Citer

Jad Darrous, Shadi Ibrahim. Understanding the Performance of Erasure Codes in Hadoop Distributed File System. CHEOPS 22 - Proceedings of the Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems, Apr 2022, Rennes, France. pp.24-32, ⟨10.1145/3503646.3524296⟩. ⟨hal-03890398⟩
131 Consultations
892 Téléchargements

Altmetric

Partager

More