Understanding the Performance of Erasure Codes in Hadoop Distributed File System - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Understanding the Performance of Erasure Codes in Hadoop Distributed File System

Abstract

Replication has been successfully employed and practiced to ensure high data availability in large-scale distributed storage systems. However, with the relentless growth of gen- erated and collected data, replication has become expensive not only in terms of storage cost but also in terms of network cost and hardware cost. Traditionally, erasure coding (EC) is employed as a cost-efficient alternative to replication when high access latency to the data can be tolerated. However, with the continuous reduction in its CPU overhead, EC is performed on the critical path of data access. For instance, EC has been integrated into the last major release of Hadoop Distributed File System (HDFS) which is the primary storage backend for data analytic frameworks (e.g., Hadoop, Spark, etc.). In this work, we measure and compare the performance of data accesses in HDFS under both replication and EC. Our analysis indicates that EC is a feasible solution for data- intensive applications and it can outperform replication in many scenarios. Furthermore, we demonstrate that it is the block placement algorithm in HDFS that mostly impacts the performance under EC.
Fichier principal
Vignette du fichier
cheops-2022-CR.pdf (749.91 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03890398 , version 1 (26-12-2022)

Licence

Identifiers

Cite

Jad Darrous, Shadi Ibrahim. Understanding the Performance of Erasure Codes in Hadoop Distributed File System. CHEOPS 22 - Proceedings of the Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems, Apr 2022, Rennes, France. pp.24-32, ⟨10.1145/3503646.3524296⟩. ⟨hal-03890398⟩
104 View
580 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More