Optimization of Real-World MapReduce Applications With Flame-MR: Practical Use Cases - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles IEEE Access Year : 2018

Optimization of Real-World MapReduce Applications With Flame-MR: Practical Use Cases

Abstract

Apache Hadoop is a widely used MapReduce framework for storing and processing large amounts of data. However, it presents some performance issues that hinder its utilization in many practical use cases. Although existing alternatives like Spark or Hama can outperform Hadoop, they require to rewrite the source code of the applications due to API incompatibilities. This paper studies the use of Flame-MR, an in-memory processing architecture for MapReduce applications, to improve the performance of real-world use cases in a transparent way while keeping application compatibility. Flame-MR adapts to the characteristics of the workloads, managing efficiently the use of custom data formats and iterative computations, while also reducing workload imbalance. The experimental evaluation, conducted in high performance clusters and the Microsoft Azure cloud, shows a clear outperformance of Flame-MR over Hadoop. In most cases, Flame-MR reduces the execution times by more than a half.
Fichier principal
Vignette du fichier
2_FINAL_Article-2.pdf (606.5 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-01955503 , version 1 (14-12-2018)

Identifiers

Cite

Jorge Veiga, Roberto R Expósito, Bruno Raffin, Juan Tourino. Optimization of Real-World MapReduce Applications With Flame-MR: Practical Use Cases. IEEE Access, 2018, 6, pp.69750-69762. ⟨10.1109/ACCESS.2018.2880842⟩. ⟨hal-01955503⟩
124 View
132 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More