Self-configuration of the Number of Concurrently Running MapReduce Jobs in a Hadoop Cluster - Inria - Institut national de recherche en sciences et technologies du numérique
Conference Papers Year : 2015

Self-configuration of the Number of Concurrently Running MapReduce Jobs in a Hadoop Cluster

Abstract

There is a trade-off between the number of concurrently running MapReduce jobs and their corresponding map and reduce tasks within a node in a Hadoop cluster. Leaving this trade-off statically configured to a single value can significantly reduce job response times leaving only suboptimal resource usage. To overcome this problem, we propose a feedback control loop based approach that dynamically adjusts the Hadoop resource manager configuration based on the current state of the cluster. The preliminary assessment based on workloads synthesized from real-world traces shows that the system performance can be improved by about 30% compared to default Hadoop setup.
Fichier principal
Vignette du fichier
icac15-paper.pdf (264.52 Ko) Télécharger le fichier
Origin Files produced by the author(s)
Loading...

Dates and versions

hal-01143157 , version 1 (06-05-2015)

Identifiers

  • HAL Id : hal-01143157 , version 1

Cite

Bo Zhang, Filip Křikava, Romain Rouvoy, Lionel Seinturier. Self-configuration of the Number of Concurrently Running MapReduce Jobs in a Hadoop Cluster. ICAC 2015, Jul 2015, Grenoble, France. pp.149-150. ⟨hal-01143157⟩
371 View
301 Download

Share

More