Self-configuration of the Number of Concurrently Running MapReduce Jobs in a Hadoop Cluster - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2015

Self-configuration of the Number of Concurrently Running MapReduce Jobs in a Hadoop Cluster

Résumé

There is a trade-off between the number of concurrently running MapReduce jobs and their corresponding map and reduce tasks within a node in a Hadoop cluster. Leaving this trade-off statically configured to a single value can significantly reduce job response times leaving only suboptimal resource usage. To overcome this problem, we propose a feedback control loop based approach that dynamically adjusts the Hadoop resource manager configuration based on the current state of the cluster. The preliminary assessment based on workloads synthesized from real-world traces shows that the system performance can be improved by about 30% compared to default Hadoop setup.
Fichier principal
Vignette du fichier
icac15-paper.pdf (264.52 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01143157 , version 1 (06-05-2015)

Identifiants

  • HAL Id : hal-01143157 , version 1

Citer

Bo Zhang, Filip Křikava, Romain Rouvoy, Lionel Seinturier. Self-configuration of the Number of Concurrently Running MapReduce Jobs in a Hadoop Cluster. ICAC 2015, Jul 2015, Grenoble, France. pp.149-150. ⟨hal-01143157⟩
374 Consultations
305 Téléchargements

Partager

More