On the Usability of Shortest Remaining Time First Policy in Shared Hadoop Clusters - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2016

On the Usability of Shortest Remaining Time First Policy in Shared Hadoop Clusters

Abstract

Hadoop has been recently used to process a diverse variety of applications, sharing the same execution infrastructure. A practical problem facing the Hadoop community is how to reduce job makespans by reducing job waiting times and ex- ecution times. Previous Hadoop schedulers have focused on improving job execution times, by improving data locality but not considering job waiting times. Even worse, enforcing data locality according to the job input sizes can be ineffi- cient: it can lead to long waiting times for small yet short jobs when sharing the cluster with jobs with smaller input sizes but higher execution complexity. This paper presents hSRTF, an adaption of the well-known Shortest Remaining Time First scheduler (i.e., SRTF) in shared Hadoop clus- ters. hSRTF embraces a simple model to estimate the re- maining time of a job and a preemption primitive (i.e., kill) to free the resources when needed. We have implemented hSRTF and performed extensive evaluations with Hadoop on the Grid’5000 testbed. The results show that hSRTF can significantly reduce the waiting times of small jobs and therefore improves their makespans, but at the cost of a rel- atively small increase in the makespans of large jobs. For instance, a time-based proportional share mode of hSRTF (i.e., hSRTF-Pr) speeds up small jobs by (on average) 45% and 26% while introducing a performance degradation for large jobs by (on average) 10% and 0.2% compared to Fifo and Fair schedulers, respectively.
Fichier principal
Vignette du fichier
paper.pdf (164.61 Ko) Télécharger le fichier
Origin Files produced by the author(s)
Loading...

Dates and versions

hal-01239341 , version 1 (03-02-2016)

Identifiers

Cite

Nathanaël Cheriere, Pierre Donat-Bouillud, Shadi Ibrahim, Matthieu Simonin. On the Usability of Shortest Remaining Time First Policy in Shared Hadoop Clusters. SAC 2016-The 31st ACM/SIGAPP Symposium on Applied Computing, Apr 2016, Pisa, Italy. ⟨10.1145/2851613.2851626⟩. ⟨hal-01239341⟩
538 View
438 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More