Fair multi-agent task allocation for large datasets analysis - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles Knowledge and Information Systems (KAIS) Year : 2017

Fair multi-agent task allocation for large datasets analysis

Abstract

MapReduce is a design pattern for processing large datasets distributed on a cluster. Its performances are linked to the data structure and the runtime environment. Indeed, data skew can yield an unfair task allocation, but even when the initial allocation produced by the partition function is well balanced, an unfair allocation can occur during the reduce phase due to the heterogeneous performance of nodes. For these reasons, we propose an adaptive multi-agent system. In our approach, the reducer agents interact during the job and the task reallocation is based on negotiation in order to decrease the workload of the most loaded reducer and so the runtime. In this paper, we propose and evaluate two negotiation strategies. Finally, we experiment our multi-agent system with real-world datasets over heterogeneous runtime environment.
Fichier principal
Vignette du fichier
morge17kais.pdf (2.42 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-01567428 , version 1 (29-09-2023)

Identifiers

Cite

Quentin Baert, Anne-Cécile Caron, Maxime Morge, Jean-Christophe Routier. Fair multi-agent task allocation for large datasets analysis. Knowledge and Information Systems (KAIS), 2017, 54 (3), pp.591-615. ⟨10.1007/s10115-017-1087-4⟩. ⟨hal-01567428⟩
223 View
2 Download

Altmetric

Share

Gmail Facebook X LinkedIn More