Mapping Large Memory-constrained Workflows onto Heterogeneous Platforms - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2024

Mapping Large Memory-constrained Workflows onto Heterogeneous Platforms

Résumé

Scientificworkflows are often represented as directed acyclic graphs (DAGs), where vertices correspond to tasks and edges represent the dependencies between them. Since these graphs are often large in both the number of tasks and their resource requirements, it is important to schedule them efficiently on parallel or distributed compute systems. Typically, each task requires a certain amount of memory to be executed and needs to communicate data to its successor tasks. The goal is thus to execute the workflow as fast as possible (i.e., to minimize its makespan) while satisfying the memory constraints. Hence, we investigate the partitioning and mapping of DAGshaped workflows onto heterogeneous platforms where each processor can have a different speed and a different memory size. We first propose a baseline algorithm in the absence of existing memoryaware solutions. As our main contribution, we then present a fourstep heuristic. Its first step is to partition the input DAG into smaller blocks with an existing DAG partitioner. The next two steps adapt the resulting blocks of the DAG to fit the processor memories and optimize for the overall makespan by further splitting and merging these blocks. Finally, we use local search via block swaps to further improve the makespan. Our experimental evaluation on real-world and simulated workflows with up to 30,000 tasks shows that exploiting the heterogeneity with the four-step heuristic reduces the makespan by a factor of 2.44 on average (even more on large workflows), compared to the baseline that ignores heterogeneity.
Fichier principal
Vignette du fichier
2407.09077v1.pdf (1.09 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04767107 , version 1 (12-11-2024)

Licence

Identifiants

Citer

Svetlana Kulagina, Henning Meyerhenke, Anne Benoit. Mapping Large Memory-constrained Workflows onto Heterogeneous Platforms. ICPP 2024 - 53rd International Conference on Parallel Processing, Aug 2024, Gotland, Sweden. pp.305-316, ⟨10.1145/3673038.3673068⟩. ⟨hal-04767107⟩
13 Consultations
6 Téléchargements

Altmetric

Partager

More