Efficient Iterative Programs with Distributed Data Collections - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Preprints, Working Papers, ... Year : 2023

Efficient Iterative Programs with Distributed Data Collections

Abstract

Big data programming frameworks have become increasingly important for the development of applications for which performance and scalability are critical. In those complex frameworks, optimizing code by hand is hard and time-consuming, making automated optimization particularly necessary. In order to automate optimization, a prerequisite is to find suitable abstractions to represent programs; for instance, algebras based on monads or monoids to represent distributed data collections. Currently, however, such algebras do not represent recursive programs in a way which allows for analyzing or rewriting them. In this paper, we extend a monoid algebra with a fixpoint operator for representing recursion as a first class citizen and show how it enables new optimizations. Experiments with the Spark platform illustrate performance gains brought by these systematic optimizations.
Fichier principal
Vignette du fichier
paper.pdf (964.62 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04108082 , version 1 (26-05-2023)

Licence

Identifiers

  • HAL Id : hal-04108082 , version 1

Cite

Sarah Chlyah, Nils Gesbert, Pierre Genevès, Nabil Layaïda. Efficient Iterative Programs with Distributed Data Collections. 2023. ⟨hal-04108082⟩
25 View
24 Download

Share

Gmail Mastodon Facebook X LinkedIn More