On the Optimization of Iterative Programming with Distributed Data Collections
Résumé
Big data programming frameworks are becoming increasingly important
for the development of applications for which performance and
scalability are critical. In those complex frameworks, optimizing
code by hand is hard and time-consuming, making automated
optimization particularly necessary. In order to automate
optimization, a prerequisite is to find suitable abstractions to
represent programs; for instance, algebras based on monads or
monoids to represent distributed data collections. Currently,
however, such algebras do not represent recursive programs in a way
which allows for analyzing or rewriting them. In this paper, we extend a
monoid algebra with a fixpoint operator for representing recursion
as a first class citizen and show how it enables new optimizations.
Experiments with the Spark platform illustrate performance gains
brought by these systematic optimizations.
Origine | Fichiers produits par l'(les) auteur(s) |
---|