Toward Stream Processing Elasticity in Realistic Geo-Distributed Environments - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2024

Toward Stream Processing Elasticity in Realistic Geo-Distributed Environments

Résumé

Stream data processing is a widely used technology for analysing IoT-generated data shortly after being produced, and delivering timely insights about them. Executing such analysis in geo-distributed platforms enables shorter delays between data production and processing and fewer disturbances due to potential instability of long-distance networks, while retaining the ability to scale the processing capacity up and down according to the demand. However, current stream processing engines were designed for environments made of homogeneous servers connected together using high-speed network links. We experimentally study the performance of Apache Flink coupled with the Gesscale auto-scaler in conditions which resemble those of geo-distributed platforms. We demonstrate that Flink’s backpressure mechanism should not be used as the only trigger for rescaling operations in heterogeneous network conditions. Raw performance, as well as performance predictability, also degrade quickly in the presence of stateful data processing operators and/or high network latency between the processing nodes.
Fichier principal
Vignette du fichier
main.pdf (990.16 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04655408 , version 1 (22-07-2024)
hal-04655408 , version 2 (19-08-2024)

Licence

Identifiants

  • HAL Id : hal-04655408 , version 2

Citer

Khaled Arsalane, Guillaume Pierre, Shadi Ibrahim. Toward Stream Processing Elasticity in Realistic Geo-Distributed Environments. IC2E 2024 - 12th IEEE International Conference on Cloud Engineering, IEEE, Sep 2024, Paphos, Cyprus. pp.1-9. ⟨hal-04655408v2⟩
175 Consultations
123 Téléchargements

Partager

More