KerA: Scalable Data Ingestion for Stream Processing - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2018

KerA: Scalable Data Ingestion for Stream Processing


Big Data applications are increasingly moving from batch-oriented execution models to stream-based models that enable them to extract valuable insights close to real-time. To support this model, an essential part of the streaming processing pipeline is data ingestion, i.e., the collection of data from various sources (sensors, NoSQL stores, filesystems, etc.) and their delivery for processing. Data ingestion needs to support high throughput, low latency and must scale to a large number of both data producers and consumers. Since the overall performance of the whole stream processing pipeline is limited by that of the ingestion phase, it is critical to satisfy these performance goals. However, state-of-art data ingestion systems such as Apache Kafka build on static stream partitioning and offset-based record access, trading performance for design simplicity. In this paper we propose KerA, a data ingestion framework that alleviate the limitations of state-of-art thanks to a dynamic partitioning scheme and to lightweight indexing, thereby improving throughput, latency and scalability. Experimental evaluations show that KerA outperforms Kafka up to 4x for ingestion throughput and up to 5x for the overall stream processing throughput. Furthermore, they show that KerA is capable of delivering data fast enough to saturate the big data engine acting as the consumer.
Fichier principal
Vignette du fichier
ICDCS_2018_paper_732.pdf (654.62 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-01773799 , version 1 (23-04-2018)





Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, María S Pérez-Hernández, Bogdan Nicolae, et al.. KerA: Scalable Data Ingestion for Stream Processing. ICDCS 2018 - 38th IEEE International Conference on Distributed Computing Systems, Jul 2018, Vienna, Austria. pp.1480-1485, ⟨10.1109/ICDCS.2018.00152⟩. ⟨hal-01773799⟩
688 View
1033 Download



Gmail Mastodon Facebook X LinkedIn More