Observatorio de I+D+i UPM

Memorias de investigación
Ponencias en congresos:
KerA: Scalable Data Ingestion for Stream Processing
Año:2018
Áreas de investigación
  • Ciencias de la computación y tecnología informática
Datos
Descripción
Big Data applications are increasingly moving from batch-oriented execution models to stream-based models that enable them to extract valuable insights close to real-time. To support this model, an essential part of the streaming processing pipeline is data ingestion, i.e., the collection of data from various sources (sensors, NoSQL stores, filesystems, etc.) and their delivery for processing. Data ingestion needs to support high throughput, low latency and must scale to a large number of both data producers and consumers. Since the overall performance of the whole stream processing pipeline is limited by that of the ingestion phase, it is critical to satisfy these performance goals. However, state-of-art data ingestion systems such as Apache Kafka build on static stream partitioning and offset-based record access, trading performance for design simplicity. In this paper we propose KerA, a data ingestion framework that alleviate the limitations of state-of-art thanks to a dynamic partitioning scheme and to lightweight indexing, thereby improving throughput, latency and scalability. Experimental evaluations show that KerA outperforms Kafka up to 4x for ingestion throughput and up to 5x for the overall stream processing throughput. Furthermore, they show that KerA is capable of delivering data fast enough to saturate the big data engine acting as the consumer.
Internacional
Si
Nombre congreso
38th IEEE International Conference on Distributed Computing Systems (ICDCS)
Tipo de participación
960
Lugar del congreso
Viena, Austria
Revisores
Si
ISBN o ISSN
2575-8411
DOI
10.1109/ICDCS.2018.00152
Fecha inicio congreso
02/07/2018
Fecha fin congreso
05/07/2018
Desde la página
1480
Hasta la página
1485
Título de las actas
Proceedings 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)
Esta actividad pertenece a memorias de investigación
Participantes
  • Autor: Maria de los Santos Perez Hernandez (UPM)
Grupos de investigación, Departamentos, Centros e Institutos de I+D+i relacionados
  • Creador: Grupo de Investigación: Ontology Engineering Group
  • Departamento: Arquitectura y Tecnología de Sistemas Informáticos
S2i 2021 Observatorio de investigación @ UPM con la colaboración del Consejo Social UPM
Cofinanciación del MINECO en el marco del Programa INNCIDE 2011 (OTR-2011-0236)
Cofinanciación del MINECO en el marco del Programa INNPACTO (IPT-020000-2010-22)