Observatorio de I+D+i UPM

Memorias de investigación
Communications at congresses:
Towards a Unified Storage and Ingestion Architecture for Stream Processing
Year:2017
Research Areas
  • Information technology and adata processing
Information
Abstract
Big Data applications are rapidly moving from a batch-oriented execution model to a streaming execution model in order to extract value from the data in real-time. However, processing live data alone is often not enough: in many cases, such applications need to combine the live data with previously archived data to increase the quality of the extracted insights. Current streaming-oriented runtimes and middlewares are not flexible enough to deal with this trend, as they address ingestion (collection and pre-processing of data streams) and persistent storage (archival of intermediate results) using separate services. This separation often leads to I/O redundancy (e.g., write data twice to disk or transfer data twice over the network) and interference (e.g., I/O bottlenecks when collecting data streams and writing archival data simultaneously). In this position paper, we argue for a unified ingestion and storage architecture for streaming data that addresses the aforementioned challenge. We identify a set of constraints and benefits for such a unified model, while highlighting the important architectural aspects required to implement it in real life. Based on these aspects, we briefly sketch our plan for future work that develops the position defended in this paper.
International
Si
Congress
Second workshop on Real-time and stream processing in Big Data, IEEE International Conference on Big Data
960
Place
Boston, Estados Unidos de América
Reviewers
Si
ISBN/ISSN
CDP08UPM
10.1109/BigData.2017.8258196
Start Date
11/12/2017
End Date
14/12/2017
From page
2402
To page
2407
Proceedings of the Second workshop on Real-time and stream processing in Big Data, IEEE International Conference on Big Data.
Participants
  • Autor: Maria de los Santos Perez Hernandez (UPM)
Research Group, Departaments and Institutes related
  • Creador: Grupo de Investigación: Ontology Engineering Group
  • Departamento: Arquitectura y Tecnología de Sistemas Informáticos
S2i 2019 Observatorio de investigación @ UPM con la colaboración del Consejo Social UPM
Cofinanciación del MINECO en el marco del Programa INNCIDE 2011 (OTR-2011-0236)
Cofinanciación del MINECO en el marco del Programa INNPACTO (IPT-020000-2010-22)