Observatorio de I+D+i UPM

Memorias de investigación
Communications at congresses:
Exploring Shared State in Key-Value Store for Window-Based Multi-Pattern Streaming
Year:2017
Research Areas
  • Information technology and adata processing
Information
Abstract
We are now witnessing an unprecedented growth of data that needs to be processed at always increasing rates in order to extract valuable insights. Big Data streaming analytics tools have been developed to cope with the online dimension of data processing: they enable real-time handling of live data sources by means of stateful aggregations (operators). Current state-of-art frameworks (e.g. Apache Flink [1]) enable each operator to work in isolation by creating data copies, at the expense of increased memory utilization. In this paper, we explore the feasibility of deduplication techniques to address the challenge of reducing memory footprint for window-based stream processing without significant impact on performance. We design a deduplication method specifically for windowbased operators that rely on key-value stores to hold a shared state. We experiment with a synthetically generated workload while considering several deduplication scenarios and based on the results, we identify several potential areas of improvement. Our key finding is that more fine-grained interactions between streaming engines and (key-value) stores need to be designed in order to better respond to scenarios that have to overcome memory scarcity.
International
Si
Congress
17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing
960
Place
Madrid, España
Reviewers
Si
ISBN/ISSN
978-1-5090-6610-0
10.1109/CCGRID.2017.126
Start Date
14/05/2017
End Date
17/05/2017
From page
1044
To page
1052
Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing
Participants
  • Autor: Maria de los Santos Perez Hernandez (UPM)
Research Group, Departaments and Institutes related
  • Creador: Grupo de Investigación: Ontology Engineering Group
  • Departamento: Arquitectura y Tecnología de Sistemas Informáticos
S2i 2019 Observatorio de investigación @ UPM con la colaboración del Consejo Social UPM
Cofinanciación del MINECO en el marco del Programa INNCIDE 2011 (OTR-2011-0236)
Cofinanciación del MINECO en el marco del Programa INNPACTO (IPT-020000-2010-22)