Observatorio de I+D+i UPM

Memorias de investigación
Communications at congresses:
Diarchy: An Optimized Management Approach for MapReduce Masters
Year:2015
Research Areas
  • Mathematics,
  • Information technology and adata processing
Information
Abstract
The MapReduce community is progressively replacing the classic Hadoop with Yarn, the second- generation Hadoop (MapReduce 2.0). This transition is being made due to many reasons, but primarily because of some scalability drawbacks of the classic Hadoop. The new framework has appropriately addressed this issue and is being praised for its multi-functionality. In this paper we carry out a probabilistic analysis that emphasizes some reliability concerns of Yarn at the job master level. This is a critical point, since the failures of a job master involves the failure of all the workers managed by such a master. In this paper, we propose Diarchy, a novel system for the management of job masters. Its aim is to increase the reliability of Yarn, based on the sharing and backup of responsibilities between two masters working as peers. The evaluation results show that Diarchy outperforms the reliability performance of Yarn in different setups, regardless of cluster size, type of job, or average failure rate and suggest a positive impact of this approach compared to the traditional, single-master Hadoop architecture.
International
Si
Congress
International Conference On Computational Science (ICCS)
960
Place
Reykjavík, Iceland
Reviewers
Si
ISBN/ISSN
1877-0509
10.1016/j.procs.2015.05.179
Start Date
01/06/2015
End Date
03/06/2015
From page
9
To page
18
International Conference On Computational Science, ICCS 2015 ? Computational Science at the Gates of Nature
Participants
  • Autor: Bunjamin Memishi (UPM)
  • Autor: Maria de los Santos Perez Hernandez (UPM)
  • Autor: Gabriel Antoniu (INRIA)
Research Group, Departaments and Institutes related
  • Creador: Grupo de Investigación: Ontology Engineering Group
  • Departamento: Arquitectura y Tecnología de Sistemas Informáticos
S2i 2019 Observatorio de investigación @ UPM con la colaboración del Consejo Social UPM
Cofinanciación del MINECO en el marco del Programa INNCIDE 2011 (OTR-2011-0236)
Cofinanciación del MINECO en el marco del Programa INNPACTO (IPT-020000-2010-22)