Observatorio de I+D+i UPM

Memorias de investigación
Ponencias en congresos:
Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review (CORE A*)
Año:2018
Áreas de investigación
  • Ciencias de la computación y tecnología informática
Datos
Descripción
Background: Statistical concepts and techniques are often applied incorrectly, even in mature disciplines such as medicine or psychology. Surprisingly, there are very few works that study statistical problems in software engineering (SE). Aim: Assess the existence of statistical errors in SE experiments. Method: Compile the most common statistical errors in experimental disciplines. Survey experiments published in ICSE to assess whether errors occur in high quality SE publications. Results: The same errors identified in others disciplines were found in ICSE experiments, exhibiting rather large prevalences, over 30% of the reviewed papers in several types of errors such as: a) Missing statistical hypotheses, b) missing sample size calculation, c) failure to assess statistical tests assumptions, and d) uncorrected multiple testing. When experiments restrict to the validation section of a larger research paper, the prevalence of errors increases. The origin of the errors can be traced back to: a) Researchers? inadequate statistical training, and, b) abundance of exploratory research. Conclusions: This paper provides preliminary evidence that SE research suffers the same statistical problems than other experimental disciplines. However, SE community does not seem to be aware of the existence of shortcomings in their experiments, whereas other disciplines work hard to avoid them. Further research is necessary to find the underlying causes and set corrective measures, but at the outset some actions could be effective: a) Improve the statistical training of SE researchers, and b) enforce quality assessment and reporting guidelines in SE publications.
Internacional
Si
Nombre congreso
The International Conference on Software Engineering (ICSE) - CORE A*
Tipo de participación
730
Lugar del congreso
Gothenburg, Sweden
Revisores
Si
ISBN o ISSN
978-1-4503-3900-1
DOI
Fecha inicio congreso
27/05/2018
Fecha fin congreso
03/06/2018
Desde la página
Hasta la página
Título de las actas
Esta actividad pertenece a memorias de investigación
Participantes
  • Autor: Rolando Patricio Reyes Chicango (UPM)
  • Autor: Oscar Dieste Tubio (UPM)
  • Autor: Efraín Rodrigo Fonseca Carrera (UPM)
  • Autor: Natalia Juristo Juzgado (UPM)
Grupos de investigación, Departamentos, Centros e Institutos de I+D+i relacionados
  • Creador: Departamento: Lenguajes y Sistemas Informáticos e Ingeniería de Software
S2i 2021 Observatorio de investigación @ UPM con la colaboración del Consejo Social UPM
Cofinanciación del MINECO en el marco del Programa INNCIDE 2011 (OTR-2011-0236)
Cofinanciación del MINECO en el marco del Programa INNPACTO (IPT-020000-2010-22)