Observatorio de I+D+i UPM

Memorias de investigación
Communications at congresses:
Difficulties in Running Experiments in the Software Industry: Experiences from the Trenches
Year:2015
Research Areas
  • Computer systems
Information
Abstract
Controlled experiments in laboratory settings are relatively commonplace in software engineering, but experiments in industry are thin on the ground. Of the few existing cases, most are 1-1 (running one experiment at one company), just a few are n-1 (running n experiments at one company) and still fewer are 1-n (running one and the same experiment at n companies). In this paper we report the difficulties that we experienced running the same experiment at several companies. We ran the same experiment in five different settings at three companies, and the results were transferred to each company so that they could exploit the resulting evidence in their decision-making process. We have found that: 1) it was relatively easy to get companies involved; 2) they did not cooperate as much as they had agreed to in the project proposal; 3) our industrial environments imposed many more constraints on the experimental design than laboratory environments; 4) professionals were less motivated than students; 5) the reliability of the results could be compromised by subject characteristics and behaviour; and 6) experiment findings could not be transferred using just the standard reporting guidelines that are used for scientific articles.
International
Si
Congress
2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry (CESI)
960
Place
Florence, Italy
Reviewers
Si
ISBN/ISSN
978-1-4673-7028-8
10.1109/CESI.2015.8
Start Date
18/05/2015
End Date
18/05/2015
From page
3
To page
9
Difficulties in Running Experiments in the Software Industry: Experiences from the Trenches
Participants
  • Autor: Sira Vegas Hernandez (UPM)
  • Autor: Oscar Dieste Tubio (UPM)
  • Autor: Natalia Juristo Juzgado (UPM)
Research Group, Departaments and Institutes related
  • Creador: Grupo de Investigación: Ingeniería del Software
  • Departamento: Lenguajes y Sistemas Informáticos e Ingeniería de Software
S2i 2019 Observatorio de investigación @ UPM con la colaboración del Consejo Social UPM
Cofinanciación del MINECO en el marco del Programa INNCIDE 2011 (OTR-2011-0236)
Cofinanciación del MINECO en el marco del Programa INNPACTO (IPT-020000-2010-22)