2017/08/15 by Marian Daun, Daun, Marian, Carolin Hübscher +3
Computer Science · #Computers and Society (cs.CY) #FOS: Computer and information sciences #Open Source Software Innovations #Software Engineering (cs.SE) #Software Engineering Research #Software Engineering Techniques and Practices #cs.CY #cs.SE
paper · pdf · doi:10.48550/arxiv.1708.04662
arxiv created 2017/08/15 · openalex publication_date 2017/08/15 · arxiv updated 2017/08/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
[Context] In software engineering research, emphasis is given to sound evaluations of new approaches. While industry surveys or industrial case studies are preferred to evaluate industrial applicability, controlled experiments with student participants are commonly used to determine measurements such as effectiveness and efficiency of a proposed approach. [Objectives] In this paper, we elaborate on the current state of the art of controlled experiments using student participants. As student participants are commonly only reluctantly accepted in scientific communities and threats regarding the generalizability are quite obvious, we want to determine how widespread controlled experiments with student participants are and in which settings they are used. [Methods] This paper reports on a systematic mapping study using high-quality journals and conferences from the software engineering field as data sources. We scanned all papers published between 2010 and 2014 and investigated all papers reporting student experiments in detail. [Results] From 2788 papers under investigation 175 report results from controlled experiments. 109 (62.29%) of these controlled experiments have been conducted with student participants. Most experiments used undergraduate student participants, recruited students on a voluntary basis, and set them tasks to measure their comprehension. However, many experiments lack information regarding the students' recruitment and other important factors. [Conclusions] In conclusion, student participation in software engineering experiments can be seen as a common evaluation approach. In contrast, there seems to be little knowledge about the threats to validity in student experiments, as major drivers such as the recruitment are not reported at all.