2024/03/02 by Martin P. Robillard, Deeksha M. Arya, Neil A. Ernst +9 · 3 citations
Computer Science · Decision Sciences · #Scientific Computing and Data Management #Software Engineering Research #Software Engineering Techniques and Practices
paper · doi:10.1145/3649598
openalex publication_date 2024/03/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30
Reflecting on the limitations of a study is a crucial part of the research process. In software engineering studies, this reflection is typically conveyed through discussions of study limitations or threats to validity. In current practice, such discussions seldom provide sufficient insight to understand the rationale for decisions taken before and during the study, and their implications. We revisit the practice of discussing study limitations and threats to validity and identify its weaknesses. We propose to refocus this practice of self-reflection to a discussion centered on the notion of trade-offs . We argue that documenting trade-offs allows researchers to clarify how the benefits of their study design decisions outweigh the costs of possible alternatives. We present guidelines for reporting trade-offs in a way that promotes a fair and dispassionate assessment of researchers’ work.