2022/10/11 by Rasmus Ros, Ros, Rasmus, Elizabeth Bjarnason +3
Computer Science · #FOS: Computer and information sciences #Open Source Software Innovations #Software Engineering (cs.SE) #Software Engineering Research #Software Engineering Techniques and Practices
paper · pdf · doi:10.48550/arxiv.2210.05192
openalex publication_date 2022/10/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Continuous experimentation (CE) is used by many companies with internet-facing products to improve their software based on user data. Some companies deliberately adopt an experiment-driven approach to software development while some companies use CE in a more ad-hoc fashion. The goal of the study is to identify factors that explain the variations in the utility and efficacy of CE between different companies. We conducted a multi-case study of 12 companies involved with CE and performed 27 interviewees with practitioners at these companies. Based on that empirical data, we then built a theory of factors at play in CE. We introduce a theory of Factors Affecting Continuous Experimentation (FACE). The theory includes three factors, namely 1) processes and infrastructure for CE, 2) the user problem complexity of the product offering, and 3) incentive structures for CE. It explains how these factors affect the effectiveness of CE and its ability to achieve problem-solution and product-market fit. Our theory can be used by practitioners to assess an organisation's potential for adopting CE, as well as, identifying factors which pose challenges in gaining value from CE practices. Our results also provide a starting point for further research on how contextual factors affect CE and how these may be mitigated.