2022/12/29 by Carlos Ansótegui, Ansotegui, Carlos, Eduard Torres +1
Computer Science · #Advanced Software Engineering Methodologies #Artificial Intelligence (cs.AI) #Distributed systems and fault tolerance #FOS: Computer and information sciences #Software Engineering (cs.SE) #Software Testing and Debugging Techniques
paper · pdf · doi:10.48550/arxiv.2301.08134
openalex publication_date 2022/12/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Combinatorial Testing (CT) tools are essential to test properly a wide range of systems (train systems, Graphical User Interfaces (GUIs), autonomous driving systems, etc). While there is an active research community working on developing CT tools, paradoxically little attention has been paid to making available enough resources to test the CT tools themselves. In particular, the set of available benchmarks to asses their correctness, effectiveness and efficiency is rather limited. In this paper, we introduce a new generator of CT benchmarks that essentially borrows the structure contained in the plethora of available Combinatorial Problems from other research communities in order to create meaningful benchmarks. We additionally perform an extensive evaluation of CT tools with these new benchmarks. Thanks to this study we provide some insights on under which circumstances a particular CT tool should be used.