2019/03/15 by David A. Tomassi, Tomassi, David A., Naji Dmeiri +13 · 3 citations
Computer Science · #FOS: Computer and information sciences #Software Engineering (cs.SE) #Software Engineering Research #Software System Performance and Reliability #Software Testing and Debugging Techniques
paper · pdf · doi:10.48550/arxiv.1903.06725
openalex publication_date 2019/03/15 · openalex created_date 2022/07/13 · openalex updated_date 2026/07/28
Fault-detection, localization, and repair methods are vital to software\nquality; but it is difficult to evaluate their generality, applicability, and\ncurrent effectiveness. Large, diverse, realistic datasets of\ndurably-reproducible faults and fixes are vital to good experimental evaluation\nof approaches to software quality, but they are difficult and expensive to\nassemble and keep current. Modern continuous-integration (CI) approaches, like\nTravis-CI, which are widely used, fully configurable, and executed within\ncustom-built containers, promise a path toward much larger defect datasets. If\nwe can identify and archive failing and subsequent passing runs, the containers\nwill provide a substantial assurance of durable future reproducibility of build\nand test. Several obstacles, however, must be overcome to make this a practical\nreality. We describe BugSwarm, a toolset that navigates these obstacles to\nenable the creation of a scalable, diverse, realistic, continuously growing set\nof durably reproducible failing and passing versions of real-world, open-source\nsystems. The BugSwarm toolkit has already gathered 3,091 fail-pass pairs, in\nJava and Python, all packaged within fully reproducible containers.\nFurthermore, the toolkit can be run periodically to detect fail-pass\nactivities, thus growing the dataset continually.\n