2024/05/01 by Daniel Sokolowski, David Spielmann, Guido Salvaneschi · 13 citations
Computer Science · #Code coverage #Computer science #Fuzz testing #Generator (circuit theory) #Operating system #Oracle #Programming language #Python (programming language) #Software #Software Engineering Research #Software Reliability and Analysis Research #Software Testing and Debugging Techniques #Software engineering #Test case #Test suite #Toolchain #TypeScript
paper · doi:10.1109/tse.2024.3393070
published in IEEE Transactions on Software Engineering 50(6), 1585-1599 (IEEE Computer Society)
openalex publication_date 2024/05/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
Infrastructure as Code (IaC) enables efficient deployment and operation, which are crucial to releasing software quickly. As setups can be complex, developers implement IaC programs in general-purpose programming languages like TypeScript and Python, using PL-IaC solutions like Pulumi and AWS CDK. The reliability of such IaC programs is even more relevant than in traditional software because a bug in IaC impacts the whole system. Yet, even though testing is a standard development practice, it is rarely used for IaC programs. For instance, in August 2022, less than 1% of the public Pulumi IaC programs on GitHub implemented tests. Available IaC program testing techniques severely limit the development velocity or require much development effort.To solve these issues, we propose Automated Configuration Testing (ACT), a methodology to test IaC programs in many configurations quickly and with low effort. ACT automatically mocks all resource definitions in the IaC program and uses generator and oracle plugins for test generation and validation. We implement ACT inProTI, a testing tool for Pulumi TypeScript with a type-based generator and oracle, and support for application specifications. Our evaluation with 6 081 programs from GitHub and artificial benchmarks shows thatProTIcan directly be applied to existing IaC programs, quickly finds bugs where current techniques are infeasible, and enables reusing existing generators and oracles thanks to its pluggable architecture.