Observation-based unit test generation at Meta

2024/02/09 by Nadia Alshahwan, Alshahwan, Nadia, Mark Harman +7 · 3 citations
Engineering · #Engineering and Test Systems #FOS: Computer and information sciences #Real-time simulation and control systems #Software Engineering (cs.SE)

paper · pdf · doi:10.48550/arxiv.2402.06111

openalex publication_date 2024/02/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

TestGen automatically generates unit tests, carved from serialized observations of complex objects, observed during app execution. We describe the development and deployment of TestGen at Meta. In particular, we focus on the scalability challenges overcome during development in order to deploy observation-based test carving at scale in industry. So far, TestGen has landed 518 tests into production, which have been executed 9,617,349 times in continuous integration, finding 5,702 faults. Meta is currently in the process of more widespread deployment. Our evaluation reveals that, when carving its observations from 4,361 reliable end-to-end tests, TestGen was able to generate tests for at least 86% of the classes covered by end-to-end tests. Testing on 16 Kotlin Instagram app-launch-blocking tasks demonstrated that the TestGen tests would have trapped 13 of these before they became launch blocking.

Cited by

• Industrial Practice of LLM-Based Test Case Carving and Assertion Generation (Experience Paper)
• LLM Assisted Coding with Metamorphic Specification Mutation Agent
• YATE: The Role of Test Repair in LLM-Based Unit Test Generation

Related

• Automated Unit Test Improvement using Large Language Models at Meta
• Enhancing Testing at Meta with Rich-State Simulated Populations
• Mutation-Guided LLM-based Test Generation at Meta
• TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark
• TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration