Test Oracle Automation in the era of LLMs

2024/05/21 by Facundo Molina, Molina, Facundo, Alessandra Gorla +2 · 8 citations
Computer Science · Engineering · #Advanced Data Storage Technologies #FOS: Computer and information sciences #Power Systems Fault Detection #Software Engineering (cs.SE) #Technology and Security Systems

paper · pdf · doi:10.48550/arxiv.2405.12766

openalex publication_date 2024/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The effectiveness of a test suite in detecting faults highly depends on the correctness and completeness of its test oracles. Large Language Models (LLMs) have already demonstrated remarkable proficiency in tackling diverse software testing tasks, such as automated test generation and program repair. This paper aims to enable discussions on the potential of using LLMs for test oracle automation, along with the challenges that may emerge during the generation of various types of oracles. Additionally, our aim is to initiate discussions on the primary threats that SE researchers must consider when employing LLMs for oracle automation, encompassing concerns regarding oracle deficiencies and data leakages.

Cited by

• SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models
• State Field Coverage: A Metric for Oracle Quality
• How well LLM-based test generation techniques perform with newer LLM versions?
• How Students Use Generative AI for Software Testing: An Observational Study
• Large Language Models for Software Testing: A Research Roadmap
• Mut4All: Fuzzing Compilers via LLM-Synthesized Mutators Learned from Bug Reports
• Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges
• Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context Awareness

Related

• The Oracle Problem in Software Testing: A Survey
• Do LLMs generate test oracles that capture the actual or the expected program behaviour?
• AugmenTest: Enhancing Tests with LLM-Driven Oracles
• Automated Discovery of Test Oracles for Database Management Systems Using LLMs
• Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMs