The Oracle Problem in Software Testing: A Survey

2014/11/20 by Earl T. Barr, Mark Harman, Phil McMinn +2 · 1,050 citations
Computer Science · #Computer science #Oracle #Programming language #Regression testing #Software #Software Engineering Research #Software Reliability and Analysis Research #Software Testing and Debugging Techniques #Software construction #Software development #Software engineering #Software performance testing #Software reliability testing #Software testing

paper · pdf · doi:10.1109/tse.2014.2372785

published in IEEE Transactions on Software Engineering 41(5), 507-525 (IEEE Computer Society)

openalex publication_date 2014/11/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.

Citations

• On Testing Non-Testable Programs

Cited by

• From Particles to Perils: SVGD-Based Hazardous Scenario Generation for Autonomous Driving Systems Testing
• Exploring the integration of large language models in industrial test maintenance processes
• Software Testing With Large Language Models: Survey, Landscape, and Vision
• How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study
• TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems
• PatchGuru: Patch Oracle Inference from Natural Language Artifacts
• Toward Systematic Counterfactual Fairness Evaluation of Large Language Models: The CAFFE Framework
• Teralizer: Semantics-Based Test Generalization from Conventional Unit Tests to Property-Based Tests
• PSALM: applying Proportional SAmpLing strategy in Metamorphic testing
• LLMCFG-TGen: Using LLM-Generated Control Flow Graphs to Automatically Create Test Cases from Use Cases
• Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations
• Synthesizing Precise Protocol Specs from Natural Language for Effective Test Generation
• Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation
• A Review on Oracle Issues in Machine Learning
• Formalizing Regression Testing for Agile and Continuous Integration Environments
• Metamorphic Testing of Large Language Models for Natural Language Processing
• Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis
• Input Relation Prompting for Metamorphic Testing on Query-Based Systems
• State Field Coverage: A Metric for Oracle Quality
• WES: Agent-based User Interaction Simulation on Real Infrastructure
• Trustworthy AI: From Principles to Practices
• STADS: Software Testing as Species Discovery
• Enhancing Code Review through Fuzzing and Likely Invariants
• A Digital Twin Framework for Metamorphic Testing of Autonomous Driving Systems Using Generative Model
• Automated Discovery of Test Oracles for Database Management Systems Using LLMs
• Large Language Models for Software Testing: A Research Roadmap
• DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Pattern
• Navigating the Labyrinth: Path-Sensitive Unit Test Generation with Large Language Models
• Argonaut: Interactive Visual Exploration for Distributed Optimization
• Intramorphic Testing: A New Approach to the Test Oracle Problem
• Arguzz: Testing zkVMs for Soundness and Completeness Bugs
• SATORI: Static Test Oracle Generation for REST APIs
• Automated Test Validators for Flaky Cyber-Physical System Simulators: Approach and Evaluation
• Object-based Metamorphic Testing through Image Structuring
• A software engineering perspective on engineering machine learning systems: State of the art and challenges
• Multi-population GAN Training: Analyzing Co-Evolutionary Algorithms
• YATE: The Role of Test Repair in LLM-Based Unit Test Generation
• Machine Learning Testing: Survey, Landscapes and Horizons
• On Introducing Automatic Test Case Generation in Practice: A Success Story and Lessons Learned
• Using Metamorphic Relations to Verify and Enhance Artcode Classification
• Detecting Critical Bugs in SMT Solvers Using Blackbox Mutational Fuzzing
• Empirical Study of Phased Model of Software Change
• Extremal Testing for Network Software using LLMs
• Graph-Based Fuzz Testing for Deep Learning Inference Engine
• Prompting for Performance: Exploring LLMs for Configuring Software
• Improving Deep Learning Framework Testing with Model-Level Metamorphic Testing
• Repository-Aware Metamorphic Relation Generation for Augmented Reality Applications using Large Language Models
• Meta-Fair: AI-Assisted Fairness Testing of Large Language Models
• Alternatives for Testing of Context-Aware Contemporary Software Systems in industrial settings: Results from a Rapid review
• Mutation Testing framework for Machine Learning
• Use Property-Based Testing to Bridge LLM Code Generation and Validation
• DISTINCT: A Description-Guided Branch-Consistency Analysis Framework for Non-Regressive Test Case Generation
• Large Language Models for Unit Testing: A Systematic Literature Review
• Isolating Noisy Labelled Test Cases in Human-in-the-Loop Oracle Learning
• IDOL: Improved Different Optimization Levels Testing for Solidity Compilers
• Documentation vs. Code Patterns: What Drives LLM-Based Exception Oracle Generation?
• A Systematic Mapping Study on Testing of Machine Learning Programs
• CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval
• Adversarial Reasoning for Repair Based on Inferred Program Intent
• Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation
• Towards the Systematic Testing of Virtual Reality Programs (extended version)
• Enhancing REST API Fuzzing with Access Policy Violation Checks and Injection Attacks
• Intent-Level Quantum Programming with Assertion-Guided Execution and Inspectable Intermediate Representation
• Automated Test Generation from Program Documentation Encoded in Code Comments
• Test It Before You Trust It: Applying Software Testing for Trustworthy In-context Learning
• Tracking the Moving Target: A Framework for Continuous Evaluation of LLM Test Generation in Industry
• Just-in-Time Catching Test Generation at Meta
• Assessing Behavioral Validation in UI Component Test Suites Using Inferred Metamorphic Relations
• ROSA: Finding Backdoors with Fuzzing
• Technical Challenges in Maintaining Tax Prep Software with Large Language Models
• Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges
• Automated patch assessment for program repair at scale
• Using Causal Inference to Test Systems with Hidden and Interacting Variables: An Evaluative Case Study
• RESTful API Automated Test Case Generation
• Building Better Quality Predictors Using "ε-Dominance"
• Robustness Testing of Intermediate Verifiers
• Tratto: A Neuro-Symbolic Approach to Deriving Axiomatic Test Oracles
• Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context Awareness
• Identifying implementation bugs in machine learning based image classifiers using metamorphic testing
• Expectations vs Reality -- A Secondary Study on AI Adoption in Software Testing

Related

• Test Oracle Automation in the era of LLMs
• A Brief Survey on Oracle-based Test Adequacy Metrics
• Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMs
• State Field Coverage: A Metric for Oracle Quality
• Intramorphic Testing: A New Approach to the Test Oracle Problem