A Survey on Metamorphic Testing

2016/02/29 by Sergio Segura, Gordon Fraser, Ana B. Sánchez +3 · 545 citations
Computer Science · #Advanced Malware Detection Techniques #Computer science #Geology #Metamorphic rock #Programming language #Software #Software Reliability and Analysis Research #Software Testing and Debugging Techniques #Software engineering #Software testing

paper · doi:10.1109/tse.2016.2532875

published in IEEE Transactions on Software Engineering 42(9), 805-824 (IEEE Computer Society)

openalex publication_date 2016/02/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

A test oracle determines whether a test execution reveals a fault, often by comparing the observed program output to the expected output. This is not always practical, for example when a program's input-output relation is complex and difficult to capture formally. Metamorphic testing provides an alternative, where correctness is not determined by checking an individual concrete output, but by applying a transformation to a test input and observing how the program output “morphs” into a different one as a result. Since the introduction of such metamorphic relations in 1998, many contributions on metamorphic testing have been made, and the technique has seen successful applications in a variety of domains, ranging from web services to computer graphics. This article provides a comprehensive survey on metamorphic testing: It summarises the research results and application areas, and analyses common practice in empirical studies of metamorphic testing as well as the main open challenges.

Citations

• On Testing Non-Testable Programs

Cited by

• Metamorphic Testing for Clinical ML Models: A Framework Proposal and Pilot Study
• PSALM: applying Proportional SAmpLing strategy in Metamorphic testing
• Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations
• LLM Assisted Coding with Metamorphic Specification Mutation Agent
• A Review on Oracle Issues in Machine Learning
• Risk Estimation in Differential Fuzzing via Extreme Value Theory
• Input Relation Prompting for Metamorphic Testing on Query-Based Systems
• Structure-Invariant Testing for Machine Translation
• Testing Machine Translation via Referential Transparency
• When Knowledge Changes: Metamorphic Testing of RAG Systems with Mutations
• A Digital Twin Framework for Metamorphic Testing of Autonomous Driving Systems Using Generative Model
• Evaluating and Pricing Advertisements in AI-Generated Responses
• Intramorphic Testing: A New Approach to the Test Oracle Problem
• Arguzz: Testing zkVMs for Soundness and Completeness Bugs
• MetaRAG: Metamorphic Testing for Hallucination Detection in RAG Systems
• SATORI: Static Test Oracle Generation for REST APIs
• Metamorphic Testing of Multimodal Human Trajectory Prediction
• A software engineering perspective on engineering machine learning systems: State of the art and challenges
• A Set-Theoretic Approach to Detecting Logic Bugs in DBMS Inner Join Optimizations
• Using Metamorphic Relations to Verify and Enhance Artcode Classification
• Maintaining Smart Contracts on Ethereum: Issues, Techniques, and Future Challenges
• MT-Teql: Evaluating and Augmenting Consistency of Text-to-SQL Models with Metamorphic Testing
• MT4DP: Data Poisoning Attack Detection for DL-based Code Search Models via Metamorphic Testing
• Metamorphic Testing: Testing the Untestable
• TigAug: Data Augmentation for Testing Traffic Light Detection in Autonomous Driving Systems
• Repository-Aware Metamorphic Relation Generation for Augmented Reality Applications using Large Language Models
• Meta-Fair: AI-Assisted Fairness Testing of Large Language Models
• Generate Realistic Test Scenes for V2X Communication Systems
• An Empirical Study of Bugs in Data Visualization Libraries
• Use of Metamorphic Relations as Knowledge Carriers to Train Deep Neural Networks
• Memory Reward Inflation in Self-Improving LLM Agents
• The Rise of Language Models in Mining Software Repositories: A Survey
• HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks
• DeepRoad: GAN-based Metamorphic Autonomous Driving System Testing
• Test It Before You Trust It: Applying Software Testing for Trustworthy In-context Learning
• Assessing Behavioral Validation in UI Component Test Suites Using Inferred Metamorphic Relations
• ROSA: Finding Backdoors with Fuzzing
• Software Development Processes in Ocean System Modeling
• Robustness Testing of Intermediate Verifiers
• T. H. Tse [wikipedia]

Related

• Metamorphic Testing: Testing the Untestable
• Metamorphic Testing: A New Approach for Generating Next Test Cases
• Metamorphic Relation Generation: State of the Art and Visions for Future Research
• Test Adequacy for Metamorphic Testing: Criteria, Measurement, and Implication
• Input Relation Prompting for Metamorphic Testing on Query-Based Systems