Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective

2022/04/26 by Edoardo Manino, Julia Rozanova, Manino, Edoardo +7 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #FOS: Computer and information sciences #Software Testing and Debugging Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2204.12316

openalex publication_date 2022/04/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Metamorphic testing has recently been used to check the safety of neural NLP models. Its main advantage is that it does not rely on a ground truth to generate test cases. However, existing studies are mostly concerned with robustness-like metamorphic relations, limiting the scope of linguistic properties they can test. We propose three new classes of metamorphic relations, which address the properties of systematicity, compositionality and transitivity. Unlike robustness, our relations are defined over multiple source inputs, thus increasing the number of test cases that we can produce by a polynomial factor. With them, we test the internal consistency of state-of-the-art NLP models, and show that they do not always behave according to their expected linguistic properties. Lastly, we introduce a novel graphical notation that efficiently summarises the inner structure of metamorphic relations.

Cited by

• Metamorphic Testing of Large Language Models for Natural Language Processing

Related

• Metamorphic Testing of Large Language Models for Natural Language Processing
• Use of Metamorphic Relations as Knowledge Carriers to Train Deep Neural Networks
• The paradox of the compositionality of natural language: a neural machine translation case study
• Compositionality decomposed: how do neural networks generalise?
• HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks