No Gold Standard, No Problem: Reference-Free Evaluation of Taxonomies

2025/05/16 by Wullschleger, Pascal, Zarharan, Majid, Daly, Donnacha +2
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.11470

Abstract

We introduce two reference-free metrics for quality evaluation of taxonomies. The first metric evaluates robustness by calculating the correlation between semantic and taxonomic similarity, covering a type of error not handled by existing metrics. The second uses Natural Language Inference to assess logical adequacy. Both metrics are tested on five taxonomies and are shown to correlate well with F1 against gold-standard taxonomies.

Citations

• CodeTaxo: Enhancing Taxonomy Expansion with Limited Examples via Code Language Prompts
• Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Models
• RaTE: a Reproducible automatic Taxonomy Evaluation by Filling the Gap
• Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI
• OntoZSL: Ontology-enhanced Zero-shot Learning
• Taxonomy Completion via Triplet Matching Network
• Ontology-guided Semantic Composition for Zero-Shot Learning
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Text Summarization with Pretrained Encoders
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• EL Embeddings: Geometric construction of models for the Description Logic EL ++
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
• GRAM: Graph-based Attention Model for Healthcare Representation Learning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books

Related

• RaTE: a Reproducible automatic Taxonomy Evaluation by Filling the Gap
• LITE: LLM-Impelled efficient Taxonomy Evaluation
• TaxoAlign: Scholarly Taxonomy Generation Using Language Models
• Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References
• Semantic F1 Scores: Fair Evaluation Under Fuzzy Class Boundaries