On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference

2018/04/25 by Poliak, Adam, Belinkov, Yonatan, Glass, James +1 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1804.09779

Abstract

We propose a process for investigating the extent to which sentence representations arising from neural machine translation (NMT) systems encode distinct semantic phenomena. We use these representations as features to train a natural language inference (NLI) classifier based on datasets recast from existing semantic annotations. In applying this process to a representative NMT system, we find its encoder appears most suited to supporting inferences at the syntax-semantics interface, as compared to anaphora resolution requiring world-knowledge. We conclude with a discussion on the merits and potential deficiencies of the existing process, and how it may be improved and extended as a broader framework for evaluating semantic coverage.

Cited by

• Survey of NLU Benchmarks Diagnosing Linguistic Phenomena: Why not Standardize Diagnostics Benchmarks?

Related

• Better Neural Machine Translation by Extracting Linguistic Information\n from BERT
• Uncertainty-Aware Semantic Augmentation for Neural Machine Translation
• On the Linguistic Representational Power of Neural Machine Translation\n Models
• Inducing Grammars with and for Neural Machine Translation
• Categorizing Semantic Representations for Neural Machine Translation