Natural Language Inference in Context -- Investigating Contextual Reasoning over Long Texts

2020/11/10 by Hanmeng Liu, Liu, Hanmeng, Leyang Cui +6 · 4 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2011.04864

arxiv created 2020/11/10 · openalex publication_date 2020/11/10 · arxiv updated 2020/11/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Natural language inference (NLI) is a fundamental NLP task, investigating the entailment relationship between two texts. Popular NLI datasets present the task at sentence-level. While adequate for testing semantic representations, they fall short for testing contextual reasoning over long texts, which is a natural part of the human inference process. We introduce ConTRoL, a new dataset for ConTextual Reasoning over Long texts. Consisting of 8,325 expert-designed "context-hypothesis" pairs with gold labels, ConTRoL is a passage-level NLI dataset with a focus on complex contextual reasoning types such as logical reasoning. It is derived from competitive selection and recruitment test (verbal reasoning test) for police recruitment, with expert level quality. Compared with previous NLI benchmarks, the materials in ConTRoL are much more challenging, involving a range of reasoning types. Empirical results show that state-of-the-art language models perform by far worse than educated humans. Our dataset can also serve as a testing-set for downstream tasks like Checking Factual Correctness of Summaries.

Citations

• Dialogue Natural Language Inference
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• WinoGrande: An Adversarial Winograd Schema Challenge at Scale
• Multi-hop Question Answering via Reasoning Chains
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Longformer: The Long-Document Transformer

Cited by

• Extractive Fact Decomposition for Interpretable Natural Language Inference in one Forward Pass
• DeKeyNLU: Enhancing Natural Language to SQL Generation through Task Decomposition and Keyword Extraction
• Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
• SEval-Ex: A Statement-Level Framework for Explainable Summarization Evaluation

Related

• Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
• Do Large Language Models Understand Logic or Just Mimick Context?
• DocNLI: A Large-scale Dataset for Document-level Natural Language Inference
• Looking Beyond Sentence-Level Natural Language Inference for Downstream Tasks
• Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation