2019/01/11 by Abhilasha Ravichander, Aakanksha Naik, Ravichander, Abhilasha +5 · 1 citation
Computer Science · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.1901.03735
openalex publication_date 2019/01/11 · openalex created_date 2022/07/30 · openalex updated_date 2026/07/28
Quantitative reasoning is a higher-order reasoning skill that any intelligent\nnatural language understanding system can reasonably be expected to handle. We\npresent EQUATE (Evaluating Quantitative Understanding Aptitude in Textual\nEntailment), a new framework for quantitative reasoning in textual entailment.\nWe benchmark the performance of 9 published NLI models on EQUATE, and find that\non average, state-of-the-art methods do not achieve an absolute improvement\nover a majority-class baseline, suggesting that they do not implicitly learn to\nreason with quantities. We establish a new baseline Q-REAS that manipulates\nquantities symbolically. In comparison to the best performing NLI model, it\nachieves success on numerical reasoning tests (+24.2%), but has limited verbal\nreasoning capabilities (-8.1%). We hope our evaluation framework will support\nthe development of models of quantitative reasoning in language understanding.\n