Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models

2021/07/05 by Mingyue Han, Han, Mingyue, Yinglin Wang +1
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · doi:10.48550/arxiv.2107.01791

openalex publication_date 2021/07/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Pretrained language models (PLM) achieve surprising performance on the Choice of Plausible Alternatives (COPA) task. However, whether PLMs have truly acquired the ability of causal reasoning remains a question. In this paper, we investigate the problem of semantic similarity bias and reveal the vulnerability of current COPA models by certain attacks. Previous solutions that tackle the superficial cues of unbalanced token distribution still encounter the same problem of semantic bias, even more seriously due to the utilization of more training data. We mitigate this problem by simply adding a regularization loss and experimental results show that this solution not only improves the model's generalization ability, but also assists the models to perform more robustly on a challenging dataset, BCOPA-CE, which has unbiased token distribution and is more difficult for models to distinguish cause and effect.

Citations

• Understanding Neural Networks through Representation Erasure
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Related

• Improving Commonsense Causal Reasoning by Adversarial Training and Data Augmentation
• Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference
• Causal Reasoning and Large Language Models: Opening a New Frontier for Causality
• A Critical Review of Causal Reasoning Benchmarks for Large Language Models
• Causality can systematically address the monsters under the bench(marks)