Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models

2019/08/19 by Zhi-Xiu Ye, Chen Qian, Ye, Zhi-Xiu +6 · 70 citations
Computer Science · #Artificial intelligence #Commonsense knowledge #Commonsense reasoning #Computer science #Construct (python library) #Inference #Knowledge representation and reasoning #Language model #Language understanding #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language #Natural language processing #Natural language understanding #Programming language #Question answering #Representation (politics) #Sentence #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.1908.06725

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2019/08/19 · arxiv created 2020/05/06 · arxiv updated 2020/05/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The state-of-the-art pre-trained language representation models, such as Bidirectional Encoder Representations from Transformers (BERT), rarely incorporate commonsense knowledge or other knowledge explicitly. We propose a pre-training approach for incorporating commonsense knowledge into language representation models. We construct a commonsense-related multi-choice question answering dataset for pre-training a neural language representation model. The dataset is created automatically by our proposed "align, mask, and select" (AMS) method. We also investigate different pre-training tasks. Experimental results demonstrate that pre-training models using the proposed approach followed by fine-tuning achieve significant improvements over previous state-of-the-art models on two commonsense-related benchmarks, including CommonsenseQA and Winograd Schema Challenge. We also observe that fine-tuned models after the proposed pre-training approach maintain comparable performance on other NLP tasks, such as sentence classification and natural language inference tasks, compared to the original BERT models. These results verify that the proposed approach, while significantly improving commonsense-related NLP tasks, does not degrade the general language representation capabilities.

Citations

• A Simple Method for Commonsense Reasoning

Cited by

• Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering
• A Review of Winograd Schema Challenge Datasets and Approaches
• Combining pre-trained language models and structured knowledge
• Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
• On Commonsense Cues in BERT for Solving Commonsense Tasks
• Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph
• QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
• Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering
• Teaching Pretrained Models with Commonsense Reasoning: A Preliminary KB-Based Approach
• Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources
• Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks
• Beyond Language: Learning Commonsense from Images for Reasoning

Related

• Evaluating Commonsense in Pre-trained Language Models
• Commonsense Knowledge Transfer for Pre-trained Language Models
• Why Do Neural Language Models Still Need Commonsense Knowledge to Handle Semantic Variations in Question Answering?
• A Simple Method for Commonsense Reasoning
• Improving Question Answering by Commonsense-Based Pre-Training