Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing

2021/09/22 by Qian Liu, Liu, Qian, Dejian Yang +9 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2109.10540

openalex publication_date 2021/09/22 · openalex created_date 2021/09/27 · openalex updated_date 2026/07/28

Abstract

Recent years pretrained language models (PLMs) hit a success on several downstream tasks, showing their power on modeling language. To better understand and leverage what PLMs have learned, several techniques have emerged to explore syntactic structures entailed by PLMs. However, few efforts have been made to explore grounding capabilities of PLMs, which are also essential. In this paper, we highlight the ability of PLMs to discover which token should be grounded to which concept, if combined with our proposed erasing-then-awakening approach. Empirical studies on four datasets demonstrate that our approach can awaken latent grounding which is understandable to human experts, even if it is not exposed to such labels during training. More importantly, our approach shows great potential to benefit downstream semantic parsing models. Taking text-to-SQL as a case study, we successfully couple our approach with two off-the-shelf parsers, obtaining an absolute improvement of up to 9.8%.

Citations

• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Decoupled Weight Decay Regularization
• A Comprehensive Exploration on WikiSQL with Table-Aware Word\n Contextualization
• Grammar-based Neural Text-to-SQL Generation
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

Cited by

• RepCali: High Efficient Fine-tuning Via Representation Calibration in Latent Space for Pre-trained Language Models

Related

• Towards Computationally Verifiable Semantic Grounding for Language Models
• Unveiling the Black Box of PLMs with Semantic Anchors: Towards Interpretable Neural Semantic Parsing
• Improving Generalization in Language Model-Based Text-to-SQL Semantic Parsing: Two Simple Semantic Boundary-Based Techniques
• Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training
• Parsing as Pretraining