All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typing

2024/10/22 by Deshmukh, Advait, Ashwin Umadi, Dananjay Srinivas +4
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2410.17355

openalex publication_date 2024/10/22 · openalex created_date 2024/11/13 · openalex updated_date 2026/07/28

Abstract

Due to their capacity to acquire world knowledge from large corpora, pre-trained language models (PLMs) are extensively used in ultra-fine entity typing tasks where the space of labels is extremely large. In this work, we explore the limitations of the knowledge acquired by PLMs by proposing a novel heuristic to approximate the pre-training distribution of entities when the pre-training data is unknown. Then, we systematically demonstrate that entity-typing approaches that rely solely on the parametric knowledge of PLMs struggle significantly with entities at the long tail of the pre-training distribution, and that knowledge-infused approaches can account for some of these shortcomings. Our findings suggest that we need to go beyond PLMs to produce solutions that perform well for infrequent entities.

Related

• Interpretable Entity Representations through Large-Scale Typing
• Ultra-fine Entity Typing with Indirect Supervision from Natural Language Inference
• EnCore: Fine-Grained Entity Typing by Pre-Training Entity Encoders on Coreference Chains
• A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models
• Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy