2019/08/29 by Peng-Hsuan Li, Li, Peng-Hsuan, Tsu-Jui Fu +3 · 2 citations
Computer Science · Decision Sciences · Engineering · #Artificial intelligence #Biology #Computation and Language (cs.CL) #Computer science #Computer security #Context (archaeology) #Data Quality and Management #Engineering #FOS: Computer and information sciences #Identification (biology) #Linguistics #Named-entity recognition #Natural Language Processing Techniques #Natural language processing #Security token #Sequence (biology) #Task (project management) #Topic Modeling #Word (group theory) #cs.CL
paper · pdf · doi:10.48550/arxiv.1908.11046
published in arXiv (Cornell University) (Cornell University) · In proceedings of AAAI 2020
openalex publication_date 2019/08/29 · openalex created_date 2019/11/22 · arxiv created 2020/07/03 · arxiv updated 2020/07/06 · openalex updated_date 2026/08/05
BiLSTM has been prevalently used as a core module for NER in a sequence-labeling setup. State-of-the-art approaches use BiLSTM with additional resources such as gazetteers, language-modeling, or multi-task supervision to further improve NER. This paper instead takes a step back and focuses on analyzing problems of BiLSTM itself and how exactly self-attention can bring improvements. We formally show the limitation of (CRF-)BiLSTM in modeling cross-context patterns for each word -- the XOR limitation. Then, we show that two types of simple cross-structures -- self-attention and Cross-BiLSTM -- can effectively remedy the problem. We test the practical impacts of the deficiency on real-world NER datasets, OntoNotes 5.0 and WNUT 2017, with clear and consistent improvements over the baseline, up to 8.7% on some of the multi-token entity mentions. We give in-depth analyses of the improvements across several aspects of NER, especially the identification of multi-token mentions. This study should lay a sound foundation for future improvements on sequence-labeling NER. (Source codes: https://github.com/jacobvsdanniel/cross-ner)