2020/10/10 by John Pougué Biyong, Biyong, John Pougue, Bo Wang +5
Computer Science · #60L10 #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2010.04897
openalex publication_date 2020/10/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Relying on large pretrained language models such as Bidirectional Encoder\nRepresentations from Transformers (BERT) for encoding and adding a simple\nprediction layer has led to impressive performance in many clinical natural\nlanguage processing (NLP) tasks. In this work, we present a novel extension to\nthe Transformer architecture, by incorporating signature transform with the\nself-attention model. This architecture is added between embedding and\nprediction layers. Experiments on a new Swedish prescription data show the\nproposed architecture to be superior in two of the three information extraction\ntasks, comparing to baseline models. Finally, we evaluate two different\nembedding approaches between applying Multilingual BERT and translating the\nSwedish text to English then encode with a BERT model pretrained on clinical\nnotes.\n