2021/09/08 by Casimiro Pio Carrino, Carrino, Casimiro Pio, Jordi Armengol-Estapé +11
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Text Readability and Simplification
paper · pdf · doi:10.48550/arxiv.2109.03570
This work presents biomedical and clinical language models for Spanish by\nexperimenting with different pretraining choices, such as masking at word and\nsubword level, varying the vocabulary size and testing with domain data,\nlooking for better language representations. Interestingly, in the absence of\nenough clinical data to train a model from scratch, we applied mixed-domain\npretraining and cross-domain transfer approaches to generate a performant\nbio-clinical model suitable for real-world clinical data. We evaluated our\nmodels on Named Entity Recognition (NER) tasks for biomedical documents and\nchallenging hospital discharge reports. When compared against the competitive\nmBERT and BETO models, we outperform them in all NER tasks by a significant\nmargin. Finally, we studied the impact of the model's vocabulary on the NER\nperformances by offering an interesting vocabulary-centric analysis. The\nresults confirm that domain-specific pretraining is fundamental to achieving\nhigher performances in downstream NER tasks, even within a mid-resource\nscenario. To the best of our knowledge, we provide the first biomedical and\nclinical transformer-based pretrained language models for Spanish, intending to\nboost native Spanish NLP applications in biomedicine. Our best models are\nfreely available in the HuggingFace hub: https://huggingface.co/BSC-TeMU.\n