gaBERT -- an Irish Language Model

2021/07/27 by Barry, James, Wagner, Joachim, Cassidy, Lauren +5 · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2107.12930

Abstract

The BERT family of neural language models have become highly popular due to their ability to provide sequences of text with rich context-sensitive token encodings which are able to generalise well to many NLP tasks. We introduce gaBERT, a monolingual BERT model for the Irish language. We compare our gaBERT model to multilingual BERT and the monolingual Irish WikiBERT, and we show that gaBERT provides better representations for a downstream parsing task. We also show how different filtering criteria, vocabulary size and the choice of subword tokenisation model affect downstream performance. We compare the results of fine-tuning a gaBERT model with an mBERT model for the task of identifying verbal multiword expressions, and show that the fine-tuned gaBERT model also performs better at this task. We release gaBERT and related code to the community.

Cited by

• Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting
• Qomhra: A Bilingual Irish and English Large Language Model

Related

• What the [MASK]? Making Sense of Language-Specific BERT Models
• RobBERT: a Dutch RoBERTa-based Language Model
• BERTje: A Dutch BERT Model
• Evaluating Multilingual BERT for Estonian
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks