2021/03/24 by Jakub Sido, Sido, Jakub, Ondřej Pražák +9 · 2 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #Text Readability and Simplification
paper · pdf · doi:10.48550/arxiv.2103.13031
This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss properties of monolingual and multilingual models based upon our results. We publish all the pre-trained and fine-tuned models freely for the research community.