SwissBERT: The Multilingual Language Model for Switzerland

2023/03/23 by Jannis Vamvas, Vamvas, Jannis, Johannes Graën +3 · 2 citations
Arts and Humanities · Computer Science · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Gender Studies in Language #Linguistic research and analysis #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2303.13310

openalex publication_date 2023/03/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present SwissBERT, a masked language model created specifically for processing Switzerland-related text. SwissBERT is a pre-trained model that we adapted to news articles written in the national languages of Switzerland -- German, French, Italian, and Romansh. We evaluate SwissBERT on natural language understanding tasks related to Switzerland and find that it tends to outperform previous models on these tasks, especially when processing contemporary news and/or Romansh Grischun. Since SwissBERT uses language adapters, it may be extended to Swiss German dialects in future work. The model and our open-source code are publicly released at https://github.com/ZurichNLP/swissbert.

Cited by

• The Mediomatix Corpus: Parallel Data for Romansh Idioms via Comparable Schoolbooks
• CodeNER: Code Prompting for Named Entity Recognition

Related

• Fine-tuning the SwissBERT Encoder Model for Embedding Sentences and Documents
• SwissDial: Parallel Multidialectal Corpus of Spoken Swiss German
• Dialectal Speech Recognition and Translation of Swiss German Speech to Standard German Text: Microsoft's Submission to SwissText 2021
• Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect
• Machine Translation of Low-Resource Spoken Dialects: Strategies for Normalizing Swiss German