FaBERT: Pre-training BERT on Persian Blogs

2024/02/09 by Mostafa Masumi, Seyed Soroush Majd, Masumi, Mostafa +5 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Peer-to-Peer Network Technologies #Web Data Mining and Analysis

paper · pdf · doi:10.48550/arxiv.2402.06617

openalex publication_date 2024/02/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce FaBERT, a Persian BERT-base model pre-trained on the HmBlogs corpus, encompassing both informal and formal Persian texts. FaBERT is designed to excel in traditional Natural Language Understanding (NLU) tasks, addressing the intricacies of diverse sentence structures and linguistic styles prevalent in the Persian language. In our comprehensive evaluation of FaBERT on 12 datasets in various downstream tasks, encompassing Sentiment Analysis (SA), Named Entity Recognition (NER), Natural Language Inference (NLI), Question Answering (QA), and Question Paraphrasing (QP), it consistently demonstrated improved performance, all achieved within a compact model size. The findings highlight the importance of utilizing diverse and cleaned corpora, such as HmBlogs, to enhance the performance of language models like BERT in Persian Natural Language Processing (NLP) applications. FaBERT is openly accessible at https://huggingface.co/sbunlp/fabert

Cited by

• PBBQ: A Persian Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models
• Exploring Subjective Tasks in Farsi: A Survey Analysis and Evaluation of Language Models
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey

Related

• TookaBERT: A Step Forward for Persian NLU
• HmBlogs: A big general Persian corpus
• Beheshti-NER: Persian Named Entity Recognition Using BERT
• Hakim: Farsi Text Embedding Model
• SINA-BERT: A pre-trained Language Model for Analysis of Medical Texts in Persian