Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models

2023/08/30 by Neha Sengupta, Sunil Kumar Sahu, Sengupta, Neha +51 · 7 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Machine Learning in Healthcare

paper · pdf · doi:10.48550/arxiv.2308.16149

Abstract

We introduce Jais and Jais-chat, new state-of-the-art Arabic-centric foundation and instruction-tuned open generative large language models (LLMs). The models are based on the GPT-3 decoder-only architecture and are pretrained on a mixture of Arabic and English texts, including source code in various programming languages. With 13 billion parameters, they demonstrate better knowledge and reasoning capabilities in Arabic than any existing open Arabic and multilingual models by a sizable margin, based on extensive evaluation. Moreover, the models are competitive in English compared to English-centric open models of similar size, despite being trained on much less English data. We provide a detailed description of the training, the tuning, the safety alignment, and the evaluation of the models. We release two open versions of the model -- the foundation Jais model, and an instruction-tuned Jais-chat variant -- with the aim of promoting research on Arabic LLMs. Available at https://huggingface.co/inception-mbzuai/jais-13b-chat

Cited by

• AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models
• AI Text Detectors and the Misclassification of Slightly Polished Arabic Text
• Factuality challenges in the era of large language models and opportunities for fact-checking
• Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
• Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale
• Saudi-Dialect-ALLaM: LoRA Fine-Tuning for Dialectal Arabic Generation

Related

• AceGPT, Localizing Large Language Models in Arabic
• Large Language Models and Arabic Content: A Review
• A Large and Diverse Arabic Corpus for Language Modeling
• ALLaM: Large Language Models for Arabic and English
• ArabianGPT: Native Arabic GPT-based Large Language Model