Qomhra: A Bilingual Irish and English Large Language Model

2025/10/20 by Joseph McInerney, Tran, Khanh-Tung, McInerney, Joseph +4
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.17652

openalex publication_date 2025/10/20 · openalex created_date 2025/10/22 · openalex updated_date 2026/07/30

Abstract

Large language model (LLM) research and development has overwhelmingly focused on the world's major languages, leading to under-representation of low-resource languages such as Irish. This paper introduces Qomhrá, a bilingual Irish and English LLM, developed under extremely low-resource constraints. A complete pipeline is outlined spanning bilingual continued pre-training, instruction tuning, and the synthesis of human preference data for future alignment training. We focus on the lack of scalable methods to create human preference data by proposing a novel method to synthesise such data by prompting an LLM to generate ``accepted'' and ``rejected'' responses, which we validate as aligning with L1 Irish speakers. To select an LLM for synthesis, we evaluate the top closed-weight LLMs for Irish language generation performance. Gemini-2.5-Pro is ranked highest by L1 and L2 Irish-speakers, diverging from LLM-as-a-judge ratings, indicating a misalignment between current LLMs and the Irish-language community. Subsequently, we leverage Gemini-2.5-Pro to translate a large scale English-language instruction tuning dataset to Irish and to synthesise a first-of-its-kind Irish-language human preference dataset. We comprehensively evaluate Qomhrá across several benchmarks, testing translation, gender understanding, topic identification, and world knowledge; these evaluations show gains of up to 29% in Irish and 44% in English compared to the existing open-source Irish LLM baseline, UCCIX. The results of our framework provide insight and guidance to developing LLMs for both Irish and other low-resource languages.

Citations

• Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh
• Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat
• Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities
• LLMs Are Few-Shot In-Context Low-Resource Language Learners
• Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
• SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• QLoRA: Efficient Finetuning of Quantized LLMs
• LIMA: Less Is More for Alignment
• Finetuned Language Models Are Zero-Shot Learners
• gaBERT -- an Irish Language Model
• LoRA: Low-Rank Adaptation of Large Language Models
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
• A Study of BFLOAT16 for Deep Learning Training
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• XNLI: Evaluating Cross-lingual Sentence Representations
• Deep reinforcement learning from human preferences

Related

• UCCIX: Irish-eXcellence Large Language Model
• Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting
• IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation
• High-quality Data-to-Text Generation for Severely Under-Resourced Languages with Out-of-the-box Large Language Models
• Fotheidil: an Automatic Transcription System for the Irish Language