NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark

2025/04/10 by Mikhailov, Vladislav, Enstad, Tita, Samuel, David +4 · 3 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2504.07749

Abstract

This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of 24 high-quality human-created datasets -- of which five are created from scratch. In contrast to existing benchmarks for Norwegian, NorEval covers a broad spectrum of task categories targeting Norwegian language understanding and generation, establishes human baselines, and focuses on both of the official written standards of the Norwegian language: Bokmål and Nynorsk. All our datasets and a collection of over 100 human-written prompts are integrated into LM Evaluation Harness, ensuring flexible and reproducible evaluation. We describe the NorEval design and present the results of benchmarking 19 open-source pre-trained and instruction-tuned LMs for Norwegian in various scenarios. Our benchmark, evaluation framework, and annotation materials are publicly available.

Cited by

• FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models
• Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages
• HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models

Related

• NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian
• NorBench -- A Benchmark for Norwegian Language Models
• Large-Scale Contextualised Language Modelling for Norwegian
• ScandEval: A Benchmark for Scandinavian Natural Language Processing
• NoCoLA: The Norwegian Corpus of Linguistic Acceptability