OpenEthics: A Comprehensive Ethical Evaluation of Open-Source Generative Large Language Models

2025/05/21 by Yıldırım Özen, Çetin, Burak Erinç, Burak Erinç Çetin +7
Computer Science · Medicine · Social Sciences · #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.16036

openalex publication_date 2025/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

Generative large language models present significant potential but also raise critical ethical concerns, including issues of safety, fairness, robustness, and reliability. Most existing ethical studies, however, are limited by their narrow focus, a lack of language diversity, and an evaluation of a restricted set of models. To address these gaps, we present a broad ethical evaluation of 29 recent open-source LLMs using a novel dataset that assesses four key ethical dimensions: robustness, reliability, safety, and fairness. Our analysis includes both a high-resource language, English, and a low-resource language, Turkish, providing a comprehensive assessment and a guide for safer model development. Using an LLM-as-a-Judge methodology, our experimental results indicate that many open-source models demonstrate strong performance in safety, fairness, and robustness, while reliability remains a key concern. Ethical evaluation shows cross-linguistic consistency, and larger models generally exhibit better ethical performance. We also show that jailbreak templates are ineffective for most of the open-source models examined in this study. We share all materials including data and scripts at https://github.com/metunlp/openethics

Citations

• ELAB: Extensive LLM Alignment Benchmark in Persian Language
• Preference Leakage: A Contamination Problem in LLM-as-a-judge
• Qwen2.5 Technical Report
• Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
• A Survey on LLM-as-a-Judge
• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• Decoding Multilingual Moral Preferences: Unveiling LLM's Biases Through the Moral Machine Experiment
• Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle
• Exploring Human-LLM Conversations: Mental Models and the Originator of Toxicity
• The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm
• MoralBench: Moral Evaluation of LLMs
• Aya 23: Open Weight Releases to Further Multilingual Progress
• Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them in
• ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming
• On Protecting the Data Privacy of Large Language Models (LLMs): A Survey
• HypoTermQA: Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLMs
• SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• MERA: A Comprehensive LLM Evaluation in Russian
• LLaMA Beyond English: An Empirical Study on Language Capability Transfer
• MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
• Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Certifying LLM Safety against Adversarial Prompting
• Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
• Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
• Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
• BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
• Jailbroken: How Does LLM Safety Training Fail?
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
• SafeText: A Benchmark for Exploring Physical Safety in Language Models
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• BBQ: A Hand-Built Bias Benchmark for Question Answering
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• A Primer on Pretrained Multilingual Language Models
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language\n Models
• LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies

Related

• Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey
• Exploring and steering the moral compass of Large Language Models
• Ethical Considerations and Policy Implications for Large Language Models: Guiding Responsible Development and Deployment
• The Only Way is Ethics: A Guide to Ethical Research with Large Language Models
• The Open Source Advantage in Large Language Models (LLMs)