LocalValueBench: A Collaboratively Built and Extensible Benchmark for Evaluating Localized Value Alignment and Ethical Safety in Large Language Models

2024/07/27 by Gwenyth Isobel Meadows, Meadows, Gwenyth Isobel, Lau, Nicholas Wai Long +6 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2408.01460

openalex publication_date 2024/07/27 · openalex created_date 2024/10/14 · openalex updated_date 2026/07/28

Abstract

The proliferation of large language models (LLMs) requires robust evaluation of their alignment with local values and ethical standards, especially as existing benchmarks often reflect the cultural, legal, and ideological values of their creators. LocalValueBench, introduced in this paper, is an extensible benchmark designed to assess LLMs' adherence to Australian values, and provides a framework for regulators worldwide to develop their own LLM benchmarks for local value alignment. Employing a novel typology for ethical reasoning and an interrogation approach, we curated comprehensive questions and utilized prompt engineering strategies to probe LLMs' value alignment. Our evaluation criteria quantified deviations from local values, ensuring a rigorous assessment process. Comparative analysis of three commercial LLMs by USA vendors revealed significant insights into their effectiveness and limitations, demonstrating the critical importance of value alignment. This study offers valuable tools and methodologies for regulators to create tailored benchmarks, highlighting avenues for future research to enhance ethical AI development.

Cited by

• CALMA: A Process for Deriving Context-aligned Axes for Language Model Alignment
• Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysis
• The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs

Related

• Diverse Human Value Alignment for Large Language Models via Ethical Reasoning
• LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Models
• Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learning
• VAL-Bench: Belief Consistency as a measure for Value Alignment in Language Models
• Exploring and steering the moral compass of Large Language Models