Probabilistic Aggregation and Targeted Embedding Optimization for Collective Moral Reasoning in Large Language Models

2025/06/17 by Yuan, Chenchen, Zhang, Zheyu, Yang, Shuo +2
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.14625

Abstract

Large Language Models (LLMs) have shown impressive moral reasoning abilities. Yet they often diverge when confronted with complex, multi-factor moral dilemmas. To address these discrepancies, we propose a framework that synthesizes multiple LLMs' moral judgments into a collectively formulated moral judgment, realigning models that deviate significantly from this consensus. Our aggregation mechanism fuses continuous moral acceptability scores (beyond binary labels) into a collective probability, weighting contributions by model reliability. For misaligned models, a targeted embedding-optimization procedure fine-tunes token embeddings for moral philosophical theories, minimizing JS divergence to the consensus while preserving semantic integrity. Experiments on a large-scale social moral dilemma dataset show our approach builds robust consensus and improves individual model fidelity. These findings highlight the value of data-driven moral alignment across multiple models and its potential for safer, more consistent AI systems.

Citations

• The Llama 3 Herd of Models
• Values, Ethics, Morals? On the Use of Moral Concepts in NLP Research
• Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Towards Theory-based Moral AI: Moral AI with Aggregating Models Based on Normative Ethical Theory
• Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis
• ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
• Affective Coherence Monitoring for Transformer-Based Language Models
• Mass-Editing Memory in a Transformer
• When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment
• PaLM: Scaling Language Modeling with Pathways
• Mapping Topics in 100,000 Real-life Moral Dilemmas
• Do Multilingual Language Models Capture Differing Moral Norms?
• Training language models to follow instructions with human feedback
• Locating and Editing Factual Associations in GPT
• Fast Model Editing at Scale
• LoRA: Low-Rank Adaptation of Large Language Models
• Social Chemistry 101: Learning to Reason about Social and Moral Norms
• Aligning AI With Shared Human Values
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Correlation Coefficients: Appropriate Use and Interpretation
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word\n Embeddings
• Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Can Machines Learn Morality? The Delphi Experiment

Related

• The Moral Mind(s) of Large Language Models
• MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning
• The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models
• The Morality of Probability: How Implicit Moral Biases in LLMs May Shape the Future of Human-AI Symbiosis
• Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework