MoralCLIP: Contrastive Alignment of Vision-and-Language Representations with Moral Foundations Theory

2025/06/06 by Condez, Ana Carolina, Tavares, Diogo, Magalhães, João
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.05696

Abstract

Recent advances in vision-language models have enabled rich semantic understanding across modalities. However, these encoding methods lack the ability to interpret or reason about the moral dimensions of content-a crucial aspect of human cognition. In this paper, we address this gap by introducing MoralCLIP, a novel embedding representation method that extends multimodal learning with explicit moral grounding based on Moral Foundations Theory (MFT). Our approach integrates visual and textual moral cues into a unified embedding space, enabling cross-modal moral alignment. MoralCLIP is grounded on the multi-label dataset Social-Moral Image Database to identify co-occurring moral foundations in visual content. For MoralCLIP training, we design a moral data augmentation strategy to scale our annotated dataset to 15,000 image-text pairs labeled with MFT-aligned dimensions. Our results demonstrate that explicit moral supervision improves both unimodal and multimodal understanding of moral content, establishing a foundation for morally-aware AI systems capable of recognizing and aligning with human moral values.

Citations

• MM-MoralBench: A MultiModal Moral Evaluation Benchmark for Large Vision-Language Models
• Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
• E2MoCase: A Dataset for Emotional, Event and Moral Observations in News Articles on High-impact Legal Cases
• PaliGemma: A versatile 3B VLM for transfer
• VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values
• Is CLIP the main roadblock for fine-grained open-world perception?
• Long-CLIP: Unlocking the Long-Text Capability of CLIP
• MOKA: Moral Knowledge Augmentation for Moral Event Extraction
• Measuring Moral Dimensions in Social Media with Mformer
• Blind Dates: Examining the Expression of Temporality in Historical Photographs
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• A Data Fusion Framework for Multi-Domain Morality Learning
• Sigmoid Loss for Language Image Pre-Training
• DeAR: Debiasing Vision-Language Models with Additive Residuals
• MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models
• Zero-shot Visual Commonsense Immorality Prediction
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Aligning AI With Shared Human Values
• Sampling Matters in Deep Embedding Learning
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge
• Liberals and conservatives rely on different sets of moral foundations.
• The development of the picture‐superiority effect
• Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Models
• Visual moral inference and communication
• A Data Fusion Framework for Multi-Domain Morality Learning
• Do Language Models Understand Morality? Towards a Robust Detection of Moral Content
• MM-MoralBench: A MultiModal Moral Evaluation Benchmark for Large Vision-Language Models