From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images

2025/11/27 by Chen, Yiming, Han, Junlin, Bai, Tianyi +3 · 1 citation
Computer Science · #Benchmark (surveying) #Cognition #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image (mathematics) #Machine Learning (cs.LG) #Multimedia (cs.MM) #Multimodal Machine Learning Applications #Perception #Pipeline (software) #Pixel #Process (computing)

paper · open access · doi:10.48550/arxiv.2511.22805

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/11/27 · openalex created_date 2025/12/03 · openalex updated_date 2026/07/28

Abstract

While Multimodal Large Language Models (MLLMs) are adept at answering what is in an image-identifying objects and describing scenes-they often lack the ability to understand how an image feels to a human observer. This gap is most evident when considering subjective cognitive properties, such as what makes an image memorable, funny, aesthetically pleasing, or emotionally evocative. To systematically address this challenge, we introduce CogIP-Bench, a comprehensive benchmark for evaluating MLLMs on such image cognitive properties. Our evaluation reveals a significant gap: current models are poorly aligned with human perception of these nuanced properties. We then demonstrate that a post-training phase can effectively bridge this gap, significantly enhancing the model's alignment with human judgments. Furthermore, we show that this learned cognitive alignment is not merely predictive but also transferable to downstream creative tasks. By integrating our cognitively-aligned MLLM into an image generation pipeline, we can guide the synthesis process to produce images that better embody desired traits, such as being more memorable or visually appealing. Our work provides a benchmark to measure this human-like perception, a post-training pipeline to enhance it, and a demonstration that this alignment unlocks more human-centric AI.

Citations

• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
• Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning
• Scaling Language-Free Visual Representation Learning
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• Unforgettable Lessons from Forgettable Images: Intra-Class Memorability Matters in Computer Vision
• MetaMorph: Multimodal Understanding and Generation via Instruction Tuning
• Cross-modal Information Flow in Multimodal Large Language Models
• CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM
• Discovering cognitive strategies with tiny recurrent neural networks
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
• A Survey on Benchmarks of Multimodal Large Language Models
• LLaVA-OneVision: Easy Visual Task Transfer
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• FindingEmo: An Image Dataset for Emotion Recognition in the Wild
• MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning
• The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective
• Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
• Visual cognition in multimodal large language models
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• Improved Baselines with Visual Instruction Tuning
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• Decoding Natural Images from EEG for Object Recognition
• SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
• OxfordTVG-HIC: Can Machine Make Humorous Captions from Images?
• MMBench: Is Your Multi-modal Model an All-around Player?
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
• Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
• EEG2IMAGE: Image Reconstruction from EEG Brain Signals
• What Images are More Memorable to Machines?
• Modeling, Quantifying, and Predicting Subjectivity of Image Aesthetics
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• Learning Transferable Visual Models From Natural Language Supervision
• DocVQA: A Dataset for VQA on Document Images
• Unified Vision-Language Pre-Training for Image Captioning and VQA
• GANalyze: Toward Visual Definitions of Cognitive Image Properties
• A Diagram Is Worth A Dozen Images

Cited by

• Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Related

• Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
• Position: Reasoning After Perception Means Reasoning Without Vision
• Do MLLMs Exhibit Human-like Perceptual Behaviors? HVSBench: A Benchmark for MLLM Alignment with Human Perceptual Behavior
• HueManity: Probing Fine-Grained Visual Perception in MLLMs
• Seeing is Not Understanding: A Benchmark on Perception-Cognition Disparities in Large Language Models